The online journal eLife has created its first “computationally reproducible” article, in which figures are integrated with the software, data and computational environment required to produce them.
Readers can change the underlying code and produce modified figures in order to better understand, validate or build on the work. For example, they could rerun the code to see what a figure looks like without outliers, or use a different type of plot to reveal trends more effectively.
Researchers say that the reproducible article, which eLife first published in its conventional format last year1, is a significant step towards a future in which papers are much more than just static pages.
“What eLife is doing is making this commitment to upgrading the research article so that it is not just the written word, but it is this multifaceted communication medium that includes the data and the code as an integral part,” says Lorena Barba, a mechanical and aerospace engineer and reproducibility specialist at the George Washington University in Washington DC.
According to Giuliano Maciocci, head of product and user experience at eLife in Cambridge, UK, the article is a prototype of technologies the journal now plans to scale up; authors who would like to exploit similar features can contact the journal for consideration.
The technologies could in future make it easier for researchers to reuse each other's code. Users cannot upload their own data and add them to figures, but Maciocci says that the plan is for them to eventually download such articles and run them on their own computers in order to, for instance, analyse their own findings using the authors' code.
Reusing such software is often surprising difficult, involving confusing sets of interdependent tools, each of which must be downloaded and installed. eLife’s proof-of-concept publication allows users to view and execute code in the body of the article itself, with no installation required. “The reader has zero friction to execute the code and play with the data,” explains Barba.
On Twitter, neurobiologist Cori Bargmann of Rockefeller University in New York City described the development as “the paper of the future”. “The new eLife format provides more opportunities to share data and analysis, and thereby build on each other’s work,” Bargmann added in a follow-up e-mail.
The ability to make reproducible documents is not new, notes Titus Brown, a bioinformatician at the University of California, Davis. Researchers can do it themselves by combining tools such as Jupyter Notebook, which provides an interactive lab notebook, with the cloud-based software Binder that allows others to execute the code. “What’s been lacking is the integration with the publisher side of things,” says Brown.
Some journals, including F1000Research, GigaScience and titles from Cell Press, already allow authors to embed executable ‘compute capsules’ from the cloud-based platform Code Ocean in their articles, with the code and execution environment rendered as an interactive widget. And in August 2018, Nature Methods, Nature Biotechnology and Nature Machine Intelligence launched an ongoing pilot programme with Code Ocean to use the company’s compute capsules for peer review.
eLife is the first journal to make the code a native part of the article itself.
Published on 20 February, the enhanced eLife article describes an attempt to replicate a 2012 paper2 about how a gene that is often mutated in cancer cells impacts the expression of other genes. The study was conducted as part of the Reproducibility Project: Cancer Biology — a systematic effort to assess the reproducibility of research findings in oncology — and led by the Center for Open Science in Charlottesville, Virginia, and the Science Exchange in Palo Alto, California.
Created using a collection of open-source tools called the eLife Reproducible Document Stack, which includes Binder and a dedicated document editor called Stencila Desktop, the reproducible article looks like any other, except that each figure is adorned with a small blue arrow. When the user clicks that icon, the programming code used to produce the figure is revealed in a live, inline text editor. As the user alters the code, the figure updates, allowing the user to adjust the presentation of data (see 'Computational reproducibility') or test the effects of removing outliers.
Casey Greene, a bioinformatician at the University of Pennsylvania’s Perelman School of Medicine in Philadelphia, notes, for instance, that the article summarizes some of its data using a bar chart with error bars. It's a configuration sometimes disparagingly referred to as a dynamite plot because of its resemblance to a Wild West-style explosives detonator — and its potential to obscure the underlying data. Using this reproducible article, a reader could recreate that plot in another style to reveal trends more effectively. However, they cannot yet share such a modification with other readers, a feature that Greene would like to see. “That, to me, would be a clear win for this technology,” he says.
According to Maciocci, the blog post announcing the article was the publisher’s most-read yet, with more than 7,500 unique views since 20 February; the article itself has had just under 2,000 users. “We definitely have a mandate to pursue the content moving forward.”
He notes that all the software underlying the article is open-source and available to any interested publisher, although none has yet contacted the journal.
As a prototype, the article is missing references and supplementary figures, but Maciocci says those will be available in future such articles.
For study author Tim Errington, director of research at the Center for Open Science, the document represents an evolution of the research article itself. “This is potentially a step to move towards me not having to, as a researcher, move my content out of my reproducible workflow to a flattened version so you can review it — that there’s actually the potential to just expose everything I’ve done.”
It’s not there yet, he admits, but it’s a start. “The question is, great, now what do we do? How do we keep making this better?”
Nature 567, 17-18 (2019)
Updates & Corrections
Correction 28 February 2019: An earlier version of this story inaccurately stated Tim Errington's job title.
Lewis, L. M. et al. eLife 7, e30274 (2018).
Lin, C. Y. et al. Cell 151, 56–57 (2012).