Nature Biotechnology is reevaluating editorial oversight of papers centered on computational analyses in anticipation of the 'big data' world.
Computational biology papers pose particular challenges to the peer review process. Often, a computational approach or its software implementation may be insufficiently documented or missing. The version of the software may not match the algorithm described in a paper or produce the published results. And source code associated with software central to the main claims of a paper may not be made available. These issues have prompted Nature Biotechnology's editors to ask whether we can improve the peer review of computational analysis papers as a whole. This is especially important as biology moves toward the analysis of increasingly large datasets.
Reproducibility of biological papers has been the subject of much debate of late. But what about the reproducibility of papers that focus primarily on computation? What can we as editors do to facilitate review and evaluation of reproducibility? What responsibility do referees hold to run a piece of software, review provided source code or implement the methods provided? Can referees be expected to download supplementary datasets, reproduce new analyses, rerun competing methods? Should authors always provide source code for methods developed? And what minimum set of descriptors should be provided for previously published software (version number, creation/modification history data, author, special hardware requirements)?
Nature Biotechnology and other Nature journals are taking steps to facilitate peer review and reproducibility, given the complexity of these questions. Since last October, all Nature journals have required that authors declare the location and accessibility of any custom code and software central to the main claims in a paper (Nature 514, 536, 2014). Nature Biotechnology will specifically raise the issue of code availability with referees with the aim of enforcing this policy in a manner consistent with community expectations. We recommend hosting source code on GitHub (https://github.com/) together with a backup on a DOI-minting repository like Zenodo (http://zenodo.org) or figshare (http://figshare.com). We are now also systematically obtaining feedback from referees on their ability to test a tool in an online environment that retains their anonymity; whether a software implementation matches the mathematical algorithm in a paper; whether it can be run on operating systems (e.g., Windows or UNIX) widely available to the research community; and whether code is sufficiently documented and clear to allow another researcher to follow the algorithm.
These are practical steps that can be taken now. But going forward, several emerging platforms across the field of computational biology promise to help facilitate documenting software and sharing software and code between authors, referees and readers. Fit-for-purpose computing environments, such as Sage Bionetworks' Synapse, allow hosting of data and software, enable appropriate accreditation to researcher contributions, and facilitate integration, analysis and publication of data-intensive science in real time as research is performed. Electronic notebooks, such as IPython Notebooks, Sweave or knitr, that record all steps of a computational protocol can help authors submit a notebook that eases reproducibility by a referee or a reader. When these tools cannot capture the complexity of custom-written pipelines and manual steps, 'software containerization' tools such as Docker allows storing, sharing and reopening of custom environments where code can be rerun in the context of custom software tools, libraries and datasets.
In this context, the current issue contains updates on two related methods—that of Barzel et al. (Nat. Biotechnol. 31, 720–725, 2013) and that of Feizi et al. (Nat. Biotechnol. 31, 726–733, 2013). Both methods analyze networks of links, such as regulatory interactions among genes, physical contacts among amino acids in a protein or social interactions between co-authors of a scientific paper. Both methods fell short of full reproducibility in their original publication.
For Barzel & Barabási (Nat. Biotechnol. 31, 720–725, 2013), the broad applicability of their method and the approaches used to compare it to existing methods were challenged by Bastiaens et al. (p. 336), who also note its conceptual similarity to modular response analysis (MRA). In reply (p. 339), Barzel & Barabási acknowledge the common roots between their silencing method and MRA but emphasize that the former improves predictions based on correlations and other statistical similarity measures, which MRA cannot address. They also now present both the original and an updated version of the code for their silencing method.
For the Feizi et al. paper (Nat. Biotechnol. 31, 726–733, 2013), insufficient detail was provided regarding parameter selection for the three application domains and regarding the correction to their online supplementary materials after the paper appeared online. Feizi et al. now provide these details in a peer-reviewed addendum to their paper, along with a new analysis demonstrating the robustness of their method performance when the network density and eigenvalue scaling parameters are both varied. The authors also make available additional scripts and resources to support the reproducibility of their method, including a video running these scripts to generate their key results in all three application domains. Finally, the authors provide a peer-reviewed corrigendum that rectifies their reported performance values, showing that their method performs better than had been initially reported (p. 424).
The higher standards of reproducibility that we are implementing will, we hope, help authors, editors and referees catch potential mistakes earlier in the review process. If source code, software and data are not made freely available and user friendly, then all in the research community will lose. As biological research increasingly wrestles with large datasets, and the methods for analyzing them become more complex, our current idea of rigorous peer review is likely to change. We recognize that changes to our editorial process are just a beginning. But we expect that close attention to the issues surrounding peer review of computational research, in conjunction with evolving community standards, will lead to continued improvement in the accessibility and reproducibility of the research we publish.
About this article
Cite this article
Rebooting review. Nat Biotechnol 33, 319 (2015). https://doi.org/10.1038/nbt.3202
This article is cited by
HiC-bench: comprehensive and reproducible Hi-C data analysis designed for parameter exploration and benchmarking
BMC Genomics (2017)
Nature Biotechnology (2017)
Nature Neuroscience (2017)
Nature Neuroscience (2017)
Nature Biotechnology (2017)