Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Rule rewrite aims to clean up scientific software

Nature Biotechnology asks peer reviewers to check accessibility of code used in computational studies.

Computer systems have been prone to error since the early days. Credit: USAF/Getty

The finding seemed counterintuitive: warming in North America was driving plant species to lower elevations — not towards higher, cooler climes, as ecologists had long predicted. But the research published in Global Change Biology indeed turned out to be wrong. In February, the journal retracted the paper after its intriguing conclusion was found to be the result of errant software code1.

Worried about a rising tide of results that fail to measure up, journals are starting to take action. In the latest such move, Nature Biotechnology announced on 7 April a plan to prevent such embarrassing episodes in its pages (Nature Biotechnol. 33, 319; 2015). Its peer reviewers will now be asked to assess the availability of documentation and algorithms used in computational analyses, not just the description of the work. The journal is also exploring whether peer reviewers can test complex code using services such as Docker, a piece of software that allows study authors to create a shareable representation of their computing environment.

Researchers say that such measures are badly needed. They note that the increasing size of data sets and complexity of analysis software makes errors harder to detect. “This is a big step forward,” says Ciera Martinez, a plant biologist at the University of California, Davis. “A large journal focusing on reproducibility is desperately needed.” Computational experts often raise issues about code quality or availability during peer review, she adds, but such concerns are often ignored because many journals do not require examination of code.

The result can be errors or irregularities that lead to retractions, corrections and divisive debates. In announcing its policy, Nature Biotechnology cited two of its studies that were called into question by scientists who could not replicate the conclusions. Both papers2,3 had reported new methods for analysing connections within networks, but neither provided sufficient documentation of their tools or approach. The journal has now published more information about how software was used in each analysis.

“We are simply seeking to make our editorial evaluation of computational tools more consistent,” says Nature Biotechnology editor Andrew Marshall, who adds that other journals that publish computational-biology research have taken similar steps.

But several issues complicate the drive for software reproducibility. One is the difficulty of finding qualified reviewers for papers in disciplines that cross departmental boundaries. “The research is collaborative, but the review process is stuck in a disciplinary mindset,” says Lior Pachter, a computational biologist at the University of California, Berkeley.

Another is social: there is no etiquette governing how those who wish to replicate results should behave towards those whose work they examine. If authors of erroneous studies face public embarrassment and shaming, that can discourage other researchers from submitting to the same scrutiny. “It’s like taking your clothes off; you don’t want to be embarrassed by someone pointing at you because you have a lot of body hair,” says Ben Marwick, an archaeologist at the University of Washington in Seattle.

Mindful of such concerns, advocates of software reproducibility are placing less emphasis on publications. Instead they argue that published tools should be able to be used by other researchers. They say that this approach acknowledges the iterative nature of science.

“When we say ‘open science’ or ‘open research’, it’s not just about accessibility and availability of content or material,” says Kaitlin Thaney, director of the non-profit Mozilla Science Lab in New York. “It’s taking it one step further to think about use and reuse, so someone can carry that forward.”

An increasing number of initiatives aim to encourage scientists to ensure that their software is replicable. Courses run by organizations such as the non-profit Software Carpentry Foundation teach the value of writing and sharing solid scientific code, as well as the principles of constructing it. Software packages such as iPython and knitr make it easier to document code creation transparently and in its research context. The Mozilla Science Lab has experimented with training researchers in the scientific-coding process, and universities such as the University of California, Berkeley, are creating courses that train graduate students to code in a way that advances the cause of open and reproducible science.

The cause has been slow to catch on in the upper echelons of research. But those pushing for great replicability hope that a combination of incentives could begin to make a difference. Measures aimed at the publication process, such as those announced by Nature Biotechnology, will hit home for many researchers. Others may be lured by the notion that replicable work is more likely to stand the test of time. “The incentive for me, as a young researcher, is simple,” says Martinez. “Better science.”

References

  1. Harsch, M. A. & Hille Ris Lambers, J. Glob. Change Biol. 21, 1376 (2015).

    Article  Google Scholar 

  2. Barzel, B. & Barabási, A.-L. Nature Biotechnol. 31, 720–725 (2013).

    CAS  Article  Google Scholar 

  3. Feizi, S., Marbach, D., Médard, M. & Kellis, M. Nature Biotechnol. 31, 726–733 (2013).

    CAS  Article  Google Scholar 

Download references

Authors

Related links

Related links

Related links in Nature Research

Programming: Pick up Python 2015-Feb-04

Interactive notebooks: Sharing the code 2014-Nov-05

My digital toolbox: Ecologist Ethan White on interactive notebooks 2014-Sep-30

'Boot camps' teach scientists computing skills 2014-Sep-03

Mozilla plan seeks to debug scientific code 2013-Sep-24

Publish your computer code: it is good enough 2010-Oct-13

Computational science: ...Error 2010-Oct-13

Related external links

Mozilla science lab

Data Sciences for the 21st Century: Environment and Society

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Check Hayden, E. Rule rewrite aims to clean up scientific software. Nature 520, 276–277 (2015). https://doi.org/10.1038/520276a

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/520276a

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing