Code is at the heart of computational social and behavioural science. To increase code reliability and reproducibility, we are implementing formal peer review of the code behind computational models whenever they are essential to the research we publish. We ask our authors to prepare and store their code with readability, transparency and future replicability in mind.
In today’s behavioural research ecosystem, the use of computational tools and algorithms extends far beyond their secondary role of empirical data analysis. The code itself is often both the source of data and the tool of hypothesis testing.
Individual-based simulations, machine learning tools, population-genetic and social-network models have been critical to the computational social science revolution of the past two decades. Landmark studies in sociology, network science, psychology, economics, neuroscience and evolution have relied on computational models and simulations. More recently, some of the most influential papers on behavioural strategies for taming the COVID-19 pandemic — many published in our pages — have relied on epidemiological models and custom-built simulations implemented in C, Java, Python, R and other languages.
At Nature Human Behaviour, we recognize the importance of the theoretical advances that computational studies can offer. But computational science, like many other disciplines, faces reproducibility issues. This is often because the code is unavailable or cannot be accessed, because it is difficult to read and interpret, or because other researchers find it difficult to compile, re-use or extend it.
One of our editorial missions is to maintain high standards of reliability and reproducibility of the findings that we publish, and this includes computational models. We already ask our authors to include Code Availability statements with their submissions, describing how their code or software can be accessed. But we also believe that to become a part of the publication record alongside the paper itself, computer code needs to be properly documented, reviewed and stored.
Taking a more active editorial role in ensuring code quality and reproducibility, this month Nature Human Behaviour joins a subset of other Nature Research journals that undertake peer review of custom code, whenever it is central to the main findings of the study.
To inform our approach to code review, we spoke to researchers working with code and computational tools in social and human behavioural sciences. Respondents noted that there is currently no clear community standard governing code quality and readability, and that code sharing practices often depend on tradition and researchers’ backgrounds. Code reliability suffers, and errors in code can lead to false results and article retractions.
Beyond simple code verification, researchers said that code peer review would encourage the scientific community to write more readable and user-friendly code, and it would increase trust in published computational results.
What does the introduction of systematic code review mean for our authors? From now on, we will require our authors to prepare their code strictly following Nature Research Guidelines for authors submitting code & software. For peer review, the authors will have to either submit their code as a Supplementary File or deposit it in a community-recognized repository such as GitHub. They will also have to include a README file that describes system requirements and provides detailed instructions for installation and use, a licence of use, and a test dataset needed to reproduce reported results.
As a condition of publication, we will ask our authors to ensure continual access to the reviewed version of code. At acceptance, we will expect our authors to deposit the peer-reviewed version of the code in a DOI-granting repository. Statements of code availability from the authors upon reasonable request will not be accepted, except in special circumstances, and any restrictions on code availability will have to be shared with the editors at submission.
Our move towards systematic code review also means that we will be asking our peer reviewers to verify that the code is functional, that it reproduces the reported findings, and that it is appropriately presented and documented. We recognize that code verification and review places extra demands on reviewers, and so we ask authors to do all they can to simplify the process. We are exploring ways to ease this burden further, for instance through future collaboration with cloud-based platforms that allow users to create shareable ready-to-run representations of their entire computing environments.
Experience with code review in other Nature Research journals shows that the benefits for code quality and reproducibility are well worth the extra effort, and the scientists we spoke to agree. Code review also presents an excellent opportunity to include more junior researchers and members of more diverse communities in the review process of our articles. Computational models will continue to revolutionize social and behavioural sciences, and we hope that beyond just improving computational reproducibility, code review will advance our mission of strengthening the ties between computational and the behavioural research communities.
About this article
Cite this article
Supporting computational reproducibility through code review. Nat Hum Behav 5, 965–966 (2021). https://doi.org/10.1038/s41562-021-01190-w
This article is cited by
Nature Human Behaviour (2022)