As Nature Reviews Methods Primers publishes its first articles, we outline the journal’s aims and scope and our contribution to the pursuit of reproducibility.
Given the scale of the problem, it’s evident that many scientists don’t even realize that they are making mistakes.
So writes Richard Harris, in his book Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions, in which he discusses how common and even incentivized poor experimental design is in research. This problem makes it difficult for researchers to reproduce published results. That science has a reproducibility crisis is no surprise. In a 2016 survey of 1,500 researchers, Nature found that one-third of respondents thought daily about the reproducibility of their own research, and more than two-thirds thought about it at least once a month1. Facilitating reproducibility and encouraging data sharing must be an integral part of the experimental design process, not an after-thought at the manuscript writing stage. Part of the problems we face as a discipline stem from flaws in experimental design and protocol, and from not recognizing when mistakes are being made in the process. There are as many ways to use a method or protocol as there are investigators who use it. So how do we know if we are using the most suitable approach? From today, Nature Reviews Methods Primers aims to contribute solutions to this pressing question!
Nature Reviews Methods Primers expands on the Primers concept of our sister journal Nature Reviews Disease Primers. There is a family resemblance between the journals: like Disease Primers, Methods Primers aims to provide an authoritative introductory overview of a given method or technique for non-experts. Each Methods Primer consists of the same major sections, outlined below. Each section will address a different aspect of using a given method or technique to help limit unintentional mistakes that make results difficult to reproduce.
The hardest part of using a new method is figuring out how it works and how to interpret the raw data. In the Experimentation section, authors will pull together the best information and their years of experience in using the method to guide readers in conducting experiments. This section will include best practices for setting up equipment, samples and controls. All of the important considerations — from sample preparation to safety, ethics, inclusivity and collaboration — will be outlined here.
The Results section will provide a plan for how to go from raw output data to processed and analysed data. This section will describe tools for data analysis and where to find them, and will outline selection of the best analytical tools and approaches for various purposes.
To tackle the reproducibility crisis, the Reproducibility and data deposition section will help researchers — new and established — consider the most important aspects of making their science open, sharable and reproducible. This section will act as a roadmap from experimental design to writing up the manuscript. We put an emphasis on data deposition in the title of the section to encourage those fields without strong data sharing practices and policies to consider and establish these.
Another important aspect of the reproducibility crisis is translation — how a technique developed in one field is used in another without clear understanding of whether the method is fit for purpose. Primers include a Limitations and optimizations section, laying out what the method is not yet able to do, as well as important considerations for optimizing the method and working around its limitations. These limitations can range from efficiency in individual experiments to automation and the ability to scale up to meet industrial demands.
“We put an emphasis on data deposition … to encourage those fields without strong data sharing practices and policies to consider and establish these”
The Applications section facilitates outside-the-box thinking about the range of research questions that can be addressed using the method across a range of disciplines. Discipline-specific and field-specific considerations are included to enable tailoring of the method to specific needs. The Outlook section looks at the potential development, optimization and application of the method in the next 5–10 years. Each Primer is accompanied by a graphical summary, called a PrimeView.
No one journal can solve the reproducibility crisis and offer every possible solution. At Nature Reviews Methods Primers, we are focused on highlighting these issues across the life, physical and social sciences, aiming to bridge the gap between methods that straddle different disciplines and provide a platform for discussions with and among researchers. Although definition of a scientific method is nebulous, our Primers will range from standard analytical methods such as solid-state NMR, to applied methods such as multiplex automated genome engineering and chemical vapour deposition techniques, as well as statistical and theoretical methods. Regardless of your research background or career stage, we hope that you will find Nature Reviews Methods Primers a useful addition to your research toolbox.
Reality check on reproducibility. Nature 533, 437 (2016).