Special |

Challenges in irreproducible research

Science moves forward by corroboration – when researchers verify others’ results. Science advances faster when people waste less time pursuing false leads. No research paper can ever be considered to be the final word, but there are too many that do not stand up to further study.

There is growing alarm about results that cannot be reproduced.  Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community. We hope that the articles contained in this collection will help.

Key reads

The solutions adopted by the high-energy physics community to foster reproducible research are examples of best practices that could be embraced more widely. This first experience suggests that reproducibility requires going beyond openness.

Perspective | Open Access | | Nature Physics

Replicating our work took four years and 100,000 worms but brought surprising discoveries, explain Gordon J. Lithgow, Monica Driscoll and Patrick Phillips.

Comment | | Nature News

A survey of Nature readers revealed a high level of concern about the problem of irreproducible results. Researchers, funders and journals need to work together to make research more reliable.

Editorial | | Nature News

Latest

Software that uncovers suspicious papers will do little for a community that does not confront organized research fraud, says Jennifer Byrne.

World View | | Nature

As the imaging technique produces ever-sharper protein structures, researchers are racing to develop tools to assess how accurate they are.

Technology Feature | | Nature

All articles

Software that uncovers suspicious papers will do little for a community that does not confront organized research fraud, says Jennifer Byrne.

World View | | Nature

Customize the experiment for the setting instead of adjusting the setting to fit a classical design.

This Month | | Nature Methods

The publishing system builds in resistance to replication. Paul Gertler, Sebastian Galiani and Mauricio Romero surveyed economics journals to find out how to fix it.

Comment | | Nature

As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves.

Comment | | Nature

Chemical probes and screening libraries can easily get mixed up or messed up, causing misleading results for unwary biologists.

Technology Feature | | Nature

Efforts to reduce irreproducibility in research must also tackle the temptation to cheat, argue Donald S. Kornfeld and Sandra L. Titus.

Comment | | Nature News

Start-up firms say robotics and software that autonomously record every detail of an experiment can transform the efficiency and reliability of research.

Toolbox | | Nature News

Papers in Nature journals should make computer code accessible where possible.

Editorial | | Nature News

Scientists and journals must work together to ensure that eye-catching artefacts are not trumpeted as genomic insights, says Daniel MacArthur.

Comment | | Nature

Biologists must realize the pitfalls of work on massive amounts of data.

Editorial | | Nature

Experimental biologists, their reviewers and their publishers must grasp basic statistics, urges David L. Vaux, or sloppy science will continue to grow.

Comment | | Nature

Many of the studies that use animals to model human diseases are too small and too prone to bias to be trusted, says Malcolm Macleod.

World View | | Nature

Research & Reviews

Customize the experiment for the setting instead of adjusting the setting to fit a classical design.

This Month | | Nature Methods

Most high-throughput target discovery screens for glioblastoma have been limited to in vitro models with uncertain physiological relevance. Here, Jeremy Rich and colleagues perform two parallel RNA interference screens for transcriptional regulators, comparing an in vitro screen in cell lines to an in vivo screen that recapitulates the tumour microenvironment. They find several transcriptional elongation factors that are specifically required for glioblastoma cell survival in vivo, particularly the transcriptional pause release factor JMJD6 which is highly expressed in gliomas. This type of in vivo functional screen has the potential to uncover novel therapeutic targets for cancer that have not been identified in previous in vitro approaches.

News & Views | | Nature

The National Institutes of Health Human Microbiome Project, published in 2012, provided a broad overview of the baseline microbiome in healthy individuals using samples from 18 different body sites. In this second installment, the authors expand this dataset with new whole-metagenome sequences and additional time points to assess the diversity and spatiotemporal distributions of the microbiota at six of these body sites. Using a combination of strain profiling, species-level metagenomic functional profiling and longitudinal analyses, this study delivers deeper insights into human microbial communities and provides an important resource for understanding what constitutes a 'healthy' microbiota.

Article | Open Access | | Nature

The development of optogenetics as a specific tool to probe the function of genetically defined neural circuits in the execution of specific behaviours has been a prominent field of recent growth in neuroscience. However, many of these studies disregard the potential for indirect effects of circuit manipulation on other downstream circuits operating independently in separate functions. Here, Bence Ölveczky and colleagues reveal how transient inactivation of specific circuits in mammals and songbirds can severely impair task-specific responses that otherwise spontaneously recover after permanent lesions of the same brain areas. This suggests that additional considerations must be taken into account when interpreting data from transient circuit manipulations of behaviour.

News & Views | | Nature

Quality control of cell lines used in biomedical research is essential to ensure reproducibility. Although cell line authentication has been widely recommended for many years, misidentification, including cross-contamination, remains a serious problem. We outline a multi-stakeholder, incremental approach and policy-related recommendations to facilitate change in the culture of cell line authentication.

Commentary | | Nature Methods

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.

Commentary | | Nature Methods

'Irreproducibility' is symptomatic of a broader challenge in measurement in biomedical research. From the US National Institute of Standards and Technology (NIST) perspective of rigorous metrology, reproducibility is only one aspect of establishing confidence in measurements. Appropriate controls, reference materials, statistics and informatics are required for a robust measurement process. Research is required to establish these tools for biological measurements, which will lead to greater confidence in research results.

Commentary | | Nature Methods

Low-powered studies lead to overestimates of effect size and low reproducibility of results. In this Analysis article, Munafò and colleagues show that the average statistical power of studies in the neurosciences is very low, discuss ethical implications of low-powered studies and provide recommendations to improve research practices.

Analysis | | Nature Reviews Neuroscience

A wealth of microarray gene expression data and a growing volume of RNA sequencing data are now available in public databases. The authors look at how these data are being used and discuss considerations for how such data should be analysed and deposited and how data reuse could be improved.

Review Article | | Nature Reviews Genetics

Animal studies have contributed immensely to our understanding of diseases and assist the development of new therapies, but inadequate experimental reporting can sometimes render such studies difficult to reproduce and to translate into the clinic. This year, a US National Institute of Neurological Disorders and Stroke workshop addressed this issue, and its conclusions are discussed in a Perspective piece in this issue of Nature. The main workshop recommendation is that at a minimum, studies should report on randomization, blinding, sample-size estimation and how the data were handled.

Perspective | Open Access | | Nature

Most scientific papers published today rely on computer programs for data collection and manipulation. Writing in the Perspective pages in this issue of Nature, Darrel Ince and colleagues argue that the policies of most journals and funding bodies towards the release of computer codes as part of the publication process are obsolete. They say that the full release of actual source code should be the norm for any scientific results dependent on computation, with an agreed list of exceptions applicable only to rare cases. Current policies range from a requirement for release of the relevant computer programs on request to Nature's less stringent stipulation of a 'natural language' description of computer algorithms.

Perspective | | Nature

Too many sloppy mistakes are creeping into scientific papers. Lab heads must look more rigorously at the data — and at themselves.

Editorial | | Nature