The replication crisis won’t be solved with broad brushstrokes

Alarm about a ‘replication crisis’ launched a wave of projects that aimed to quantitatively evaluate scientific reproducibility: statistical analyses, mass replications and surveys. Such efforts, collectively called metascience, have grown into a social movement advocating broad reforms: open-science mandates, preregistration of experiments and new incentives for careful research. It has drawn attention to the need for improvements, and caused rancour.

Philosophers, historians and sociologists no longer accept a single, unified definition of science. Instead, they document how scientists in different fields have developed unique practices of producing, communicating and evaluating evidence, guided loosely by a set of shared values. However, this diversity and underlying scholarship are often overlooked by metascience activists.

Over the past three years, Aaron Panofsky, a sociologist at the University of California, Los Angeles, and I have interviewed 60 senior biologists, chemists, geologists and physicists who are reviewing editors at Science, plus another 83 scientists seeking science-wide reforms. These highly recognized researchers saw growing interest in making science more open and robust — but also expressed scepticism.

Senior researchers bristled at the idea that their fields were in ‘crisis’, and suspected that activists were seeking recognition for themselves. A frustrated biologist argued that people running mass replication studies “were not motivated to find reproducibility” and benefited from finding it lacking. Others said metascientists dismissed replication work done to further a line of research rather than assess the state of the literature. Another saw data deposition as a frustrating, externally imposed mandate: “We’re already drowning in all the bureaucratic crap.”

Even those who acknowledged the potential value of reforms, such as those for data sharing, felt that there was no discussion about the costs. “If you add up all of the things that only take ten minutes, it’s a huge chunk of your day.”

Reformers counter that such complaints represent objections from a privileged elite, and point to perverse incentives, such as pressure to publish, that apply across academia. Marcus Munafo, a biological psychologist at the University of Bristol, UK, who co-founded the UK Reproducibility Network, hopes to change the system. He told me that science should move from a nineteenth-century “artisanal” practice towards one with structures to “audit or evaluate processes”. Brian Nosek, executive director of the Center for Open Science in Charlottesville, Virginia, feels progress has slowed because of “resistance” by people “doing quite well with the system as it is”.

I think that some reluctance does stem from entrenched interests. (It would be interesting to know whether early-career researchers would have answered differently.) But some is based on experience and knowledge. Many interviewees accepted the need for targeted improvements, but objected to blanket decrees. One physicist blamed data-deposition mandates on a bureaucracy that cannot distinguish differences in how scientific fields work. A plant biologist thought preregistration was appropriate for large, long-term experiments — such as clinical trials — but not short-term, iterative experiments such as hers, where each experiment depends on the previous one’s results.

Part of the problem is that many reformers come from a narrow swathe of academia. The authors of ‘A manifesto for reproducible science’, an influential perspective commissioned for the inaugural issue of Nature Human Behaviour, are predominantly from psychology, social and behavioural sciences (M. R. Munafò et al. Nature Hum. Behav. 1, 0021; 2017). Of the 39 authors introducing the widely adopted transparency and openness guidelines, most come from social sciences, and the rest are funders, employees at open-science institutes, editors of scientific journals and a science-policy scholar (B. A. Nosek et al. Science 348, 1422–1425; 2015).

The researchers we spoke to emphasized different norms across fields. A trained chemist who has also done research in biology explained that, although models in chemistry tend to have very high precision and very high reproducibility, “The reverse of that is true for biology.” This can lead to systematically different interpretations of failed replications, and evaluations of reproducibility. Are experiments poorly designed, or technically difficult?

My interviewees praised cases of overhaul that originated in the community they applied to. These included TERRINet, a program developed to coordinate research in robotics to code validation exercises in seismography. In another much-lauded effort, thousands of cell biologists studying autophagy came together to standardize definitions and protocols (D. J. Klionsky Mol. Biol. Cell 27, 733–738; 2015).

If reformers want to make diverse scientific fields more robust, they need to demonstrate that they understand how specific fields operate before they push a set of practices about how science overall should operate. Rather than aligning practices with an overarching ideal, reformers should focus efforts on challenges in specific fields — for instance by working with scientific societies. Otherwise, efforts could be resisted as extensions of bureaucracy, rather than embraced as routes to more robust research.

Nature 594, 151 (2021)

doi: https://doi.org/10.1038/d41586-021-01509-7

Editor’s note: Nature journals have embraced practices to promote robust practices and full, open reporting.

Competing Interests

The author declares no competing interests.


Nature Briefing

An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday.

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing