Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reproducibility will not cure what ails science

Subjects

A bill to make data for environmental regulation more transparent reveals the fuzzy boundary between science and ideology, argues Daniel Sarewitz.

Leaders of the scientific community, nudged by the media (including Nature), are acknowledging that a culture of science focused on rewarding eye-catching and positive findings may have resulted in major bodies of knowledge that cannot be reproduced.

Private-sector, academic and non-profit groups are leading multiple efforts to replicate selected published findings, and so far the results do not make happy reading. Several high-profile endeavours have been unable to reproduce the large majority of peer-reviewed studies that they examined. Meanwhile, the US National Academies is preparing to publish a high-profile report on scientific integrity that will flag irreproducibility as a key concern for the research enterprise.

As the spotlight shines on reproducibility, uncomfortable issues will emerge at the interface of research and 'evidence-based' policy.

Nature special: Challenges in irreproducible research

Consider, for example, the Secret Science Reform Act of 2015, a US bill that would “prohibit the Environmental Protection Agency from proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible”. Passed in March by the House of Representatives essentially along party lines (Republicans in favour, Democrats opposed) and now awaiting action by the Senate, the bill has been vigorously opposed by many scientific and environmental organizations.

They argue, probably correctly, that the bill's intent is to block and even roll back environmental regulations by requiring that all data on which the rules are based be made publicly available for independent replication. One of the main objections is that a lot of the scientific research that informs regulatory decisions is not of the sort that can be replicated. For example, a statement of opposition from numerous scientific societies and universities explains that: “With respect to reproducibility of research, some scientific research, especially in areas of public health, involves longitudinal studies that are so large and of great duration that they could not realistically be reproduced. Rather, these studies are replicated utilizing statistical modeling.”

Precisely. Replication of the sort that can be done with tightly controlled laboratory experiments is indeed often impossible when you are studying the behaviour of dynamic, complex systems, for example at the intersection of human health, the natural environment and technological risks. But it is hard to see how this amounts to an argument against mandating open access to the data from these studies. Growing concerns about the quality of published scientific results have often singled out bad statistical practices and modelling assumptions, and have typically focused on the very types of science that often underlie regulations, such as efforts to quantify the population-wide health effects of a single chemical.

Quality assurance will increasingly become a matter of political interpretation.

Although concerns about the bill's consequences are reasonable, the idea that it would be bad to make public the data underlying environmental regulations seems to contradict science's fundamental claims to objectivity and legitimacy. In June, a commentary in Science by an array of leading voices, including the current and future heads of the National Academies, flagged “increased transparency” and “increased data disclosure” as crucial elements of science's “self-correcting norm” that can help to address “the disconcerting rise in irreproducible findings” ( B. Alberts et al. Science 348, 1420–1422; 2015). This is more or less the position taken by the Secret Science bill's sponsor, Representative Lamar Smith (Republican, Texas): “The bill requires the EPA to use data that is available to the public when the Agency writes its regulations. This allows independent researchers to evaluate the studies that the EPA uses to justify its regulations. This is the scientific method.”

This battle for the soul of science is almost surreal in its avoidance of the true issue, which is ideological. One side believes that the government should introduce stricter environmental regulations; the other wants fewer restrictions on the marketplace. Science is the battleground, but it cannot adjudicate this dispute. At its core, the disagreement is about values, not facts. But just as importantly, the facts themselves are inevitably incomplete, uncertain, contested and, as we have been learning, often unreliable.

Like a divorced couple bitterly fighting over the custody of their child, both sides in the Secret Science debate insist that they have only the interests of science at heart. Republicans are using a narrow, idealized portrayal of science — that it produces clear and reproducible findings — as a weapon to undercut environmental and public-health regulation of the private sector. But many scientists, environmentalists and Democrats have long used similar portrayals to justify the same regulations, and to bash Republicans as anti-scientific when they did not agree.

More and more, science is tackling questions that are relevant to society and politics. The reliability of such science is often not testable with textbook methods of replication. This means that quality assurance will increasingly become a matter of political interpretation. It also means that the 'self-correcting norm' that has served science well for the past 500 years is no longer enough to protect science's special place in society. Scientists must have the self-awareness to recognize and openly acknowledge the relationship between their political convictions and how they assess scientific evidence.

Credit: Michael Temchine

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Daniel Sarewitz.

Additional information

Tweet Follow @NatureNews

Related links

Related links

Related links in Nature Research

Robust research: Institutions must do their part for reproducibility 2015-Sep-01

Over half of psychology studies fail reproducibility test 2015-Aug-27

Registered clinical trials make positive findings vanish 2015-Aug-13

Announcement: Time to tackle cells’ mistaken identity 2015-Apr-15

Metascience could rescue the ‘replication crisis’ 2014-Nov-04

Replication studies: Bad copy 2012-May-16

Related external links

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sarewitz, D. Reproducibility will not cure what ails science. Nature 525, 159 (2015). https://doi.org/10.1038/525159a

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/525159a

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing