Leaders of the scientific community, nudged by the media (including Nature), are acknowledging that a culture of science focused on rewarding eye-catching and positive findings may have resulted in major bodies of knowledge that cannot be reproduced.

Private-sector, academic and non-profit groups are leading multiple efforts to replicate selected published findings, and so far the results do not make happy reading. Several high-profile endeavours have been unable to reproduce the large majority of peer-reviewed studies that they examined. Meanwhile, the US National Academies is preparing to publish a high-profile report on scientific integrity that will flag irreproducibility as a key concern for the research enterprise.

As the spotlight shines on reproducibility, uncomfortable issues will emerge at the interface of research and 'evidence-based' policy.

Nature special: Challenges in irreproducible research

Consider, for example, the Secret Science Reform Act of 2015, a US bill that would “prohibit the Environmental Protection Agency from proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible”. Passed in March by the House of Representatives essentially along party lines (Republicans in favour, Democrats opposed) and now awaiting action by the Senate, the bill has been vigorously opposed by many scientific and environmental organizations.

They argue, probably correctly, that the bill's intent is to block and even roll back environmental regulations by requiring that all data on which the rules are based be made publicly available for independent replication. One of the main objections is that a lot of the scientific research that informs regulatory decisions is not of the sort that can be replicated. For example, a statement of opposition from numerous scientific societies and universities explains that: “With respect to reproducibility of research, some scientific research, especially in areas of public health, involves longitudinal studies that are so large and of great duration that they could not realistically be reproduced. Rather, these studies are replicated utilizing statistical modeling.”

Precisely. Replication of the sort that can be done with tightly controlled laboratory experiments is indeed often impossible when you are studying the behaviour of dynamic, complex systems, for example at the intersection of human health, the natural environment and technological risks. But it is hard to see how this amounts to an argument against mandating open access to the data from these studies. Growing concerns about the quality of published scientific results have often singled out bad statistical practices and modelling assumptions, and have typically focused on the very types of science that often underlie regulations, such as efforts to quantify the population-wide health effects of a single chemical.

Quality assurance will increasingly become a matter of political interpretation.

Although concerns about the bill's consequences are reasonable, the idea that it would be bad to make public the data underlying environmental regulations seems to contradict science's fundamental claims to objectivity and legitimacy. In June, a commentary in Science by an array of leading voices, including the current and future heads of the National Academies, flagged “increased transparency” and “increased data disclosure” as crucial elements of science's “self-correcting norm” that can help to address “the disconcerting rise in irreproducible findings” ( B. Alberts et al. Science 348, 1420–1422; 2015). This is more or less the position taken by the Secret Science bill's sponsor, Representative Lamar Smith (Republican, Texas): “The bill requires the EPA to use data that is available to the public when the Agency writes its regulations. This allows independent researchers to evaluate the studies that the EPA uses to justify its regulations. This is the scientific method.”

This battle for the soul of science is almost surreal in its avoidance of the true issue, which is ideological. One side believes that the government should introduce stricter environmental regulations; the other wants fewer restrictions on the marketplace. Science is the battleground, but it cannot adjudicate this dispute. At its core, the disagreement is about values, not facts. But just as importantly, the facts themselves are inevitably incomplete, uncertain, contested and, as we have been learning, often unreliable.

Like a divorced couple bitterly fighting over the custody of their child, both sides in the Secret Science debate insist that they have only the interests of science at heart. Republicans are using a narrow, idealized portrayal of science — that it produces clear and reproducible findings — as a weapon to undercut environmental and public-health regulation of the private sector. But many scientists, environmentalists and Democrats have long used similar portrayals to justify the same regulations, and to bash Republicans as anti-scientific when they did not agree.

More and more, science is tackling questions that are relevant to society and politics. The reliability of such science is often not testable with textbook methods of replication. This means that quality assurance will increasingly become a matter of political interpretation. It also means that the 'self-correcting norm' that has served science well for the past 500 years is no longer enough to protect science's special place in society. Scientists must have the self-awareness to recognize and openly acknowledge the relationship between their political convictions and how they assess scientific evidence.

Credit: Michael Temchine