Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Irreproducible biology research costs put at $28 billion per year

Study calculates cost of flawed biomedical research in the United States.

Scientists in the United States spend $28 billion each year on basic biomedical research that cannot be repeated successfully. That is the conclusion of a study published on 9 June in PLoS Biology1 that attempts to quantify the causes, and costs, of irreproducibility.

John Ioannidis, an epidemiologist at Stanford University in California who studies scientific robustness, says that the analysis is sure to prompt discussion about the problem — but should be taken with a pinch of salt, given that its estimates carry great uncertainty.

But Len Freedman, the study’s lead author and head of the non-profit Global Biological Standards Institute in Washington DC, says that the work is of value, even though it cannot pin down the size of the problem. “Clearly, there are tremendous inefficiences [in research], and this is putting a spotlight on that,” says Freedman, whose group seeks to develop best practices for biological experiments.

Tracing trouble

Freedman and his colleagues defined irreproducibility broadly, as being any errors or omissions that prevent attempts to replicate experimental findings. The researchers sought to determine the relative contributions of four sources of irreproducibility identified in an earlier analysis2: study design; laboratory protocols; biological reagents and reference materials; and data analysis and reporting.

The team did not examine whether the results of any individual paper were reproducible, however. Instead, the researchers surveyed existing analyses of factors that contribute to irreproducibility, using these to estimate an overall reproducibility rate — and to weigh the relative influences of various factors.

Iain Cockburn, an economist at Boston University in Massachusetts and a co-author of the study, says that the analysis was limited by the quality and amount of data available. For example, assessments of poor materials were based on estimates of how often cell lines are misidentified. In other cases, the team extrapolated from reports from clinical work.

Overall, the team found that poor materials made the largest contribution to reproducibility problems, at 36%, followed by study design at 28% and data analysis at 26%. The team estimates the overall rate of irreproducibility at 53%, but cautions that the true rate could be anywhere between 18% and 89%. That puts the potential economic cost of irreproducibility anywhere from $10 billion to $50 billion per year.

“The four categories are decent, but the estimates are off,” says Ioannidis. “I would put a much higher rate on the data analysis and reporting component.”

The analysis is further limited by the fact that the four contributing factors that it uses are interdependent, says Melissa Haendel, an information scientist at Oregon Health and Science University in Portland. For example, research materials and the controls used to assess them are part of laboratory protocols, and study design affects data analysis.

Seeking solutions

Concerns about reproducibility in biomedical research have increased over the past decade, partly in response to a 2005 study by Ioannidis that found scientific journals to be biased towards publishing flashy, positive results3. And researchers at pharmaceutical companies have reported that their attempts to replicate the conclusions of peer-reviewed papers fail at rates upwards of 75%4,5.

Several institutions are taking steps to address the problem. In November, for example, the US National Institutes of Health urged journals to adopt guidelines aimed at boosting reproducibility. (Nature Publishing Group is among many scientific publishers that have done so.)

The major cost of irreproducibility does not lie with a faulty initial study, but with the wasted follow-on work that it inspires, says Lee Ellis, a cancer biologist at the University of Texas MD Anderson Cancer Center in Houston. Scientists may lose time pursuing false leads, which can delay discoveries that lead to new therapies.

Freedman hopes that his latest work will help to convince scientists that taking small steps, such as better documenting of protocols and the use of certified reagents, could produce large gains in reproducibility. “The message is less about ‘Oh my god, we’re flushing $20 billion down the toilet’,” he says, “and more about ‘Here is an opportunity to increase efficiencies to get more bang for the buck’.”


  1. Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. PLoS Biol. 13, e1002165 (2015).

    Article  Google Scholar 

  2. Freedman, L. P. & Inglese, J. Cancer Res. 74, 4024–4029 (2014).

    CAS  Article  Google Scholar 

  3. Ioannidis, J. P. A. PLoS Med. 2, e124 (2005).

    Article  Google Scholar 

  4. Begley, C. G. & Ellis, L. M. Nature 483, 531–533 (2012).

    ADS  CAS  Article  Google Scholar 

  5. Prinz, F., Schlange, T. & Asadullah, K. Nature Rev. Drug Discov. 10, 712 (2011).

    CAS  Article  Google Scholar 

Download references


Related links

Related links

Related links in Nature Research

Researchers argue for standard format to cite lab resources 2015-May-29

US societies push back against NIH reproducibility guidelines 2015-Apr-17

Policy: NIH plans to enhance reproducibility 2014-Jan-27

Independent labs to verify high-profile papers 2012-Aug-14

Drug development: Raise standards for preclinical cancer research 2012-Mar-28

Related external links

Challenges in Irreproducible Research

Global Biological Standards Institute

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Baker, M. Irreproducible biology research costs put at $28 billion per year. Nature (2015).

Download citation

  • Published:

  • DOI:

Further reading


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing