Scientists in the United States spend $28 billion each year on basic biomedical research that cannot be repeated successfully. That is the conclusion of a study published on 9 June in PLoS Biology1 that attempts to quantify the causes, and costs, of irreproducibility.
John Ioannidis, an epidemiologist at Stanford University in California who studies scientific robustness, says that the analysis is sure to prompt discussion about the problem — but should be taken with a pinch of salt, given that its estimates carry great uncertainty.
But Len Freedman, the study’s lead author and head of the non-profit Global Biological Standards Institute in Washington DC, says that the work is of value, even though it cannot pin down the size of the problem. “Clearly, there are tremendous inefficiences [in research], and this is putting a spotlight on that,” says Freedman, whose group seeks to develop best practices for biological experiments.
Freedman and his colleagues defined irreproducibility broadly, as being any errors or omissions that prevent attempts to replicate experimental findings. The researchers sought to determine the relative contributions of four sources of irreproducibility identified in an earlier analysis2: study design; laboratory protocols; biological reagents and reference materials; and data analysis and reporting.
The team did not examine whether the results of any individual paper were reproducible, however. Instead, the researchers surveyed existing analyses of factors that contribute to irreproducibility, using these to estimate an overall reproducibility rate — and to weigh the relative influences of various factors.
Iain Cockburn, an economist at Boston University in Massachusetts and a co-author of the study, says that the analysis was limited by the quality and amount of data available. For example, assessments of poor materials were based on estimates of how often cell lines are misidentified. In other cases, the team extrapolated from reports from clinical work.
Overall, the team found that poor materials made the largest contribution to reproducibility problems, at 36%, followed by study design at 28% and data analysis at 26%. The team estimates the overall rate of irreproducibility at 53%, but cautions that the true rate could be anywhere between 18% and 89%. That puts the potential economic cost of irreproducibility anywhere from $10 billion to $50 billion per year.
“The four categories are decent, but the estimates are off,” says Ioannidis. “I would put a much higher rate on the data analysis and reporting component.”
The analysis is further limited by the fact that the four contributing factors that it uses are interdependent, says Melissa Haendel, an information scientist at Oregon Health and Science University in Portland. For example, research materials and the controls used to assess them are part of laboratory protocols, and study design affects data analysis.
Concerns about reproducibility in biomedical research have increased over the past decade, partly in response to a 2005 study by Ioannidis that found scientific journals to be biased towards publishing flashy, positive results3. And researchers at pharmaceutical companies have reported that their attempts to replicate the conclusions of peer-reviewed papers fail at rates upwards of 75%4, 5.
Several institutions are taking steps to address the problem. In November, for example, the US National Institutes of Health urged journals to adopt guidelines aimed at boosting reproducibility. (Nature Publishing Group is among many scientific publishers that have done so.)
The major cost of irreproducibility does not lie with a faulty initial study, but with the wasted follow-on work that it inspires, says Lee Ellis, a cancer biologist at the University of Texas MD Anderson Cancer Center in Houston. Scientists may lose time pursuing false leads, which can delay discoveries that lead to new therapies.
Freedman hopes that his latest work will help to convince scientists that taking small steps, such as better documenting of protocols and the use of certified reagents, could produce large gains in reproducibility. “The message is less about ‘Oh my god, we’re flushing $20 billion down the toilet’,” he says, “and more about ‘Here is an opportunity to increase efficiencies to get more bang for the buck’.”
- Journal name: