Comment | Published:

Medical research

Trial unpredictability yields predictable therapy gains

Nature volume 500, pages 395396 (22 August 2013) | Download Citation

In decades of clinical-trial data, new treatments are better than standard ones just over half the time. That's as it should be, say Benjamin Djulbegovic and colleagues.

The effects of a new treatment are hard to predict. In the 1940s, pathologist Sidney Farber theorized that folate might help children with acute lymphoblastic leukaemia by stimulating blood cells; instead, he was surprised to find that the leukaemia cells proliferated. That 'failure' led him to try anti-folate drugs. These did lead to remissions, and were the first example of successful chemotherapy for cancer. In the 2000s, physicians thought that corticosteroids might help to reduce brain swelling following trauma, but randomized trials showed that they actually increased mortality. Surprisingly, however, corticosteroids were found to reduce the death rate in meningitis1.

Image: CLAIRE WELSH/NATURE

These are just a few examples of the steady therapeutic advances delivered over the past half century by the randomized controlled trial (RCT) system. But these improvements can feel frustratingly slow for patients and physicians. There is much soul searching in the drug-discovery community about how progress could be made more quickly2.

Here we provide empirical evidence that the system's success rate is optimal. We analysed hundreds of trials, published and unpublished, public and industry funded, involving hundreds of thousands of people over several decades. We find that just over half the time, RCTs show that new treatments are better than existing ones.

This success rate is incremental, but maintains a system that has served us well and is founded on the ethical and scientific necessity that the results of individual RCTs should not be predictable. We contend that the use of RCTs for assessing the effects of new treatments should not be fundamentally altered without analogous evidence that replacement systems will, on average, outperform them.

Genuine uncertainty

Better drugs and therapies have come about because people participating in phase III trials are willing to be randomly allocated to new or existing treatments. Phase III trials are typically the final step in evaluating treatment efficacy. They are usually preceded by phase I trials that assess how a drug is metabolized, excreted and tolerated, and phase II trials that gather preliminary data on efficacy. Although phases I and II can occasionally identify new treatments with dramatic effects, thus obviating the need for further testing3, phase III trials are usually required to judge whether new treatments are superior to existing ones.

On ethical as well as scientific grounds, RCTs should be done only when there are genuine uncertainties about the relative merits of alternative treatments4. If there were a high likelihood (say, more than 80%) that one of the treatments in a comparison was better than the other, it would be ethically unsound to deny some patients access to the superior treatment, and even if such a trial got past an ethics committee, well-informed patients would probably refuse to participate. In other words, if the results were predictable, the system of RCTs as we know it would cease4. Progress in therapeutics has occurred precisely because science and ethics require that the results of individual RCTs are not predictable.

Because this 'uncertainty requirement' — variously referred to as 'equipoise', 'the uncertainty principle' or 'the indifference principle'4 — is insufficiently appreciated by the public, patients, research funders and investigators, we set out to test its long-term impact by calculating the average likelihood of a proposed new treatment being superior to established ones5.

We conducted an analysis of 860 published and unpublished phase III RCTs performed by academics or pharmaceutical companies in six consecutive series of trials with a total of more than 350,000 patients: four series of 743 publicly sponsored trials over the past 50 years6, and two series of 117 publicly and commercially sponsored clinical trials over the past 30 years7 (see 'The best medicine'). Our results show that the probability of finding that a new treatment is better than a standard treatment is about 50–60%, confirming the theoretical predictions we made more than 15 years ago4,5.

We found that in publicly sponsored RCTs, the likelihood that new treatments would work better than existing ones ranges from 57% to 63% for patient survival and from 55% to 66% for all primary outcomes (such as survival without recurrence of disease, response to treatment, symptom frequency and measures of disability). The only available comparable rates for industry-sponsored RCTs show that, overall, new treatments are superior to existing treatments for measures of morbidity (nausea, for example) in 75% of trials, but similar (53%) for survival7. Over time, the pattern in all trials has converged at around 50% (probably because earlier studies used inferior comparators) and applies across various clinical fields and types of treatment6,7.

Maximum gain

Philosophers of science have suggested that discovery in science happens most rapidly when only one or a few hypotheses are tested at a time8. The RCT system is paradigmatic of this approach. It has generated incremental advances that, together, translate into important improvements in health and lifespan. For example, five decades of controlled experimentation have seen cure rates for childhood leukaemia improve from 0% to more than 80% (ref. 6), yet in testing, only 2–5% of novel treatments have provided a breakthrough.

There is still room for improving existing practices for clinical trials. There is substantial avoidable waste in designing, conducting and reporting medical research9. For example, the results of only around 50% of RCTs are published — negative results and most industry trials remain hidden. The rigour of randomized trials can also be improved, for example by systematically taking into account all relevant previous research.

But our results show that the development of new treatments has been possible because the trials were done when unpredictability was greatest — in other words, when there was the most to gain6,7. The observed distribution of treatment successes is not an accident. There is a predictable relationship between the uncertainty requirement (the moral principle) on which trials are based and the outcomes of clinical trials4.

In summary, our retrospective view of more than 50 years of randomized trials shows that they remain the 'indispensable ordeals' through which biomedical researchers' responsibility to patients and the public is manifested10. These trials may need a tweak and polish, but they're not broken.

References

  1. 1.

    , , & Testing Treatments: Better Research for Better Healthcare 2nd edn (Pinter & Martin, 2011).

  2. 2.

    , & (eds) Evolution of Translational Omics: Lessons Learned and the Path Forward (National Academies Press, 2012).

  3. 3.

    , , & Br. Med. J. 334, 349–351 (2007).

  4. 4.

    J. Med. Philos. 32, 79–98 (2007).

  5. 5.

    Br. Med. J. 314, 74–75 (1997).

  6. 6.

    et al. Cochrane Database Syst. Rev. 10, MR000024 (2012).

  7. 7.

    et al. PLoS ONE 8, e58711 (2013).

  8. 8.

    Science 146, 347–353 (1964).

  9. 9.

    & The Lancet 374, 86–89, (2009).

  10. 10.

    Control. Clin. Trials 1, 263–267 (1980).

Download references

Author information

Affiliations

  1. Benjamin Djulbegovic is professor in the Department of Internal Medicine, University of South Florida (USF), Tampa, and at the H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida, USA.

    • Benjamin Djulbegovic
  2. Ambuj Kumar is associate professor at the Department of Internal Medicine, USF and at the H. Lee Moffitt Cancer Center and Research Institute.

    • Ambuj Kumar
  3. Paul Glasziou is professor of evidence-based medicine at Bond University, Robina, Australia.

    • Paul Glasziou
  4. Branko Miladinovic is assistant professor in the Department of Internal Medicine, USF.

    • Branko Miladinovic
  5. Iain Chalmers is coordinator of the James Lind Initiative, Oxford, UK.

    • Iain Chalmers

Authors

  1. Search for Benjamin Djulbegovic in:

  2. Search for Ambuj Kumar in:

  3. Search for Paul Glasziou in:

  4. Search for Branko Miladinovic in:

  5. Search for Iain Chalmers in:

Corresponding author

Correspondence to Benjamin Djulbegovic.

About this article

Publication history

Published

DOI

https://doi.org/10.1038/500395a

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Newsletter Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing