Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Nature Reviews Drug Discovery volume 21, pages 915–931 (2022)Cite this article

Subjects

Abstract

Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which ‘predictive validity’ is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Pharmaceutical R&D depends on selections made using a set of decision tools.
Fig. 2: Predictive validity, throughput and research and development decision performance.
Fig. 3: R&D productivity is very sensitive to predictive validity.
Fig. 4: Financial value of decision tool evaluation in pharmaceutical R&D.

Similar content being viewed by others

Code availability

The Mathematica code used to generate the decision-theoretical analyses shown in Figs. 2 and 3 is available in Supplementary Box 4.

References

  1. Morgan, P. et al. Impact of a five-dimensional framework on R&D productivity at AstraZeneca. Nat. Rev. Drug Discov. 17, 167–181 (2018).

    CAS  PubMed  Google Scholar 

  2. Plenge, R. M., Scolnick, E. M. & Altshuler, D. Validating therapeutic targets through human genetics. Nat. Rev. Drug Discov. 12, 581–594 (2013).

    CAS  PubMed  Google Scholar 

  3. Cook, D. et al. Lessons learned from the fate of AstraZeneca’s drug pipeline: a five-dimensional framework. Nat. Rev. Drug Discov. 13, 419–431 (2014).

    CAS  PubMed  Google Scholar 

  4. Emmerich, C. H. et al. Improving target assessment in biomedical research: the GOT-IT recommendations. Nat. Rev. Drug Discov. 20, 64–81 (2021).

    CAS  PubMed  Google Scholar 

  5. Edwards, A. M. et al. Preclinical target validation using patient-derived cells. Nat. Rev. Drug Discov. 14, 149–150 (2015).

    PubMed  Google Scholar 

  6. Scannell, J. W. & Bosley, J. When quality beats quantity: decision theory, drug discovery, and the reproducibility crisis. PLoS ONE 11, e0147215 (2016).

    PubMed  PubMed Central  Google Scholar 

  7. Paul, S. M. et al. How to improve R&D productivity: the pharmaceutical industry’s grand challenge. Nat. Rev. Drug Discov. 9, 203–214 (2010).

    CAS  PubMed  Google Scholar 

  8. Bender, A. & Cortes-Ciriano, I. Artificial intelligence in drug discovery: what is realistic, what are illusions? Part 2: a discussion of chemical and biological data. Drug Discov. Today 26, 1040–1052 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  9. Bender, A. & Cortés-Ciriano, I. Artificial intelligence in drug discovery: what is realistic, what are illusions? Part 1: ways to make an impact, and why we are not there yet. Drug Discov. Today 26, 511–524 (2021).

    CAS  PubMed  Google Scholar 

  10. Bycroft, C. et al. The UK Biobank resource with deep phenotyping and genomic data. Nature 562, 203–209 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  11. Bloom, N., Jones, C. I., van Reenen, J. & Webb, M. Are ideas getting harder to find? Am. Economic Rev. 110, 1104–1144 (2020).

    Google Scholar 

  12. Scannell, J. W., Blanckley, A., Boldon, H. & Warrington, B. Diagnosing the decline in pharmaceutical R&D efficiency. Nat. Rev. Drug Discov. 11, 191–200 (2012).

    CAS  PubMed  Google Scholar 

  13. Woodcock, J. & Woosley, R. The FDA critical path initiative and its influence on new drug development. Annu. Rev. Med. 59, 1–12 (2008).

    CAS  PubMed  Google Scholar 

  14. Horrobin, D. F. Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat. Rev. Drug Discov. 2, 151–154 (2003).

    CAS  PubMed  Google Scholar 

  15. Steward, F. & Wibberley, G. Drug innovation — what’s slowing it down? Nature 284, 118–120 (1980).

    CAS  PubMed  Google Scholar 

  16. Munos, B. Lessons from 60 years of pharmaceutical innovation. Nat. Rev. Drug Discov. 8, 959–968 (2009).

    CAS  PubMed  Google Scholar 

  17. Le Fanu, J. The Rise and Fall of Modern Medicine (Little Brown, 1999).

  18. Pammolli, F., Magazzini, L. & Riccaboni, M. The productivity crisis in pharmaceutical R&D. Nat. Rev. Drug Discov. 10, 428–438 (2011).

    CAS  PubMed  Google Scholar 

  19. Gordian, M., Singh, N., Zemmel, R. & Elias, T. Why products fail in phase III. Vivo 24, 49–56 (2006).

    Google Scholar 

  20. Kola, I. & Landis, J. Can the pharmaceutical industry reduce attrition rates? Nat. Rev. Drug Discov. 3, 711–715 (2004).

    CAS  PubMed  Google Scholar 

  21. Duncan, W. A. M. & Parsons, M. E. Reminiscences of the development of cimetidine. Gastroenterology 78, 620–625 (1980).

    CAS  PubMed  Google Scholar 

  22. Shih, H. P., Zhang, X. & Aronov, A. M. Drug discovery effectiveness from the standpoint of therapeutic mechanisms and indications. Nat. Rev. Drug Discov. 17, 19–33 (2018).

    CAS  PubMed  Google Scholar 

  23. Paull, K., Hodes, L. & Simon, R. M. Efficiency of antitumor screening relative to activity criteria. J. Natl Cancer Inst. 76, 1137–1142 (1986).

    CAS  PubMed  Google Scholar 

  24. Chabner, B. A. NCI-60 cell line screening: a radical departure in its time. J. Natl Cancer Inst. 108, djv388 (2016).

    PubMed  Google Scholar 

  25. Dykes, D. J. & Waud, W. R. In Tumor Models in Cancer Research (ed. Teicher, B. A.) 23–40 (Humana Press, 2002).

  26. Wong, C. H., Siah, K. W. & Lo, A. W. Estimation of clinical trial success rates and related parameters. Biostatistics 20, 273–286 (2019).

    PubMed  Google Scholar 

  27. DiMasi, J. A. Risks in new drug development: approval success rates for investigational drugs. Clin. Pharmacol. Ther. 69, 297–307 (2001).

    CAS  PubMed  Google Scholar 

  28. Baillar, J. C. & Gornik, H. L. Cancer undefeated. N. Engl. J. Med. 336, 1569–1574 (1997).

    Google Scholar 

  29. Morgan, G. W., Ward, R. & Barton, M. The contribution of cytotoxic chemotherapy to 5-year survival in adult malignancies. Clin. Oncol. 16, 549–560 (2004).

    Google Scholar 

  30. Cairns, J. The treatment of diseases and the war against cancer. Sci. Am. 253, 51–59 (1985).

    CAS  PubMed  Google Scholar 

  31. Leaf, C. The Truth in Small Doses: Why We’re Losing the War on Cancer – and How to Win It (Simon & Shuster, 2013).

  32. Baillar, J. C. & Smith, E. M. Progress against cancer? N. Engl. J. Med. 314, 1226–1232 (1986).

    Google Scholar 

  33. Schilsky, R. L. & Schnipper, L. E. Hans Christian Andersen and the value of new cancer treatments. J. Natl Cancer Inst. 110, 441–442 (2017).

    Google Scholar 

  34. Marquart, J., Chen, E. Y. & Prasad, V. Estimation of the percentage of US patients with cancer who benefit from genome-driven oncology. JAMA Oncol. 4, 1093–1098 (2018).

    PubMed  PubMed Central  Google Scholar 

  35. Lewis, K. Platforms for antibiotic discovery. Nat. Rev. Drug. Discov. 12, 371–387 (2013).

    CAS  PubMed  Google Scholar 

  36. Bentley, R. Different roads to discovery: prontosil (hence sulfa drugs) and penicillin (hence β-lactams). J. Ind. Microbiol. Biotechnol. 36, 775–786 (2009).

    CAS  PubMed  Google Scholar 

  37. da Cunha, B. R., Fonseca, L. P. & Calado, C. R. C. Antibiotic discovery: where have we come from, where do we go? Antibiotics 8, 45 (2019).

    Google Scholar 

  38. Payne, D. J., Gwynn, M. N., Holmes, D. J. & Pompliano, D. L. Drugs for bad bugs: confronting the challenges of antibacterial discovery. Nat. Rev. Drug Discov. 6, 29–40 (2007).

    CAS  PubMed  Google Scholar 

  39. Silver, L. L. Challenges of antibacterial discovery. Clin. Microbiol. Rev. 24, 71–109 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  40. Brown, E. D. & Wright, G. D. Antibacterial drug discovery in the resistance era. Nature 529, 336–343 (2016).

    CAS  PubMed  Google Scholar 

  41. Tommasi, R., Brown, D. G., Walkup, G. K., Manchester, J. I. & Miller, A. A. ESKAPEing the labyrinth of antibacterial discovery. Nat. Rev. Drug Discov. 14, 529–542 (2015).

    CAS  PubMed  Google Scholar 

  42. Lohmann, V. & Bartenschlager, R. On the history of hepatitis C virus cell culture systems. J. Med. Chem. 57, 1627–1642 (2014).

    CAS  PubMed  Google Scholar 

  43. Bartenschlager, R. Hepatitis C virus replicons: potential role for drug development. Nat. Rev. Drug Discov. 1, 911–916 (2002).

    CAS  PubMed  Google Scholar 

  44. Lohmann, V. Hepatitis C virus cell culture models: an encomium on basic research paving the road to therapy development. Med. Microbiol. Immunol. 208, 3–24 (2019).

    PubMed  Google Scholar 

  45. Meanwell, N. A. 2015 Philip S. Portoghese medicinal chemistry lectureship. Curing hepatitis C virus infection with direct-acting antiviral agents: the arc of a medicinal chemistry triumph. J. Med. Chem. 59, 7311–7351 (2016).

    CAS  PubMed  Google Scholar 

  46. Ringel, M. S., Scannell, J. W., Baedeker, M. & Schulze, U. Breaking Eroom’s Law. Nat. Rev. Drug Discov. 19, 833–834 (2020).

    CAS  PubMed  Google Scholar 

  47. Wu, S. S. et al. Reviving an R&D pipeline: a step change in the phase II success rate. Drug Discov. Today 26, 308–314 (2021).

    CAS  PubMed  Google Scholar 

  48. Morgan, P. et al. Can the flow of medicines be improved? Fundamental pharmacokinetic and pharmacological principles toward improving phase II survival. Drug Discov. Today 17, 419–424 (2012).

    CAS  PubMed  Google Scholar 

  49. Spiegelhalter, D. J., Abrams, K. R. & Myles, J. P. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Bayesian Approaches to Clinical Trials and Health-Care Evaluation (Wiley, 2004).

  50. O’Hagan, A., Stevens, J. W. & Campbell, M. J. Assurance in clinical trial design. Pharm. Stat. 4, 187–201 (2005).

    Google Scholar 

  51. Mauro, G. W., di Scala, L., Bretz, F. & Racine-Poon, A. Predictive probability of success in clinical drug development. Epidemiol. Biostat. Public Health https://doi.org/10.2427/8760 (2013).

    Article  Google Scholar 

  52. Senn, S. Statistical Issues in Drug Development. 2nd Edn (Wiley, 2008).

  53. Willan, A. R. & Pinto, E. M. The value of information and optimal clinical trial design. Stat. Med. 24, 1791–1806 (2005).

    PubMed  Google Scholar 

  54. Bacchetti, P., McCulloch, C. E. & Segal, M. R. Simple, defensible sample sizes based on cost efficiency. Biometrics 64, 577–585 (2008).

    PubMed  PubMed Central  Google Scholar 

  55. Bacchetti, P., Deeks, S. G. & McCune, J. M. Breaking free of sample size dogma to perform innovative translational research. Sci. Transl Med. 3, 87ps24 (2011).

    PubMed  PubMed Central  Google Scholar 

  56. Detsky, A. S. Using cost-effectiveness analysis to improve the efficiency of allocating funds to clinical trials. Stat. Med. 9, 173–184 (1990).

    CAS  PubMed  Google Scholar 

  57. Berry, D. A. A guide to drug discovery: Bayesian clinical trials. Nat. Rev. Drug Discov. 5, 27–36 (2006).

    CAS  PubMed  Google Scholar 

  58. Leach, A. R. & Gillet, V. J. An Introduction to Chemoinformatics (Springer, 2007).

  59. Ajay, A., Walters, W. P. & Murcko, M. A. Can we learn to distinguish between “drug-like” and “nondrug-like” molecules? J. Med. Chem. 41, 3314–3324 (1998).

    CAS  PubMed  Google Scholar 

  60. Sadowski, J. & Kubinyi, H. A scoring scheme for discriminating between drugs and nondrugs. J. Med. Chem. 41, 3325–3329 (1998).

    CAS  PubMed  Google Scholar 

  61. Zhang, J. H., Chung, T. D. Y. & Oldenburg, K. R. A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J. Biomol. Screen. 4, 67–73 (1999).

    CAS  PubMed  Google Scholar 

  62. Lipinski, C. A., Lombardo, F., Dominy, B. W. & Feeney, P. J. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Adv. Drug Deliv. Rev. 23, 3–25 (1997).

    CAS  Google Scholar 

  63. Lipinski, C. A. Rule of five in 2015 and beyond: target and ligand structural limitations, ligand chemistry structure and drug discovery project decisions. Adv. Drug Deliv. Rev. 101, 34–41 (2016).

    CAS  PubMed  Google Scholar 

  64. Walters, W. P. & Namchuk, M. Designing screens: how to make your hits a hit. Nat. Rev. Drug Discov. 2, 259–266 (2003).

    CAS  PubMed  Google Scholar 

  65. Bender, A. et al. Which aspects of HTS are empirically correlated with downstream success? Curr. Opin. Drug Discov. Dev. 11, 327–337 (2008).

    CAS  Google Scholar 

  66. Bickerton, G. R., Paolini, G. V., Besnard, J., Muresan, S. & Hopkins, A. L. Quantifying the chemical beauty of drugs. Nat. Chem. 4, 90–98 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  67. Cumming, J. G., Davis, A. M., Muresan, S., Haeberlein, M. & Chen, H. Chemical predictive modelling to improve compound quality. Nat. Rev. Drug Discov. 12, 948–962 (2013).

    CAS  PubMed  Google Scholar 

  68. Hopkins, A. L., Keserü, G. M., Leeson, P. D., Rees, D. C. & Reynolds, C. H. The role of ligand efficiency metrics in drug discovery. Nat. Rev. Drug. Discov. 13, 105–121 (2014).

    CAS  PubMed  Google Scholar 

  69. Drug Baron. Monte Carlo models of drug R&D focus attention on cutting costs – Part 1. DRUGBARON BLOG https://drugbaron.com/monte-carlo-models-of-drug-rd-focus-attention-on-cutting-costs-part-1/ (2013).

  70. Peck, R. W., Lendrem, D. W., Grant, I., Lendrem, B. C. & Isaacs, J. D. Why is it hard to terminate failing projects in pharmaceutical R&D? Nat. Rev. Drug Discov. 14, 663–664 (2015).

    CAS  PubMed  Google Scholar 

  71. Satopää, V. A., Salikhov, M., Tetlock, P. E. & Mellers, B. Bias, information, noise: the BIN model of forecasting. Manag. Sci. https://doi.org/10.1287/mnsc.2020.3882 (2021).

    Article  Google Scholar 

  72. Hinkle, D. E., Wiersma, W. & Jurs, S. G. Applied Statistics for the Behavioral Sciences (Houghton Mifflin, 2003).

  73. Akoglu, H. User’s guide to correlation coefficients. Turkish J. Emerg. Med. 18, 91–93 (2018).

    Google Scholar 

  74. StatisticsSolutions. Pearson’s Correlation Coefficient. https://www.statisticssolutions.com/pearsons-correlation-coefficient/ (2021).

  75. NCSS. Confidence intervals for Pearson’s correlation. PASS Sample Size Software https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Confidence_Intervals_for_Pearsons_Correlation.pdf.

  76. Plenge, R. M. Disciplined approach to drug discovery and early development. Sci. Transl Med. 8, 349ps15 (2016).

    PubMed  Google Scholar 

  77. Finan, C. et al. The druggable genome and support for target identification and validation in drug development. Sci. Transl Med. 9, eaag1166 (2017).

    PubMed  PubMed Central  Google Scholar 

  78. King, E. A., Wade Davis, J. & Degner, J. F. Are drug targets with genetic support twice as likely to be approved? Revised estimates of the impact of genetic support for drug mechanisms on the probability of drug approval. PLoS Genet. 15, e1008489 (2019).

    PubMed  PubMed Central  Google Scholar 

  79. Nelson, M. R. et al. The support of human genetic evidence for approved drug indications. Nat. Genet. 47, 856–860 (2015).

    CAS  PubMed  Google Scholar 

  80. Hingorani, A. D. et al. Improving the odds of drug development success through human genomics: modelling study. Sci. Rep. 9, 18911 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  81. Sittampalam, G. et al. Assay Guidance Manual (NIH, 2016).

  82. Williams, M., Mullane, K. & Curtis, M. J. In Research in the Biomedical Sciences: Transparent and Reproducible (eds Williams, M., Curtis, M. J. & Mullane, K.) 197–306 (Academic Press, 2018).

  83. Mullane, K. & Williams, M. Enhancing reproducibility: failures from reproducibility initiatives underline core challenges. Biochem. Pharmacol. 138, 7–18 (2017).

    CAS  PubMed  Google Scholar 

  84. Vollert, J. et al. Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals. BMJ Open. Sci. 4, e100046 (2020).

    PubMed  PubMed Central  Google Scholar 

  85. Perrin, S. Make mouse studies work. Nature 507, 423–425 (2014).

    PubMed  Google Scholar 

  86. Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015).

    PubMed  PubMed Central  Google Scholar 

  87. Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the arrive guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).

    PubMed  PubMed Central  Google Scholar 

  88. Snyder, H. M. et al. Guidelines to improve animal study design and reproducibility for Alzheimer’s disease and related dementias: for funders and researchers. Alzheimer’s Dement. 12, 1177–1185 (2016).

    Google Scholar 

  89. Lapchak, P. A., Zhang, J. H. & Noble-Haeusslein, L. J. RIGOR guidelines: escalating STAIR and STEPS for effective translational research. Transl. Stroke Res. 4, 279–285 (2013).

    PubMed  Google Scholar 

  90. Hooijmans, C. R. et al. SYRCLE’s risk of bias tool for animal studies. BMC Med. Res. Methodol. 14, 1–9 (2014).

    Google Scholar 

  91. Macleod, M. R. et al. Risk of bias in reports of in vivo research: a focus for improvement. PLoS Biol. 13, e1002273 (2015).

    PubMed  PubMed Central  Google Scholar 

  92. Horvath, P. et al. Screening out irrelevant cell-based models of disease. Nat. Rev. Drug Discov. 15, 751–769 (2016).

    CAS  PubMed  Google Scholar 

  93. Belzung, C. & Lemoine, M. Criteria of validity for animal models of psychiatric disorders: focus on anxiety disorders and depression. Biol. Mood Anxiety Disord. 1, 9 (2011).

    PubMed  PubMed Central  Google Scholar 

  94. Colquhoun, D. An investigation of the false discovery rate and the misinterpretation of p-values. R. Soc. Open Sci. 1, 140216 (2014).

    PubMed  PubMed Central  Google Scholar 

  95. Ioannidis, J. P. A. Why most published research findings are false modeling the framework for false positive findings. PLoS Med. 2, e124 (2005).

    PubMed  PubMed Central  Google Scholar 

  96. Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2006).

    Google Scholar 

  97. Chuang-Stein, C. et al. A quantitative approach for making Go/No-Go decisions in drug development. Ther. Innov. Regul. Sci. 45, 187–202 (2011).

    Google Scholar 

  98. Ferreira, G. S. et al. A standardised framework to identify optimal animal models for efficacy assessment in drug development. PLoS ONE 14, e0218014 (2019).

    CAS  PubMed  Google Scholar 

  99. Veening-Griffioen, D. H. et al. Tradition, not science, is the basis of animal model selection in translational and applied research. ALTEX 38, 49–62 (2021).

    PubMed  Google Scholar 

  100. Prescod-Weinstein, C. What does dark matter even do? N. Sci. 247, 24 (2020).

    Google Scholar 

  101. Hubbard, D. How to Measure Anything: Finding the Value of “Intangibles” in Business (Wiley, 2014).

  102. Funtowicz, S. O. & Ravetz, J. R. Uncertainty and Quality in Science for Policy (Kluwer, 1990).

  103. Dias, L., Morton, A. & Quigley, J. (eds) Elicitation: The Science and Art of Structuring Judgement International Series in Operations Research and Management Science vol. 261 (Springer, 2018).

  104. Tetlock, P. E. & Gardner, D. Superforecasting: the Art and Science of Prediction (Random House, 2015).

  105. Baudy, A. R. et al. Liver microphysiological systems development guidelines for safety risk assessment in the pharmaceutical industry. Lab Chip 20, 215–225 (2020).

    CAS  PubMed  Google Scholar 

  106. Gurusamy, K. S. et al. Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanation. PeerJ 9, e10673 (2021).

    PubMed  PubMed Central  Google Scholar 

  107. Collins, A., Ross, J. & Lang, S. H. A systematic review of the asymmetric inheritance of cellular organelles in eukaryotes: a critique of basic science validity and imprecision. PLoS ONE 12, e0178645 (2017).

    PubMed  PubMed Central  Google Scholar 

  108. Ekert, J. E. et al. Recommended guidelines for developing, qualifying, and implementing complex in vitro models (CIVMs) for drug discovery. SLAS Discov. 25, 1174–1190 (2020).

    CAS  PubMed  Google Scholar 

  109. Friedrich, C. M. A model qualification method for mechanistic physiological QSP models to support model-informed drug development. CPT Pharmacomet. Syst. Pharmacol. 5, 43–53 (2016).

    CAS  Google Scholar 

  110. Wehling, M. Assessing the translatability of drug projects: what needs to be scored to predict success? Nat. Rev. Drug Discov. 8, 541–546 (2009).

    CAS  PubMed  Google Scholar 

  111. Collins, A. T. & Lang, S. H. A systematic review of the validity of patient derived xenograft (PDX) models: the implications for translational research and personalised medicine. PeerJ 2018, e5981 (2018).

    Google Scholar 

  112. Willner, P. The validity of animal models of depression. Psychopharmacology 83, 1–16 (1984).

    CAS  PubMed  Google Scholar 

  113. Morgan, M. G. Use (and abuse) of expert elicitation in support of decision making for public policy. Proc. Natl Acad. Sci. USA 111, 7176–7184 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  114. Aspinall, W. A route to more tractable expert advice. Nature 463, 294–295 (2010).

    CAS  PubMed  Google Scholar 

  115. Cooke, R. M. Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, 1991).

  116. Chalmers, J. & Armour, M. In Handbook of Research Methods in Health Social Sciences (ed. Liamputtong, P.) 715–735 (Springer, 2019).

  117. Kahneman, D., Slovic, P. & Tversky, A. Judgment under Uncertainty: Heuristics and Biases (Cambridge University Press, 1982).

  118. Katsagounos, I., Thomakos, D. D., Litsiou, K. & Nikolopoulos, K. Superforecasting reality check: evidence from a small pool of experts and expedited identification. Eur. J. Oper. Res. 289, 107–117 (2021).

    PubMed  Google Scholar 

  119. Mellers, B. et al. Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psychol. Sci. 10, 267–281 (2015).

    PubMed  Google Scholar 

  120. Bar-Hillel, M. The base-rate fallacy in probability judgments. Acta Psychol. 44, 211–233 (1980).

    Google Scholar 

  121. Murphy, A. H. & Daan, H. Impacts of feedback and experience on the quality of subjective probability forecasts. Comparison of results from the first and second years of the Zierikzee experiment. Mon. Weather. Rev. 112, 413–423 (1984).

    Google Scholar 

  122. Montibeller, G. & von Winterfeldt, D. In Elicitation (eds Dias, L., Morton, A. & Quigley, J.) 377–392 International Series in Operations Research and Management Science vol. 261 (Springer, 2018).

  123. Gawande, A. The Checklist Manifesto (Henry Holt and Company, 2009).

  124. Kleinmuntz, D. N. Decomposition and the Control of Erros in Decision Analytic Models (Sloan School of Management, Massachusetts Institute of Technology, 1988).

  125. Henrion, M., Fischer, G. W. & Mullin, T. Divide and conquer? effects of decomposition on the accuracy and calibration of subjective probability distributions. Organ. Behav. Hum. Decis. Process. 55, 207–227 (1993).

    Google Scholar 

  126. Andradóttir, S. & Bier, V. M. An analysis of decomposition for subjective estimation in decision analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 28, 443–453 (1998).

    Google Scholar 

  127. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

    CAS  PubMed  Google Scholar 

  128. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712–713 (2011).

    CAS  PubMed  Google Scholar 

  129. Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. Br. Med. J. 334, 197–200 (2007).

    CAS  Google Scholar 

  130. Sena, E., Wheble, P., Sandercock, P. & Macleod, M. Systematic review and meta-analysis of the efficacy of tirilazad in experimental stroke. Stroke 38, 388–394 (2007).

    CAS  PubMed  Google Scholar 

  131. Howells, D. W., Sena, E. S. & Macleod, M. R. Bringing rigour to translational medicine. Nat. Rev. Neurol. 10, 37–43 (2014).

    CAS  PubMed  Google Scholar 

  132. Atkinson, M. A. Evaluating preclinical efficacy. Sci. Transl Med. 3, 96cm22 (2011).

    PubMed  Google Scholar 

  133. Reichlin, T. S., Vogt, L. & Würbel, H. The researchers’ view of scientific rigor-survey on the conduct and reporting of in vivo research. PLoS ONE 11, e0165999 (2016).

    PubMed  PubMed Central  Google Scholar 

  134. Landis, S. C. et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490, 187–191 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  135. Smith, A. J., Clutton, R. E., Lilley, E., Hansen, K. E. A. & Brattelid, T. PREPARE: guidelines for planning animal research and testing. Lab. Anim. 52, 135–141 (2018).

    CAS  PubMed  Google Scholar 

  136. Wendler, A. & Wehling, M. Translatability score revisited: differentiation for distinct disease areas. J. Transl Med. 15, 226 (2017).

    PubMed  PubMed Central  Google Scholar 

  137. Wendler, A. & Wehling, M. Translatability scoring in drug development: eight case studies. J. Transl Med. 10, 39 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  138. Voelkl, B., Vogt, L., Sena, E. S. & Würbel, H. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol. 16, e2003693 (2018).

    PubMed  PubMed Central  Google Scholar 

  139. Bodden, C. et al. Heterogenising study samples across testing time improves reproducibility of behavioural data. Sci. Rep. 9, 8247 (2019).

    PubMed  PubMed Central  Google Scholar 

  140. Helene Richter, S. Systematic heterogenization for better reproducibility in animal experimentation. Lab. Anim. 46, 343–349 (2017).

    Google Scholar 

  141. Ferreira, G. S. Tools to Enable Animal to Human Translation: Assessing the Value of Disease Models (Utrecht University, 2021).

  142. Okamoto, K. & Aoki, K. Development of a strain of spontaneously hypertensive rats. Jan. J. Circ. 27, 282–293 (1963).

    CAS  Google Scholar 

  143. Veening-Griffioen, D. H. et al. Are some animal models more equal than others? A case study on the translational value of animal models of efficacy for Alzheimer’s disease. Eur. J. Pharmacol. 859, 172524 (2019).

    CAS  PubMed  Google Scholar 

  144. Kahneman, D. & Tversky, A. On the psychology of prediction. Psychol. Rev. 80, 237–251 (1973).

    Google Scholar 

  145. Fabre, K. et al. Introduction to a manuscript series on the characterization and use of microphysiological systems (MPS) in pharmaceutical safety and ADME applications. Lab Chip 20, 1049–1057 (2020).

    PubMed  Google Scholar 

  146. Ewart, L. et al. Qualifying a human Liver-Chip for predictive toxicology: Performance assessment and economic implications. Preprint at BioRxiv https://doi.org/10.1101/2021.12.14.472674 (2022).

  147. Proctor, W. R. et al. Utility of spherical human liver microtissues for prediction of clinical drug-induced liver injury. Arch. Toxicol. 91, 2849–2863 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  148. Tetlock, P. E. Expert Political Judgment: how Good is it? How can we know? (Princeton University Press, 2005).

  149. Swinney, D. C. & Anthony, J. How were new medicines discovered? Nat. Rev. Drug Discov. 10, 507–519 (2011).

    CAS  PubMed  Google Scholar 

  150. Eder, J., Sedrani, R. & Wiesmann, C. The discovery of first-in-class drugs: origins and evolution. Nat. Rev. Drug Discov. 13, 577–587 (2014).

    CAS  PubMed  Google Scholar 

  151. Beck, H. & Yee, D. Minireview: were the IGF signaling inhibitors all bad? Mol. Endocrinol. 29, 1549–1557 (2015).

    Google Scholar 

  152. Baserga, R. The decline and fall of the IGF-I receptor. J. Cell. Physiol. 228, 675–679 (2013).

    CAS  PubMed  Google Scholar 

  153. Brookes, P. The early history of the biological alkylating agents, 1918–1968. Mutat. Res. 233, 3–14 (1990).

    CAS  PubMed  Google Scholar 

  154. De Vita, V. T. & Chu, E. A history of cancer chemotherapy. Cancer Res. 68, 8643–8653 (2008).

    CAS  PubMed  Google Scholar 

  155. Chabner, B. A. & Roberts, T. G. Chemotherapy and the war on cancer. Nat. Rev. Cancer 5, 65–72 (2005).

    CAS  PubMed  Google Scholar 

  156. Stehelin, D., Varmus, H. E., Bishop, J. M. & Vogt, P. K. DNA related to the transforming gene(s) of avian sarcoma viruses is present in normal avian DNA. Nature 260, 170–173 (1976).

    CAS  PubMed  Google Scholar 

  157. Varmus, H. E. The molecular genetics of cellular oncogenes. Annu. Rev. Genet. 18, 553–612 (1984).

    CAS  PubMed  Google Scholar 

  158. Morange, M. From the regulatory vision of cancer to the oncogene paradigm, 1975–1985. J. Hist. Biol. 30, 1–29 (1997).

    CAS  PubMed  Google Scholar 

  159. Bazell, R. Her-2: The Making of Herceptin, a Revolutionary Treatment for Breast Cancer (Random House, 1998).

  160. Mukherjee, S. The Emperor of All Maladies: A Biography of Cancer (Scribner, 2010).

  161. Weinstein, I. B. Cancer: addiction to oncogenes - the Achilles heal of cancer. Science 297, 63–64 (2002).

    CAS  PubMed  Google Scholar 

  162. Sawyers, C. L. Shifting paradigms: the seeds of oncogene addiction. Nat. Med. 15, 1158–1161 (2009).

    CAS  PubMed  Google Scholar 

  163. Kersten, K., Visser, K. E., Miltenburg, M. H. & Jonkers, J. Genetically engineered mouse models in oncology research and cancer medicine. EMBO Mol. Med. 9, 137–153 (2017).

    CAS  PubMed  Google Scholar 

  164. Hidalgo, M. et al. Patient-derived Xenograft models: an emerging platform for translational cancer research. Cancer Discov. 4, 998–1013 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  165. Bhimani, J., Ball, K. & Stebbing, J. Patient-derived xenograft models — the future of personalised cancer treatment. Br. J. Cancer 122, 601–602 (2020).

    PubMed  PubMed Central  Google Scholar 

  166. Bonekamp, N. A. et al. Small-molecule inhibitors of human mitochondrial DNA transcription. Nature 588, 712–716 (2020).

    CAS  PubMed  Google Scholar 

  167. Jin, X. et al. A metastasis map of human cancer cell lines. Nature 588, 331–336 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  168. Shawver, L. K., Slamon, D. & Ullrich, A. Smart drugs: tyrosine kinase inhibitors in cancer therapy. Cancer Cell 1, 117–123 (2002).

    CAS  PubMed  Google Scholar 

  169. Davis, C. et al. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009–13. BMJ 359, j4530 (2017).

    PubMed  PubMed Central  Google Scholar 

  170. Tannock, I. F. & Hickman, J. A. Limits to personalized cancer medicine. N. Engl. J. Med. 375, 1289–1294 (2016).

    PubMed  Google Scholar 

  171. Booth, C. M. & Del Paggio, J. C. Approvals in 2016: questioning the clinical benefit of anticancer therapies. Nat. Rev. Clin. Oncol. 14, 135–136 (2017).

    PubMed  Google Scholar 

  172. Hwang, T. J. et al. Efficacy, safety, and regulatory approval of food and drug administration–designated breakthrough and nonbreakthrough cancer medicines. J. Clin. Oncol. 36, 1805–1812 (2018).

    CAS  PubMed  Google Scholar 

  173. Prasad, V. Our best weapons against cancer are not magic bullets. Nature 577, 451 (2020).

    CAS  PubMed  Google Scholar 

  174. Haslam, A. & Prasad, V. Estimation of the percentage of US patients with cancer who are eligible for and respond to checkpoint inhibitor immunotherapy drugs. JAMA Netw. Open 2, e192535 (2019).

    PubMed  PubMed Central  Google Scholar 

  175. Middleton, G., Robbins, H., Andre, F. & Swanton, C. A state-of-the-art review of stratified medicine in cancer: towards a future precision medicine strategy in cancer. Ann. Oncol. 33, 143–157 (2022).

    CAS  PubMed  Google Scholar 

  176. Cherny, N. I. An appraisal of FDA approvals for adult solid tumours in 2017–2021: has the eagle landed? Nat. Rev. Clin. Oncol. 19, 486–492 (2022).

    PubMed  Google Scholar 

  177. Dagogo-Jack, I. & Shaw, A. T. Tumour heterogeneity and resistance to cancer therapies. Nat. Rev. Clin. Oncol. 15, 81–94 (2018).

    CAS  PubMed  Google Scholar 

  178. Hochhaus, A. et al. Long-term outcomes of imatinib treatment for chronic Myeloid leukemia. N. Engl. J. Med. 376, 917–927 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  179. Kesselheim, A. S. & Avorn, J. The most transformative drugs of the past 25 years: a survey of physicians. Nat. Rev. Drug Discov. 12, 425–431 (2013).

    CAS  PubMed  Google Scholar 

  180. Elliott, J. et al. ALK inhibitors for non-small cell lung cancer: a systematic review and network meta-analysis. PLoS ONE 15, e0229179 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  181. Shah, R. & Lester, J. F. Tyrosine kinase inhibitors for the treatment of EGFR mutation-positive non–small-cell lung cancer: a clash of the generations. Clin. Lung Cancer 21, e216–e228 (2020).

    CAS  PubMed  Google Scholar 

  182. Comen, E., Gilewski, T. A. & Norton, L. In Holland-Frei Cancer Medicine (eds Bast, R. C. et al.) 589–600 (Wiley, 2016).

  183. Skipper, H. E. The effects of chemotherapy on the kinetics of leukemic cell behavior. Cancer Res. 25, 1544–1550 (1965).

    CAS  PubMed  Google Scholar 

  184. Skipper, H. E. Kinetics of mammary tumor cell growth and implications for therapy. Cancer 28, 1479–1499 (1971).

    CAS  PubMed  Google Scholar 

  185. Kerr, J. F. R., Wyllie, A. H. & Currie, A. R. Apoptosis: a basic biological phenomenon with wide-ranging implications in tissue kinetics. Br. J. Cancer 26, 239–257 (1972).

    CAS  PubMed  PubMed Central  Google Scholar 

  186. Hickman, J. A. Apoptosis induced by anticancer drugs. Cancer Metastasis Rev. 11, 121–139 (1992).

    CAS  PubMed  Google Scholar 

  187. Strasser, A. & Vaux, D. L. Cell death in the origin and treatment of cancer. Mol. Cell 78, 1045–1054 (2020).

    CAS  PubMed  Google Scholar 

  188. Nowell, P. C. The clonal evolution of tumor cell populations. Science 194, 23–28 (1976).

    CAS  PubMed  Google Scholar 

  189. Loeb, L. A. Human cancers express mutator phenotypes: origin, consequences and targeting. Nat. Rev. Cancer 11, 450–457 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  190. Vendramin, R., Litchfield, K. & Swanton, C. Cancer evolution: Darwin and beyond. EMBO J. 40, e108389 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  191. Turajlic, S. & Swanton, C. Implications of cancer evolution for drug development. Nat. Rev. Drug Discov. 16, 441–442 (2017).

    CAS  PubMed  Google Scholar 

  192. Gerlinger, M. et al. Intratumor heterogeneity and branched evolution revealed by multiregion sequencing. N. Engl. J. Med. 366, 883–892 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  193. McGranahan, N. & Swanton, C. Clonal heterogeneity and tumor evolution: past, present, and the future. Cell 168, 613–628 (2017).

    CAS  PubMed  Google Scholar 

  194. Muthuswamy, S. K. Self-organization in cancer: implications for histopathology, cancer cell biology, and metastasis. Cancer Cell 39, 443–446 (2021).

    CAS  PubMed  Google Scholar 

  195. Hill, W., Caswell, D. R. & Swanton, C. Capturing cancer evolution using genetically engineered mouse models (GEMMs). Trends Cell Biol. 31, 1007–1018 (2021).

    CAS  PubMed  Google Scholar 

  196. Sotillo, R., Schvartzman, J. M., Socci, N. D. & Benezra, R. Mad2-induced chromosome instability leads to lung tumour relapse after oncogene withdrawal. Nature 464, 436–440 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  197. Takenaka, T., Yamazaki, K., Miura, N., Mori, R. & Takeo, S. The prognostic impact of tumor volume in patients with clinical stage IA non-small cell lung cancer. J. Thorac. Oncol. 11, 1074–1080 (2016).

    PubMed  Google Scholar 

  198. Jung, J., Seol, H. S. & Chang, S. The generation and application of patient-derived xenograft model for cancer research. Cancer Res. Treat. 50, 1–10 (2018).

    PubMed  Google Scholar 

  199. Eirew, P. et al. Dynamics of genomic clones in breast cancer patient xenografts at single-cell resolution. Nature 518, 422–426 (2015).

    CAS  PubMed  Google Scholar 

  200. Ben-David, U. et al. Patient-derived xenografts undergo mouse-specific tumor evolution. Nat. Genet. 49, 1567–1575 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  201. Greaves, M. & Maley, C. C. Clonal evolution in cancer. Nature 481, 306–313 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  202. EMA. ICH Guideline S9 on Nonclinical Evalution for Anticancer Pharmaceuticals (2010).

  203. FDA. Guidance for Industry: S9 Nonclinical Evaluation for Anticancer Pharmaceuticals (2010).

  204. Wieschowski, S. et al. Preclinical efficacy studies in investigator brochures: do they enable risk–benefit assessment? PLoS Biol. 16, e2004879 (2018).

    PubMed  PubMed Central  Google Scholar 

  205. Hickman, J. A. et al. Three-dimensional models of cancer for pharmacology and cancer cell biology: capturing tumor complexity in vitro/ex vivo. Biotechnol. J. 9, 1115–1128 (2014).

    CAS  PubMed  Google Scholar 

  206. Misra, S. et al. Ex vivo organotypic culture system of precision-cut slices of human pancreatic ductal adenocarcinoma. Sci. Rep. 9, 2133 (2019).

    PubMed  PubMed Central  Google Scholar 

  207. Sontheimer-Phelps, A., Hassell, B. A. & Ingber, D. E. Modelling cancer in microfluidic human organs-on-chips. Nat. Rev. Cancer 19, 65–81 (2019).

    CAS  PubMed  Google Scholar 

  208. Burian, R. M. How the choice of experimental organism matters: epistemological reflections on an aspect of biological practice. J. Hist. Biol. 26, 351–367 (1993).

    CAS  PubMed  Google Scholar 

  209. Vorrink, S. U., Zhou, Y., Ingelman-Sundberg, M. & Lauschke, V. M. Prediction of drug-induced hepatotoxicity using long-term stable primary hepatic 3D spheroid cultures in chemically defined conditions. Toxicol. Sci. 163, 655–665 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  210. Geeleher, P., Gamazon, E. R., Seoighe, C., Cox, N. J. & Huang, R. S. Consistency in large pharmacogenomic studies. Nature 540, E1–E2 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  211. Haibe-Kains, B. et al. Revisiting inconsistency in large pharmacogenomic studies. F1000Res. 5, 2333 (2017).

    PubMed Central  Google Scholar 

  212. Haibe-Kains, B. et al. Inconsistency in large pharmacogenomic studies. Nature 504, 389–393 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  213. Haverty, P. M. et al. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 533, 333–337 (2016).

    CAS  PubMed  Google Scholar 

  214. Mayr, L. M. & Bojanic, D. Novel trends in high-throughput screening. Curr. Opin. Pharmacol. 9, 580–588 (2009).

    CAS  PubMed  Google Scholar 

  215. Pedró-Rosa, L. et al. Identification of potent inhibitors of the trypanosoma brucei methionyl-tRNA synthetase via high-throughput orthogonal screening. J. Biomol. Screen. 20, 122–130 (2015).

    PubMed  Google Scholar 

  216. Mondritzki, T. Prädiktive Wertigkeit verschiedener präklinischer Outcome-Parameter für eine erfolgreiche versus nicht-erfolgreiche klinische Entwicklung von Arzneimitteln zur Behandlung der Herzinsuffizienz (Donau-Universität-Krems, 2014).

  217. Moser, J. & Verdin, P. Burgeoning oncology pipeline raises questions about sustainability. Nat. Rev. Drug Discov. 17, 698–699 (2018).

    CAS  PubMed  Google Scholar 

  218. Ten Years On: Measuring the Return from Pharmaceutical Innovation (Deloitte, 2019); https://www2.deloitte.com/content/dam/Deloitte/uk/Documents/life-sciences-health-care/deloitte-uk-ten-years-on-measuring-return-on-pharma-innovation-report-2019.pdf.

  219. Skardal, A. et al. Multi-tissue interactions in an integrated three-tissue organ-on-a-chip platform. Sci. Rep. 7, 8837 (2017).

    PubMed  PubMed Central  Google Scholar 

  220. Maurer, T. Model-based discovery and development of novel therapies for type-2 diabetes mellitus. In Bridging Bench and Bedside with Quantitative Model-Based Translational Pharmacology symposium (New York Academy of Science, 2012).

  221. Bertau, M., Mosekilde, E. & Westerhoff, H. V. Biosimulation in Drug Development (Wiley-VCH, 2008).

  222. Mager, D. E. & Kimko, H. H. C. Systems Pharmacology and Pharmacodynamics. AAPS Advances in the Pharmaceutical Sciences Series (Springer International Publishing, 2016).

  223. Helmlinger, G. et al. Quantitative systems pharmacology: an exemplar model-building workflow with applications in cardiovascular, metabolic, and oncology drug development. CPT Pharmacomet. Syst. Pharmacol. 8, 380–395 (2019).

    CAS  Google Scholar 

  224. Mardinoglu, A. et al. The potential use of metabolic cofactors in treatment of NAFLD. Nutrients 11, 1578 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  225. Jørgensen, P. G. et al. Cardiac adaptation in hibernating, free-ranging Scandinavian Brown Bears (Ursus arctos). Sci. Rep. 10, 247 (2020).

    PubMed  PubMed Central  Google Scholar 

  226. Peretti, D. et al. RBM3 mediates structural plasticity and protective effects of cooling in neurodegeneration. Nature 518, 236–239 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  227. Krystal, A. D. et al. A randomized proof-of-mechanism trial applying the ‘fast-fail’ approach to evaluating κ-opioid antagonism as a treatment for anhedonia. Nat. Med. 26, 760–768 (2020).

    CAS  PubMed  Google Scholar 

  228. Grabb, M. C., Hillefors, M. & Potter, W. Z. The NIMH ‘Fast-Fail Trials’ (FAST) initiative: rationale, promise, and progress. Pharm. Med. 34, 233–245 (2020).

    Google Scholar 

  229. Krystal, A. D. et al. The first implementation of the NIMH FAST-FAIL approach to psychiatric drug development. Nat. Rev. Drug. Discov. 18, 82–84 (2018).

    PubMed  PubMed Central  Google Scholar 

  230. Cuthbert, B. N. & Insel, T. R. Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med. 11, 126 (2013).

    PubMed  PubMed Central  Google Scholar 

  231. Insel, T. et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am. J. Psychiatry 167, 748–751 (2010).

    PubMed  Google Scholar 

  232. Cuthbert, B. N. The PRISM project: social withdrawal from an RDoC perspective. Neurosci. Biobehav. Rev. 97, 34–37 (2019).

    PubMed  Google Scholar 

  233. Sullivan, P. F. et al. A mega-analysis of genome-wide association studies for major depressive disorder. Mol. Psychiatry 18, 497–511 (2013).

    PubMed  Google Scholar 

  234. Healy, D. The Antidepressant Era (Harvard University Press, 1997).

  235. Liu, K. S. et al. Is bigger better for depression trials? J. Psychiatr. Res. 42, 622–630 (2008).

    PubMed  Google Scholar 

  236. FDA. Major Depressive Disorder: Developing Drugs for Treatment Guidance for Industry (2018).

  237. Nelson, R. R. The simple economics of basic scientific research. J. Polit. Econ. 67, 297–306 (1959).

    Google Scholar 

  238. Arrow, K. J. In Readings in Industrial Economics (ed. Rowley, C. K.) 219–236 (Palgrave, 1972).

  239. Dobson, C. M. Chemical space and biology. Nature 432, 824–828 (2004).

    CAS  PubMed  Google Scholar 

  240. Lipinski, C. & Hopkins, A. Navigating chemical space for biology and medicine. Nature 432, 855–861 (2004).

    CAS  PubMed  Google Scholar 

  241. Seymore, S. B. Making patents useful. Minn. Law Rev. 98, 1046–1109 (2014).

    Google Scholar 

  242. Billette de Villemeur, E. & Versaevel, B. One lab, two firms, many possibilities: on R&D outsourcing in the biopharmaceutical industry. J. Health Econ. 65, 260–283 (2019).

    PubMed  Google Scholar 

  243. Hoofnagle, J. H. & Sherker, A. H. Therapy for hepatitis C — the costs of success. N. Engl. J. Med. 370, 1552–1553 (2014).

    CAS  PubMed  Google Scholar 

  244. WSJ. Senate Committee Is Investigating Pricing of Hepatitis C Drug. Wall Street Journal https://www.wsj.com/articles/senate-finance-committee-is-investigating-pricing-of-hepatitis-c-drug-1405109206 (2014).

  245. Morrison, C. 2019 biotech IPOs: party on. Nat. Rev. Drug Discov. 19, 6–9 (2020).

    CAS  PubMed  Google Scholar 

  246. Morrison, C. Boom: 2018’s biotech IPOs. Nat. Rev. Drug Discov. 18, 3–6 (2018).

    PubMed  Google Scholar 

  247. Williamson, A. R. Creating a structural genomics consortium. Nat. Struct. Biol. 7, 953 (2000).

    CAS  PubMed  Google Scholar 

  248. Vaudano, E. The innovative medicines initiative: a public private partnership model to foster drug discovery. Comput. Struct. Biotechnol. J. 6, e201303017 (2013).

    PubMed  PubMed Central  Google Scholar 

  249. Ochoa, D. et al. Open targets platform: supporting systematic drug-target identification and prioritisation. Nucleic Acids Res. 49, D1302–D1310 (2021).

    CAS  PubMed  Google Scholar 

  250. Dolgin, E. Massive NIH–industry project opens portals to target validation. Nat. Rev. Drug Discov. 18, 240–242 (2019).

    Google Scholar 

  251. Holden, A. L. The SNP consortium: summary of a private consortium effort to develop an applied map of the human genome. BioTechniques 32, S22–S26 (2002).

    Google Scholar 

  252. Holden, A. L., Contreras, J. L., John, S. & Nelson, M. R. The international serious adverse events consortium. Nat. Rev. Drug Discov. 13, 795–796 (2014).

    CAS  PubMed  Google Scholar 

  253. Contreras, J. L. & Vertinsky, L. S. Pre-competition. North Carol. Law Rev. 95, 67–131 (2016).

    Google Scholar 

  254. Lundqvist, B. Joint research and development collaborations under competition law, with a layman’s economic viewpoint. SSRN Electron. J. https://doi.org/10.2139/ssrn.2913840 (2017).

    Article  Google Scholar 

  255. Sams-Dodd, F. Strategies to optimize the validity of disease models in the drug discovery process. Drug Discov. Today 11, 355–363 (2006).

    CAS  PubMed  Google Scholar 

  256. Hooijmans, C. R., De Vries, R., Leenaars, M., Curfs, J. & Ritskes-Hoitinga, M. Improving planning, design, reporting and scientific quality of animal experiments by using the Gold Standard Publication Checklist, in addition to the ARRIVE guidelines. Br. J. Pharmacol. 162, 1259–1260 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  257. Fisher, M. Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 30, 2752–2758 (1999).

    Google Scholar 

  258. Vettoretti, M., Facchinetti, A., Sparacino, G. & Cobelli, C. Predicting insulin treatment scenarios with the net effect method: domain of validity. Diabetes Technol. Ther. 18, 694–704 (2016).

    CAS  PubMed  Google Scholar 

  259. Sacca, L., Toffolo, G. & Cobelli, C. V-A and A-V modes in whole body and regional kinetics: Domain of validity from a physiological model. Am. J. Physiol. Endocrinol. Metab. 263, E597–E606 (1992).

    CAS  Google Scholar 

  260. Rudolf, A. F., Skovgaard, T., Knapp, S., Jensen, L. J. & Berthelsen, J. A comparison of protein kinases inhibitor screening methods using both enzymatic activity and binding affinity determination. PLoS ONE 9, e98800 (2014).

    PubMed  PubMed Central  Google Scholar 

  261. Brown, D. G., May-Dracka, T. L., Gagnon, M. M. & Tommasi, R. Trends and exceptions of physical properties on antibacterial activity for Gram-positive and Gram-negative pathogens. J. Med. Chem. 57, 10144–10161 (2014).

    CAS  PubMed  Google Scholar 

  262. Zoffmann, S. et al. Machine learning-powered antibiotics phenotypic drug discovery. Sci. Rep. 9, 5013 (2019).

    PubMed  PubMed Central  Google Scholar 

  263. Farha, M. A. & Brown, E. D. Unconventional screening approaches for antibiotic discovery. Ann. NY Acad. Sci. 1354, 54–66 (2015).

    PubMed  Google Scholar 

  264. Yokokawa, F. Recent Progress on the Development of Novel Antitubercular Agents from Whole-Cell Screening Hits. J. Synth. Org. Chem. Jpn. 72, 1239–1249 (2014).

    CAS  Google Scholar 

  265. Horscroft, N. et al. Replicon cell culture system as a valuable tool in antiviral drug discovery against hepatitis C virus. Antivir. Chem. Chemother. 16, 1–12 (2005).

    CAS  PubMed  Google Scholar 

  266. Woerz, I., Lohmann, V. & Bartenschlager, R. Hepatitis C virus replicons: dinosaurs still in business? J. Viral Hepat. 16, 1–9 (2009).

    PubMed Central  Google Scholar 

  267. Kaplan, G. & Racaniello, V. R. Construction and characterization of poliovirus subgenomic replicons. J. Virol. 62, 1687–1696 (1988).

    CAS  PubMed  PubMed Central  Google Scholar 

  268. Khromykh, A. A. & Westaway, E. G. Subgenomic replicons of the flavivirus Kunjin: construction and applications. J. Virol. 71, 1497–1505 (1997).

    CAS  PubMed  PubMed Central  Google Scholar 

  269. Behrens, S.-E., Grassmann, C. W., Thiel, H.-J., Meyers, G. & Tautz, N. Characterization of an autonomous subgenomic pestivirus RNA replicon. J. Virol. 72, 2364–2372 (1998).

    CAS  PubMed  PubMed Central  Google Scholar 

  270. Lohmann, V. et al. Replication of subgenomic hepatitis C virus RNAs in a hepatoma cell line. Science 285, 110–113 (1999).

    CAS  PubMed  Google Scholar 

  271. Chung, R. T. & Baumert, T. F. Curing chronic hepatitis C — the arc of a medical triumph. N. Engl. J. Med. 370, 1576–1578 (2014).

    CAS  PubMed  Google Scholar 

  272. Sena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R. & Howells, D. W. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J. Cereb. Blood Flow. Metab. 34, 737–742 (2014).

    PubMed  PubMed Central  Google Scholar 

  273. Macleod, M. R., O’Collins, T., Howells, D. W. & Donnan, G. A. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 35, 1203–1208 (2004).

    PubMed  Google Scholar 

  274. MacLeod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).

    PubMed  Google Scholar 

  275. Macleod, M. R. et al. Good laboratory practice: preventing introduction of bias at the bench. Stroke 40, e50–e52 (2009).

    PubMed  Google Scholar 

  276. Dirnagl, U., Iadecola, C. & Moskowitz, M. A. Pathobiology of ischaemic stroke: an integrated view. Trends Neurosci. 22, 391–397 (1999).

    CAS  PubMed  Google Scholar 

  277. Dronne, M. A., Grenier, E., Chapuisat, G., Hommel, M. & Boissel, J. P. A modelling approach to explore some hypotheses of the failure of neuroprotective trials in ischemic stroke patients. Prog. Biophys. Mol. Biol. 97, 60–78 (2008).

    PubMed  Google Scholar 

  278. Choi, D. W. Excitotoxicity: still hammering the Ischemic brain in 2020. Front. Neurosci. 14, 579953 (2020).

    PubMed  PubMed Central  Google Scholar 

  279. Orset, C. et al. Efficacy of Alteplase in a mouse model of acute Ischemic stroke: a retrospective pooled analysis. Stroke 47, 1312–1318 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  280. Phipps, M. S. & Cronin, C. A. Management of acute ischemic stroke. BMJ 368, l6983 (2020).

    PubMed  Google Scholar 

Download references

Acknowledgements

The authors thank A. Morton and A. Colson at the Department of Management Science at Strathclyde University for their help on the decision tool evaluation section; B. Versaevel and E. Billet de Villemeur, at EMLYON Business School and at Université des Sciences et Technologies de Lille, respectively, for their help on the economics of decision tool-related innovation; and L. Ewart, EVP Science at Emulate, for her help with toxicology-related content. They thank M. Todd at the UCL School of Pharmacy for helpful comments on an earlier draft of the paper. Finally, they thank the reviewers, whose thoughtful comments substantially improved the paper.

Author information

Authors and Affiliations

  1. Science, Technology, and Innovation Studies, University of Edinburgh, Edinburgh, UK

    Jack W. Scannell

  2. JW Scannell Analytics Ltd, Edinburgh, UK

    Jack W. Scannell

  3. NovaDiscovery SA, Lyon, France

    James Bosley

  4. School of Biological Sciences, Manchester University, Manchester, UK

    John A. Hickman

  5. P1vital Ltd, Wallingford, UK

    Gerard R. Dawson

  6. The Knowledge House GmbH, Dusseldorf, Germany

    Hubert Truebel

  7. 3D-PharmXchange, Tilden, Netherlands

    Guilherme S. Ferreira

  8. Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK

    Duncan Richards

  9. Talisman Therapeutics Ltd, Cambridge, UK

    J. Mark Treherne

Authors
  1. Jack W. Scannell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James Bosley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John A. Hickman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerard R. Dawson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hubert Truebel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guilherme S. Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Duncan Richards
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. J. Mark Treherne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jack W. Scannell.

Ethics declarations

Competing interests

J.W.S. is a director and shareholder of JW Scannell Analytics, which sells consulting services to the biopharmaceutical and financial services sectors, including to firms that are commercializing screening and disease models; is CEO of Unify Pharmaceuticals Corp; is Director of Etheros Pharmaceuticals Corp; and holds equity options in Ochre Bio. J.B. is an employee of Novadiscovery and holds equity options in the firm. G.R.D. is a director and major shareholder of P1vital and P1vital Products Ltd. P1vital provides clinical research services to the pharmaceutical industry. P1vital Products provides data management and digital technology to the pharmaceutical industry and health-care providers. H.T. is employed by AiCuris AG, owns shares in Bayer AG, Evotec SE, Cyclerion Therapeutics, Inc. and recently incorporated www.knowledge-house.com. G.S.F. has received fees from Merck KGaA and Curare Consulting; has been employed by the Access to Medicine Foundation, 3D PharmXchange and Janssen Biologics; and is now employed by GlaxoSmithKline. D.R. is a consultant to OMASS Therapeutics Ltd and is on the scientific advisory board of Avicenna Biosciences, Inc. J.M.T. is a director and/or shareholder of Talisman Therapeutics, Gen2 Neuroscience, Avilex Pharma, Domain Therapeutics, Cellesce, Ubiquigent and Monument Therapeutics. J.A.H. declares no competing interests.

Peer review

Peer review information

Nature Reviews Drug Discovery thanks Neil Carragher, Matthew Nelson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

DataFAIRy Bioassay Annotation: https://www.pistoiaalliance.org/projects/current-projects/datafairy-bioassay-annotation/

Sample size calculator from Statistics Kingdom: https://www.statskingdom.com/sample_size_regression.html

Springer Nature Experiments: https://experiments.springernature.com/

The Collaborative Adverse Outcome Pathway Wiki: https://aopwiki.org/

Supplementary information

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Scannell, J.W., Bosley, J., Hickman, J.A. et al. Predictive validity in drug discovery: what it is, why it matters and how to improve it. Nat Rev Drug Discov 21, 915–931 (2022). https://doi.org/10.1038/s41573-022-00552-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41573-022-00552-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research