Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which ‘predictive validity’ is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.
This is a preview of subscription content, access via your institution
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $6.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Morgan, P. et al. Impact of a five-dimensional framework on R&D productivity at AstraZeneca. Nat. Rev. Drug Discov. 17, 167–181 (2018).
Plenge, R. M., Scolnick, E. M. & Altshuler, D. Validating therapeutic targets through human genetics. Nat. Rev. Drug Discov. 12, 581–594 (2013).
Cook, D. et al. Lessons learned from the fate of AstraZeneca’s drug pipeline: a five-dimensional framework. Nat. Rev. Drug Discov. 13, 419–431 (2014).
Emmerich, C. H. et al. Improving target assessment in biomedical research: the GOT-IT recommendations. Nat. Rev. Drug Discov. 20, 64–81 (2021).
Edwards, A. M. et al. Preclinical target validation using patient-derived cells. Nat. Rev. Drug Discov. 14, 149–150 (2015).
Scannell, J. W. & Bosley, J. When quality beats quantity: decision theory, drug discovery, and the reproducibility crisis. PLoS ONE 11, e0147215 (2016).
Paul, S. M. et al. How to improve R&D productivity: the pharmaceutical industry’s grand challenge. Nat. Rev. Drug Discov. 9, 203–214 (2010).
Bender, A. & Cortes-Ciriano, I. Artificial intelligence in drug discovery: what is realistic, what are illusions? Part 2: a discussion of chemical and biological data. Drug Discov. Today 26, 1040–1052 (2021).
Bender, A. & Cortés-Ciriano, I. Artificial intelligence in drug discovery: what is realistic, what are illusions? Part 1: ways to make an impact, and why we are not there yet. Drug Discov. Today 26, 511–524 (2021).
Bycroft, C. et al. The UK Biobank resource with deep phenotyping and genomic data. Nature 562, 203–209 (2018).
Bloom, N., Jones, C. I., van Reenen, J. & Webb, M. Are ideas getting harder to find? Am. Economic Rev. 110, 1104–1144 (2020).
Scannell, J. W., Blanckley, A., Boldon, H. & Warrington, B. Diagnosing the decline in pharmaceutical R&D efficiency. Nat. Rev. Drug Discov. 11, 191–200 (2012).
Woodcock, J. & Woosley, R. The FDA critical path initiative and its influence on new drug development. Annu. Rev. Med. 59, 1–12 (2008).
Horrobin, D. F. Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat. Rev. Drug Discov. 2, 151–154 (2003).
Steward, F. & Wibberley, G. Drug innovation — what’s slowing it down? Nature 284, 118–120 (1980).
Munos, B. Lessons from 60 years of pharmaceutical innovation. Nat. Rev. Drug Discov. 8, 959–968 (2009).
Le Fanu, J. The Rise and Fall of Modern Medicine (Little Brown, 1999).
Pammolli, F., Magazzini, L. & Riccaboni, M. The productivity crisis in pharmaceutical R&D. Nat. Rev. Drug Discov. 10, 428–438 (2011).
Gordian, M., Singh, N., Zemmel, R. & Elias, T. Why products fail in phase III. Vivo 24, 49–56 (2006).
Kola, I. & Landis, J. Can the pharmaceutical industry reduce attrition rates? Nat. Rev. Drug Discov. 3, 711–715 (2004).
Duncan, W. A. M. & Parsons, M. E. Reminiscences of the development of cimetidine. Gastroenterology 78, 620–625 (1980).
Shih, H. P., Zhang, X. & Aronov, A. M. Drug discovery effectiveness from the standpoint of therapeutic mechanisms and indications. Nat. Rev. Drug Discov. 17, 19–33 (2018).
Paull, K., Hodes, L. & Simon, R. M. Efficiency of antitumor screening relative to activity criteria. J. Natl Cancer Inst. 76, 1137–1142 (1986).
Chabner, B. A. NCI-60 cell line screening: a radical departure in its time. J. Natl Cancer Inst. 108, djv388 (2016).
Dykes, D. J. & Waud, W. R. In Tumor Models in Cancer Research (ed. Teicher, B. A.) 23–40 (Humana Press, 2002).
Wong, C. H., Siah, K. W. & Lo, A. W. Estimation of clinical trial success rates and related parameters. Biostatistics 20, 273–286 (2019).
DiMasi, J. A. Risks in new drug development: approval success rates for investigational drugs. Clin. Pharmacol. Ther. 69, 297–307 (2001).
Baillar, J. C. & Gornik, H. L. Cancer undefeated. N. Engl. J. Med. 336, 1569–1574 (1997).
Morgan, G. W., Ward, R. & Barton, M. The contribution of cytotoxic chemotherapy to 5-year survival in adult malignancies. Clin. Oncol. 16, 549–560 (2004).
Cairns, J. The treatment of diseases and the war against cancer. Sci. Am. 253, 51–59 (1985).
Leaf, C. The Truth in Small Doses: Why We’re Losing the War on Cancer – and How to Win It (Simon & Shuster, 2013).
Baillar, J. C. & Smith, E. M. Progress against cancer? N. Engl. J. Med. 314, 1226–1232 (1986).
Schilsky, R. L. & Schnipper, L. E. Hans Christian Andersen and the value of new cancer treatments. J. Natl Cancer Inst. 110, 441–442 (2017).
Marquart, J., Chen, E. Y. & Prasad, V. Estimation of the percentage of US patients with cancer who benefit from genome-driven oncology. JAMA Oncol. 4, 1093–1098 (2018).
Lewis, K. Platforms for antibiotic discovery. Nat. Rev. Drug. Discov. 12, 371–387 (2013).
Bentley, R. Different roads to discovery: prontosil (hence sulfa drugs) and penicillin (hence β-lactams). J. Ind. Microbiol. Biotechnol. 36, 775–786 (2009).
da Cunha, B. R., Fonseca, L. P. & Calado, C. R. C. Antibiotic discovery: where have we come from, where do we go? Antibiotics 8, 45 (2019).
Payne, D. J., Gwynn, M. N., Holmes, D. J. & Pompliano, D. L. Drugs for bad bugs: confronting the challenges of antibacterial discovery. Nat. Rev. Drug Discov. 6, 29–40 (2007).
Silver, L. L. Challenges of antibacterial discovery. Clin. Microbiol. Rev. 24, 71–109 (2011).
Brown, E. D. & Wright, G. D. Antibacterial drug discovery in the resistance era. Nature 529, 336–343 (2016).
Tommasi, R., Brown, D. G., Walkup, G. K., Manchester, J. I. & Miller, A. A. ESKAPEing the labyrinth of antibacterial discovery. Nat. Rev. Drug Discov. 14, 529–542 (2015).
Lohmann, V. & Bartenschlager, R. On the history of hepatitis C virus cell culture systems. J. Med. Chem. 57, 1627–1642 (2014).
Bartenschlager, R. Hepatitis C virus replicons: potential role for drug development. Nat. Rev. Drug Discov. 1, 911–916 (2002).
Lohmann, V. Hepatitis C virus cell culture models: an encomium on basic research paving the road to therapy development. Med. Microbiol. Immunol. 208, 3–24 (2019).
Meanwell, N. A. 2015 Philip S. Portoghese medicinal chemistry lectureship. Curing hepatitis C virus infection with direct-acting antiviral agents: the arc of a medicinal chemistry triumph. J. Med. Chem. 59, 7311–7351 (2016).
Ringel, M. S., Scannell, J. W., Baedeker, M. & Schulze, U. Breaking Eroom’s Law. Nat. Rev. Drug Discov. 19, 833–834 (2020).
Wu, S. S. et al. Reviving an R&D pipeline: a step change in the phase II success rate. Drug Discov. Today 26, 308–314 (2021).
Morgan, P. et al. Can the flow of medicines be improved? Fundamental pharmacokinetic and pharmacological principles toward improving phase II survival. Drug Discov. Today 17, 419–424 (2012).
Spiegelhalter, D. J., Abrams, K. R. & Myles, J. P. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Bayesian Approaches to Clinical Trials and Health-Care Evaluation (Wiley, 2004).
O’Hagan, A., Stevens, J. W. & Campbell, M. J. Assurance in clinical trial design. Pharm. Stat. 4, 187–201 (2005).
Mauro, G. W., di Scala, L., Bretz, F. & Racine-Poon, A. Predictive probability of success in clinical drug development. Epidemiol. Biostat. Public Health https://doi.org/10.2427/8760 (2013).
Senn, S. Statistical Issues in Drug Development. 2nd Edn (Wiley, 2008).
Willan, A. R. & Pinto, E. M. The value of information and optimal clinical trial design. Stat. Med. 24, 1791–1806 (2005).
Bacchetti, P., McCulloch, C. E. & Segal, M. R. Simple, defensible sample sizes based on cost efficiency. Biometrics 64, 577–585 (2008).
Bacchetti, P., Deeks, S. G. & McCune, J. M. Breaking free of sample size dogma to perform innovative translational research. Sci. Transl Med. 3, 87ps24 (2011).
Detsky, A. S. Using cost-effectiveness analysis to improve the efficiency of allocating funds to clinical trials. Stat. Med. 9, 173–184 (1990).
Berry, D. A. A guide to drug discovery: Bayesian clinical trials. Nat. Rev. Drug Discov. 5, 27–36 (2006).
Leach, A. R. & Gillet, V. J. An Introduction to Chemoinformatics (Springer, 2007).
Ajay, A., Walters, W. P. & Murcko, M. A. Can we learn to distinguish between “drug-like” and “nondrug-like” molecules? J. Med. Chem. 41, 3314–3324 (1998).
Sadowski, J. & Kubinyi, H. A scoring scheme for discriminating between drugs and nondrugs. J. Med. Chem. 41, 3325–3329 (1998).
Zhang, J. H., Chung, T. D. Y. & Oldenburg, K. R. A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J. Biomol. Screen. 4, 67–73 (1999).
Lipinski, C. A., Lombardo, F., Dominy, B. W. & Feeney, P. J. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Adv. Drug Deliv. Rev. 23, 3–25 (1997).
Lipinski, C. A. Rule of five in 2015 and beyond: target and ligand structural limitations, ligand chemistry structure and drug discovery project decisions. Adv. Drug Deliv. Rev. 101, 34–41 (2016).
Walters, W. P. & Namchuk, M. Designing screens: how to make your hits a hit. Nat. Rev. Drug Discov. 2, 259–266 (2003).
Bender, A. et al. Which aspects of HTS are empirically correlated with downstream success? Curr. Opin. Drug Discov. Dev. 11, 327–337 (2008).
Bickerton, G. R., Paolini, G. V., Besnard, J., Muresan, S. & Hopkins, A. L. Quantifying the chemical beauty of drugs. Nat. Chem. 4, 90–98 (2012).
Cumming, J. G., Davis, A. M., Muresan, S., Haeberlein, M. & Chen, H. Chemical predictive modelling to improve compound quality. Nat. Rev. Drug Discov. 12, 948–962 (2013).
Hopkins, A. L., Keserü, G. M., Leeson, P. D., Rees, D. C. & Reynolds, C. H. The role of ligand efficiency metrics in drug discovery. Nat. Rev. Drug. Discov. 13, 105–121 (2014).
Drug Baron. Monte Carlo models of drug R&D focus attention on cutting costs – Part 1. DRUGBARON BLOG https://drugbaron.com/monte-carlo-models-of-drug-rd-focus-attention-on-cutting-costs-part-1/ (2013).
Peck, R. W., Lendrem, D. W., Grant, I., Lendrem, B. C. & Isaacs, J. D. Why is it hard to terminate failing projects in pharmaceutical R&D? Nat. Rev. Drug Discov. 14, 663–664 (2015).
Satopää, V. A., Salikhov, M., Tetlock, P. E. & Mellers, B. Bias, information, noise: the BIN model of forecasting. Manag. Sci. https://doi.org/10.1287/mnsc.2020.3882 (2021).
Hinkle, D. E., Wiersma, W. & Jurs, S. G. Applied Statistics for the Behavioral Sciences (Houghton Mifflin, 2003).
Akoglu, H. User’s guide to correlation coefficients. Turkish J. Emerg. Med. 18, 91–93 (2018).
StatisticsSolutions. Pearson’s Correlation Coefficient. https://www.statisticssolutions.com/pearsons-correlation-coefficient/ (2021).
NCSS. Confidence intervals for Pearson’s correlation. PASS Sample Size Software https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Confidence_Intervals_for_Pearsons_Correlation.pdf.
Plenge, R. M. Disciplined approach to drug discovery and early development. Sci. Transl Med. 8, 349ps15 (2016).
Finan, C. et al. The druggable genome and support for target identification and validation in drug development. Sci. Transl Med. 9, eaag1166 (2017).
King, E. A., Wade Davis, J. & Degner, J. F. Are drug targets with genetic support twice as likely to be approved? Revised estimates of the impact of genetic support for drug mechanisms on the probability of drug approval. PLoS Genet. 15, e1008489 (2019).
Nelson, M. R. et al. The support of human genetic evidence for approved drug indications. Nat. Genet. 47, 856–860 (2015).
Hingorani, A. D. et al. Improving the odds of drug development success through human genomics: modelling study. Sci. Rep. 9, 18911 (2019).
Sittampalam, G. et al. Assay Guidance Manual (NIH, 2016).
Williams, M., Mullane, K. & Curtis, M. J. In Research in the Biomedical Sciences: Transparent and Reproducible (eds Williams, M., Curtis, M. J. & Mullane, K.) 197–306 (Academic Press, 2018).
Mullane, K. & Williams, M. Enhancing reproducibility: failures from reproducibility initiatives underline core challenges. Biochem. Pharmacol. 138, 7–18 (2017).
Vollert, J. et al. Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals. BMJ Open. Sci. 4, e100046 (2020).
Perrin, S. Make mouse studies work. Nature 507, 423–425 (2014).
Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015).
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the arrive guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).
Snyder, H. M. et al. Guidelines to improve animal study design and reproducibility for Alzheimer’s disease and related dementias: for funders and researchers. Alzheimer’s Dement. 12, 1177–1185 (2016).
Lapchak, P. A., Zhang, J. H. & Noble-Haeusslein, L. J. RIGOR guidelines: escalating STAIR and STEPS for effective translational research. Transl. Stroke Res. 4, 279–285 (2013).
Hooijmans, C. R. et al. SYRCLE’s risk of bias tool for animal studies. BMC Med. Res. Methodol. 14, 1–9 (2014).
Macleod, M. R. et al. Risk of bias in reports of in vivo research: a focus for improvement. PLoS Biol. 13, e1002273 (2015).
Horvath, P. et al. Screening out irrelevant cell-based models of disease. Nat. Rev. Drug Discov. 15, 751–769 (2016).
Belzung, C. & Lemoine, M. Criteria of validity for animal models of psychiatric disorders: focus on anxiety disorders and depression. Biol. Mood Anxiety Disord. 1, 9 (2011).
Colquhoun, D. An investigation of the false discovery rate and the misinterpretation of p-values. R. Soc. Open Sci. 1, 140216 (2014).
Ioannidis, J. P. A. Why most published research findings are false modeling the framework for false positive findings. PLoS Med. 2, e124 (2005).
Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2006).
Chuang-Stein, C. et al. A quantitative approach for making Go/No-Go decisions in drug development. Ther. Innov. Regul. Sci. 45, 187–202 (2011).
Ferreira, G. S. et al. A standardised framework to identify optimal animal models for efficacy assessment in drug development. PLoS ONE 14, e0218014 (2019).
Veening-Griffioen, D. H. et al. Tradition, not science, is the basis of animal model selection in translational and applied research. ALTEX 38, 49–62 (2021).
Prescod-Weinstein, C. What does dark matter even do? N. Sci. 247, 24 (2020).
Hubbard, D. How to Measure Anything: Finding the Value of “Intangibles” in Business (Wiley, 2014).
Funtowicz, S. O. & Ravetz, J. R. Uncertainty and Quality in Science for Policy (Kluwer, 1990).
Dias, L., Morton, A. & Quigley, J. (eds) Elicitation: The Science and Art of Structuring Judgement International Series in Operations Research and Management Science vol. 261 (Springer, 2018).
Tetlock, P. E. & Gardner, D. Superforecasting: the Art and Science of Prediction (Random House, 2015).
Baudy, A. R. et al. Liver microphysiological systems development guidelines for safety risk assessment in the pharmaceutical industry. Lab Chip 20, 215–225 (2020).
Gurusamy, K. S. et al. Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanation. PeerJ 9, e10673 (2021).
Collins, A., Ross, J. & Lang, S. H. A systematic review of the asymmetric inheritance of cellular organelles in eukaryotes: a critique of basic science validity and imprecision. PLoS ONE 12, e0178645 (2017).
Ekert, J. E. et al. Recommended guidelines for developing, qualifying, and implementing complex in vitro models (CIVMs) for drug discovery. SLAS Discov. 25, 1174–1190 (2020).
Friedrich, C. M. A model qualification method for mechanistic physiological QSP models to support model-informed drug development. CPT Pharmacomet. Syst. Pharmacol. 5, 43–53 (2016).
Wehling, M. Assessing the translatability of drug projects: what needs to be scored to predict success? Nat. Rev. Drug Discov. 8, 541–546 (2009).
Collins, A. T. & Lang, S. H. A systematic review of the validity of patient derived xenograft (PDX) models: the implications for translational research and personalised medicine. PeerJ 2018, e5981 (2018).
Willner, P. The validity of animal models of depression. Psychopharmacology 83, 1–16 (1984).
Morgan, M. G. Use (and abuse) of expert elicitation in support of decision making for public policy. Proc. Natl Acad. Sci. USA 111, 7176–7184 (2014).
Aspinall, W. A route to more tractable expert advice. Nature 463, 294–295 (2010).
Cooke, R. M. Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, 1991).
Chalmers, J. & Armour, M. In Handbook of Research Methods in Health Social Sciences (ed. Liamputtong, P.) 715–735 (Springer, 2019).
Kahneman, D., Slovic, P. & Tversky, A. Judgment under Uncertainty: Heuristics and Biases (Cambridge University Press, 1982).
Katsagounos, I., Thomakos, D. D., Litsiou, K. & Nikolopoulos, K. Superforecasting reality check: evidence from a small pool of experts and expedited identification. Eur. J. Oper. Res. 289, 107–117 (2021).
Mellers, B. et al. Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psychol. Sci. 10, 267–281 (2015).
Bar-Hillel, M. The base-rate fallacy in probability judgments. Acta Psychol. 44, 211–233 (1980).
Murphy, A. H. & Daan, H. Impacts of feedback and experience on the quality of subjective probability forecasts. Comparison of results from the first and second years of the Zierikzee experiment. Mon. Weather. Rev. 112, 413–423 (1984).
Montibeller, G. & von Winterfeldt, D. In Elicitation (eds Dias, L., Morton, A. & Quigley, J.) 377–392 International Series in Operations Research and Management Science vol. 261 (Springer, 2018).
Gawande, A. The Checklist Manifesto (Henry Holt and Company, 2009).
Kleinmuntz, D. N. Decomposition and the Control of Erros in Decision Analytic Models (Sloan School of Management, Massachusetts Institute of Technology, 1988).
Henrion, M., Fischer, G. W. & Mullin, T. Divide and conquer? effects of decomposition on the accuracy and calibration of subjective probability distributions. Organ. Behav. Hum. Decis. Process. 55, 207–227 (1993).
Andradóttir, S. & Bier, V. M. An analysis of decomposition for subjective estimation in decision analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 28, 443–453 (1998).
Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712–713 (2011).
Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. Br. Med. J. 334, 197–200 (2007).
Sena, E., Wheble, P., Sandercock, P. & Macleod, M. Systematic review and meta-analysis of the efficacy of tirilazad in experimental stroke. Stroke 38, 388–394 (2007).
Howells, D. W., Sena, E. S. & Macleod, M. R. Bringing rigour to translational medicine. Nat. Rev. Neurol. 10, 37–43 (2014).
Atkinson, M. A. Evaluating preclinical efficacy. Sci. Transl Med. 3, 96cm22 (2011).
Reichlin, T. S., Vogt, L. & Würbel, H. The researchers’ view of scientific rigor-survey on the conduct and reporting of in vivo research. PLoS ONE 11, e0165999 (2016).
Landis, S. C. et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490, 187–191 (2012).
Smith, A. J., Clutton, R. E., Lilley, E., Hansen, K. E. A. & Brattelid, T. PREPARE: guidelines for planning animal research and testing. Lab. Anim. 52, 135–141 (2018).
Wendler, A. & Wehling, M. Translatability score revisited: differentiation for distinct disease areas. J. Transl Med. 15, 226 (2017).
Wendler, A. & Wehling, M. Translatability scoring in drug development: eight case studies. J. Transl Med. 10, 39 (2012).
Voelkl, B., Vogt, L., Sena, E. S. & Würbel, H. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol. 16, e2003693 (2018).
Bodden, C. et al. Heterogenising study samples across testing time improves reproducibility of behavioural data. Sci. Rep. 9, 8247 (2019).
Helene Richter, S. Systematic heterogenization for better reproducibility in animal experimentation. Lab. Anim. 46, 343–349 (2017).
Ferreira, G. S. Tools to Enable Animal to Human Translation: Assessing the Value of Disease Models (Utrecht University, 2021).
Okamoto, K. & Aoki, K. Development of a strain of spontaneously hypertensive rats. Jan. J. Circ. 27, 282–293 (1963).
Veening-Griffioen, D. H. et al. Are some animal models more equal than others? A case study on the translational value of animal models of efficacy for Alzheimer’s disease. Eur. J. Pharmacol. 859, 172524 (2019).
Kahneman, D. & Tversky, A. On the psychology of prediction. Psychol. Rev. 80, 237–251 (1973).
Fabre, K. et al. Introduction to a manuscript series on the characterization and use of microphysiological systems (MPS) in pharmaceutical safety and ADME applications. Lab Chip 20, 1049–1057 (2020).
Ewart, L. et al. Qualifying a human Liver-Chip for predictive toxicology: Performance assessment and economic implications. Preprint at BioRxiv https://doi.org/10.1101/2021.12.14.472674 (2022).
Proctor, W. R. et al. Utility of spherical human liver microtissues for prediction of clinical drug-induced liver injury. Arch. Toxicol. 91, 2849–2863 (2017).
Tetlock, P. E. Expert Political Judgment: how Good is it? How can we know? (Princeton University Press, 2005).
Swinney, D. C. & Anthony, J. How were new medicines discovered? Nat. Rev. Drug Discov. 10, 507–519 (2011).
Eder, J., Sedrani, R. & Wiesmann, C. The discovery of first-in-class drugs: origins and evolution. Nat. Rev. Drug Discov. 13, 577–587 (2014).
Beck, H. & Yee, D. Minireview: were the IGF signaling inhibitors all bad? Mol. Endocrinol. 29, 1549–1557 (2015).
Baserga, R. The decline and fall of the IGF-I receptor. J. Cell. Physiol. 228, 675–679 (2013).
Brookes, P. The early history of the biological alkylating agents, 1918–1968. Mutat. Res. 233, 3–14 (1990).
De Vita, V. T. & Chu, E. A history of cancer chemotherapy. Cancer Res. 68, 8643–8653 (2008).
Chabner, B. A. & Roberts, T. G. Chemotherapy and the war on cancer. Nat. Rev. Cancer 5, 65–72 (2005).
Stehelin, D., Varmus, H. E., Bishop, J. M. & Vogt, P. K. DNA related to the transforming gene(s) of avian sarcoma viruses is present in normal avian DNA. Nature 260, 170–173 (1976).
Varmus, H. E. The molecular genetics of cellular oncogenes. Annu. Rev. Genet. 18, 553–612 (1984).
Morange, M. From the regulatory vision of cancer to the oncogene paradigm, 1975–1985. J. Hist. Biol. 30, 1–29 (1997).
Bazell, R. Her-2: The Making of Herceptin, a Revolutionary Treatment for Breast Cancer (Random House, 1998).
Mukherjee, S. The Emperor of All Maladies: A Biography of Cancer (Scribner, 2010).
Weinstein, I. B. Cancer: addiction to oncogenes - the Achilles heal of cancer. Science 297, 63–64 (2002).
Sawyers, C. L. Shifting paradigms: the seeds of oncogene addiction. Nat. Med. 15, 1158–1161 (2009).
Kersten, K., Visser, K. E., Miltenburg, M. H. & Jonkers, J. Genetically engineered mouse models in oncology research and cancer medicine. EMBO Mol. Med. 9, 137–153 (2017).
Hidalgo, M. et al. Patient-derived Xenograft models: an emerging platform for translational cancer research. Cancer Discov. 4, 998–1013 (2014).
Bhimani, J., Ball, K. & Stebbing, J. Patient-derived xenograft models — the future of personalised cancer treatment. Br. J. Cancer 122, 601–602 (2020).
Bonekamp, N. A. et al. Small-molecule inhibitors of human mitochondrial DNA transcription. Nature 588, 712–716 (2020).
Jin, X. et al. A metastasis map of human cancer cell lines. Nature 588, 331–336 (2020).
Shawver, L. K., Slamon, D. & Ullrich, A. Smart drugs: tyrosine kinase inhibitors in cancer therapy. Cancer Cell 1, 117–123 (2002).
Davis, C. et al. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009–13. BMJ 359, j4530 (2017).
Tannock, I. F. & Hickman, J. A. Limits to personalized cancer medicine. N. Engl. J. Med. 375, 1289–1294 (2016).
Booth, C. M. & Del Paggio, J. C. Approvals in 2016: questioning the clinical benefit of anticancer therapies. Nat. Rev. Clin. Oncol. 14, 135–136 (2017).
Hwang, T. J. et al. Efficacy, safety, and regulatory approval of food and drug administration–designated breakthrough and nonbreakthrough cancer medicines. J. Clin. Oncol. 36, 1805–1812 (2018).
Prasad, V. Our best weapons against cancer are not magic bullets. Nature 577, 451 (2020).
Haslam, A. & Prasad, V. Estimation of the percentage of US patients with cancer who are eligible for and respond to checkpoint inhibitor immunotherapy drugs. JAMA Netw. Open 2, e192535 (2019).
Middleton, G., Robbins, H., Andre, F. & Swanton, C. A state-of-the-art review of stratified medicine in cancer: towards a future precision medicine strategy in cancer. Ann. Oncol. 33, 143–157 (2022).
Cherny, N. I. An appraisal of FDA approvals for adult solid tumours in 2017–2021: has the eagle landed? Nat. Rev. Clin. Oncol. 19, 486–492 (2022).
Dagogo-Jack, I. & Shaw, A. T. Tumour heterogeneity and resistance to cancer therapies. Nat. Rev. Clin. Oncol. 15, 81–94 (2018).
Hochhaus, A. et al. Long-term outcomes of imatinib treatment for chronic Myeloid leukemia. N. Engl. J. Med. 376, 917–927 (2017).
Kesselheim, A. S. & Avorn, J. The most transformative drugs of the past 25 years: a survey of physicians. Nat. Rev. Drug Discov. 12, 425–431 (2013).
Elliott, J. et al. ALK inhibitors for non-small cell lung cancer: a systematic review and network meta-analysis. PLoS ONE 15, e0229179 (2020).
Shah, R. & Lester, J. F. Tyrosine kinase inhibitors for the treatment of EGFR mutation-positive non–small-cell lung cancer: a clash of the generations. Clin. Lung Cancer 21, e216–e228 (2020).
Comen, E., Gilewski, T. A. & Norton, L. In Holland-Frei Cancer Medicine (eds Bast, R. C. et al.) 589–600 (Wiley, 2016).
Skipper, H. E. The effects of chemotherapy on the kinetics of leukemic cell behavior. Cancer Res. 25, 1544–1550 (1965).
Skipper, H. E. Kinetics of mammary tumor cell growth and implications for therapy. Cancer 28, 1479–1499 (1971).
Kerr, J. F. R., Wyllie, A. H. & Currie, A. R. Apoptosis: a basic biological phenomenon with wide-ranging implications in tissue kinetics. Br. J. Cancer 26, 239–257 (1972).
Hickman, J. A. Apoptosis induced by anticancer drugs. Cancer Metastasis Rev. 11, 121–139 (1992).
Strasser, A. & Vaux, D. L. Cell death in the origin and treatment of cancer. Mol. Cell 78, 1045–1054 (2020).
Nowell, P. C. The clonal evolution of tumor cell populations. Science 194, 23–28 (1976).
Loeb, L. A. Human cancers express mutator phenotypes: origin, consequences and targeting. Nat. Rev. Cancer 11, 450–457 (2011).
Vendramin, R., Litchfield, K. & Swanton, C. Cancer evolution: Darwin and beyond. EMBO J. 40, e108389 (2021).
Turajlic, S. & Swanton, C. Implications of cancer evolution for drug development. Nat. Rev. Drug Discov. 16, 441–442 (2017).
Gerlinger, M. et al. Intratumor heterogeneity and branched evolution revealed by multiregion sequencing. N. Engl. J. Med. 366, 883–892 (2012).
McGranahan, N. & Swanton, C. Clonal heterogeneity and tumor evolution: past, present, and the future. Cell 168, 613–628 (2017).
Muthuswamy, S. K. Self-organization in cancer: implications for histopathology, cancer cell biology, and metastasis. Cancer Cell 39, 443–446 (2021).
Hill, W., Caswell, D. R. & Swanton, C. Capturing cancer evolution using genetically engineered mouse models (GEMMs). Trends Cell Biol. 31, 1007–1018 (2021).
Sotillo, R., Schvartzman, J. M., Socci, N. D. & Benezra, R. Mad2-induced chromosome instability leads to lung tumour relapse after oncogene withdrawal. Nature 464, 436–440 (2010).
Takenaka, T., Yamazaki, K., Miura, N., Mori, R. & Takeo, S. The prognostic impact of tumor volume in patients with clinical stage IA non-small cell lung cancer. J. Thorac. Oncol. 11, 1074–1080 (2016).
Jung, J., Seol, H. S. & Chang, S. The generation and application of patient-derived xenograft model for cancer research. Cancer Res. Treat. 50, 1–10 (2018).
Eirew, P. et al. Dynamics of genomic clones in breast cancer patient xenografts at single-cell resolution. Nature 518, 422–426 (2015).
Ben-David, U. et al. Patient-derived xenografts undergo mouse-specific tumor evolution. Nat. Genet. 49, 1567–1575 (2017).
Greaves, M. & Maley, C. C. Clonal evolution in cancer. Nature 481, 306–313 (2012).
EMA. ICH Guideline S9 on Nonclinical Evalution for Anticancer Pharmaceuticals (2010).
FDA. Guidance for Industry: S9 Nonclinical Evaluation for Anticancer Pharmaceuticals (2010).
Wieschowski, S. et al. Preclinical efficacy studies in investigator brochures: do they enable risk–benefit assessment? PLoS Biol. 16, e2004879 (2018).
Hickman, J. A. et al. Three-dimensional models of cancer for pharmacology and cancer cell biology: capturing tumor complexity in vitro/ex vivo. Biotechnol. J. 9, 1115–1128 (2014).
Misra, S. et al. Ex vivo organotypic culture system of precision-cut slices of human pancreatic ductal adenocarcinoma. Sci. Rep. 9, 2133 (2019).
Sontheimer-Phelps, A., Hassell, B. A. & Ingber, D. E. Modelling cancer in microfluidic human organs-on-chips. Nat. Rev. Cancer 19, 65–81 (2019).
Burian, R. M. How the choice of experimental organism matters: epistemological reflections on an aspect of biological practice. J. Hist. Biol. 26, 351–367 (1993).
Vorrink, S. U., Zhou, Y., Ingelman-Sundberg, M. & Lauschke, V. M. Prediction of drug-induced hepatotoxicity using long-term stable primary hepatic 3D spheroid cultures in chemically defined conditions. Toxicol. Sci. 163, 655–665 (2018).
Geeleher, P., Gamazon, E. R., Seoighe, C., Cox, N. J. & Huang, R. S. Consistency in large pharmacogenomic studies. Nature 540, E1–E2 (2016).
Haibe-Kains, B. et al. Revisiting inconsistency in large pharmacogenomic studies. F1000Res. 5, 2333 (2017).
Haibe-Kains, B. et al. Inconsistency in large pharmacogenomic studies. Nature 504, 389–393 (2013).
Haverty, P. M. et al. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 533, 333–337 (2016).
Mayr, L. M. & Bojanic, D. Novel trends in high-throughput screening. Curr. Opin. Pharmacol. 9, 580–588 (2009).
Pedró-Rosa, L. et al. Identification of potent inhibitors of the trypanosoma brucei methionyl-tRNA synthetase via high-throughput orthogonal screening. J. Biomol. Screen. 20, 122–130 (2015).
Mondritzki, T. Prädiktive Wertigkeit verschiedener präklinischer Outcome-Parameter für eine erfolgreiche versus nicht-erfolgreiche klinische Entwicklung von Arzneimitteln zur Behandlung der Herzinsuffizienz (Donau-Universität-Krems, 2014).
Moser, J. & Verdin, P. Burgeoning oncology pipeline raises questions about sustainability. Nat. Rev. Drug Discov. 17, 698–699 (2018).
Ten Years On: Measuring the Return from Pharmaceutical Innovation (Deloitte, 2019); https://www2.deloitte.com/content/dam/Deloitte/uk/Documents/life-sciences-health-care/deloitte-uk-ten-years-on-measuring-return-on-pharma-innovation-report-2019.pdf.
Skardal, A. et al. Multi-tissue interactions in an integrated three-tissue organ-on-a-chip platform. Sci. Rep. 7, 8837 (2017).
Maurer, T. Model-based discovery and development of novel therapies for type-2 diabetes mellitus. In Bridging Bench and Bedside with Quantitative Model-Based Translational Pharmacology symposium (New York Academy of Science, 2012).
Bertau, M., Mosekilde, E. & Westerhoff, H. V. Biosimulation in Drug Development (Wiley-VCH, 2008).
Mager, D. E. & Kimko, H. H. C. Systems Pharmacology and Pharmacodynamics. AAPS Advances in the Pharmaceutical Sciences Series (Springer International Publishing, 2016).
Helmlinger, G. et al. Quantitative systems pharmacology: an exemplar model-building workflow with applications in cardiovascular, metabolic, and oncology drug development. CPT Pharmacomet. Syst. Pharmacol. 8, 380–395 (2019).
Mardinoglu, A. et al. The potential use of metabolic cofactors in treatment of NAFLD. Nutrients 11, 1578 (2019).
Jørgensen, P. G. et al. Cardiac adaptation in hibernating, free-ranging Scandinavian Brown Bears (Ursus arctos). Sci. Rep. 10, 247 (2020).
Peretti, D. et al. RBM3 mediates structural plasticity and protective effects of cooling in neurodegeneration. Nature 518, 236–239 (2015).
Krystal, A. D. et al. A randomized proof-of-mechanism trial applying the ‘fast-fail’ approach to evaluating κ-opioid antagonism as a treatment for anhedonia. Nat. Med. 26, 760–768 (2020).
Grabb, M. C., Hillefors, M. & Potter, W. Z. The NIMH ‘Fast-Fail Trials’ (FAST) initiative: rationale, promise, and progress. Pharm. Med. 34, 233–245 (2020).
Krystal, A. D. et al. The first implementation of the NIMH FAST-FAIL approach to psychiatric drug development. Nat. Rev. Drug. Discov. 18, 82–84 (2018).
Cuthbert, B. N. & Insel, T. R. Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med. 11, 126 (2013).
Insel, T. et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am. J. Psychiatry 167, 748–751 (2010).
Cuthbert, B. N. The PRISM project: social withdrawal from an RDoC perspective. Neurosci. Biobehav. Rev. 97, 34–37 (2019).
Sullivan, P. F. et al. A mega-analysis of genome-wide association studies for major depressive disorder. Mol. Psychiatry 18, 497–511 (2013).
Healy, D. The Antidepressant Era (Harvard University Press, 1997).
Liu, K. S. et al. Is bigger better for depression trials? J. Psychiatr. Res. 42, 622–630 (2008).
FDA. Major Depressive Disorder: Developing Drugs for Treatment Guidance for Industry (2018).
Nelson, R. R. The simple economics of basic scientific research. J. Polit. Econ. 67, 297–306 (1959).
Arrow, K. J. In Readings in Industrial Economics (ed. Rowley, C. K.) 219–236 (Palgrave, 1972).
Dobson, C. M. Chemical space and biology. Nature 432, 824–828 (2004).
Lipinski, C. & Hopkins, A. Navigating chemical space for biology and medicine. Nature 432, 855–861 (2004).
Seymore, S. B. Making patents useful. Minn. Law Rev. 98, 1046–1109 (2014).
Billette de Villemeur, E. & Versaevel, B. One lab, two firms, many possibilities: on R&D outsourcing in the biopharmaceutical industry. J. Health Econ. 65, 260–283 (2019).
Hoofnagle, J. H. & Sherker, A. H. Therapy for hepatitis C — the costs of success. N. Engl. J. Med. 370, 1552–1553 (2014).
WSJ. Senate Committee Is Investigating Pricing of Hepatitis C Drug. Wall Street Journal https://www.wsj.com/articles/senate-finance-committee-is-investigating-pricing-of-hepatitis-c-drug-1405109206 (2014).
Morrison, C. 2019 biotech IPOs: party on. Nat. Rev. Drug Discov. 19, 6–9 (2020).
Morrison, C. Boom: 2018’s biotech IPOs. Nat. Rev. Drug Discov. 18, 3–6 (2018).
Williamson, A. R. Creating a structural genomics consortium. Nat. Struct. Biol. 7, 953 (2000).
Vaudano, E. The innovative medicines initiative: a public private partnership model to foster drug discovery. Comput. Struct. Biotechnol. J. 6, e201303017 (2013).
Ochoa, D. et al. Open targets platform: supporting systematic drug-target identification and prioritisation. Nucleic Acids Res. 49, D1302–D1310 (2021).
Dolgin, E. Massive NIH–industry project opens portals to target validation. Nat. Rev. Drug Discov. 18, 240–242 (2019).
Holden, A. L. The SNP consortium: summary of a private consortium effort to develop an applied map of the human genome. BioTechniques 32, S22–S26 (2002).
Holden, A. L., Contreras, J. L., John, S. & Nelson, M. R. The international serious adverse events consortium. Nat. Rev. Drug Discov. 13, 795–796 (2014).
Contreras, J. L. & Vertinsky, L. S. Pre-competition. North Carol. Law Rev. 95, 67–131 (2016).
Lundqvist, B. Joint research and development collaborations under competition law, with a layman’s economic viewpoint. SSRN Electron. J. https://doi.org/10.2139/ssrn.2913840 (2017).
Sams-Dodd, F. Strategies to optimize the validity of disease models in the drug discovery process. Drug Discov. Today 11, 355–363 (2006).
Hooijmans, C. R., De Vries, R., Leenaars, M., Curfs, J. & Ritskes-Hoitinga, M. Improving planning, design, reporting and scientific quality of animal experiments by using the Gold Standard Publication Checklist, in addition to the ARRIVE guidelines. Br. J. Pharmacol. 162, 1259–1260 (2011).
Fisher, M. Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 30, 2752–2758 (1999).
Vettoretti, M., Facchinetti, A., Sparacino, G. & Cobelli, C. Predicting insulin treatment scenarios with the net effect method: domain of validity. Diabetes Technol. Ther. 18, 694–704 (2016).
Sacca, L., Toffolo, G. & Cobelli, C. V-A and A-V modes in whole body and regional kinetics: Domain of validity from a physiological model. Am. J. Physiol. Endocrinol. Metab. 263, E597–E606 (1992).
Rudolf, A. F., Skovgaard, T., Knapp, S., Jensen, L. J. & Berthelsen, J. A comparison of protein kinases inhibitor screening methods using both enzymatic activity and binding affinity determination. PLoS ONE 9, e98800 (2014).
Brown, D. G., May-Dracka, T. L., Gagnon, M. M. & Tommasi, R. Trends and exceptions of physical properties on antibacterial activity for Gram-positive and Gram-negative pathogens. J. Med. Chem. 57, 10144–10161 (2014).
Zoffmann, S. et al. Machine learning-powered antibiotics phenotypic drug discovery. Sci. Rep. 9, 5013 (2019).
Farha, M. A. & Brown, E. D. Unconventional screening approaches for antibiotic discovery. Ann. NY Acad. Sci. 1354, 54–66 (2015).
Yokokawa, F. Recent Progress on the Development of Novel Antitubercular Agents from Whole-Cell Screening Hits. J. Synth. Org. Chem. Jpn. 72, 1239–1249 (2014).
Horscroft, N. et al. Replicon cell culture system as a valuable tool in antiviral drug discovery against hepatitis C virus. Antivir. Chem. Chemother. 16, 1–12 (2005).
Woerz, I., Lohmann, V. & Bartenschlager, R. Hepatitis C virus replicons: dinosaurs still in business? J. Viral Hepat. 16, 1–9 (2009).
Kaplan, G. & Racaniello, V. R. Construction and characterization of poliovirus subgenomic replicons. J. Virol. 62, 1687–1696 (1988).
Khromykh, A. A. & Westaway, E. G. Subgenomic replicons of the flavivirus Kunjin: construction and applications. J. Virol. 71, 1497–1505 (1997).
Behrens, S.-E., Grassmann, C. W., Thiel, H.-J., Meyers, G. & Tautz, N. Characterization of an autonomous subgenomic pestivirus RNA replicon. J. Virol. 72, 2364–2372 (1998).
Lohmann, V. et al. Replication of subgenomic hepatitis C virus RNAs in a hepatoma cell line. Science 285, 110–113 (1999).
Chung, R. T. & Baumert, T. F. Curing chronic hepatitis C — the arc of a medical triumph. N. Engl. J. Med. 370, 1576–1578 (2014).
Sena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R. & Howells, D. W. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J. Cereb. Blood Flow. Metab. 34, 737–742 (2014).
Macleod, M. R., O’Collins, T., Howells, D. W. & Donnan, G. A. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 35, 1203–1208 (2004).
MacLeod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).
Macleod, M. R. et al. Good laboratory practice: preventing introduction of bias at the bench. Stroke 40, e50–e52 (2009).
Dirnagl, U., Iadecola, C. & Moskowitz, M. A. Pathobiology of ischaemic stroke: an integrated view. Trends Neurosci. 22, 391–397 (1999).
Dronne, M. A., Grenier, E., Chapuisat, G., Hommel, M. & Boissel, J. P. A modelling approach to explore some hypotheses of the failure of neuroprotective trials in ischemic stroke patients. Prog. Biophys. Mol. Biol. 97, 60–78 (2008).
Choi, D. W. Excitotoxicity: still hammering the Ischemic brain in 2020. Front. Neurosci. 14, 579953 (2020).
Orset, C. et al. Efficacy of Alteplase in a mouse model of acute Ischemic stroke: a retrospective pooled analysis. Stroke 47, 1312–1318 (2016).
Phipps, M. S. & Cronin, C. A. Management of acute ischemic stroke. BMJ 368, l6983 (2020).
The authors thank A. Morton and A. Colson at the Department of Management Science at Strathclyde University for their help on the decision tool evaluation section; B. Versaevel and E. Billet de Villemeur, at EMLYON Business School and at Université des Sciences et Technologies de Lille, respectively, for their help on the economics of decision tool-related innovation; and L. Ewart, EVP Science at Emulate, for her help with toxicology-related content. They thank M. Todd at the UCL School of Pharmacy for helpful comments on an earlier draft of the paper. Finally, they thank the reviewers, whose thoughtful comments substantially improved the paper.
J.W.S. is a director and shareholder of JW Scannell Analytics, which sells consulting services to the biopharmaceutical and financial services sectors, including to firms that are commercializing screening and disease models; is CEO of Unify Pharmaceuticals Corp; is Director of Etheros Pharmaceuticals Corp; and holds equity options in Ochre Bio. J.B. is an employee of Novadiscovery and holds equity options in the firm. G.R.D. is a director and major shareholder of P1vital and P1vital Products Ltd. P1vital provides clinical research services to the pharmaceutical industry. P1vital Products provides data management and digital technology to the pharmaceutical industry and health-care providers. H.T. is employed by AiCuris AG, owns shares in Bayer AG, Evotec SE, Cyclerion Therapeutics, Inc. and recently incorporated www.knowledge-house.com. G.S.F. has received fees from Merck KGaA and Curare Consulting; has been employed by the Access to Medicine Foundation, 3D PharmXchange and Janssen Biologics; and is now employed by GlaxoSmithKline. D.R. is a consultant to OMASS Therapeutics Ltd and is on the scientific advisory board of Avicenna Biosciences, Inc. J.M.T. is a director and/or shareholder of Talisman Therapeutics, Gen2 Neuroscience, Avilex Pharma, Domain Therapeutics, Cellesce, Ubiquigent and Monument Therapeutics. J.A.H. declares no competing interests.
Peer review information
Nature Reviews Drug Discovery thanks Neil Carragher, Matthew Nelson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
DataFAIRy Bioassay Annotation: https://www.pistoiaalliance.org/projects/current-projects/datafairy-bioassay-annotation/
Sample size calculator from Statistics Kingdom: https://www.statskingdom.com/sample_size_regression.html
Springer Nature Experiments: https://experiments.springernature.com/
The Collaborative Adverse Outcome Pathway Wiki: https://aopwiki.org/
About this article
Cite this article
Scannell, J.W., Bosley, J., Hickman, J.A. et al. Predictive validity in drug discovery: what it is, why it matters and how to improve it. Nat Rev Drug Discov 21, 915–931 (2022). https://doi.org/10.1038/s41573-022-00552-x