Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

External validation of AI models in health should be replaced with recurring local validation

Nature Medicine volume 29, pages 2686–2687 (2023)Cite this article

Subjects

Clinical prediction models follow a standard development pipeline: model development and internal validation; external validation; and clinical impact studies. External validation studies should be followed by real-world studies evaluating the deployed models’ usefulness1. However, the latter are rarely performed. Instead, external validation ends up being the de facto test for evaluating machine learning (ML) models before deployment.

External validation is often considered the ultimate test to conclusively judge an ML model’s safety, reliability and generalizability2,3,4: a model that passes an external validation test on one or a few datasets is deemed generalizable, safe and reliable. However, external validation does not guarantee generalizability or equate to model usefulness, which should be the true goal of any clinical decision-support tool. Many have demonstrated the unreliability of clinical models when tested across multiple clinical sites3,5. Some even argued that there is no such thing as a truly validated model4. We summarize the limitations of external validation in Table 1.

Table 1 Limitations of external validation and advantages of recurring local validation
Full size table

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Shah, N. H., Milstein, A. & Bagley, S. C. JAMA 322, 1351–1352 (2019).

    Article  PubMed  Google Scholar 

  2. Yu, A. C. & Mohajer, B. J. Eng. Radiol. Artif. Intell. 4, e210064 (2022).

    Article  PubMed  Google Scholar 

  3. Singh, H., Mhasawade, V. & Chunara, R. PLoS Digit. Health 1, e0000023 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Van Calster, B., Steyerberg, E. W., Wynants, L. & van Smeden, M. BMC Med 21, 70 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C. & van Diepen, M. Clin. Kidney J 14, 49–58 (2021).

    Article  PubMed  Google Scholar 

  6. Habib, A. R., Lin, A. L. & Grant, R. W. JAMA Intern. Med. 181, 1040–1041 (2021).

    Article  PubMed  Google Scholar 

  7. Shankar, S., Garcia, R., Hellerstein, J. M. & Parameswaran, A. G. Preprint at arXiv https://arxiv.org/abs/2209.09125 (2022).

  8. Bedoya, A. D. et al. J. Am. Med. Inform. Assoc. 29, 1631–1636 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Youssef, A. et al. Preprint at medRxiv https://medrxiv.org/content/10.1101/2022.08.09.22278600v2 (2022).

  10. Granlund, T., Stirbu, V. & Mikkonen, T. SN Comput. Sci. 2, 342 (2021).

    Article  Google Scholar 

Download references

Acknowledgements

D.A.C. is funded by an RAEng Research Chair and a UK National Institute for Health and Care Research (NIHR) Research Professorship, the NIHR Oxford Biomedical Research Centre, the InnoHK Centre for Cerebro-cardiovascular Engineering and the Oxford Pandemic Sciences Institute. T.Z. was supported by the Royal Academy of Engineering under the Research Fellowship scheme.

Author information

Authors and Affiliations

  1. Stanford Bioengineering Department, Stanford University, Stanford, CA, USA

    Alexey Youssef

  2. Department of Engineering Science, University of Oxford, Oxford, UK

    Alexey Youssef, Anshul Thakur, Tingting Zhu & David Clifton

  3. Duke University School of Medicine, Durham, NC, USA

    Michael Pencina

  4. Oxford–Suzhou Centre for Advanced Research, Suzhou, China

    David Clifton

  5. Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA

    Nigam H. Shah

  6. Technology and Digital Solutions, Stanford Medicine, Stanford, CA, USA

    Nigam H. Shah

  7. Clinical Excellence Research Center, Stanford Medicine, Stanford, CA, USA

    Nigam H. Shah

Authors
  1. Alexey Youssef
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Pencina
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anshul Thakur
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tingting Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Clifton
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nigam H. Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexey Youssef.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Youssef, A., Pencina, M., Thakur, A. et al. External validation of AI models in health should be replaced with recurring local validation. Nat Med 29, 2686–2687 (2023). https://doi.org/10.1038/s41591-023-02540-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-023-02540-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing