A comprehensive overview of medical AI devices approved by the US Food and Drug Administration sheds new light on limitations of the evaluation process that can mask vulnerabilities of devices when they are deployed on patients.
This is a preview of subscription content, access via your institution
Relevant articles
Open Access articles citing this article.
-
Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis
npj Digital Medicine Open Access 09 April 2024
-
The clinician-AI interface: intended use and explainability in FDA-cleared AI devices for medical image interpretation
npj Digital Medicine Open Access 26 March 2024
-
Developing a Framework for Self-regulatory Governance in Healthcare AI Research: Insights from South Korea
Asian Bioethics Review Open Access 25 March 2024
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
References
Liu, X. et al. Nat. Med. 26, 1364–1374 (2020).
Norgeot, B. et al. Nat. Med. 26, 1320–1324 (2020).
Cruz Rivera, S., Liu, X. & Chan, A. W. et al. Nat. Med. 26, 1351–1363 (2020).
U.S. Food & Drug Administration Center for Devices and Radiological Health. https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm514737.pdf (October 2017).
U.S. Food & Drug Administration Center for Devices and Radiological Health. https://www.fda.gov/media/145022/download (January 2021).
U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/denovo.cfm (accessed 14 December 2020).
U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/pma.cfm (accessed 14 December 2020).
U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm (accessed 14 December 2020).
Benjamens, S., Dhunnoo, P. & Meskó, B. Npj Digit. Med. 3, 1–8 (2020).
American College of Radiology. https://www.acrdsi.org/DSI-Services/FDA-Cleared-AI-Algorithms (accessed 29 November 2020).
Kaushal, A., Altman, R. & Langlotz, C. J. Am. Med. Assoc. 324, 1212–1213 (2020).
Wang, X. et al. 2017 IEEE Conference on Computer Vision and Pattern Recognition 3462–3471 (Institute of Electrical and Electronics Engineers, 2017).
Irvin, J. et al. The Thirty-Third AAAI Conference on Artificial Intelligence 590–597 (Association for the Advancement of Artificial Intelligence, 2019).
Johnson, A. E. W. et al. Sci. Data 6, 317 (2019).
Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.Q. 2017 IEEE Conference on Computer Vision and Pattern Recognition 4700–4708 (Institute of Electrical and Electronics Engineers, 2017).
Seyyed-Kalantari, L., Liu, G., McDermott, M., Chen, I. Y. & Ghassemi, M. Pac. Symp. Biocomput. 26, 232–243 (2021).
U.S. Food & Drug Administration. https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/postmarket-requirements-devices (2018).
Ferryman, K. J. Am. Med. Inform. Assoc. 27, 2016–2019 (2020).
Acknowledgements
J.Z. is supported by the National Science Foundation (CCF 1763191 and CAREER 1942926), the US National Institutes of Health (P30AG059307 and U01MH098953) and grants from the Silicon Valley Foundation and the Chan-Zuckerberg Initiative. R.D. is supported by the US National Institutes of Health (T32 5T32AR007422-38). Our compiled database of approved medical AI devices, analysis code, and models used for the case study are all available at https://ericwu09.github.io/medical-ai-evaluation.
Author information
Authors and Affiliations
Contributions
E.W., K.W. and J.Z. designed the study. E.W., K.W. conducted research with help from R.D. and D.O. All the authors contributed to interpretation of the results and writing of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Rights and permissions
About this article
Cite this article
Wu, E., Wu, K., Daneshjou, R. et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med 27, 582–584 (2021). https://doi.org/10.1038/s41591-021-01312-x
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41591-021-01312-x
This article is cited by
-
Demographic bias in misdiagnosis by computational pathology models
Nature Medicine (2024)
-
Artificial intelligence in the neonatal intensive care unit: the time is now
Journal of Perinatology (2024)
-
Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis
npj Digital Medicine (2024)
-
The clinician-AI interface: intended use and explainability in FDA-cleared AI devices for medical image interpretation
npj Digital Medicine (2024)
-
Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance
Drug Safety (2024)