TECHNOLOGY FEATURE
09 January 2023
Correction 12 January 2023

The reproducibility issues that haunt health-care AI

Health-care systems are rolling out artificial-intelligence tools for diagnosis and monitoring. But how reliable are the models?

Emily Sohn⁰

Emily Sohn
1. Emily Sohn is a freelance journalist in Minneapolis, Minnesota.
View author publications

You can also search for this author in PubMed Google Scholar

Access through your institution

Buy or subscribe

Each day, around 350 people in the United States die from lung cancer. Many of those deaths could be prevented by screening with low-dose computed tomography (CT) scans. But scanning millions of people would produce millions of images, and there aren’t enough radiologists to do the work. Even if there were, specialists regularly disagree about whether images show cancer or not. The 2017 Kaggle Data Science Bowl set out to test whether machine-learning algorithms could fill the gap.

Access options

Access through your institution

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Nature 613, 402-403 (2023)

doi: https://doi.org/10.1038/d41586-023-00023-2

Updates & Corrections

Correction 12 January 2023: An earlier version of this feature erroneously stated that Sayash Kapoor discovered reproducibility failures and pitfalls in 329 studies across 17 fields. In fact, those studies had themselves reported reproducibility failures and pitfalls.

References

Yu, K. H. et al. J. Med. Internet Res. 22, e16709 (2020).
Article PubMed Google Scholar
Roberts, M. et al. Nature Mach. Intell. 3, 199–217 (2021).
Article Google Scholar
McDermott, M. B. A. et al. Sci. Transl. Med. 13, eabb1655 (2021).
Article PubMed Google Scholar
Wong, A. et al. JAMA Intern. Med. 181, 1065–1070 (2021).
Article PubMed Google Scholar
Mongan, J., Moy, L. & Kahn, C. E. Radiol. Artif. Intell. 2, e200029 (2020).
Article PubMed Google Scholar