Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • OUTLOOK

Breaking into the black box of artificial intelligence

Scientists are finding ways to explain the inner workings of complex machine-learning models.
  1. Neil Savage

    1. Neil Savage is a freelance writer in Lowell, Massachusetts.

    View author publications

    You can also search for this author in PubMed  Google Scholar

Cartoon of a human figure standing in front of a huge black box for which the sides are becoming opaque

Illustration: Sandro Rybak

In February 2020, with COVID-19 spreading rapidly around the globe and antigen tests hard to come by, some physicians turned to artificial intelligence (AI) to try to diagnose cases1. Some researchers tasked deep neural networks — complex systems that are adept at finding subtle patterns in images — with looking at X-rays and chest computed tomography (CT) scans to quickly distinguish between people with COVID-based pneumonia and those without2. “Early in the COVID-19 pandemic, there was a race to build tools, especially AI tools, to help out,” says Alex DeGrave, a computer engineer at the University of Washington in Seattle. But in that rush, researchers did not notice that many of the AI models had decided to take a few shortcuts.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-022-00858-1

This article is part of Nature Outlook: Robotics and artificial intelligence, an editorially independent supplement produced with the financial support of third parties. About this content.

References

  1. Wang, L. & Wong, A. Preprint at https://arxiv.org/abs/2003.09871 (2020).

  2. Ai, T. et al. Radiology 296, E32–E40 (2020).

    Article  PubMed  Google Scholar 

  3. DeGrave, A. J., Janizek, J. D. & Lee, S.-I. Nature Mach. Intell. 3, 610–619 (2021).

    Article  Google Scholar 

  4. Petsiuk, V. et al. Preprint at https://arxiv.org/abs/2006.03204 (2020).

  5. Leino, K., Sen, S., Datta, A., Fredrikson, M. & Li, L. Preprint at https://arxiv.org/abs/1802.03788 (2018).

  6. Bau, D. et al. Preprint at https://arxiv.org/abs/1811.10597 (2018).

  7. Gardiner, L.-J. et al. Proc. Natl Acad. Sci. USA 118, e2103070118 (2021).

    Article  PubMed  Google Scholar 

  8. Yeh, C.-K. & Ravikumar, P. Appl. AI Lett. 2, e57 (2021).

    Article  Google Scholar 

Download references

Subjects

Latest on:

Nature Careers

Jobs

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing

Search

Advanced search

Quick links