Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

Potential sources of dataset bias complicate investigation of underdiagnosis by machine learning algorithms

Nature Medicine volume 28, pages 1157–1158 (2022)Cite this article

Subjects

Matters Arising to this article was published on 16 June 2022

The Original Article was published on 10 December 2021

arising from L. Seyyed-Kalantari et al. Nat. Med. https://doi.org/10.1038/s41591-021-01595-0 (2021).

An increasing number of reports raise concerns about the risk that machine learning algorithms could amplify health disparities due to biases embedded in the training data1,2. Seyyed-Kalantari et al.3 find that models trained on three chest X-ray datasets yield disparities in false-positive rates (FPRs) across subgroups on the ‘no-finding’ label (indicating the absence of disease). The models consistently yield higher FPRs on subgroups known to be historically underserved, and the study concludes that the models exhibit and potentially even amplify systematic underdiagnosis. We argue that the experimental setup in the study is insufficient to study algorithmic underdiagnosis. In the absence of specific knowledge (or assumptions) about the extent and nature of the dataset bias, it is difficult to investigate model bias. Importantly, their use of test data exhibiting the same bias as the training data (due to random splitting) severely complicates the interpretation of the reported disparities.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Adamson, A. S. & Smith, A. Machine learning and health care disparities in dermatology. JAMA Dermatol. 154, 1247–1248 (2018).

    Article  Google Scholar 

  2. Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).

    Article  CAS  Google Scholar 

  3. Seyyed-Kalantari, L., Zhang, H., McDermott, M., Chen, I. Y. & Ghassemi, M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat. Med. 27, 2176–2182 (2021).

  4. Castro, D. C., Walker, I. & Glocker, B. Causality matters in medical imaging. Nat. Commun. 11, 1–10 (2020).

    Article  Google Scholar 

  5. Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. N. Engl. J. Med. 385, 283–286 (2021).

  6. He, H. & Garcia, E. A. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21, 1263–1284 (2009).

    Article  Google Scholar 

  7. Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002).

    Article  Google Scholar 

  8. Johnson, J. M. & Khoshgoftaar, T. M. Survey on deep learning with class imbalance. J. Big Data 6, 1–54 (2019).

    Article  Google Scholar 

  9. Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V. & Herrera, F. A unifying view on dataset shift in classification. Pattern Recognit. 45, 521–530 (2012).

    Article  Google Scholar 

  10. Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 590–597 (2019).

Download references

Acknowledgements

We thank X. Liu, A. Denniston and M. McCradden for helpful discussions. M.B. is funded through an Imperial College London President’s PhD Scholarship. C.J. is supported by Microsoft Research and EPSRC through the Microsoft PhD Scholarship Programme. B.G. is supported through funding from the European Research Council under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 757173, Project MIRA, ERC-2017-STG).

Author information

Author notes

  1. The authors contributed equally: Mélanie Bernhardt, Charles Jones, Ben Glocker.

Authors and Affiliations

  1. Department of Computing, Imperial College London, London, United Kingdom

    Mélanie Bernhardt, Charles Jones & Ben Glocker

Authors
  1. Mélanie Bernhardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ben Glocker
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to this work in terms of formulating the arguments, interpreting the available evidence and cowriting the manuscript.

Corresponding author

Correspondence to Ben Glocker.

Ethics declarations

Competing interests

B.G. is a part-time employee of HeartFlow and Kheiron Medical Technologies and holds stock options with both as part of the standard compensation package. M.B. and C.J. declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bernhardt, M., Jones, C. & Glocker, B. Potential sources of dataset bias complicate investigation of underdiagnosis by machine learning algorithms. Nat Med 28, 1157–1158 (2022). https://doi.org/10.1038/s41591-022-01846-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-022-01846-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing