Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Tackling bias in AI health datasets through the STANDING Together initiative

Nature Medicine volume 28, pages 2232–2233 (2022)Cite this article

Subjects

To the Editor — As of June 2022, a wide range of Artificial Intelligence (AI) as a Medical Device (AIaMDs) have received regulatory clearance internationally, with at least 343 devices cleared by the US Food and Drug Administration (FDA)1. Despite the enormous potential of AIaMDs, their rapid growth in healthcare has been accompanied by concerns that AI models may learn biases engrained in medical practice and exacerbate health inequalities. This has been exemplified by several AI systems that have shown the ability of algorithms to systematically misrepresent and exacerbate health problems in minority groups2,3. This raises concerns that, without appropriate safeguarding, AI models may perpetuate existing health inequality and mistrust.

Tackling bias in AI requires a multifaceted approach. A recent report by the US National Institute of Standards and Technology on bias in AI emphasized that algorithmic development does not occur by engineering decisions alone, but embeds a myriad of values and behaviors within the data and the humans who interact with them. The report calls for a sociotechnical approach that considers how different biases interact and the social contexts within which AI systems are built and used4. Although there is an expanding field of research dedicated to fairness in machine learning, many AIaMDs that receive regulatory clearance have not appropriately accounted for biases that disadvantage certain populations. There are also ethical challenges around algorithmic fairness methods (computational techniques that seek to ensure outputs are not unjustifiably influenced by bias), given that these methods are aimed at making predictions fair, rather than enabling the fair treatment of individuals5. Furthermore, current approaches to satisfy regulatory requirements are focused on aggregate-level performance, which can mask stratification across subpopulations.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. FDA Center for Devices & Radiological Health. https://go.nature.com/3AG0McN (2021).

  2. Obermeyer, Z. et al. Science 366, 447–453 (2019).

    Article  CAS  PubMed  Google Scholar 

  3. Seyyed-Kalantari, L. et al. Nat. Med. 27, 2176–2182 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Schwartz, R. et al. National Institute of Standards and Technology; https://go.nature.com/3Q6rjpj (2022).

  5. McCradden, M. D. et al. Lancet Digit Health 2, e221–e223 (2020).

    Article  PubMed  Google Scholar 

  6. Khan, S. M. et al. Lancet Digit Health 3, e51–e66 (2021).

    Article  PubMed  Google Scholar 

  7. Wen, D. et al. Lancet Digit Health 4, e64–e74 (2022).

    Article  PubMed  Google Scholar 

  8. Rostamzadeh, N. et al. FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency https://doi.org/10.1145/3531146.3533239 (2022).

  9. Gebru, T. et al. Preprint at https://doi.org/10.48550/arXiv.1803.09010 (2018).

  10. Medicines and Healthcare products Regulatory Agency. https://go.nature.com/3RsijvS (2021).

Download references

Acknowledgements

This project is funded by The NHS AI Lab at the NHS Transformation Directorate and the Health Foundation and managed by the National Institute for Health and Care Research (AI_HI200014). The views expressed in this publication are those of the author(s) and not necessarily those of the NHS Transformation Directorate, the Health Foundation or the National Institute for Health and Care Research.

Author information

Authors and Affiliations

  1. College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK

    Shaswath Ganapathi

  2. University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK

    Jo Palmer, Joseph E. Alderman, Elizabeth Sapey, Alastair K. Denniston & Xiaoxuan Liu

  3. Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK

    Joseph E. Alderman, Elizabeth Sapey, Alastair K. Denniston & Xiaoxuan Liu

  4. Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK

    Melanie Calvert, Alastair K. Denniston & Xiaoxuan Liu

  5. Centre for Patient Reported Outcome Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK

    Melanie Calvert

  6. NIHR Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK

    Melanie Calvert & Alastair K. Denniston

  7. NIHR Surgical Reconstruction and Microbiology Research Centre, University of Birmingham, Birmingham, UK

    Melanie Calvert

  8. NIHR Applied Research Collaborative West Midlands University of Birmingham, Birmingham, UK

    Melanie Calvert

  9. Patient Partner, Birmingham, UK

    Cyrus Espinoza & Jacqui Gath

  10. Patient Partner, Sheffield, UK

    Jacqui Gath

  11. Department of Electrical Engineering and Computer Science; Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Marzyeh Ghassemi

  12. Google Research, Mountain View, California, USA

    Katherine Heller & Stephen R. Pfohl

  13. The Ethox Centre and the Wellcome Centre for Ethics and Humanities, Nuffield Department of Population Health, University of Oxford, Oxford, UK

    Francis Mckay

  14. Google Research, London, UK

    Alan Karthikesalingam

  15. Institute of Women’s Health, University College London, London, UK

    Stephanie Kuku

  16. Hardian Health, London, UK

    Stephanie Kuku

  17. Genomics England, London, UK

    Maxine Mackintosh

  18. Health Data Research, London, UK

    Sinduja Manohar, Neil Sebire & Alastair K. Denniston

  19. Institute of Health Informatics, University College London, London, UK

    Bilal A. Mateen

  20. The Wellcome Trust, London, UK

    Bilal A. Mateen

  21. Oxford University Hospitals NHS Foundation Trust, Oxford, UK

    Rubeta Matin

  22. Department of Bioethics, Hospital for Sick Children, Toronto, Ontario, Canada

    Melissa McCradden

  23. Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

    Melissa McCradden

  24. Australian Institute for Machine Learning, University of Adelaide, Adelaide, South Australia, Australia

    Lauren Oakden-Rayner

  25. Medicines and Healthcare Products Regulatory Agency, London, UK

    Johan Ordish & Russell Pearson

  26. Google Research, Montreal, Canada

    Negar Rostamzadeh

  27. Great Ormond Street Hospital for Children, London, UK

    Neil Sebire

  28. Institute of Global Health Innovation, Imperial College London, London, UK

    Viknesh Sounderajah

  29. Department of Surgery and Cancer, Imperial College London, London, UK

    Viknesh Sounderajah

  30. Wolfson Lung Injury Unit, Heart and Lung Research Institute, University of Cambridge, Cambrdige, UK

    Charlotte Summers

  31. Cambridge University Hospitals NHS Foundation Trust, Cambridge, UK

    Charlotte Summers

  32. Leeds Teaching Hospitals NHS Trust, Leeds, UK

    Darren Treanor

  33. University of Leeds, Leeds, UK

    Darren Treanor

  34. Department of Clinical Pathology, and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden

    Darren Treanor

  35. Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden

    Darren Treanor

Authors
  1. Shaswath Ganapathi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jo Palmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joseph E. Alderman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Melanie Calvert
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cyrus Espinoza
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jacqui Gath
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marzyeh Ghassemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Katherine Heller
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Francis Mckay
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Alan Karthikesalingam
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Stephanie Kuku
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Maxine Mackintosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Sinduja Manohar
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Bilal A. Mateen
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Rubeta Matin
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Melissa McCradden
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Lauren Oakden-Rayner
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Johan Ordish
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Russell Pearson
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Stephen R. Pfohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Negar Rostamzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Elizabeth Sapey
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Neil Sebire
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Viknesh Sounderajah
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Charlotte Summers
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Darren Treanor
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Alastair K. Denniston
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Xiaoxuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoxuan Liu.

Ethics declarations

Competing interests

K.H., A.K., N.R. and S.R.P. are employees of Google. S.K. is a consultant for Hardian Health. D.T. and F.M. are funded by National Pathology Imaging Co-operative (NPIC, Pproject no. 104687), which is supported by a £50 million investment from the Data to Early Diagnosis and Precision Medicine strand of the government’s Industrial Strategy Challenge Fund, managed and delivered by UK Research and Innovation (UKRI). X.L., A.K.D., J.E.A. and J.P. are funded by NIHR, the NHS Transformation Directorate and the Health Foundation (AI_HI200014). M.J.C. is Director of the Birmingham Health Partners Centre for Regulatory Science and Innovation, Director of the Centre for the Centre for Patient Reported Outcomes Research and is a National Institute for Health and Care Research (NIHR) Senior Investigator. M.J.C. receives funding from the NIHR, UK Research and Innovation (UKRI), NIHR Birmingham Biomedical Research Centre, the NIHR Surgical Reconstruction and Microbiology Research Centre, NIHR ARC West Midlands, UK SPINE, European Regional Development Fund – Demand Hub and Health Data Research UK at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Innovate UK (part of UKRI), Macmillan Cancer Support, UCB Pharma, Janssen, GSK and Gilead, has received personal fees from Astellas, Aparito Ltd, CIS Oncology, Takeda, Merck, Daiichi Sankyo, Glaukos, GSK and the Patient-Centered Outcomes Research Institute (PCORI) outside the submitted work; a family member of M.J.C. owns shares in GSK. ES receives research funding from UKRI (MR/V033654/1 and MR/S002782/1), the British Lung Foundation, and Alpha 1 Foundation and NIHR. C.S. receives research funding from the National Institute for Health and Care Research (NIHR133788), UKRI (MR/P502091/1 and MR/X005070/1), the Wellcome Trust, and the NIHR Cambridge Biomedical Research Centre (BRC1215-20014). All other authors declare no conflicts.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ganapathi, S., Palmer, J., Alderman, J.E. et al. Tackling bias in AI health datasets through the STANDING Together initiative. Nat Med 28, 2232–2233 (2022). https://doi.org/10.1038/s41591-022-01987-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-022-01987-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research