Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Transparency and reproducibility in artificial intelligence

Nature volume 586, pages E14–E16 (2020)Cite this article

Subjects

Matters Arising to this article was published on 14 October 2020

The Original Article was published on 01 January 2020

arising from S. M. McKinney et al. Nature https://doi.org/10.1038/s41586-019-1799-6 (2020)

Your institute does not have access to this article

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

Buy

All prices are NET prices.

Data availability

No data have been generated as part of this manuscript.

References

  1. McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020).

    ADS  CAS  Article  PubMed  Google Scholar 

  2. Bluemke, D. A. et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers—from the Radiology editorial board. Radiology 293, 315–316 (2019).

    Article  Google Scholar 

  3. Gundersen, O. E., Gil, Y. & Aha, D. W. On reproducible AI: towards reproducible research, open science, and digital scholarship in AI publications. AI Mag. 39, 56–68 (2018).

    Article  Google Scholar 

  4. Crane, M. Questionable answers in question answering research: reproducibility and variability of published results. Trans. Assoc. Comput. Linguist. 6, 241–252 (2018).

    Article  Google Scholar 

  5. Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).

  6. Stodden, V. et al. Enhancing reproducibility for computational methods. Science 354, 1240–1241 (2016).

    ADS  CAS  Article  PubMed  Google Scholar 

  7. Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).

    ADS  Article  PubMed  Google Scholar 

  8. Bzdok, D. & Ioannidis, J. P. A. Exploration, inference, and prediction in neuroscience and biomedicine. Trends Neurosci. 42, 251–262 (2019).

    CAS  Article  PubMed  Google Scholar 

  9. Gundersen, O. E. & Kjensmo, S. State of the art: Reproducibility in artificial intelligence. In Thirty-second AAAI Conference on Artificial Intelligence (AAAI-18) 1644–1651 (2018).

  10. Shorten, C. & Khoshgoftaar, T. M. A survey on Image Data Augmentation for Deep Learning. J. Big Data 6, 60 (2019).

    Article  Google Scholar 

  11. Lee, R. S. et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 170177 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  12. Wallach, J. D., Boyack, K. W. & Ioannidis, J. P. A. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol. 16, e2006930 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  13. Amann, R. I. et al. Toward unrestricted use of public genomic data. Science 363, 350–352 (2019).

    ADS  CAS  Article  PubMed  Google Scholar 

  14. Carlson, B. Putting oncology patients at risk. Biotechnol. Healthc. 9, 17–21 (2012).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank S. McKinney and colleagues for their prompt and open communication regarding the materials and methods of their study. This work was supported in part by the National Cancer Institute (R01 CA237170).

Author information

Authors and Affiliations

  1. Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada

    Benjamin Haibe-Kains, Farnoosh Khodakarami & Michael M. Hoffman

  2. Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada

    Benjamin Haibe-Kains, Farnoosh Khodakarami, Bo Wang, Chris McIntosh & Michael M. Hoffman

  3. Department of Computer Science, University of Toronto, Toronto, Ontario, Canada

    Benjamin Haibe-Kains, George Alexandru Adam, Bo Wang, Anna Goldenberg & Michael M. Hoffman

  4. Ontario Institute for Cancer Research, Toronto, Ontario, Canada

    Benjamin Haibe-Kains

  5. Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada

    Benjamin Haibe-Kains, George Alexandru Adam, Bo Wang, Chris McIntosh, Anna Goldenberg & Michael M. Hoffman

  6. Artificial Intelligence in Medicine (AIM) Program, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

    Ahmed Hosny & Hugo J. W. L. Aerts

  7. Radiation Oncology and Radiology, Dana-Farber Cancer Institute, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

    Ahmed Hosny & Hugo J. W. L. Aerts

  8. Department of Epidemiology and Biostatistics and Institute for Implementation Science in Population Health, CUNY Graduate School of Public Health and Health Policy, New York, NY, USA

    Levi Waldron

  9. Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada

    Bo Wang & Chris McIntosh

  10. Department of Laboratory Medicine and Pathobiology, University of Toronto, Ontario, Canada

    Bo Wang

  11. SickKids Research Institute, Toronto, Ontario, Canada

    Anna Goldenberg

  12. Child and Brain Development Program, CIFAR, Toronto, Ontario, Canada

    Anna Goldenberg

  13. Department of Genetics, Stanford University School of Medicine, Stanford, CA, USA

    Anshul Kundaje

  14. Department of Computer Science, Stanford University, Stanford, CA, USA

    Anshul Kundaje

  15. Dept. of Systems Pharmacology and Translational Therapeutics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    Casey S. Greene

  16. Childhood Cancer Data Lab, Alex’s Lemonade Stand Foundation, Philadelphia, PA, USA

    Casey S. Greene

  17. Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Tamara Broderick

  18. Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Jeffrey T. Leek

  19. Department of Statistics, University of British Columbia, Vancouver, British Columbia, Canada

    Keegan Korthauer

  20. BC Children’s Hospital Research Institute, Vancouver, British Columbia, Canada

    Keegan Korthauer

  21. European Molecular Biology Laboratory, Genome Biology Unit, Heidelberg, Germany

    Wolfgang Huber

  22. European Molecular Biology Laboratory, European Bioinformatics Institute, EMBL-EBI, Hinxton, UK

    Alvis Brazma

  23. McGill University, Montreal, Quebec, Canada

    Joelle Pineau

  24. Montreal Institute for Learning Algorithms, Quebec, Canada

    Joelle Pineau

  25. Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA

    Robert Tibshirani, Trevor Hastie & John P. A. Ioannidis

  26. Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA

    Robert Tibshirani, Trevor Hastie & John P. A. Ioannidis

  27. Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA

    John P. A. Ioannidis

  28. Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA

    John P. A. Ioannidis

  29. Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA

    John P. A. Ioannidis

  30. Department of Biostatistics, Harvard T.H Chan School of Public Health, Boston, MA, USA

    John Quackenbush

  31. Channing Division of Network Medicine, Brigham and Women’s Hospital, Boston, MA, USA

    John Quackenbush

  32. Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA

    John Quackenbush

  33. Radiology and Nuclear Medicine, Maastricht University, Maastricht, The Netherlands

    Hugo J. W. L. Aerts

  34. Cardiovascular Imaging Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA

    Hugo J. W. L. Aerts

  35. National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, USA

    Thakkar Shraddha & Weida Tong

  36. Immuneering Corporation, Cambridge, MA, USA

    Rebecca Kusko

  37. Engineering Science Department, Oxford e-Research Centre, University of Oxford, Oxford, UK

    Susanna-Assunta Sansone

  38. SAS Institute Inc, Cary, NC, USA

    Russ D. Wolfinger

  39. Weill Cornell Medicine, New York, NY, USA

    Christopher E. Mason

  40. Q2 Solutions, Morrisville, NC, USA

    Wendell Jones

  41. Hospital Virgen del Rocio, Sevilla, Spain

    Joaquin Dopazo

  42. Fondazione Bruno Kessler, Trento, Italy

    Cesare Furlanello

Authors
  1. Benjamin Haibe-Kains
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Alexandru Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Hosny
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Farnoosh Khodakarami
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Levi Waldron
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chris McIntosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Anna Goldenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Anshul Kundaje
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Casey S. Greene
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Tamara Broderick
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Michael M. Hoffman
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Jeffrey T. Leek
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Keegan Korthauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Wolfgang Huber
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Alvis Brazma
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Joelle Pineau
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Robert Tibshirani
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Trevor Hastie
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. John P. A. Ioannidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. John Quackenbush
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Hugo J. W. L. Aerts
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

Massive Analysis Quality Control (MAQC) Society Board of Directors

  • Thakkar Shraddha
  • , Rebecca Kusko
  • , Susanna-Assunta Sansone
  • , Weida Tong
  • , Russ D. Wolfinger
  • , Christopher E. Mason
  • , Wendell Jones
  • , Joaquin Dopazo
  •  & Cesare Furlanello

Contributions

B.H.-K. and G.A.A. wrote the first draft of the manuscript. B.H.-K. and H.J.W.L.A. designed and supervised the study. A.H., F.K., T.S., R.K., S.-A.S., W.T., R.D.W., C.E.M., W.J., J.D., C.F., L.W., B.W., C. McIntosh, A.G., A.K., C.S.G., T.B., M.M.H., J.T.L., K.K., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Benjamin Haibe-Kains.

Ethics declarations

Competing interests

A.H. is a shareholder of and receives consulting fees from Altis Labs. M.M.H. received a GPU Grant from Nvidia. H.J.W.L.A. is a shareholder of and receives consulting fees from Onc.AI. B.H.K. is a scientific advisor for Altis Labs. C.M. holds an equity position in Bridge7Oncology and receives royalties from RaySearch Laboratories. A.K. is on the SAB of ImmuneAI Inc, a consultant for Biogen Inc., a scientific co-founder of RavelBio Inc. and a shareholder of Freenome Inc. G.A.A., F.K., L.W., B.W., C.S.G., J.T.L., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. declare no other competing interests related to the manuscript.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Haibe-Kains, B., Adam, G.A., Hosny, A. et al. Transparency and reproducibility in artificial intelligence. Nature 586, E14–E16 (2020). https://doi.org/10.1038/s41586-020-2766-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-020-2766-y

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing