Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Transparency and reproducibility in artificial intelligence

Matters Arising to this article was published on 14 October 2020

The Original Article was published on 01 January 2020

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Data availability

No data have been generated as part of this manuscript.

References

  1. 1.

    McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020).

    ADS  CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Bluemke, D. A. et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers—from the Radiology editorial board. Radiology 293, 315–316 (2019).

    Article  Google Scholar 

  3. 3.

    Gundersen, O. E., Gil, Y. & Aha, D. W. On reproducible AI: towards reproducible research, open science, and digital scholarship in AI publications. AI Mag. 39, 56–68 (2018).

    Article  Google Scholar 

  4. 4.

    Crane, M. Questionable answers in question answering research: reproducibility and variability of published results. Trans. Assoc. Comput. Linguist. 6, 241–252 (2018).

    Article  Google Scholar 

  5. 5.

    Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).

  6. 6.

    Stodden, V. et al. Enhancing reproducibility for computational methods. Science 354, 1240–1241 (2016).

    ADS  CAS  Article  PubMed  Google Scholar 

  7. 7.

    Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).

    ADS  Article  PubMed  Google Scholar 

  8. 8.

    Bzdok, D. & Ioannidis, J. P. A. Exploration, inference, and prediction in neuroscience and biomedicine. Trends Neurosci. 42, 251–262 (2019).

    CAS  Article  PubMed  Google Scholar 

  9. 9.

    Gundersen, O. E. & Kjensmo, S. State of the art: Reproducibility in artificial intelligence. In Thirty-second AAAI Conference on Artificial Intelligence (AAAI-18) 1644–1651 (2018).

  10. 10.

    Shorten, C. & Khoshgoftaar, T. M. A survey on Image Data Augmentation for Deep Learning. J. Big Data 6, 60 (2019).

    Article  Google Scholar 

  11. 11.

    Lee, R. S. et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 170177 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Wallach, J. D., Boyack, K. W. & Ioannidis, J. P. A. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol. 16, e2006930 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Amann, R. I. et al. Toward unrestricted use of public genomic data. Science 363, 350–352 (2019).

    ADS  CAS  Article  PubMed  Google Scholar 

  14. 14.

    Carlson, B. Putting oncology patients at risk. Biotechnol. Healthc. 9, 17–21 (2012).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank S. McKinney and colleagues for their prompt and open communication regarding the materials and methods of their study. This work was supported in part by the National Cancer Institute (R01 CA237170).

Author information

Affiliations

Authors

Consortia

Contributions

B.H.-K. and G.A.A. wrote the first draft of the manuscript. B.H.-K. and H.J.W.L.A. designed and supervised the study. A.H., F.K., T.S., R.K., S.-A.S., W.T., R.D.W., C.E.M., W.J., J.D., C.F., L.W., B.W., C. McIntosh, A.G., A.K., C.S.G., T.B., M.M.H., J.T.L., K.K., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Benjamin Haibe-Kains.

Ethics declarations

Competing interests

A.H. is a shareholder of and receives consulting fees from Altis Labs. M.M.H. received a GPU Grant from Nvidia. H.J.W.L.A. is a shareholder of and receives consulting fees from Onc.AI. B.H.K. is a scientific advisor for Altis Labs. C.M. holds an equity position in Bridge7Oncology and receives royalties from RaySearch Laboratories. A.K. is on the SAB of ImmuneAI Inc, a consultant for Biogen Inc., a scientific co-founder of RavelBio Inc. and a shareholder of Freenome Inc. G.A.A., F.K., L.W., B.W., C.S.G., J.T.L., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. declare no other competing interests related to the manuscript.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Haibe-Kains, B., Adam, G.A., Hosny, A. et al. Transparency and reproducibility in artificial intelligence. Nature 586, E14–E16 (2020). https://doi.org/10.1038/s41586-020-2766-y

Download citation

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing