Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Guidelines for study protocols describing predefined validations of prediction models in medical deep learning and beyond

Nature Machine Intelligence volume 6, pages 2–3 (2024)Cite this article

Subjects

In a recent issue of Nature Machine Intelligence, Dhiman et al.1 highlight the importance of planning evaluations of deep learning systems in advance by predefining study protocols. We applaud this objective as such planning should prevent the evaluation of multiple systems and presentation of only the best result, which would almost certainly be overoptimistic for the intended application of the system. A dissemination presenting only the overoptimistic result may still conform with established reporting guidelines, such as transparent reporting of a multivariable prediction model for individual prognosis or diagnosis2 (TRIPOD). To avoid creative data dredging, it is crucial to agree on a primary analysis prior to validating the deep learning system. Formalizing this primary analysis in a study protocol that predefines all relevant aspects of the evaluation, including the development of the deep learning system, will enhance transparency and research integrity for the researchers involved and everyone else. It also enables the researchers to identify and discuss different opinions and agree on an analysis plan without being influenced by analyses of the external cohorts.

In response to this need, we previously defined a list of recommended protocol items for external cohort evaluation of a deep learning system3 (PIECES). In addition to an explicit specification of the primary analysis and any pre-planned secondary analyses that authors wish to commit themselves to report on, the PIECES recommendation requests that they describe precisely how the system was developed and will be assayed in the evaluation as well as details of the external cohorts to be evaluated, including their origin, exclusions, representation, target population, acquisition of input data, and ascertainment of target output. It also requests a declaration of the status at the point of finalizing the protocol, which should scrupulously elucidate any investigations performed before the protocol was finalized that could potentially reveal correlations between input data and target output in the external cohorts. Although the PIECES recommendation was designed with a special interest in deep learning systems predicting diagnosis, prognosis or treatment response in cancer research, it is readily applicable to other statistical and machine learning systems and for other prediction tasks.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

  1. Dhiman, P. et al. Nat. Mach. Intell. 5, 816–817 (2023).

    Article  Google Scholar 

  2. Moons, K. G. M. et al. Ann. Intern. Med. 162, W1–W73 (2015).

    Article  Google Scholar 

  3. Kleppe, A. et al. Nat. Rev. Cancer 21, 199–211 (2021).

  4. Korevaar, D. A. et al. Clin. Chem. 63, 1331–1341 (2017).

    Article  Google Scholar 

  5. Skrede, O.-J. et al. Lancet 395, 350–360 (2020).

    Article  Google Scholar 

  6. Cyll, K. et al. Cancers 13, 4291 (2021).

    Article  Google Scholar 

  7. van der Laak, J., Litjens, G. & Ciompi, F. Nat. Med. 27, 775–784 (2021).

    Article  Google Scholar 

  8. Ghaffari Laleh, N., Ligero, M., Perez-Lopez, R. & Kather, J. N. Clin. Cancer Res. 29, 316–323 (2023).

    Article  Google Scholar 

  9. Pisula, J. I. et al. Br. J. Cancer 128, 1369–1376 (2023).

    Article  Google Scholar 

  10. Nature 556, 273–274 (2018).

Download references

Author information

Author notes

  1. Deceased: Håvard E. Danielsen.

Authors and Affiliations

  1. Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway

    Andreas Kleppe, Ole-Johan Skrede, Knut Liestøl & Håvard E. Danielsen

  2. Department of Informatics, University of Oslo, Oslo, Norway

    Andreas Kleppe, Ole-Johan Skrede & Knut Liestøl

  3. Centre for Research-based Innovation Visual Intelligence, UiT The Arctic University of Norway, Tromsø, Norway

    Andreas Kleppe

  4. Nuffield Division of Clinical Laboratory Sciences, University of Oxford, Oxford, UK

    David J. Kerr & Håvard E. Danielsen

Authors
  1. Andreas Kleppe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ole-Johan Skrede
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Knut Liestøl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David J. Kerr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Håvard E. Danielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.K. and O.-J.S. drafted the manuscript, and all authors contributed to reviewing and editing the manuscript.

Corresponding author

Correspondence to Andreas Kleppe.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kleppe, A., Skrede, OJ., Liestøl, K. et al. Guidelines for study protocols describing predefined validations of prediction models in medical deep learning and beyond. Nat Mach Intell 6, 2–3 (2024). https://doi.org/10.1038/s42256-023-00774-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-023-00774-7

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research