Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Nature Cancer volume 3, pages 1026–1038 (2022)Cite this article

Subjects

Abstract

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Clinical applications of AI-based computational pathology.
Fig. 2: A CNN.
Fig. 3: Implications of computational pathology for research and diagnostics.
Fig. 4: Types of AI applications and pitfalls in histopathology.
Fig. 5: Principles of end-to-end weakly supervised prediction workflows.
Fig. 6: Robust computational pathology with focus on privacy-preserving deep learning.

References

  1. Dentro, S. C. et al. Characterizing genetic intra-tumor heterogeneity across 2,658 human cancer genomes. Cell 184, 2239–2254 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Burrell, R. A., McGranahan, N., Bartek, J. & Swanton, C. The causes and consequences of genetic heterogeneity in cancer evolution. Nature 501, 338–345 (2013).

    Article  CAS  PubMed  Google Scholar 

  3. Yates, L. R. & Campbell, P. J. Evolution of the cancer genome. Nat. Rev. Genet. 13, 795–806 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Rabbie, R., Lau, D., White, R. M. & Adams, D. J. Unraveling the cartography of the cancer ecosystem. Genome Biol. 22, 87 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Merlo, L. M. F., Pepper, J. W., Reid, B. J. & Maley, C. C. Cancer as an evolutionary and ecological process. Nat. Rev. Cancer 6, 924–935 (2006).

    Article  CAS  PubMed  Google Scholar 

  6. Cancer Genome Atlas Research Network. Integrated genomic analyses of ovarian carcinoma. Nature 474, 609–615 (2011).

    Article  Google Scholar 

  7. The Cancer Genome Atlas Network. Comprehensive molecular characterization of human colon and rectal cancer. Nature 487, 330–337 (2012).

  8. The Cancer Genome Atlas Research Network. Comprehensive molecular portraits of human breast tumours. Nature 490, 61–70 (2012).

  9. The Cancer Genome Atlas Research Network. Comprehensive genomic characterization of squamous cell lung cancers. Nature 489, 519–525 (2012).

  10. Cancer Genome Atlas Research Network et al. Integrated genomic characterization of endometrial carcinoma. Nature 497, 67–73 (2013).

    Article  Google Scholar 

  11. Cancer Genome Atlas Network. Genomic classification of cutaneous melanoma. Cell 161, 1681–1696 (2015).

    Article  Google Scholar 

  12. Chen, F. et al. Moving pan-cancer studies from basic research toward the clinic. Nat. Cancer 2, 879–890 (2021).

    Article  PubMed  Google Scholar 

  13. Echle, A. et al. Deep learning in cancer pathology: a new generation of clinical biomarkers. Br. J. Cancer 124, 686–696 (2020).

  14. Prewitt, J. M. & Mendelsohn, M. L. The analysis of cell images. Ann. N. Y. Acad. Sci. 128, 1035–1053 (1966).

    Article  CAS  PubMed  Google Scholar 

  15. Onji, K. et al. Quantitative analysis of colorectal lesions observed on magnified endoscopy images. J. Gastroenterol. 46, 1382–1390 (2011).

    Article  PubMed  Google Scholar 

  16. Irshad, H. et al. Automated mitosis detection using texture, SIFT features and HMAX biologically inspired approach. J. Pathol. Inform. 4, S12 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  17. LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems (ed. Touretzky, D.) Vol. 2 (Morgan-Kaufmann, 1990).

  18. Lecun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. In Proceedings of the IEEE 2278–2324 (IEEE, 1998).

  19. Fu, Y. et al. Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat. Cancer 1, 800–810 (2020).

  20. Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Cancer 1, 789–799 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Brockmoeller, S. et al. Deep learning identifies inflamed fat as a risk factor for lymph node metastasis in early colorectal cancer. J. Pathol. 256, 269–281 (2021).

  22. Wulczyn, E. et al. Interpretable survival prediction for colorectal cancer using deep learning. NPJ Digit. Med. 4, 71 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J. Pathol. Inform. 7, 29 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  24. Ozkan, T. A. et al. Interobserver variability in Gleason histological grading of prostate cancer. Scand. J. Urol. 50, 420–424 (2016).

    Article  CAS  PubMed  Google Scholar 

  25. Gurcan, M. N. et al. Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Sui, D. et al. A pyramid architecture-based deep learning framework for breast cancer detection. BioMed Res. Int. 2021, 2567202 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Gehrung, M. et al. Triage-driven diagnosis of Barrett’s esophagus for early detection of esophageal adenocarcinoma using deep learning. Nat. Med. 27, 833–841 (2021).

    Article  CAS  PubMed  Google Scholar 

  29. Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021).

  30. Yang, H. et al. Deep learning-based six-type classifier for lung cancer and mimics from histopathological whole slide images: a retrospective study. BMC Med. 19, 80 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  31. Lu, M. Y. et al. AI-based pathology predicts origins for cancers of unknown primary. Nature 594, 106–110 (2021).

  32. Nagpal, K. et al. Publisher Correction: Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit. Med. 2, 113 (2019).

  33. Bulten, W. et al. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol. 21, 233–241 (2020).

    Article  PubMed  Google Scholar 

  34. Ström, P. et al. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. Lancet Oncol. 21, 222–232 (2020).

    Article  PubMed  Google Scholar 

  35. Balkenhol, M. C. A. et al. Deep learning assisted mitotic counting for breast cancer. Lab. Invest. 99, 1596–1606 (2019).

    Article  PubMed  Google Scholar 

  36. Veta, M., van Diest, P. J., Jiwa, M., Al-Janabi, S. & Pluim, J. P. W. Mitosis counting in breast cancer: object-level interobserver agreement and comparison to an automatic method. PLoS ONE 11, e0161286 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  37. Calderaro, J. & Kather, J. N. Artificial intelligence-based pathology for gastrointestinal and hepatobiliary cancers. Gut 70, 1183–1193 (2021).

    Article  CAS  PubMed  Google Scholar 

  38. Saillard, C. et al. Predicting survival after hepatocellular carcinoma resection using deep-learning on histological slides. Hepatology 72, 2000–2013 (2020).

  39. Wulczyn, E. et al. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS ONE 15, e0233678 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

    Article  CAS  PubMed  Google Scholar 

  41. Heinz, C. N., Echle, A., Foersch, S., Bychkov, A. & Kather, J. N. The future of artificial intelligence in digital pathology—results of a survey across stakeholder groups. Histopathology 80, 1121–1127 (2022).

  42. Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. J. W. L. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500–510 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Bera, K., Braman, N., Gupta, A., Velcheti, V. & Madabhushi, A. Predicting cancer outcomes with radiomics and artificial intelligence in radiology. Nat. Rev. Clin. Oncol. 19, 132–146 (2021).

  44. Hughes, J. W. et al. Deep learning evaluation of biomarkers from echocardiogram videos. EBioMedicine 73, 103613 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Galon, J. et al. Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science 313, 1960–1964 (2006).

    Article  CAS  PubMed  Google Scholar 

  46. Pagès, F. et al. International validation of the consensus Immunoscore for the classification of colon cancer: a prognostic and accuracy study. Lancet 391, 2128–2139 (2018).

    Article  PubMed  Google Scholar 

  47. Kleppe, A. et al. Chromatin organisation and cancer prognosis: a pan-cancer study. Lancet Oncol. 19, 356–369 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med. 16, e1002730 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  49. Bychkov, D. et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci. Rep. 8, 3395 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Skrede, O.-J. et al. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet 395, 350–360 (2020).

    Article  CAS  PubMed  Google Scholar 

  51. Courtiol, P. et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25, 1519–1525 (2019).

  52. Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Howard, F. M. et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat. Commun. 12, 4423 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Kleppe, A. et al. Designing deep learning studies in cancer diagnostics. Nat. Rev. Cancer 21, 199–211 (2021).

  55. Madabhushi, A., Wang, X., Barrera, C. & Velcheti, V. Predicting response to immunotherapy using computer extracted features of cancer nuclei from hematoxylin and eoisin (H&E) stained images of non-small cell lung cancer (NSCLC). US Patent 11055844B2 (2019).

  56. Farahmand, S. et al. Deep learning trained on hematoxylin and eosin tumor region of interest predicts HER2 status and trastuzumab treatment response in HER2+ breast cancer. Mod. Pathol. 35, 44–51 (2021).

  57. Kather, J. N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat. Med. 25, 1054–1056 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  58. Echle, A. et al. Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning. Gastroenterology 159, 1406–1416 (2020).

    Article  CAS  PubMed  Google Scholar 

  59. Yamashita, R. et al. Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study. Lancet Oncol. 22, 132–141 (2021).

    Article  PubMed  Google Scholar 

  60. Echle, A. et al. Deep learning for the detection of microsatellite instability from histology images in colorectal cancer: a systematic literature review. ImmunoInformatics 3-4, 100008 (2021).

    Article  CAS  Google Scholar 

  61. Bilal, M., Raza, S. E. A., Azam, A., Graham, S. & Ilyas, M. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study. Lancet Digit. Health 3, e763–e772 (2021).

  62. Muti, H. S. et al. Development and validation of deep learning classifiers to detect Epstein–Barr virus and microsatellite instability status in gastric cancer: a retrospective multicentre cohort study. Lancet Digit. Health 3, e654–e664 (2021).

  63. Hong, R., Liu, W., DeLair, D., Razavian, N. & Fenyö, D. Predicting endometrial cancer subtypes and molecular features from histopathology images using multi-resolution deep learning models. Cell Rep. Med. 2, 100400 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Le, D. T. et al. PD-1 blockade in tumors with mismatch-repair deficiency. N. Engl. J. Med. 372, 2509–2520 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Harder, N. et al. Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma. Sci. Rep. 9, 7449 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Popovici, V. et al. Joint analysis of histopathology image features and gene expression in breast cancer. BMC Bioinformatics 17, 209 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Schulz, S. et al. Multimodal deep learning for prognosis prediction in renal cancer. Front. Oncol. 11, 788740 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Hao, J., Kosaraju, S. C., Tsaku, N. Z., Song, D. H. & Kang, M. PAGE-Net: interpretable and integrative deep learning for survival analysis using histopathological images and genomic data. In Biocomputing 2020 355–366 (World Scientific, 2019).

  69. Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J. & Shah, S. P. Harnessing multimodal data integration to advance precision oncology. Nat. Rev. Cancer 22, 114–126 (2021).

  70. Nam, D., Chapiro, J., Paradis, V., Seraphin, T. P. & Kather, J. N. Artificial intelligence in liver diseases: improving diagnostics, prognostics and response prediction. JHEP Rep. 4, 100443 (2022).

  71. Greenson, J. K. et al. Pathologic predictors of microsatellite instability in colorectal cancer. Am. J. Surg. Pathol. 33, 126–133 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  72. Hyde, A. et al. A histology-based model for predicting microsatellite instability in colorectal cancers. Am. J. Surg. Pathol. 34, 1820–1829 (2010).

    Article  PubMed  Google Scholar 

  73. Couture, H. D. et al. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4, 30 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  74. Schaumberg, A. J., Rubin, M. A. & Fuchs, T. J. H&E-stained whole slide image deep learning predicts SPOP mutation state in prostate cancer. Preprint at bioRxiv https://doi.org/10.1101/064279 (2017).

  75. Schrammen, P. L. et al. Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology. J. Pathol. 256, 50–60 (2021).

  76. Sirinukunwattana, K. et al. Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning. Gut 70, 544–554 (2021).

    Article  CAS  PubMed  Google Scholar 

  77. Bilal, M. et al. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study. Lancet Digit. Health 3, e763–e772 (2021).

  78. Schirris, Y., Gavves, E., Nederlof, I., Horlings, H. M. & Teuwen, J. DeepSMILE: self-supervised heterogeneity-aware multiple instance learning for DNA damage response defect classification directly from H&E whole-slide images. Preprint at https://arxiv.org/abs/2107.09405 (2021).

  79. Schmauch, B. et al. A deep learning model to predict RNA-seq expression of tumours from whole slide images. Nat. Commun. 11, 3877 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Cao, R. et al. Development and interpretation of a pathomics-based model for the prediction of microsatellite instability in colorectal cancer. Theranostics 10, 11080–11091 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Binder, A. et al. Morphological and molecular breast cancer profiling through explainable machine learning. Nat. Mach. Intell. 3, 355–366 (2021).

    Article  Google Scholar 

  82. Yu, K.-H. et al. Classifying non-small cell lung cancer types and transcriptomic subtypes using convolutional neural networks. J. Am. Med. Inform. Assoc. 27, 757–769 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  83. Loeffler, C. M. L. et al. Artificial intelligence-based detection of FGFR3 mutational status directly from routine histology in bladder cancer: a possible preselection for molecular testing? Eur. Urol. Focus 8, 472–479 (2021).

  84. Levy-Jurgenson, A., Tekpli, X., Kristensen, V. N. & Yakhini, Z. Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer. Sci. Rep. 10, 18802 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  85. Baxi, V., Edwards, R., Montalto, M. & Saha, S. Digital pathology and artificial intelligence in translational medicine and clinical practice. Mod. Pathol. 35, 23–32 (2022).

    Article  CAS  PubMed  Google Scholar 

  86. AbdulJabbar, K. et al. Geospatial immune variability illuminates differential evolution of lung adenocarcinoma. Nat. Med. 26, 1054–1062 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).

    Article  PubMed  Google Scholar 

  88. Berglund, E. et al. Spatial maps of prostate cancer transcriptomes reveal an unexplored landscape of heterogeneity. Nat. Commun. 9, 2419 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  89. Erickson, A. et al. The spatial landscape of clonal somatic mutations in benign and malignant tissue. Preprint at bioRxiv https://doi.org/10.1101/2021.07.12.452018 (2021).

  90. Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 53, 1334–1347 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  91. Thrane, K., Eriksson, H., Maaskola, J., Hansson, J. & Lundeberg, J. Spatially resolved transcriptomics enables dissection of genetic heterogeneity in stage III cutaneous malignant melanoma. Cancer Res. 78, 5970–5979 (2018).

    Article  CAS  PubMed  Google Scholar 

  92. Zhao, T. et al. Spatial genomics enables multi-modal study of clonal heterogeneity in tissues. Nature 601, 85–91 (2022).

    Article  CAS  PubMed  Google Scholar 

  93. Lomakin, A. et al. Spatial genomics maps the structure, character and evolution of cancer clones. Preprint at bioRxiv https://doi.org/10.1101/2021.04.16.439912 (2021).

  94. Wang, D., Khosla, A., Gargeya, R. & Irshad, H. Deep learning for identifying metastatic breast cancer. Preprint at https://arxiv.org/abs/1606.05718 (2016).

  95. Gecer, B. et al. Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks. Pattern Recognit. 84, 345–356 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  96. Shaban, M. et al. Context-aware convolutional neural network for grading of colorectal cancer histology images. IEEE Trans. Med. Imaging 39, 2395–2405 (2020).

    Article  PubMed  Google Scholar 

  97. Ciga, O., Xu, T. & Martel, A. L. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022).

  98. Gadermayr, M. et al. Generative adversarial networks for facilitating stain-independent supervised and unsupervised segmentation: a study on kidney histology. IEEE Trans. Med. Imaging 38, 2293–2302 (2019).

    Article  PubMed  Google Scholar 

  99. de Bel, T., Hermsen, M., Kers, J., van der Laak, J. & Litjens, G. Stain-transforming cycle-consistent generative adversarial networks for improved segmentation of renal histopathology. International Conference on Medical Imaging with Deep Learning https://openreview.net/forum?id=BkxJkgSlx4 (2018).

  100. Xu, J. et al. Stacked Sparse Autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans. Med. Imaging 35, 119–130 (2016).

    Article  PubMed  Google Scholar 

  101. Noroozi, M. & Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision—ECCV 2016 69–84 (Springer Nature, 2016).

  102. Srinidhi, C. L., Ciga, O. & Martel, A. L. Deep neural network models for computational histopathology: a survey. Med. Image Anal. 67, 101813 (2021).

    Article  PubMed  Google Scholar 

  103. Koohbanani, N. A., Unnikrishnan, B., Khurram, S. A., Krishnaswamy, P. & Rajpoot, N. Self-Path: self-supervision for classification of pathology images with limited annotations. IEEE Trans. Med. Imaging 40, 2845–2856 (2021).

    Article  PubMed  Google Scholar 

  104. Gildenblat, J. & Klaiman, E. Self-supervised similarity learning for digital pathology. Preprint at https://arxiv.org/abs/1905.08139 (2019).

  105. Srinidhi, C. L., Kim, S. W., Chen, F.-D. & Martel, A. L. Self-supervised driven consistency training for annotation efficient histopathology image analysis. Med. Image Anal. 75, 102256 (2022).

    Article  PubMed  Google Scholar 

  106. Schirris, Y. et al. WeakSTIL: weak whole-slide image level stromal tumor infiltrating lymphocyte scores are all you need. Proc. SPIE 12039, Medical Imaging 2022: Digital and Computational Pathology 120390B (4 April 2022).

  107. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 8748–8763 (PMLR, 2021).

  108. Chen, R. J., Lu, M. Y., Chen, T. Y., Williamson, D. F. K. & Mahmood, F. Synthetic data in machine learning for medicine and healthcare. Nat. Biomed. Eng. 5, 493–497 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  109. Kather, J. N., Ghaffari Laleh, N., Foersch, S. & Truhn, D. Medical domain knowledge in domain-agnostic generative AI. NPJ Digit. Med. 5, 90 (2022).

  110. Levine, A. B. et al. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J. Pathol. 252, 178–188 (2020).

  111. Krause, J. et al. Deep learning detects genetic alterations in cancer histology generated by adversarial networks. J. Pathol. 254, 70–79 (2021).

    PubMed  Google Scholar 

  112. Xu, Y., Zhu, J.-Y., Chang, E. I.-C., Lai, M. & Tu, Z. Weakly supervised histopathology cancer image segmentation and classification. Med. Image Anal. 18, 591–604 (2014).

    Article  PubMed  Google Scholar 

  113. Xu, Y., Zhang, J., Chang, E. I.-C., Lai, M. & Tu, Z. Context-constrained multiple instance learning for histopathology image segmentation. Med. Image Comput. Comput. Assist. Interv. 15, 623–630 (2012).

    PubMed  Google Scholar 

  114. Couture, H. D., Marron, J. S., Perou, C. M., Troester, M. A. & Niethammer, M. Multiple instance learning for heterogeneous images: training a CNN for histopathology. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science (eds. Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) Vol. 11071 (Springer, 2018).

  115. Ghaffari Laleh, N. et al. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Med. Image Anal. 79, 102474 (2022).

  116. Uegami, W. et al. MIXTURE of human expertise and deep learning—developing an explainable model for predicting pathological diagnosis and survival in patients with interstitial lung disease. Mod. Pathol. 35, 1083–1091 (2022).

  117. Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).

  118. Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. Preprint at https://arxiv.org/abs/2010.11929 (2020).

  119. Touvron, H. et al. Training data-efficient image transformers & distillation through attention. Preprint at https://arxiv.org/abs/2012.12877 (2020).

  120. Paul, S. & Chen, P.-Y. Vision transformers are robust learners. Proc. AAAI Conference on Artificial Intelligence 36, 2 (2022).

  121. Laleh, N. G. et al. Adversarial attacks and adversarial robustness in computational pathology. Preprint at bioRxiv https://doi.org/10.1101/2022.03.15.484515 (2022).

  122. Chen, X., Hsieh, C.-J. & Gong, B. When vision transformers outperform ResNets without pre-training or strong data augmentations. Preprint at https://arxiv.org/abs/2106.01548 (2021).

  123. Lu, M. Y. et al. Federated learning for computational pathology on gigapixel whole slide images. Med. Image Anal. 76, 102298 (2022).

  124. Warnat-Herresthal, S. et al. Swarm Learning for decentralized and confidential clinical machine learning. Nature 594, 265–270 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  125. Saldanha, O. L. et al. Swarm learning for decentralized artificial intelligence in cancer histopathology. Nat. Med. 28, 1232–1239 (2022).

  126. DuMont Schütte, A. et al. Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation. NPJ Digit. Med. 4, 141 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  127. Chen, D., Yu, N., Zhang, Y. & Fritz, M. GAN-Leaks: a taxonomy of membership inference attacks against generative models. Preprint at https://arxiv.org/abs/1909.03935 (2019).

  128. Schömig-Markiefka, B. et al. Quality control stress test for deep learning-based diagnostic model in digital pathology. Mod. Pathol. 34, 2098–2108 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  129. Janowczyk, A., Zuo, R., Gilmore, H., Feldman, M. & Madabhushi, A. HistoQC: an open-source quality control tool for digital pathology slides. JCO Clin. Cancer Inform. 3, 1–7 (2019).

    Article  PubMed  Google Scholar 

  130. Ren, J., Hacihaliloglu, I., Singer, E. A., Foran, D. J. & Qi, X. Unsupervised domain adaptation for classification of histopathology whole-slide images. Front. Bioeng. Biotechnol. 7, 102 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  131. Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H. & Ferrante, E. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl Acad. Sci. USA 117, 12592–12594 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  132. Cirillo, D. et al. Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. NPJ Digit. Med. 3, 81 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  133. Kundu, S. AI in medicine must be explainable. Nat. Med. 27, 1328 (2021).

    Article  CAS  PubMed  Google Scholar 

  134. Ghassemi, M., Oakden-Rayner, L. & Beam, A. L. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit. Health 3, e745–e750 (2021).

    Article  PubMed  Google Scholar 

  135. Foersch, S. et al. Deep learning for diagnosis and survival prediction in soft tissue sarcoma. Ann. Oncol. 32, 1178–1187 (2021).

    Article  CAS  PubMed  Google Scholar 

  136. Carter, S., Armstrong, Z., Schubert, L., Johnson, I. & Olah, C. Activation Atlas. Distill 4.3, e15 (2019); https://distill.pub/2019/activation-atlas/

  137. Goh, G. et al. Multimodal neurons in artificial neural networks. Distill https://doi.org/10.23915/distill.00030 (2021).

  138. Tjoa, E. & Guan, C. A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans. Neural Netw. Learn. Syst. 32, 4793–4813 (2021).

    Article  PubMed  Google Scholar 

  139. Gunning, D. et al. XAI—explainable artificial intelligence. Sci. Robot. 4, eaay7120 (2019).

    Article  PubMed  Google Scholar 

  140. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. Preprint at https://arxiv.org/abs/2010.00747 (2020).

  141. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  142. Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  143. van Treeck, M. et al. DeepMed: a unified, modular pipeline for end-to-end deep learning in computational pathology. Preprint at bioRxiv https://doi.org/10.1101/2021.12.19.473344 (2021).

  144. Pocock, J. et al. TIAToolbox: an end-to-end toolbox for advanced tissue image analytics. Preprint at bioRxiv https://doi.org/10.1101/2021.12.23.474029 (2021).

  145. Rosenthal, J. et al. Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the PathML toolkit for computational pathology. Mol. Cancer Res. 20, 202–206 (2021).

  146. Dolezal, J., Kochanny, S. & Howard, F. jamesdolezal/slideflow: slideflow 1.0—official public release. Zenodo https://doi.org/10.5281/zenodo.5708490 (2021).

  147. Cruz Rivera, S. et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat. Med. 26, 1351–1363 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  148. Norgeot, B. et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat. Med. 26, 1320–1324 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  149. Pantanowitz, L. et al. An artificial intelligence algorithm for prostate cancer diagnosis in whole slide images of core needle biopsies: a blinded clinical validation and deployment study. Lancet Digit. Health 2, e407–e416 (2020).

    Article  PubMed  Google Scholar 

  150. Center for Devices & Radiological Health. Good Machine Learning Practice for Medical Device Development https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles (2021).

  151. Office of the Commissioner. FDA Authorizes Software that Can Help Identify Prostate Cancer https://www.fda.gov/news-events/press-announcements/fda-authorizes-software-can-help-identify-prostate-cancer (2021).

  152. Kather, J. N. & Calderaro, J. Development of AI-based pathology biomarkers in gastrointestinal and liver cancer. Nat. Rev. Gastroenterol. Hepatol. 17, 591–592 (2020).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

J.N.K. is supported by the German Federal Ministry of Health (DEEP LIVER, ZMVI1-2520DAT111) and the Max-Eder-Programme of the German Cancer Aid (grant 70113864). No other specific funding for this work is declared by any of the authors.

Author information

Author notes

  1. These authors contributed equally: Artem Shmatko, Narmin Ghaffari Laleh.

Authors and Affiliations

  1. Division of AI in Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany

    Artem Shmatko & Moritz Gerstung

  2. European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK

    Artem Shmatko & Moritz Gerstung

  3. Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany

    Narmin Ghaffari Laleh & Jakob Nikolas Kather

  4. Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany

    Jakob Nikolas Kather

  5. Pathology and Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds, Leeds, UK

    Jakob Nikolas Kather

  6. Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany

    Jakob Nikolas Kather

Authors
  1. Artem Shmatko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Narmin Ghaffari Laleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Moritz Gerstung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jakob Nikolas Kather
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors jointly designed the outline, wrote the article and agreed to submit the manuscript for publication.

Corresponding authors

Correspondence to Moritz Gerstung or Jakob Nikolas Kather.

Ethics declarations

Competing interests

J.N.K. declares consulting services for Owkin, France, as well as honoraria from Roche, Eisai and MSD. The other authors declare no competing interests.

Peer review

Peer review information

Nature Cancer thanks Hamid Ali and Olivier Elemento for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shmatko, A., Ghaffari Laleh, N., Gerstung, M. et al. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat Cancer 3, 1026–1038 (2022). https://doi.org/10.1038/s43018-022-00436-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43018-022-00436-4

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer