Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Artificial intelligence applications in histopathology

Nature Reviews Electrical Engineering volume 1, pages 93–108 (2024)Cite this article

Subjects

Abstract

Histopathology is a vital diagnostic discipline in medicine, fundamental to our understanding, detection, assessment and treatment of conditions such as cancer, dementia and heart disease. Traditionally, the standard workflow in histopathology has primarily relied on the visual interpretation of tissue samples carried out by human experts under a light microscope. Since the 2000s, thanks to advances in scanning technologies such as whole-slide imaging, histopathology is undergoing a digital transformation. The rapid increase in digital data is fuelling the development and application of artificial intelligence (AI) methods. In this Review, we delve into the latest progress in AI methods for histopathology, which promise to yield accurate, scalable, useful and affordable support tools for clinical decision. We examine the challenges and opportunities in this domain, exploring historically important approaches and problems that have shaped the field, while also highlighting recent technological breakthroughs that are poised to redefine its future. Furthermore, we offer an overview of publicly available datasets that have been instrumental in propelling the development of AI methods in histopathology.

Key points

  • Digital pathology largely benefits from weak and self-supervision, as fine-grained supervision requires costly annotations. Foundation models trained with large pathology datasets in a self-supervised manner show high performance in downstream tasks.

  • Various model choices, such as convolutional neural networks, vision transformers and graph neural networks are used for digital pathology tasks. Different models exhibit different costs and benefits, such as the amount of data training needed and the ability to capture long-distance correlations.

  • Machine learning can be used in an array of histopathology interpretation tasks, such as cancer classification, tumour grading, genetic mutation prediction, cell classification, treatment planning and survival estimation.

  • Recent advancements show that neural networks can be trained to generate human-interpretable results, which is important to build user trust in healthcare settings.

  • Whole-slide image acquisition, data curation and annotation, and the hardware and expertise needed to train deep learning models can be difficult challenges for low-resource settings. There is considerable progress that aim to reduce the barriers in each of these directions.

  • Digital pathology slides show immense variability between different sites. Methods such as federated learning deployed on multisite data can aid in creating more generalizable models.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic diagram of the artificial intelligence integrated pathology diagnostic workflow.
Fig. 2: Various self-supervision techniques used in digital pathology.
Fig. 3: Different levels of supervision on digital pathology images.
Fig. 4: Elements of a pseudo case study of a single patient.

Similar content being viewed by others

References

  1. Kumar, N., Gupta, R. & Gupta, S. Whole slide imaging (WSI) in pathology: current perspectives and future directions. J. Digit. Imaging 33, 1034–1040 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Jahn, S. W., Plass, M. & Moinfar, F. Digital pathology: advantages, limitations and emerging perspectives. J. Clin. Med. 9, 3697 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the Camelyon dataset. GigaScience 7, giy065 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Pataki, B. Á. et al. HunCRC: annotated pathological slides to enhance deep learning applications in colorectal cancer screening. Sci. Data 9, 370 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Bulten, W. et al. Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge. Nat. Med. 28, 154–163 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Madabhushi, A. Digital pathology image analysis: opportunities and challenges. Imaging Med. 1, 7–10 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Aeffner, F. et al. Introduction to digital image analysis in whole-slide imaging: a white paper from the digital pathology association. J. Pathol. Inform. 10, 9 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  8. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  PubMed  ADS  Google Scholar 

  9. Hennessy, J. L. & Patterson, D. A. A new golden age for computer architecture. Commun. ACM 62, 48–60 (2019).

    Article  Google Scholar 

  10. Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V. & Madabhushi, A. Artificial intelligence in digital pathology — new tools for diagnosis and precision. Nat. Rev. Clin. Oncol. 16, 703–715 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019). This paper is one of the pioneers in applying multiple instance learning (MIL) with a neural-network-based aggregation method in large-scale histopathology data spanning several cancer types and demonstrating the potential of weak supervision.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Diao, J. A. et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nat. Commun. 12, 1613 (2021). This paper uses deep learning methods in conjunction with human interpretable features to predict clinically relevant markers.

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  13. Glass, B. et al. Machine learning models to quantify HER2 for real-time tissue image analysis in prospective clinical trials. J. Clin. Oncol. 39, 3061 (2021).

    Article  Google Scholar 

  14. Echle, A. et al. Deep learning in cancer pathology: a new generation of clinical biomarkers. Br. J. Cancer 124, 686–696 (2021).

    Article  PubMed  Google Scholar 

  15. Shmatko, A., Ghaffari Laleh, N., Gerstung, M. & Kather, J. N. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat. Cancer 3, 1026–1038 (2022).

    Article  PubMed  Google Scholar 

  16. Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16144–16155 (IEEE, 2022). This paper uses vision transformers in a newly defined hierarchical image pyramid transformer architecture that hierarchically combines representations from multiple fields of view, at the cell, patch and region levels, to obtain a slide-level representation.

  17. Qayyum, A., Qadir, J., Bilal, M. & Al-Fuqaha, A. Secure and robust machine learning for healthcare: a survey. IEEE Rev. Biomed. Eng. 14, 156–180 (2020). This paper is a survey of secure and robust machine learning methods for medical applications, presenting state-of-the-art methods that can aid in mitigating the risks.

    Article  Google Scholar 

  18. Tjoa, E. & Guan, C. A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans. Neural Netw. Learn. Syst. 32, 4793–4813 (2020).

    Article  Google Scholar 

  19. Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N. & Huang, J. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020). This paper implements an attention-based MIL method for classification of certain cancer types showing that attention can aid in pointing out relevant patches that contributed to the overall classification of the whole-slide image (WSI).

    Article  PubMed  Google Scholar 

  20. Hashimoto, N. et al. Multi-scale domain-adversarial multiple-instance CNN for cancer subtype classification with unannotated histopathological images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3852–3861 (IEEE, 2020). This paper implements an MIL model that benefits from the complementary information originating from different magnifications and mitigates potential performance variation across different sites with a domain-adversarial network.

  21. Abbet, C., Zlobec, I., Bozorgtabar, B. & Thiran, J.-P. Divide-and-rule: self-supervised learning for survival analysis in colorectal cancer. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2020: 23rd International Conference, Part V, 480–489 (Springer, 2020).

  22. Chang, J. C., Amershi, S. & Kamar, E. Revolt: collaborative crowdsourcing for labeling machine learning datasets. In Proc. 2017 CHI Conference on Human Factors in Computing Systems, 2334–2346 (ACM, 2017).

  23. Bertram, C. A., Aubreville, M., Marzahl, C., Maier, A. & Klopfleisch, R. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci. Data 6, 274 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Aubreville, M. et al. Mitosis domain generalization in histopathology images — the MIDOG challenge. Med. Image Anal. 84, 102699 (2023).

    Article  PubMed  Google Scholar 

  25. Karimi, D. et al. Deep learning-based Gleason grading of prostate cancer from histopathology images — role of multiscale decision aggregation and data augmentation. IEEE J. Biomed. Health Inform. 24, 1413–1426 (2019).

    Article  PubMed  Google Scholar 

  26. Zhang, Z. et al. Pathologist-level interpretable whole-slide cancer diagnosis with deep learning. Nat. Mach. Intell. 1, 236–245 (2019).

    Article  Google Scholar 

  27. Graham, S. et al. Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019). This paper implements a convolutional neural network architecture that uses distances to the nucleus as an important feature for segmenting and classifying different type of cells.

    Article  PubMed  Google Scholar 

  28. Sharmay, Y., Ehsany, L., Syed, S. & Brown, D. E. HistoTransfer: understanding transfer learning for histopathology. In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), https://doi.org/10.1109/BHI50953.2021.9508542 (IEEE, 2021).

  29. Mormont, R., Geurts, P. & Marée, R. Comparison of deep transfer learning strategies for digital pathology. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2262–2271 (IEEE, 2018).

  30. Kang, M., Song, H., Park, S., Yoo, D. & Pereira, S. Benchmarking self-supervised learning on diverse pathology datasets. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3344–3354 (IEEE, 2023). This is a recent paper that conducts an extensive study which shows that using models trained with self-supervised methods in large pathology datasets have superior performance compared to models trained with supervised methods on public computer vision datasets such as ImageNet.

  31. Maron, O. & Lozano-Pérez, T. A framework for multiple-instance learning. Adv. Neur. Inf. Proc. Syst. 10, 570–576 (1997).

    Google Scholar 

  32. Zhang, Z. & Sabuncu, M. Self-distillation as instance-specific label smoothing. Adv. Neural Inf. Process. Syst. 33, 2184–2195 (2020).

    Google Scholar 

  33. Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J. A visual–language foundation model for pathology image analysis using medical twitter. Nat. Med. 29, 2307–2316 (2023). This paper implements the contrastive language-image pretraining method on pathology image-caption pairs extracted from medical Twitter, which is then used for several downstream pathology tasks.

    Article  CAS  PubMed  Google Scholar 

  34. Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (IEEE, 2009).

  35. Srinidhi, C. L. & Martel, A. L. Improving self-supervised learning with hardness-aware dynamic curriculum learning: an application to digital pathology. In Proc. IEEE/CVF International Conference on Computer Vision, 562–571 (IEEE, 2021).

  36. Boyd, J. et al. Self-supervised representation learning using visual field expansion on digital pathology. In Proc. IEEE/CVF International Conference on Computer Vision, 639–647 (IEEE, 2021).

  37. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 1597–1607 (PMLR, 2020).

  38. Ciga, O., Xu, T. & Martel, A. L. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022). This paper conducts a large-scale study that shows models trained with self-supervised contrastive learning on a large pathology datasets can yield high performance in downstream pathology applications.

    Google Scholar 

  39. Li, J. et al. Lesion-aware contrastive representation learning for histopathology whole slide images analysis. In Proc. Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Part II, 273–282 (Springer, 2022).

  40. Yan, J., Chen, H., Li, X. & Yao, J. Deep contrastive learning based tissue clustering for annotation-free histopathology image analysis. Comput. Med. Imaging Graph. 97, 102053 (2022).

    Article  PubMed  Google Scholar 

  41. Li, J., Lin, T. & Xu, Y. Sslp: Spatial guided self-supervised learning on pathological images. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2021: 24th International Conference, Part II, 3–12 (Springer, 2021).

  42. Shen, Y., Luo, Y., Shen, D. & Ke, J. RandStainNA: learning stain-agnostic features from histology slides by bridging stain augmentation and normalization. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2022: 25th International Conference, Part II, 212–221 (Springer, 2022).

  43. Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In International Conference on Machine Learning, 2127–2136 (PMLR, 2018).

  44. Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021). This paper presents a method named constrained-attention multiple-instance learning (CLAM) that clusters patch representations in WSIs, using only informative patches in the final classification to lower the computational burden.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Zhang, H. et al. DTFD-MIL: double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18802–18812 (2022).

  46. Li, H. et al. DT-MIL: deformable transformer for multi-instance learning on histopathological image. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2021: 24th International Conference, Part VIII, 206–216 (Springer, 2021).

  47. Zhao, Y. et al. Setmil: spatial encoding transformer-based multiple instance learning for pathological image analysis. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2022: 25th International Conference, Part II, 66–76 (Springer, 2022).

  48. Shao, Z. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural Inf. Process. Syst. 34, 2136–2147 (2021). This paper implements a vision transformer in conjunction with learning an interpatch correlation matrix that yields a more sophisticated aggregation method that accounts for the correlations between different tiles in a WSI.

    Google Scholar 

  49. Chikontwe, P., Kim, M., Nam, S. J., Go, H. & Park, S. H. Multiple instance learning with center embeddings for histopathology classification. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2020: 23rd International Conference, Part V, 519–528 (Springer, 2020).

  50. Tu, C., Zhang, Y. & Ning, Z. Dual-curriculum contrastive multi-instance learning for cancer prognosis analysis with whole slide images. Adv. Neural Inf. Process. Syst. 35, 29484–29497 (2022).

    Google Scholar 

  51. Li, J. et al. A multi-resolution model for histopathology image classification and localization with multiple instance learning. Comput. Biol. Med. 131, 104253 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Ke, Z., Wang, D., Yan, Q., Ren, J. & Lau, R. W. Dual student: breaking the limits of the teacher in semi-supervised learning. In Proc. IEEE/CVF International Conference on Computer Vision, 6728–6736 (IEEE, 2019).

  53. Gou, J., Yu, B., Maybank, S. J. & Tao, D. Knowledge distillation: a survey. Int. J. Comput. Vis. 129, 1789–1819 (2021).

    Article  Google Scholar 

  54. Qu, L. et al. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Adv. Neural Inf. Process. Syst. 35, 15368–15381 (2022).

    Google Scholar 

  55. Javed, S., Mahmood, A., Qaiser, T. & Werghi, N. Knowledge distillation in histology landscape by multi-layer features supervision. IEEE J. Biomed. Health Inform. 27, 2037–2046 (2023).

    Article  Google Scholar 

  56. Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748–8763 (PMLR, 2021).

  57. Lin, W. et al. PMC-CLIP: contrastive language-image pre-training using biomedical documents. Preprint at https://doi.org/10.48550/arXiv.2303.07240 (2023).

  58. Gamper, J. & Rajpoot, N. Multiple instance captioning: learning representations from histopathology textbooks and articles. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16544–16554 (IEEE, 2021).

  59. Lu, M. Y. et al. Visual language pretrained multiple instance zero-shot transfer for histopathology images. Proc. IEEE Conference on Computer Vision and Pattern Recognition, 19764–19775 (IEEE, 2023).

  60. Moghadam, P. A. et al. A morphology focused diffusion probabilistic model for synthesis of histopathology images. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision, 2000–2009 (IEEE, 2023).

  61. Liechty, B. et al. Machine learning can aid in prediction of IDH mutation from H&E-stained histology slides in infiltrating gliomas. Sci. Rep. 12, 22623 (2022).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  62. Zanjani, F. G., Zinger, S., Bejnordi, B. E., van der Laak, J. A. & de With, P. H. Stain normalization of histopathology images using generative adversarial networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 573–577 (IEEE, 2018).

  63. Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. Adv. Neural Inf. Process. Syst. 33, 6840–6851 (2020).

    Google Scholar 

  64. Hou, L. et al. Patch-based convolutional neural network for whole slide tissue image classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2424–2433 (2016).

  65. Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations (2021).

  66. Caron,M. et al. Emerging properties in self-supervised vision transformers. In Proc. IEEE/CVF International Conference on Computer Vision, 9650–9660 (IEEE, 2021).

  67. Wang, X. et al. Transpath: transformer-based self-supervised learning for histopathological image classification. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2021: 24th International Conference, Part VIII, 186–195 (Springer, 2021).

  68. Qian, Z. et al. Transformer based multiple instance learning for weakly supervised histopathology image segmentation. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2022: 25th International Conference, Part II, 160–170 (Springer, 2022).

  69. Jaume, G. et al. Quantifying explainers of graph neural networks in computational pathology. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8102–8112 (IEEE, 2021).

  70. Lu, C. et al. Feature-driven local cell graph(FLocK): new computational pathology-based descriptors for prognosis of lung cancer and HPV status of oropharyngeal cancers. Med. Image Anal. 68, 101903 (2021).

    Article  PubMed  Google Scholar 

  71. Nakhli, R. et al. Sparse multi-modal graph transformer with shared-context processing for representation learning of giga-pixel images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11547–11557 (IEEE, 2023).

  72. Pati, P. et al. HACT-Net: a hierarchical cell-to-tissue graph neural network for histopathological image classification. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis, Vol. 12443 (eds Sudre, C. H. et al.) 208–219 (Springer, 2020).

  73. Wang, J., Chen, R. J., Lu, M. Y., Baras, A. & Mahmood, F. Weakly supervised prostate TMA classification via graph convolutional networks. https://doi.org/10.48550/arXiv.1910.13328 (2019).

  74. Lu, W., Toss, M., Rakha, E., Rajpoot, N. & Minhas, F. SlideGraph+: whole slide image level graphs to predict HER2Status in breast cancer. https://doi.org/10.48550/arXiv.2110.06042 (2021).

  75. Lu, W., Graham, S., Bilal, M., Rajpoot, N. & Minhas, F. Capturing cellular topology in multi-gigapixel pathology images. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1049–1058 (IEEE, 2020).

  76. Anklin, V. et al. Learning whole-slide segmentation from inexact and incomplete labels using tissue graphs. https://doi.org/10.48550/arXiv.2103.03129 (2021).

  77. Chen, R. J. et al. Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. https://doi.org/10.48550/arXiv.2107.13048 (2021).

  78. Zhou, J. et al. Graph neural networks: a review of methods and applications. AI Open. 1, 57–81 (2020).

    Article  Google Scholar 

  79. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018). This paper implements a framework that predicts several gene mutations from histopathology images, showing that deep learning models can mitigate the need for additional genetic testing.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Wang, X. et al. Weakly supervised deep learning for whole slide lung cancer image analysis. IEEE Trans. Cybern. 50, 3950–3962 (2020).

    Article  PubMed  ADS  Google Scholar 

  81. Kiani, A. et al. Impact of a deep learning assistant on the histopathologic classification of liver cancer. npj Digital Med. 3, 23 (2020).

    Article  Google Scholar 

  82. Bulten, W. et al. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol. 21, 233–241 (2020).

    Article  PubMed  Google Scholar 

  83. Harmon, S. A. et al. Multiresolution application of artificial intelligence in digital pathology for prediction of positive lymph nodes from primary tumors in bladder cancer. JCO Clin. Cancer Inf. 4, 367–382 (2020).

    Article  Google Scholar 

  84. Duanmu, H. et al. Spatial attention-based deep learning system for breast cancer pathological complete response prediction with serial histopathology images in multiple stains. In Medical Image Computing and Computer Assisted Intervention — MICCAI 2021 (eds de Bruijne, M. et al.) 550–560 (Springer, 2021).

  85. Wako, B. D. et al. Squamous cell carcinoma of skin cancer margin classification from digital histopathology images using deep learning. Cancer Control. 29, 10732748221132528 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  86. Oldenhuis, C., Oosting, S., Gietema, J. & De Vries, E. Prognostic versus predictive value of biomarkers in oncology. Eur. J. Cancer 44, 946–953 (2008).

    Article  CAS  PubMed  Google Scholar 

  87. Shaban, M. et al. A novel digital score for abundance of tumour infiltrating lymphocytes predicts disease free survival in oral squamous cell carcinoma. Sci. Rep. 9, 13341 (2019).

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  88. Chen, R. J. et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide Images. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 3995–4005 (IEEE, 2021). This paper implements a multimodal c-attention transformer (MCAT) that learns the relation between WSIs and genomic features which are then used for survival prediction, showing that WSIs can be used in conjunction with other data modalities to make clinical predictions.

  89. Zadeh, S. G. & Schmid, M. Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3126–3137 (2021).

    Article  PubMed  Google Scholar 

  90. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. https://doi.org/10.1126/scitranslmed.3002564 (2011).

  91. Rawat, R. R. et al. Deep learned tissue ‘fingerprints’ classify breast cancers by ER/PR/Her2 status from H&E images. Sci. Rep. 10, 7275 (2020).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  92. Schmauch, B. et al. A deep learning model to predict RNA-seq expression of tumours from whole slide images. Nat. Commun. 11, 3877 (2020).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  93. Vanguri, R. S. et al. Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non-small cell lung cancer. Nat. Cancer 3, 1151–1164 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  94. Gamper, J. et al. Pannuke dataset extension, insights and baselines. Preprint at https://doi.org/10.48550/arXiv.2003.10778 (2020).

  95. Hörst, F. et al. CellVIT: vision transformers for precise cell segmentation and classification. Preprint at https://doi.org/10.48550/arXiv.2306.15350 (2023).

  96. Bertram, C. A. et al. Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels. In Proc. Interpretable and Annotation-Efficient Learning for Medical Image Computing: 3rd International Workshop, iMIMIC 2020, Second International Workshop, MIL3ID 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, 204–213 (Springer, 2020).

  97. Piansaddhayanaon, C. et al. ReCasNet: improving consistency within the two-stage mitosis detection framework. Artif. Intell. Med. 135, 102462 (2023).

    Article  PubMed  Google Scholar 

  98. Balkenhol, M. C. A. et al. Deep learning assisted mitotic counting for breast cancer. Lab. Investig. 99, 1596–1606 (2019).

    Article  PubMed  Google Scholar 

  99. Pantanowitz, L. et al. Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses. Diagn. Pathol. 15, 80 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  100. Sirinukunwattana, K. et al. Gland segmentation in colon histology images: the GlaS challenge contest. Med. Image Anal. 35, 489–502 (2017).

    Article  PubMed  Google Scholar 

  101. Kim, Y. J. et al. PAIP 2019: liver cancer segmentation challenge. Med. Image Anal. 67, 101854 (2021).

    Article  PubMed  Google Scholar 

  102. Aresta, G. et al. BACH: grand challenge on breast cancer histology images. Med. Image Anal. 56, 122–139 (2019). This paper presents a grand challenge with breast cancer histology images (BACH), which attracted a large number of submissions, showing the importance of publicly available standardized datasets for the development and evaluation of machine learning models in the field of digital pathology.

    Article  PubMed  Google Scholar 

  103. Schmitz, R. et al. Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture. Med. Image Anal. 70, 101996 (2021).

    Article  PubMed  Google Scholar 

  104. Xu, Y. et al. Deep convolutional activation features for large scale brain tumor histopathology image classification and segmentation. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 947–951 (IEEE, 2015).

  105. Yi, F. et al. Microvessel prediction in H&E stained pathology images using fully convolutional neural networks. BMC Bioinform. 19, 64 (2018).

    Article  Google Scholar 

  106. Leiby, J. S., Hao,J., Kang, G. H., Park, J. W. & Kim, D. Attention-based multiple instance learning with self-supervision to predict microsatellite instability in colorectal cancer from histology whole-slide images. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 3068–3071 (IEEE, 2022).

  107. Van Rijthoven, M. et al. HookNet: multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images. Med. Image Anal. 68, 101890 (2021).

    Article  PubMed  Google Scholar 

  108. Ester, O. et al. Valuing vicinity: memory attention framework for context-based semantic segmentation in histopathology. Comput. Med. Imaging Graph. 107, 102238 (2023).

    Article  PubMed  Google Scholar 

  109. Macenko, M. et al. A method for normalizing histology slides for quantitative analysis. In Proc. 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 1107–1110 (IEEE, 2009). This paper developed a widely used stain normalization method called Macenko, which has been an essential tool for handling WSIs originating from different sites.

  110. Bejnordi, B. E. et al. Stain specific standardization of whole-slide histopathological images. IEEE Trans. Med. Imaging 35, 404–415 (2016).

    Article  PubMed  Google Scholar 

  111. Huang, T., Yang, G. & Tang, G. A fast two-dimensional median filtering algorithm. IEEE Trans. Acoust. Speech Signal. Process. 27, 13–18 (1979).

    Article  Google Scholar 

  112. Buades, A., Coll, B. & Morel, J.-M. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 60–65 (IEEE, 2005).

  113. Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095 (2007).

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  114. Ke, J., Deng, J. & Lu, Y. Noise reduction with image inpainting: an application in clinical data diagnosis. In ACM SIGGRAPH 2019 Posters (Association for Computing Machinery, 2019).

  115. Borovec, J. et al. ANHIR: automatic non-rigid histological image registration challenge. IEEE Trans. Med. Imaging 39, 3042–3052 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  116. Hering, A. et al. Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Trans. Med. Imaging 42, 697–712 (2022).

    Article  ADS  Google Scholar 

  117. Shao, W. et al. Prosregnet: a deep learning framework for registration of MRI and histopathology images of the prostate. Med. Image Anal. 68, 101919 (2021).

    Article  PubMed  Google Scholar 

  118. Roy, M. et al. Deep learning based registration of serial whole-slide histopathology images in different stains. J. Pathol. Inform. 14, 100311 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  119. Jeong, J. et al. Stain normalization using score-based diffusion model through stain separation and overlapped moving window patch strategies. Comput. Biol. Med. 152, 106335 (2023).

    Article  PubMed  Google Scholar 

  120. Bao, S. et al. Random multi-channel image synthesis for multiplexed immunofluorescence imaging. In MICCAI Workshop on Computational Pathology, 36–46 (PMLR, 2021).

  121. Ye, J. et al. A multi-attribute controllable generative model for histopathology image synthesis. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2021: 24th International Conference, Part VIII, 613–623 (Springer, 2021).

  122. Shrivastava, A. et al. Self-attentive adversarial stain normalization. In Proc. Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, Part I, 120–140 (Springer, 2021).

  123. Shrivastava, A. & Fletcher, P. T. NASDM: nuclei-aware semantic histopathology image generation using diffusion models. Preprint at https://doi.org/10.48550/arXiv.2303.11477 (2023).

  124. Dhariwal, P. & Nichol, A. Diffusion models beat GANs on image synthesis. Adv. Neural Inf. Process. Syst. 34, 8780–8794 (2021).

    Google Scholar 

  125. Boyce, B. Whole slide imaging: uses and limitations for surgical pathology and teaching. Biotech. Histochem. 90, 321–330 (2015).

    Article  CAS  PubMed  Google Scholar 

  126. Tellez, D., Litjens, G., van der Laak, J. & Ciompi, F. Neural image compression for gigapixel histopathology image analysis. IEEE Trans. Pattern Anal. Mach. Intell. 43, 567–578 (2019).

    Article  Google Scholar 

  127. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R. & Bengio, Y. Quantized neural networks: training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18, 6869–6898 (2017).

    MathSciNet  Google Scholar 

  128. Budd, S., Robinson, E. C. & Kainz, B. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Med. Image Anal. 71, 102062 (2021).

    Article  PubMed  Google Scholar 

  129. Menghani, G. Efficient deep learning: a survey on making deep learning models smaller, faster, and better. ACM Comput. Surv. 55, 1–37 (2023).

    Article  Google Scholar 

  130. Zeiser, F. A. et al. Deepbatch: a hybrid deep learning model for interpretable diagnosis of breast cancer in whole-slide images. Expert Syst. Appl. 185, 115586 (2021).

    Article  Google Scholar 

  131. Guo, Z. et al. A fast and refined cancer regions segmentation framework in whole-slide breast pathological images. Sci. Rep. 9, 882 (2019).

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  132. Frantar, E., Ashkboos, S., Hoefler, T. & Alistarh, D. OPTQ: accurate quantization for generative pre-trained transformers. In 11th International Conference on Learning Representations (2023).

  133. Patel, A. et al. Contemporary whole slide imaging devices and their applications within the modern pathology department: a selected hardware review. J. Pathol. Inform. 12, 50 (2021). This paper provides an extensive review of hardware technologies used in WSI acquisition and highlights potential areas in the acquisition pipeline that can benefit from advancements.

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  134. Hanna, M. G. et al. Integrating digital pathology into clinical practice. Mod. Pathol. 35, 152–164 (2022).

    Article  PubMed  Google Scholar 

  135. Bertram, C. A. & Klopfleisch, R. The pathologist 2.0: an update on digital pathology in veterinary medicine. Vet. Pathol. 54, 756–766 (2017).

    Article  PubMed  Google Scholar 

  136. McClintock, D. S., Abel, J. T. & Cornish, T. C. Whole Slide Imaging Hardware, Software, and Infrastructure (Whole Slide Imaging, 2022).

  137. Edelstein, A. D. et al. Advanced methods of microscope control using μManager software. J. Biol. Methods 1, e10 (2014).

    Article  PubMed  Google Scholar 

  138. Katare, P., Awasthi, N., Venukumar, A. & Gorthi, S. S. Low-cost, continuous motion imaging, computationally augmented whole slide imager for digital pathology. IEEE J. Sel. Top. Quantum Electron. 27, 1–7 (2021).

    Article  Google Scholar 

  139. Jiang, Y. et al. Recognizing basal cell carcinoma on smartphone-captured digital histopathology images with a deep neural network. Br. J. Dermatol. 182, 754–762 (2020).

    Article  CAS  PubMed  Google Scholar 

  140. Schömig-Markiefka, B. et al. Quality control stress test for deep learning-based diagnostic model in digital pathology. Mod. Pathol. 34, 2098–2108 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  141. Stacke, K., Eilertsen, G., Unger, J. & Lundström, C. Measuring domain shift for deep learning in histopathology. IEEE J. Biomed. Health Inform. 25, 325–336 (2020).

    Article  Google Scholar 

  142. Zhang, Y. et al. Benchmarking the robustness of deep neural networks to common corruptions in digital pathology. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2022: 25th International Conference, Part II, 242–252 (Springer, 2022).

  143. McCoy, L. G., Brenna, C. T., Chen, S. S., Vold, K. & Das, S. Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based. J. Clin. Epidemiol. 142, 252–257 (2022).

    Article  PubMed  Google Scholar 

  144. Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. 128, 336–359 (2020).

    Article  Google Scholar 

  145. Javed, S. A. et al. Additive mil: intrinsically interpretable multiple instance learning for pathology. Adv. Neural Inf. Process. Syst. 35, 20689–20702 (2022).

    Google Scholar 

  146. Chefer, H., Gur, S. & Wolf, L. Transformer interpretability beyond attention visualization. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 782–791 (2021).

  147. Kim, S., Nam, J. & Ko, B. C. ViT-NeT: interpretable vision transformers with neural tree decoder. In International Conference on Machine Learning, 11162–11172 (PMLR, 2022).

  148. L’Imperio, V. et al. Pathologist validation of a machine learning–derived feature for colon cancer risk stratification. JAMA Netw. Open. 6, e2254891 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  149. Markus, A. F., Kors, J. A. & Rijnbeek, P. R. The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. J. Biomed. Inform. 113, 103655 (2021).

    Article  PubMed  Google Scholar 

  150. Arrieta, A. B. et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion. 58, 82–115 (2020).

    Article  Google Scholar 

  151. Vodrahalli, K., Gerstenberg, T. & Zou, J. Y. Uncalibrated models can improve human–AI collaboration. Adv. Neural Inf. Process. Syst. 35, 4004–4016 (2022).

    Google Scholar 

  152. Babic, B., Gerke, S., Evgeniou, T. & Cohen, I. G. Beware explanations from AI in health care. Science 373, 284–286 (2021).

    Article  CAS  PubMed  ADS  Google Scholar 

  153. Kundu, S. AI in medicine must be explainable. Nat. Med. 27, 1328–1328 (2021).

    Article  CAS  PubMed  Google Scholar 

  154. Wang, F., Kaushal, R. & Khullar, D. Should health care demand interpretable artificial intelligence or accept ‘Black Box’ medicine? Ann. Intern. Med. 172, 59 (2020).

    Article  PubMed  Google Scholar 

  155. Chauhan, C. & Gullapalli, R. R. Ethics of AI in pathology: current paradigms and emerging issues. Am. J. Pathol. 191, 1673–1683 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  156. Schüffler, P. J. et al. Integrated digital pathology at scale: a solution for clinical diagnostics and cancer research at a large academic medical center. J. Am. Med. Inform. Assoc. 28, 1874–1884 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  157. Ghaffari Laleh, N. et al. Adversarial attacks and adversarial robustness in computational pathology. Nat. Commun. 13, 5711 (2022).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  158. Finlayson, S. G. et al. Adversarial attacks on medical machine learning. Science 363, 1287–1289 (2019).

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  159. Korpihalkola, J., Sipola, T. & Kokkonen, T. Color-optimized one-pixel attack against digital pathology images. In 2021 29th Conference of Open Innovations Association (FRUCT), 206–213 (IEEE, 2021).

  160. Hanna, M. G. et al. Validation of a digital pathology system including remote review during the COVID-19 pandemic. Mod. Pathol. 33, 2115–2127 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  161. Gerke, S., Babic, B., Evgeniou, T. & Cohen, I. G. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. npj Digit. Med. 3, 53 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  162. He, J. et al. The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25, 30–36 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  163. Lennerz, J. K., Green, U., Williamson, D. F. K. & Mahmood, F. A unifying force for the realization of medical AI. npj Digital Med. 5, 172 (2022).

    Article  Google Scholar 

  164. Henricks, W. H. et al. Pathology informatics essentials for residents. Acad. Pathol. https://doi.org/10.1177/2374289516659051 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  165. Lujan, G. et al. Dissecting the business case for adoption and implementation of digital pathology: a white paper from the Digital Pathology Association. J. Pathol. Inform. 12, 17 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  166. Andreux, M. et al. (eds.) Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, Vol. 12444, 129–139 (Springer, 2020).

  167. Lu, M. Y. et al. Federated learning for computational pathology on gigapixel whole slide images. Med. Image Anal. 76, 102298 (2022). This paper introduces a federated learning framework for multisite training of digital pathology images, for the purpose of creating robust and generalizable machine learning models.

    Article  PubMed  Google Scholar 

  168. Rieke, N. et al. The future of digital health with federated learning. npj Digital Med. 3, 119 (2020).

    Article  Google Scholar 

  169. Herrmann, M. D. et al. Implementing the DICOM standard for digital pathology. J. Pathol. Inform. 9, 37 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  170. Moore, J. et al. OME-NGFF: a next-generation file format for expanding bioimaging data-access strategies. Nat. Methods 18, 1496–1498 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  171. Karargyris, A. et al. MedPerf: open benchmarking platform for medical artificial intelligence using federated evaluation. https://doi.org/10.48550/arXiv.2110.01406 (2021).

  172. Cardoso, M. J. et al. MONAI: an open-source framework for deep learning in healthcare. https://doi.org/10.48550/arXiv.2211.02701 (2022).

  173. Levine, A. B. et al. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J. Pathol. 252, 178–188 (2020).

    Article  CAS  PubMed  Google Scholar 

  174. Morrison, D., Harris-Birtill, D. & Caie, P. D. Generative deep learning in digital pathology workflows. Am. J. Pathol. 191, 1717–1723 (2021).

    Article  PubMed  Google Scholar 

  175. Hou, L. et al. Robust histopathology image analysis: to label or to synthesize? In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8533–8542 (IEEE, 2019). This paper develops a method for quantifying the sharpness of WSIs for the purpose of quality control, as blurriness is one of the artifacts that can arise in the imaging process and can affect all downstream tasks.

  176. Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical AI. Nat. Med. https://doi.org/10.1038/s41591-022-01981-2 (2022).

  177. Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J. & Shah, S. P. Harnessing multimodal data integration to advance precision oncology. Nat. Rev. Cancer 22, 114–126 (2022).

    Article  CAS  PubMed  Google Scholar 

  178. Cheerla, A. & Gevaert, O. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinfomatics 35, i446–i454 (2019).

    Article  CAS  Google Scholar 

  179. Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878.e6 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  180. Chen, R. J. et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. Preprint at https://doi.org/10.1109/TMI.2020.3021387 (2020).

  181. Khosravi, P. et al. A deep learning approach to diagnostic classification of prostate cancer using pathology–radiology fusion. J. Magn. Reson. Imaging 54, 462–471 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  182. Cui, C. et al. Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data. In Proc. Medical Image Computing and Computer Assisted Intervention — MICCAI 2022: 25th International Conference, Part V, 626–635 (Springer, 2022).

  183. He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. PathVQA: 30000+ questions for medical visual question answering. Preprint at https://doi.org/10.48550/arXiv.2003.10286 (2020).

  184. Sun, Y. et al. PathAsst: redefining pathology through generative foundation AI assistant for pathology. Preprint at https://doi.org/10.48550/arXiv.2305.15072 (2023).

  185. Plana, D. et al. Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw. Open. 5, e2233946 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  186. He, B. et al. Blinded, randomized trial of sonographer versus AI cardiac function assessment. Nature https://doi.org/10.1038/s41586-023-05947-3 (2023).

  187. Attia, Z. I. et al. Prospective evaluation of smartwatch-enabled detection of left ventricular dysfunction. Nat. Med. https://doi.org/10.1038/s41591-022-02053-1 (2022).

  188. Adams, R. et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat. Med. 28, 1455–1460 (2022).

    Article  CAS  PubMed  Google Scholar 

  189. Bannon, D. et al. DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes. Nat. Methods 18, 43–45 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  190. Rao, P., Lopez Barron, D. E., Yarlagadda, D. V. K., Tawfik, O. & Rao, D. Scalable storage of whole slide images and fast retrieval of tiles using Apache Spark. In Medical Imaging 2018: Digital Pathology 38 (eds Gurcan, M. N. & Tomaszewski, J. E.) (SPIE, 2018).

  191. Ghahremani, P., Marino, J., Dodds, R. & Nadeem, S. DeepLIIF: an online platform for quantification of clinical pathology slides. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 21367–21373 (IEEE, 2022).

  192. Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. N. Engl. J. Med. 385, 283–286 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  193. Lee, C. S. & Lee, A. Y. Clinical applications of continual learning machine learning. Lancet Digital Health 2, e279–e281 (2020).

    Article  PubMed  Google Scholar 

  194. Vokinger, K. N., Feuerriegel, S. & Kesselheim, A. S. Continual learning in medical devices: FDA’s action plan and beyond. Lancet Digital Health 3, e337–e338 (2021).

    Article  CAS  PubMed  Google Scholar 

  195. Sculley, D. et al. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, Vol. 28 (eds Cortes, C.et al.) (Curran Assoc., 2015).

  196. Nir, G. et al. Automatic grading of prostate cancer in digitized histopathology images: learning from multiple experts. Med. Image Anal. 50, 167–180 (2018).

    Article  PubMed  Google Scholar 

  197. Weinstein, J. N. et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet. 45, 1113–1120 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  198. Roetzer-Pejrimovsky, T. et al. The Digital Brain Tumour Atlas, an open histopathology resource. Sci. Data 9, 55 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  199. Qaiser, T. et al. HER 2 challenge contest: a detailed assessment of automated HER 2 scoring algorithms in whole slide images of breast cancer tissues. Histopathology 72, 227–238 (2018).

    Article  PubMed  Google Scholar 

  200. Kumar, N. et al. A multi-organ nucleus segmentation challenge. IEEE Trans. Med. Imaging 39, 1380–1391 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  201. Gamper, J., Alemi Koohbanani, N., Benet, K., Khuram, A. & Rajpoot, N. PanNuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification. In Proc. Digital Pathology: 15th European Congress, ECDP 2019, 11–19 (Springer, 2019).

  202. Ryu, J. et al. OCELOT: overlapped cell on tissue dataset for histopathology. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23902–23912 (IEEE, 2023).

  203. Da, Q. et al. DigestPath: a benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Med. Image Anal. 80, 102485 (2022).

    Article  PubMed  Google Scholar 

  204. Hernández-Neuta, I. et al. Smartphone-based clinical diagnostics: towards democratization of evidence-based health care. J. Intern. Med. 285, 19–39 (2019). This paper presents a survey of smartphone-based diagnostics in several clinical areas including digital pathology, highlighting the importance of low-cost image-acquisition techniques in low-resource environments for democratizing healthcare.

    Article  PubMed  Google Scholar 

  205. Roy, S. et al. Smartphone adapters for digital photomicrography. J. Pathol. Inform. 5, 24 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  206. Mudenda, V., Malyangu, E., Sayed, S. & Fleming, K. Addressing the shortage of pathologists in Africa: creation of a MMed programme in pathology in Zambia. Afr. J. Lab. Med. https://doi.org/10.4102/ajlm.v9i1.974 (2020).

Download references

Acknowledgements

M.R.S. acknowledges funding support by the National Institutes of Health (NIH) grant R01AG053949, and the National Science Foundation (NSF) CAREER 1748377 grant. J.R. was supported by a Medical Scientist Training Program grant from the National Institute of General Medical Sciences of the NIH under award no. T32GM007739 to the Weill Cornell/Rockefeller/Sloan Kettering Tri-Institutional MD-PhD Program. L.M. received funding from NSF grant 2124167 and NIH-NCI grant U54CA273956.

Author information

Authors and Affiliations

  1. Biomedical Engineering, Cornell University and Cornell Tech, New York, NY, USA

    Cagla Deniz Bahadir

  2. Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, NY, USA

    Mohamed Omar, Luigi Marchionni, Benjamin Liechty & David J. Pisapia

  3. Weill Cornell/Rockefeller/Sloan Kettering Tri-Institutional MD-PhD Program, New York, NY, USA

    Jacob Rosenthal

  4. Radiology, Weill Cornell Medicine, New York, NY, USA

    Mert R. Sabuncu

  5. Electrical and Computer Engineering, Cornell University and Cornell Tech, New York, NY, USA

    Mert R. Sabuncu

Authors
  1. Cagla Deniz Bahadir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Omar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacob Rosenthal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luigi Marchionni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Benjamin Liechty
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. David J. Pisapia
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mert R. Sabuncu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.D.B., M.O. and J.R. researched data for the article. M.R.S., C.D.B., M.O., L.M., B.L., D.J.P. and J.R. contributed substantially to discussion of the content. All authors wrote the article. All authors reviewed and/or edited the manuscript before submission.

Corresponding author

Correspondence to Mert R. Sabuncu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Electrical Engineering thanks Jakob Nikolas Kather and Jens Kleesiek for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bahadir, C.D., Omar, M., Rosenthal, J. et al. Artificial intelligence applications in histopathology. Nat Rev Electr Eng 1, 93–108 (2024). https://doi.org/10.1038/s44287-023-00012-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s44287-023-00012-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research