Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

OPINION

Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology

Nature Reviews Clinical Oncology volume 16, pages 703–715 (2019)Cite this article

Subjects

Abstract

In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and ‘hand-crafted’ feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Milestones in computational pathology.
Fig. 2: Workflow and general framework for artificial intelligence (AI) approaches in digital pathology.
Fig. 3: Visual representations of hand-crafted features across cancer types.
Fig. 4: Artificial intelligence (AI) and machine learning approaches complement the expertise and support the pathologist and oncologist.

Similar content being viewed by others

References

  1. Prewitt, J. M. S. Intelligent microscopes: recent and near-future advances. Proc. SPIE https://doi.org/10.1117/12.958214 (1979).

  2. Prewitt, J. M. S. Parametric and nonparametric recognition by computer: an application to leukocyte image processing. Adv. Comput. 12, 285–414 (1972).

  3. Prewitt, J. M. S. & Mendelsohn, M. L. The analysis of cell images. Ann. NY Acad. Sci. 128, 1035–1053 (1966).

    CAS  PubMed  Google Scholar 

  4. McCarthy, J., Minsky, M. L., Rochester, N. & Shannon, C. E. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI Mag. 27, 12 (2006).

    Google Scholar 

  5. McCarthy, J. J., Minsky, M. L. & Rochester, N. Artificial intelligence. Research Laboratory of Electronics (RLE) at the Massachusetts Institute of Technology (MIT) https://dspace.mit.edu/handle/1721.1/52263 (1959).

  6. Yao, X. Evolving artificial neural networks. Proc. IEEE 87, 1423–1447 (1999).

    Google Scholar 

  7. Haykin, S. Neural Networks (Prentice Hall, 1994).

    Google Scholar 

  8. Deng, L. Deep learning: methods and applications. Found. Trends® Signal Process. 7, 197–387 (2014).

    Google Scholar 

  9. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    CAS  PubMed  Google Scholar 

  10. Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

    CAS  PubMed  Google Scholar 

  11. Krizhevsky, A., Sutskever I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Nips.cc http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (2012).

  12. LeCun, Y., Huang, F. J. & Bottou, L. in Proc. 2004 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. II–104 (IEEE, 2004).

  13. LeCun, Y. & Bengio, Y. in The handbook of brain theory and neural networks (ed. Arbib, M. A.) 255–258 (MIT Press, 1998).

    Google Scholar 

  14. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Google Scholar 

  15. Deng, J., et al. in 2009 IEEE Conf. Comput. Vis. Pattern Recognit. 248–255 (IEEE, 2009).

  16. Mukhopadhyay, S. et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: a multicenter blinded randomized noninferiority study of 1992 cases (Pivotal Study). Am. J. Surg. Pathol. 42, 39–52 (2018).

    PubMed  Google Scholar 

  17. Kujan, O. et al. Why oral histopathology suffers inter-observer variability on grading oral epithelial dysplasia: an attempt to understand the sources of variation. Oral Oncol. 43, 224–231 (2007).

    PubMed  Google Scholar 

  18. Chi, A. C., Katabi, N., Chen, H.-S. & Cheng, Y.-S. L. Interobserver variation among pathologists in evaluating perineural invasion for oral squamous cell carcinoma. Head Neck Pathol. 10, 451–464 (2016).

    PubMed  PubMed Central  Google Scholar 

  19. Evans, A. J. et al. Interobserver variability between expert urologic pathologists for extraprostatic extension and surgical margin status in radical prostatectomy specimens. Am. J. Surg. Pathol. 32, 1503–1512 (2008).

    PubMed  Google Scholar 

  20. Shanes, J. G. et al. Interobserver variability in the pathologic interpretation of endomyocardial biopsy results. Circulation 75, 401–405 (1987).

    CAS  PubMed  Google Scholar 

  21. Elmore, J. G. et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313, 1122–1132 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  22. Brimo, F., Schultz, L. & Epstein, J. I. The value of mandatory second opinion pathology review of prostate needle biopsy interpretation before radical prostatectomy. J. Urol. 184, 126–130 (2010).

    PubMed  Google Scholar 

  23. Kilgore, M. L. & Goldman, D. P. Drug costs and out-of-pocket spending in cancer clinical trials. Contemp. Clin. Trials 29, 1–8 (2008).

    PubMed  Google Scholar 

  24. Agarwal, A., Ressler, D. & Snyder, G. The current and future state of companion diagnostics. Pharmacogenomics Pers. Med. 8, 99–110 (2015).

    Google Scholar 

  25. Michor, F. & Polyak, K. The origins and implications of intratumor heterogeneity. Cancer Prev. Res. 3, 1361–1364 (2010).

    Google Scholar 

  26. Cyll, K. et al. Tumour heterogeneity poses a significant challenge to cancer biomarker research. Br. J. Cancer 117, 367–375 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  27. Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

    Google Scholar 

  28. Nagpal, K. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. npj Digital Med. 2, 48 (2019).

    Google Scholar 

  29. Bychkov, D. et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci. Rep. 8, 3395 (2018).

    PubMed  PubMed Central  Google Scholar 

  30. Wang, D., Khosla, A., Gargeya, R., Irshad, H. & Beck, A. H. Deep learning for identifying metastatic breast cancer. ArXiv.org https://arxiv.org/abs/1606.05718 (2016).

  31. Steiner, D. et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am. J. Surg. Pathol. 42, 1636–1646 (2018).

    PubMed  PubMed Central  Google Scholar 

  32. Chen, J. & Srinivas, C. Automatic lymphocyte detection in H&E images with deep neural networks. ArXiv.org https://arxiv.org/abs/1612.03217 (2016).

  33. Garcia, E., et al. in 2017 IEEE 30th Int. Symp. Comput.-Based Med. Sys. (CBMS). 200–204 (IEEE, 2017).

  34. Lu, C. et al. Multi-pass adaptive voting for nuclei detection in histopathological images. Sci. Rep. 6, 33985 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  35. Sornapudi, S. et al. Deep learning nuclei detection in digitized histology images by superpixels. J. Pathol. Inform. https://doi.org/10.4103/jpi.jpi_74_17 (2018).

    PubMed  PubMed Central  Google Scholar 

  36. Wang, H. et al. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J. Med. Imaging (Bellingham) 1, 034003 (2014).

    Google Scholar 

  37. Al-Kofahi, Y., Lassoued, W., Lee, W. & Roysam, B. Improved automatic detection and segmentation of cell nuclei in histopathology images. IEEE Trans. Biomed. Eng. 57, 841–852 (2010).

    PubMed  Google Scholar 

  38. Naik, S., et al. in 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 284–287 (IEEE, 2008).

  39. Nguyen, K., Jain, A. K. & Allen, R. L. in 2010 20th Int. Conf. Pattern Recognit. 1497–1500 (IEEE, 2010).

  40. Kothari, S., et al. in 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro 657–660 (2011, IEEE).

  41. Sirinukunwattana, K. et al. Gland segmentation in colon histology images: the glas challenge contest. Med. Image Anal. 35, 489–502 (2017).

    PubMed  Google Scholar 

  42. Lee, G. et al. Co-occurring gland angularity in localized subgraphs: predicting biochemical recurrence in intermediate-risk prostate cancer patients. PLOS ONE 9, e97954 (2014).

    PubMed  PubMed Central  Google Scholar 

  43. Lee, G. et al. Nuclear shape and architecture in benign fields predict biochemical recurrence in prostate cancer patients following radical prostatectomy: preliminary findings. Eur. Urol. Focus 3, 457–466 (2017).

    PubMed  Google Scholar 

  44. Lu, C. et al. Nuclear shape and orientation features from H&E images predict survival in early-stage estrogen receptor-positive breast cancers. Lab. Investig. J. Tech. Methods Pathol. 98, 1438–1448 (2018).

    CAS  Google Scholar 

  45. Corredor, G. et al. Spatial architecture and arrangement of tumor-infiltrating lymphocytes for predicting likelihood of recurrence in early-stage non-small cell lung cancer. Clin. Cancer Res. 25, 1526–1534 (2018).

    PubMed  PubMed Central  Google Scholar 

  46. Mungle, T. et al. MRF‐ANN: a machine learning approach for automated ER scoring of breast cancer immunohistochemical images. J. Microsc. 267, 117–129 (2017).

    CAS  PubMed  Google Scholar 

  47. Wang, X. et al. Prediction of recurrence in early stage non-small cell lung cancer using computer extracted nuclear features from digital H&E images. Sci. Rep. 7, 13543 (2017).

    PubMed  PubMed Central  Google Scholar 

  48. Rosado, B. et al. Accuracy of computer diagnosis of melanoma: a quantitative meta-analysis. Arch. Dermatol. 139, 361–367 (2003).

    PubMed  Google Scholar 

  49. Rosenbaum, B. E. et al. Computer-assisted measurement of primary tumor area is prognostic of recurrence-free survival in stage IB melanoma patients. Mod. Pathol. 30, 1402–1410 (2017).

    PubMed  PubMed Central  Google Scholar 

  50. Teramoto, A., Tsukamoto, T., Kiriyama, Y. & Fujita, H. Automated classification of lung cancer types from cytological images using deep convolutional neural networks. Biomed. Res. Int. https://doi.org/10.1155/2017/4067832 (2017).

    Google Scholar 

  51. Wu, M., Yan, C., Liu, H. & Liu, Q. Automatic classification of ovarian cancer types from cytological images using deep convolutional neural networks. Biosci. Rep. https://doi.org/10.1042/BSR20180289 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  52. Ali, S., Basavanhally, A., Ganesan, S. & Madabhushi, A. Histogram of Hosoya indices for assessing similarity across subgraph populations: breast cancer prognosis prediction from digital pathology [abstract 118]. Lab. Invest. (supplement) 95, 32A (2015).

  53. Yu, K.-H. et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 7, 12474 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  54. Cruz-Roa, A. et al. Accurate and reproducible invasive breast cancer detection in whole-slide images: a deep learning approach for quantifying tumor extent. Sci. Rep. 7, 46450 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  55. Liu, Y. et al. Artificial intelligence-based breast cancer nodal metastasis detection. Arch. Pathol. Lab. Med. 143, 859–868 (2018).

    PubMed  Google Scholar 

  56. Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7, giy065 (2018).

    PubMed Central  Google Scholar 

  57. Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. ArXiv.org https://arxiv.org/abs/1703.02442 (2017).

    Google Scholar 

  58. Madabhushi, A. & Lee, G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 33, 170–175 (2016).

    PubMed  PubMed Central  Google Scholar 

  59. Jansen, I. et al. Histopathology: ditch the slides, because digital and 3D are on show. World J. Urol. 36, 549–555 (2018).

    PubMed  PubMed Central  Google Scholar 

  60. Ali, S., Lewis, J. & Madabhushi, A. Spatially aware cell cluster(spACC1) graphs: predicting outcome in oropharyngeal pl6+ tumors. Med. Image Comput. Comput. Assist. Interv. 16, 412–419 (2013).

    PubMed  Google Scholar 

  61. Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  62. Wang, X. et al. Computer extracted features of cancer nuclei from H&E stained tissues of tumor predicts response to nivolumab in non-small cell lung cancer. J. Clin. Oncol. 36(15_suppl), 12061–12061 (2018).

    Google Scholar 

  63. Barrera, C. et al. Computer-extracted features relating to spatial arrangement of tumor infiltrating lymphocytes to predict response to nivolumab in non-small cell lung cancer (NSCLC). J. Clin. Oncol. 36, 12115–12115 (2018).

    Google Scholar 

  64. Wang, X., et al. Computerized nuclear morphometric features from H&E slide images are prognostic of recurrence and predictive of added benefit of adjuvant chemotherapy in early stage non-small cell lung cancer. Presented at the United States and Canadian Academy of Pathology’s 108th Annual Meeting. (2019).

  65. Gisselsson, D. et al. Abnormal nuclear shape in solid tumors reflects mitotic instability. Am. J. Pathol. 158, 199–206 (2001).

    CAS  PubMed  PubMed Central  Google Scholar 

  66. Malon, C. D. & Cosatto, E. Classification of mitotic figures with convolutional neural networks and seeded blob features. J. Pathol. Inform. 4, 9 (2013).

    Google Scholar 

  67. Whitney, J. et al. Quantitative nuclear histomorphometry predicts oncotype DX risk categories for early stage ER+ breast cancer. BMC Cancer 18, 610 (2018).

    PubMed  PubMed Central  Google Scholar 

  68. Simon, I., Pound, C. R., Partin, A. W., Clemens, J. Q. & Christens‐Barry, W. A. Automated image analysis system for detecting boundaries of live prostate cancer cells. Cytom. J. Int. Soc. Anal. Cytol. 31, 287–294 (1998).

    CAS  Google Scholar 

  69. Basavanhally, A. et al. Multi-field-of-view framework for distinguishing tumor grade in ER+ breast cancer from entire histopathology slides. IEEE Trans. Biomed. Eng. 60, 2089–2099 (2013).

    PubMed  PubMed Central  Google Scholar 

  70. Lewis, J. S., Ali, S., Luo, J., Thorstad, W. L. & Madabhushi, A. A quantitative histomorphometric classifier (QuHbIC) identifies aggressive versus indolent p16-positive oropharyngeal squamous cell carcinoma. Am. J. Surg. Pathol. 38, 128–137 (2014).

    PubMed  PubMed Central  Google Scholar 

  71. Barker, J., Hoogi, A., Depeursinge, A. & Rubin, D. L. Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles. Med. Image Anal. 30, 60–71 (2016).

    PubMed  Google Scholar 

  72. Kong, J. et al. Machine-based morphologic analysis of glioblastoma using whole-slide pathology images uncovers clinically relevant molecular correlates. PLOS ONE 8, e81049 (2013).

    PubMed  PubMed Central  Google Scholar 

  73. Osareh, A. & Shadgar, B. in 2010 5th Int. Symp. Health Informat. Bioinformat. 114–120 (2010, IEEE).

  74. Veta, M. et al. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Mod. Pathol. 25, 1559 (2012).

    PubMed  Google Scholar 

  75. Lu, C. et al. An oral cavity squamous cell carcinoma quantitative histomorphometric-based image classifier of nuclear morphology can risk stratify patients for disease-specific survival. Mod. Pathol. 30, 1655–1665 (2017).

    PubMed  PubMed Central  Google Scholar 

  76. Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23, 181–193.e7 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  77. Yuan, Y. Modelling the spatial heterogeneity and molecular correlates of lymphocytic infiltration in triple-negative breast cancer. J. R. Soc. Interface https://doi.org/10.1098/rsif.2014.1153 (2015).

    PubMed Central  PubMed  Google Scholar 

  78. Heindl, A., et al. Relevance of spatial heterogeneity of immune infiltration for predicting risk of recurrence after endocrine therapy of ER+ breast cancer. J. Natl Cancer. Inst. https://doi.org/10.1093/jnci/djx137 (2018).

    Google Scholar 

  79. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl Med. 3, 108ra113 (2011).

    PubMed  Google Scholar 

  80. Bhargava, H. K. et al. Computer-extracted stromal features of African-Americans versus Caucasians from H&E slides and impact on prognosis of biochemical recurrence. J. Clin. Oncol. 36(15_suppl), 12075–12075 (2018).

    Google Scholar 

  81. Vamathevan, J. et al. Applications of machine learning in drug discovery and development. Nat. Rev. Drug Discov. 18, 463–477 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  82. Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J. Pathol. Inform. 7, 29 (2016).

    PubMed  PubMed Central  Google Scholar 

  83. Araújo, T. et al. Classification of breast cancer histology images using Convolutional Neural Networks. PLOS ONE 12, e0177544 (2017).

    PubMed  PubMed Central  Google Scholar 

  84. Ehteshami Bejnordi, B. et al. Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies. Mod. Pathol. 31, 1502–1512 (2018).

    PubMed  Google Scholar 

  85. Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29, 1836–1842 (2018).

    CAS  PubMed  Google Scholar 

  86. Sheikhzadeh F., Guillaud, M. & Ward, R. K. Automatic labeling of molecular biomarkers of whole slide immunohistochemistry images using fully convolutional networks. PLOS ONE 13, e0190783 (2018).

  87. Basavanhally, A. N. et al. Computerized image-based detection and grading of lymphocytic infiltration in HER2+ breast cancer histopathology. IEEE Trans. Biomed. Eng. 57, 642–653 (2010).

    PubMed  Google Scholar 

  88. Alzubaidi, L. et al. A robust deep learning approach to detect nuclei in histopathological images. Int. J. Innov. Res. Comp. Commun. Eng. 5, 4461–4467 (2007).

    Google Scholar 

  89. Höfener, H. et al. Deep learning nuclei detection: a simple approach can deliver state-of-the-art results. Comput. Med. Imaging Graph. 70, 43–52 (2018).

    PubMed  Google Scholar 

  90. Xu, J., Luo, X., Wang, G., Gilmore, H. & Madabhushi, A. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191, 214–223 (2016).

    PubMed  PubMed Central  Google Scholar 

  91. Senaras, C., Niazi, M. K. K., Lozanski, G. & Gurcan, M. N. DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning. PLOS ONE 13, e0205387 (2018).

    PubMed  PubMed Central  Google Scholar 

  92. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  93. Tschandl, P. et al. Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study. Lancet Oncol. 20, 938–947 (2019).

    PubMed  PubMed Central  Google Scholar 

  94. Couture, H. D. et al. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. npj Breast Cancer 4, 30 (2018).

    PubMed  PubMed Central  Google Scholar 

  95. Romo-Bucheli, D., Janowczyk, A., Gilmore, H., Romero, E. & Madabhushi, A. A deep learning based strategy for identifying and associating mitotic activity with gene expression derived risk categories in estrogen receptor positive breast cancers. Cytomtery A 91, 566–573 (2017).

    CAS  Google Scholar 

  96. Geessink, O. G. F., et al. Computer aided quantification of intratumoral stroma yields an independent prognosticator in rectal cancer. Cell Oncol. 42, 331–341 (2019).

    Google Scholar 

  97. Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLOS Med. 16, e1002730 (2019).

    PubMed  PubMed Central  Google Scholar 

  98. Rodner, E. et al. Fully convolutional networks in multimodal nonlinear microscopy images for automated detection of head and neck carcinoma: pilot study. Head Neck 41, 116–121 (2019).

    PubMed  Google Scholar 

  99. Cruz-Roa, A. et al. High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: application to invasive breast cancer detection. PLOS ONE 13, e0196828 (2018).

    PubMed  PubMed Central  Google Scholar 

  100. Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

    Google Scholar 

  101. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

    CAS  PubMed  Google Scholar 

  102. Li, W., Gauci, M. & Gross, R. in Proc. 15th Ann. Conf. Genet. Evolut. Computat. 223–230. (ACM, 2013).

  103. Hou, L., et al. Unsupervised histopathology image synthesis. ArXiv.org https://arxiv.org/abs/1712.05021 (2017).

  104. Shaban, M. T., Baur, C., Navab, N. & Albarqouni, S. StainGAN: stain style transfer for digital histological images. IEEE https://doi.org/10.1109/ISBI.2019.8759152 (2018).

  105. Xu, Z., Moro, C. F., Bozóky, B. & Zhang, Q. GAN-based virtual re-staining: a promising solution for whole slide image analysis. ArXiv.org https://arxiv.org/abs/1901.04059 (2019).

  106. Gadermayr, M., Gupta, L., Klinkhammer, B. M., Boor, P. & Merhof, D. Unsupervisedly training GANs for segmenting digital pathology with automatically generated annotations. Proc. Machine Learn. 102, 175–184 (2018).

  107. Kapil, A., et al. Deep semi supervised generative learning for automated PD-L1 tumor cell scoring on NSCLC tissue needle biopsies. Sci. Rep. 8,17343 (2018).

  108. Boutros, P. C. The path to routine use of genomic biomarkers in the cancer clinic. Genome Res. 25, 1508–1513 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  109. Chatterjee, S. K. & Zetter, B. R. Cancer biomarkers: knowing the present and predicting the future. Future Oncol. 1, 37–50 (2005).

    CAS  PubMed  Google Scholar 

  110. Grigg, C. & Rizvi, N. A. PD-L1 biomarker testing for non-small cell lung cancer: truth or fiction? J. Immunother. Cancer 4, 48 (2016).

    PubMed  PubMed Central  Google Scholar 

  111. Mroz, E. A. & Rocco, J. W. Intra-tumor heterogeneity in head and neck cancer and its clinical implications. World J. Otorhinolaryngol. Head Neck Surg. 2, 60–67 (2016).

    PubMed  PubMed Central  Google Scholar 

  112. Morris, L. G. T. et al. Pan-cancer analysis of intratumor heterogeneity as a prognostic determinant of survival. Oncotarget 7, 10051–10063 (2016).

    PubMed  PubMed Central  Google Scholar 

  113. Sottoriva, A. et al. Intratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics. Proc. Natl Acad. Sci. USA 110, 4009–4014 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  114. Verma, N. et al. Image-based risk score to predict recurrence of ER+ breast cancer in ECOG-ACRIN Cancer Research Group E2197. J. Clin. Oncol. 36 (15_suppl), 540 (2018).

    Google Scholar 

  115. Ginsburg, S. B., Lee, G., Ali, S. & Madabhushi, A. Feature importance in nonlinear embeddings (FINE): applications in digital pathology. IEEE Trans. Med. Imaging. 35, 76–88 (2016).

    PubMed  Google Scholar 

  116. Cooper, L. A. D. et al. Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images. Lab. Investig. J. Tech. Methods Pathol. 95, 366–376 (2015).

    Google Scholar 

  117. Lloyd, M. C., Johnson, J. O., Kasprzak, A. & Bui, M. M. Image analysis of the tumor microenvironment. Adv. Exp. Med. Biol. 936, 1–10 (2016).

    CAS  PubMed  Google Scholar 

  118. Hamilton, P. W. et al. Digital pathology and image analysis in tissue biomarker research. Methods 70, 59–73 (2014).

    CAS  PubMed  Google Scholar 

  119. Barsoum, I., Tawedrous, E., Faragalla, H. & Yousef, G. M. Histo-genomics: digital pathology at the forefront of precision medicine. Diagnosis https://doi.org/10.1515/dx-2018-0064 (2018).

    Article  Google Scholar 

  120. FDA. Developing a software precertification program: a working model. Fda.gov https://www.fda.gov/media/119722/download (2019).

  121. Daniel, G., Silcox, C., Sharma, I. & Wright, M. Current state and near-term priorities for AI-enabled diagnostic support software in health care. Duke.edu https://healthpolicy.duke.edu/sites/default/files/atoms/files/dukemargolisaienableddxss.pdf (2019).

  122. Pesapane, F., Volonté, C., Codari, M. & Sardanelli, F. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging 9, 745–753 (2018).

    PubMed  PubMed Central  Google Scholar 

  123. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019).

    Google Scholar 

  124. Food and Drug Administration. FDA in brief: FDA proposes improvements to the de novo pathway for novel medical devices to advance safe, effective, and innovative treatments for patients. Fda.gov https://www.fda.gov/NewsEvents/Newsroom/FDAInBrief/ucm627522.htm (2018).

  125. European Commission. MDCG 2018-2 Future EU medical device nomenclature – description of requirements. Europa.eu https://ec.europa.eu/docsroom/documents/28668 (2018).

  126. European Economic Community. Council Directive 93/42/EEC of 14 June 1993 concerning medical devices. Europa.eu https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1993L0042:20071011:EN:PDF (1993).

  127. European Parliament. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending directive 2001/83/EC, regulation (EC) no 178/2002 and regulation (EC) no 1223/2009 and repealing council directives 90/385/EEC and 93/42/EEC (text with EEA relevance). Europa.eu http://data.europa.eu/eli/reg/2017/745/oj/eng (2017).

  128. European Parliament. Regulation (EU) 2017/746 of the European Parliament and of the council of 5 April 2017 on in vitro diagnostic medical devices and repealing directive 98/79/EC and commission decision 2010/227/EU (Text with EEA relevance) Europa.eu http://data.europa.eu/eli/reg/2017/746/oj/eng (2017).

  129. García-Rojo, M., et al. New European Union regulations related to whole slide image scanners and image analysis software. J. Pathol. Inform. 10, https://doi.org/10.4103/jpi.jpi_33_18 (2019).

    PubMed  PubMed Central  Google Scholar 

  130. Arterys. Medical imaging cloud AI. Arterys.com https://www.arterys.com/ (2019).

  131. Arterys Inc. Arterys receives first FDA clearance for broad oncology imaging suite with deep learning. Prnewswire.com https://www.prnewswire.com/news-releases/arterys-receives-first-fda-clearance-for-broad-oncology-imaging-suite-with-deep-learning-300599275.html (2018).

  132. Food and Drug Administration. IntelliSite Pathology Solution (PIPS, Philips Medical Systems). Fda.gov https://www.fda.gov/drugs/resources-information-approved-drugs/intellisite-pathology-solution-pips-philips-medical-systems (2017).

  133. PAIGE. PAIGE. Paige.ai https://paige.ai/ (2019).

  134. Business Wire. FDA grants breakthrough designation to Paige.AI. Businesswire.com https://www.businesswire.com/news/home/20190307005205/en/FDA-Grants-Breakthrough-Designation-Paige.AI (2019).

  135. Bueno, G., Fernández-Carrobles, M. M., Deniz, O. & García-Rojo, M. New trends of emerging technologies in digital pathology. Pathobiology 83, 61–69 (2016).

    PubMed  Google Scholar 

  136. Food and Drug Administration. Radiological computer-assisted diagnostic (CADx) software for lesions suspicious for cancer. Fda.gov. https://www.accessdata.fda.gov/cdrh_docs/pdf17/DEN170022.pdf (2017).

  137. Agendia. MammaPrint Test. Agendia.com https://www.agendia.com/our-tests/mammaprint/ (2019).

  138. Food and Drug Administration. MammaPrint 510(k) premarket notification. Fda.gov https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=k070675 (2019).

  139. OncotypeIQ. Smarter Cancer Care - Genomic Testing. Oncotypeiq.com https://www.oncotypeiq.com/en-US/ (2019).

  140. Food and Drug Administration. FDA issues warning letter to genomics lab for illegally marketing genetic test that claims to predict patients’ responses to specific medications. Fda.gov https://www.fda.gov/news-events/press-announcements/fda-issues-warning-letter-genomics-lab-illegally-marketing-genetic-test-claims-predict-patients (2019).

  141. Food and Drug Administration. Discussion paper on laboratory developed tests (LDTs) January 13, 2017. Fda.gov https://www.fda.gov/media/102367/download (2019).

  142. Paxton, A. Laboratory-developed tests CAP suggests added oversight of homebrews. Captodayonline.com http://www.captodayonline.com/Archives/1009/1009d_laboratory_developed_tests.html (2009).

  143. Higgins, C. Applications and challenges of digital pathology and whole slide imaging. Biotech. Histochem. 90, 341–347 (2015).

    CAS  PubMed  Google Scholar 

  144. Tizhoosh, H. R. & Pantanowitz, L. Artificial intelligence and digital pathology: challenges and opportunities. J. Pathol. Inform. 9, 38 (2018).

    PubMed  PubMed Central  Google Scholar 

  145. Doyle, S., Feldman, M., Tomaszewski, J. & Madabhushi, A. A boosted Bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies. IEEE Trans. Biomed. Eng. 59, 1205–1218 (2012).

    PubMed  Google Scholar 

  146. Huang, B., Bates, M. & Zhuang, X. Super resolution fluorescence microscopy. Annu. Rev. Biochem. 78, 993–1016 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  147. Kleppe, A. et al. Chromatin organisation and cancer prognosis: a pan-cancer study. Lancet Oncol. 19, 356–369 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  148. Gallas, B. D. NCIP Hub - Group: eeDAP studies ~ Wiki: High-throughput Truthing - Year 2. Ncihub.org https://nciphub.org/groups/eedapstudies/wiki/HighThroughputTruthing/HighThroughputTruthingYear2 (2018).

  149. Grand Challenge. All Challenges. Grand-challenge.org https://grand-challenge.org/challenges/ (2019).

  150. Ching, P. et al. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15, 20170387 (2018).

    Google Scholar 

  151. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

    CAS  PubMed  Google Scholar 

  152. Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Med. 15, e1002683 (2018).

    PubMed  PubMed Central  Google Scholar 

  153. Yagi, Y. & Gilbertson, J. R. Digital Imaging in Pathology: The Case for Standardization. (SAGE Publications, 2005).

    Google Scholar 

  154. Janowczyk, A., Basavanhally, A. & Madabhushi, A. Stain Normalization using Sparse AutoEncoders (StaNoSA): application to digital pathology. Comput. Med. Imaging Graph. 57, 50–61 (2017).

    PubMed  Google Scholar 

  155. Khan, A. M., Rajpoot, N., Treanor, D. & Magee, D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans. Biomed. Eng. 61, 1729–1738 (2014).

    PubMed  Google Scholar 

  156. Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging. 35, 1962–1971 (2016).

    PubMed  Google Scholar 

  157. Janowczyk, A., Zuo, R., Gilmore, H., Feldman, M. & Madabhushi, A. HistoQC: an open-source quality control tool for digital pathology slides. JCO Clin. Cancer Inform. 3, 1–7 (2019).

    PubMed  Google Scholar 

  158. choosehappy. HistoQC is an Open-source Quality Control Tool for Digital Pathology Slides: choosehappy/HistoQC. Github.com https://github.com/choosehappy/HistoQC (2019).

  159. Leo, P. et al. Stable and discriminating features are predictive of cancer presence and Gleason grade in radical prostatectomy specimens: a multi-site study. Sci. Rep. 8, 14918 (2018).

    PubMed  PubMed Central  Google Scholar 

  160. Leo, P. et al. Evaluating stability of histomorphometric features across scanner and staining variations: prostate cancer diagnosis from whole slide images. J. Med. Imaging 3, 047502 (2016).

    Google Scholar 

  161. Lowe, A. et al. Validation of digital pathology in a healthcare environment. Digitalpathologyassociation.org https://digitalpathologyassociation.org/_data/files/DPA-Healthcare-White-Paper--FINAL_v1.0.pdf (2011).

  162. Montalto, M. C. An industry perspective: an update on the adoption of whole slide imaging. J. Pathol. Inform. 7, 18 (2016).

    PubMed  PubMed Central  Google Scholar 

  163. Digital Pathology Association. Healthcare FAQs. Digitalpathologyassociation.org https://digitalpathologyassociation.org/healthcare-faqs (2019).

  164. Simon, R. M., Paik, S. & Hayes, D. F. Use of archived specimens in evaluation of prognostic and predictive biomarkers. J. Natl Cancer Inst. 101, 1446–1452 (2009).

    PubMed  PubMed Central  Google Scholar 

  165. Hayes, D. F. et al. Tumor marker utility grading system: a framework to evaluate clinical utility of tumor markers. J. Natl Cancer Inst. 88, 1456–1466 (1996).

    CAS  PubMed  Google Scholar 

  166. Goetz, M. P. et al. NCCN Guidelines insights: breast cancer, version 3.2018. J. Natl Compr. Canc. Netw. 17, 118–126 (2019).

    PubMed  Google Scholar 

  167. Sparano, J. A. et al. Adjuvant chemotherapy guided by a 21-gene expression assay in breast cancer. N. Engl. J. Med. 379, 111–121 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  168. Cardoso, F. et al. 70-gene signature as an aid to treatment decisions in early-stage breast cancer. N. Engl. J. Med. 375, 717–729 (2016).

    CAS  PubMed  Google Scholar 

  169. Paik, S. et al. A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast cancer. N. Engl. J. Med. 351, 2817–2826 (2004).

    CAS  PubMed  Google Scholar 

  170. Paik, S. et al. Gene expression and benefit of chemotherapy in women with node-negative, estrogen receptor-positive breast cancer. J. Clin. Oncol. 24, 3726–3734 (2006).

    CAS  PubMed  Google Scholar 

  171. DecipherBio. GenomeDx Biosciences: Where to get decipher prostate cancer classifier. decipherbio.com https://genomedx.com/decipher-test/get-decipher/ (2019).

  172. Erho, N. et al. Discovery and validation of a prostate cancer genomic classifier that predicts early metastasis following radical prostatectomy. PLOS ONE. 8, e66855 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  173. Carroll, P. H. & Mohler, J. L. NCCN guidelines updates: prostate cancer and prostate cancer early detection. J. Natl Compr. Canc. Netw. 16(5S), 620–623 (2018).

    PubMed  Google Scholar 

  174. Bhargava, R. & Madabhushi, A. Emerging themes in image informatics and molecular analysis for digital pathology. Annu. Rev. Biomed. Eng. 18, 387–412 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  175. Madabhushi, A. Digital pathology image analysis: opportunities and challenges. Imaging Med. 1, 7–10 (2009).

    PubMed  PubMed Central  Google Scholar 

  176. Madabhushi, A., Agner, S., Basavanhally, A., Doyle, S. & Lee, G. Computer-aided prognosis: predicting patient and disease outcome via quantitative fusion of multi-scale, multi-modal data. Comput. Med. Imaging Graph. 35, 506–514 (2011).

    PubMed  Google Scholar 

  177. Pantanowitz, L. et al. Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch. Pathol. Lab. Med. 137, 1710–1722 (2013).

    PubMed  PubMed Central  Google Scholar 

  178. Fuchs, T. J., Wild, P. J., Moch, H. & Buhmann, J. M. Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. Med. Image Comput. Comput. Assist. Interv. 11, 1–8 (2008).

    PubMed  Google Scholar 

  179. Haybäck, J. Mechanisms of Molecular Carcinogenesis – Volume 2. (Springer, 2017).

  180. Proscia. Proscia digital pathology. Proscia.com https://proscia.com (2019).

  181. Deep Lens. Digital pathology cloud platform. Deeplens.ai https://www.deeplens.ai (2019).

  182. PathAI. PathAI. Pathai.com https://www.pathai.com/ (2019).

  183. Aifora. WebMicroscope. Big pictures. Deep Diagnosis. Aiforia.com https://www.aiforia.com/ (2019).

  184. Glaser, A. K., et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nat. Biomed. Eng. 1, 0084 (2017).

  185. Fereidouni, F. et al. Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nat. Biomed. Eng. 1, 957 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  186. Fraggetta, F., Garozzo, S., Zannoni, G. F., Pantanowitz, L. & Rossi, E. D. Routine digital pathology workflow: the Catania experience. J. Pathol. Inform. 8, 51 (2017).

    PubMed  PubMed Central  Google Scholar 

  187. Cheng, C. L. et al. Enabling digital pathology in the diagnostic setting: navigating through the implementation journey in an academic medical centre. J. Clin. Pathol. 69, 784–792 (2016).

    PubMed  Google Scholar 

  188. Tomczak, K., Czerwińska, P. & Wiznerowicz, M. The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. Contemp. Oncol. 19, A68–77 (2015).

    Google Scholar 

  189. Gutman, D. A. et al. Cancer Digital Slide Archive: an informatics resource to support integrated in silico analysis of TCGA pathology data. J. Am. Med. Inform. Assoc. 20, 1091–1098 (2013).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Research discussed in this publication was supported by the Department of Defence and Department of Veterans, the National Cancer Institute of the National Institutes of Health, the National Centre for Research Resources, the Ohio Third Frontier Technology Validation Fund, and the Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering and the Clinical and Translational Science Award Program (CTSA) at Case Western Reserve University. The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the institutions named.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA

    Kaustav Bera & Anant Madabhushi

  2. Department of Pathology, Yale University School of Medicine, New Haven, CT, USA

    Kurt A. Schalper & David L. Rimm

  3. Thoracic Medical Oncology, Perlmutter Cancer Center, New York University, New York, NY, USA

    Vamsidhar Velcheti

  4. Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA

    Anant Madabhushi

Authors
  1. Kaustav Bera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kurt A. Schalper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David L. Rimm
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vamsidhar Velcheti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anant Madabhushi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.B. and A.M. researched data for the article and discussed the article contents. All authors wrote, reviewed and edited the manuscript before submission.

Corresponding author

Correspondence to Anant Madabhushi.

Ethics declarations

Competing interests

K.A.S. has received fees as a speaker for Merck and Takeda, is a consultant for Celgene, Moderna and Shattuck Labs, and receives research funding from Genoptix (Novartis), Onkaido, Pierre Fabre, Surface Oncology, Takeda, Tesaro and Tioma. D.L.R. is a consultant and research adviser to Agendia, Agilent, AstraZeneca, Biocept, BMS, Cell Signalling Technology, Cepheid, Merck, PAIGE, Perkin Elmer and Ultivue, owns equity in AstraZeneca, Cepheid, Lilly, Navigate–Novartis, NextCure, PixelGear and Ultivue, and receives research funding from Perkin Elmer. A.M. is a consultant and scientific adviser to Elucid Bioimaging and Inspirata, owns stock options in Elucid Bioimaging and Inspirata, and receives research funding from Philips. K.B. and V.V. declare no competing interests.

Additional information

Peer review information

Nature Reviews Clinical Oncology thanks J. Liu, A. Tsirigos and the other, anonymous, reviewer(s), for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bera, K., Schalper, K.A., Rimm, D.L. et al. Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 16, 703–715 (2019). https://doi.org/10.1038/s41571-019-0252-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41571-019-0252-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing