OPINION

Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology

Abstract

In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and ‘hand-crafted’ feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Milestones in computational pathology.
Fig. 2: Workflow and general framework for artificial intelligence (AI) approaches in digital pathology.
Fig. 3: Visual representations of hand-crafted features across cancer types.
Fig. 4: Artificial intelligence (AI) and machine learning approaches complement the expertise and support the pathologist and oncologist.

References

  1. 1.

    Prewitt, J. M. S. Intelligent microscopes: recent and near-future advances. Proc. SPIE https://doi.org/10.1117/12.958214 (1979).

  2. 2.

    Prewitt, J. M. S. Parametric and nonparametric recognition by computer: an application to leukocyte image processing. Adv. Comput. 12, 285–414 (1972).

  3. 3.

    Prewitt, J. M. S. & Mendelsohn, M. L. The analysis of cell images. Ann. NY Acad. Sci. 128, 1035–1053 (1966).

  4. 4.

    McCarthy, J., Minsky, M. L., Rochester, N. & Shannon, C. E. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI Mag. 27, 12 (2006).

  5. 5.

    McCarthy, J. J., Minsky, M. L. & Rochester, N. Artificial intelligence. Research Laboratory of Electronics (RLE) at the Massachusetts Institute of Technology (MIT) https://dspace.mit.edu/handle/1721.1/52263 (1959).

  6. 6.

    Yao, X. Evolving artificial neural networks. Proc. IEEE 87, 1423–1447 (1999).

  7. 7.

    Haykin, S. Neural Networks (Prentice Hall, 1994).

  8. 8.

    Deng, L. Deep learning: methods and applications. Found. Trends® Signal Process. 7, 197–387 (2014).

  9. 9.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  10. 10.

    Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

  11. 11.

    Krizhevsky, A., Sutskever I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Nips.cc http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (2012).

  12. 12.

    LeCun, Y., Huang, F. J. & Bottou, L. in Proc. 2004 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. II–104 (IEEE, 2004).

  13. 13.

    LeCun, Y. & Bengio, Y. in The handbook of brain theory and neural networks (ed. Arbib, M. A.) 255–258 (MIT Press, 1998).

  14. 14.

    LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

  15. 15.

    Deng, J., et al. in 2009 IEEE Conf. Comput. Vis. Pattern Recognit. 248–255 (IEEE, 2009).

  16. 16.

    Mukhopadhyay, S. et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: a multicenter blinded randomized noninferiority study of 1992 cases (Pivotal Study). Am. J. Surg. Pathol. 42, 39–52 (2018).

  17. 17.

    Kujan, O. et al. Why oral histopathology suffers inter-observer variability on grading oral epithelial dysplasia: an attempt to understand the sources of variation. Oral Oncol. 43, 224–231 (2007).

  18. 18.

    Chi, A. C., Katabi, N., Chen, H.-S. & Cheng, Y.-S. L. Interobserver variation among pathologists in evaluating perineural invasion for oral squamous cell carcinoma. Head Neck Pathol. 10, 451–464 (2016).

  19. 19.

    Evans, A. J. et al. Interobserver variability between expert urologic pathologists for extraprostatic extension and surgical margin status in radical prostatectomy specimens. Am. J. Surg. Pathol. 32, 1503–1512 (2008).

  20. 20.

    Shanes, J. G. et al. Interobserver variability in the pathologic interpretation of endomyocardial biopsy results. Circulation 75, 401–405 (1987).

  21. 21.

    Elmore, J. G. et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313, 1122–1132 (2015).

  22. 22.

    Brimo, F., Schultz, L. & Epstein, J. I. The value of mandatory second opinion pathology review of prostate needle biopsy interpretation before radical prostatectomy. J. Urol. 184, 126–130 (2010).

  23. 23.

    Kilgore, M. L. & Goldman, D. P. Drug costs and out-of-pocket spending in cancer clinical trials. Contemp. Clin. Trials 29, 1–8 (2008).

  24. 24.

    Agarwal, A., Ressler, D. & Snyder, G. The current and future state of companion diagnostics. Pharmacogenomics Pers. Med. 8, 99–110 (2015).

  25. 25.

    Michor, F. & Polyak, K. The origins and implications of intratumor heterogeneity. Cancer Prev. Res. 3, 1361–1364 (2010).

  26. 26.

    Cyll, K. et al. Tumour heterogeneity poses a significant challenge to cancer biomarker research. Br. J. Cancer 117, 367–375 (2017).

  27. 27.

    Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

  28. 28.

    Nagpal, K. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. npj Digital Med. 2, 48 (2019).

  29. 29.

    Bychkov, D. et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci. Rep. 8, 3395 (2018).

  30. 30.

    Wang, D., Khosla, A., Gargeya, R., Irshad, H. & Beck, A. H. Deep learning for identifying metastatic breast cancer. ArXiv.org https://arxiv.org/abs/1606.05718 (2016).

  31. 31.

    Steiner, D. et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am. J. Surg. Pathol. 42, 1636–1646 (2018).

  32. 32.

    Chen, J. & Srinivas, C. Automatic lymphocyte detection in H&E images with deep neural networks. ArXiv.org https://arxiv.org/abs/1612.03217 (2016).

  33. 33.

    Garcia, E., et al. in 2017 IEEE 30th Int. Symp. Comput.-Based Med. Sys. (CBMS). 200–204 (IEEE, 2017).

  34. 34.

    Lu, C. et al. Multi-pass adaptive voting for nuclei detection in histopathological images. Sci. Rep. 6, 33985 (2016).

  35. 35.

    Sornapudi, S. et al. Deep learning nuclei detection in digitized histology images by superpixels. J. Pathol. Inform. https://doi.org/10.4103/jpi.jpi_74_17 (2018).

  36. 36.

    Wang, H. et al. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J. Med. Imaging (Bellingham) 1, 034003 (2014).

  37. 37.

    Al-Kofahi, Y., Lassoued, W., Lee, W. & Roysam, B. Improved automatic detection and segmentation of cell nuclei in histopathology images. IEEE Trans. Biomed. Eng. 57, 841–852 (2010).

  38. 38.

    Naik, S., et al. in 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 284–287 (IEEE, 2008).

  39. 39.

    Nguyen, K., Jain, A. K. & Allen, R. L. in 2010 20th Int. Conf. Pattern Recognit. 1497–1500 (IEEE, 2010).

  40. 40.

    Kothari, S., et al. in 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro 657–660 (2011, IEEE).

  41. 41.

    Sirinukunwattana, K. et al. Gland segmentation in colon histology images: the glas challenge contest. Med. Image Anal. 35, 489–502 (2017).

  42. 42.

    Lee, G. et al. Co-occurring gland angularity in localized subgraphs: predicting biochemical recurrence in intermediate-risk prostate cancer patients. PLOS ONE 9, e97954 (2014).

  43. 43.

    Lee, G. et al. Nuclear shape and architecture in benign fields predict biochemical recurrence in prostate cancer patients following radical prostatectomy: preliminary findings. Eur. Urol. Focus 3, 457–466 (2017).

  44. 44.

    Lu, C. et al. Nuclear shape and orientation features from H&E images predict survival in early-stage estrogen receptor-positive breast cancers. Lab. Investig. J. Tech. Methods Pathol. 98, 1438–1448 (2018).

  45. 45.

    Corredor, G. et al. Spatial architecture and arrangement of tumor-infiltrating lymphocytes for predicting likelihood of recurrence in early-stage non-small cell lung cancer. Clin. Cancer Res. 25, 1526–1534 (2018).

  46. 46.

    Mungle, T. et al. MRF‐ANN: a machine learning approach for automated ER scoring of breast cancer immunohistochemical images. J. Microsc. 267, 117–129 (2017).

  47. 47.

    Wang, X. et al. Prediction of recurrence in early stage non-small cell lung cancer using computer extracted nuclear features from digital H&E images. Sci. Rep. 7, 13543 (2017).

  48. 48.

    Rosado, B. et al. Accuracy of computer diagnosis of melanoma: a quantitative meta-analysis. Arch. Dermatol. 139, 361–367 (2003).

  49. 49.

    Rosenbaum, B. E. et al. Computer-assisted measurement of primary tumor area is prognostic of recurrence-free survival in stage IB melanoma patients. Mod. Pathol. 30, 1402–1410 (2017).

  50. 50.

    Teramoto, A., Tsukamoto, T., Kiriyama, Y. & Fujita, H. Automated classification of lung cancer types from cytological images using deep convolutional neural networks. Biomed. Res. Int. https://doi.org/10.1155/2017/4067832 (2017).

  51. 51.

    Wu, M., Yan, C., Liu, H. & Liu, Q. Automatic classification of ovarian cancer types from cytological images using deep convolutional neural networks. Biosci. Rep. https://doi.org/10.1042/BSR20180289 (2018).

  52. 52.

    Ali, S., Basavanhally, A., Ganesan, S. & Madabhushi, A. Histogram of Hosoya indices for assessing similarity across subgraph populations: breast cancer prognosis prediction from digital pathology [abstract 118]. Lab. Invest. (supplement) 95, 32A (2015).

  53. 53.

    Yu, K.-H. et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 7, 12474 (2016).

  54. 54.

    Cruz-Roa, A. et al. Accurate and reproducible invasive breast cancer detection in whole-slide images: a deep learning approach for quantifying tumor extent. Sci. Rep. 7, 46450 (2017).

  55. 55.

    Liu, Y. et al. Artificial intelligence-based breast cancer nodal metastasis detection. Arch. Pathol. Lab. Med. 143, 859–868 (2018).

  56. 56.

    Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7, giy065 (2018).

  57. 57.

    Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. ArXiv.org https://arxiv.org/abs/1703.02442 (2017).

  58. 58.

    Madabhushi, A. & Lee, G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 33, 170–175 (2016).

  59. 59.

    Jansen, I. et al. Histopathology: ditch the slides, because digital and 3D are on show. World J. Urol. 36, 549–555 (2018).

  60. 60.

    Ali, S., Lewis, J. & Madabhushi, A. Spatially aware cell cluster(spACC1) graphs: predicting outcome in oropharyngeal pl6+ tumors. Med. Image Comput. Comput. Assist. Interv. 16, 412–419 (2013).

  61. 61.

    Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).

  62. 62.

    Wang, X. et al. Computer extracted features of cancer nuclei from H&E stained tissues of tumor predicts response to nivolumab in non-small cell lung cancer. J. Clin. Oncol. 36(15_suppl), 12061–12061 (2018).

  63. 63.

    Barrera, C. et al. Computer-extracted features relating to spatial arrangement of tumor infiltrating lymphocytes to predict response to nivolumab in non-small cell lung cancer (NSCLC). J. Clin. Oncol. 36, 12115–12115 (2018).

  64. 64.

    Wang, X., et al. Computerized nuclear morphometric features from H&E slide images are prognostic of recurrence and predictive of added benefit of adjuvant chemotherapy in early stage non-small cell lung cancer. Presented at the United States and Canadian Academy of Pathology’s 108th Annual Meeting. (2019).

  65. 65.

    Gisselsson, D. et al. Abnormal nuclear shape in solid tumors reflects mitotic instability. Am. J. Pathol. 158, 199–206 (2001).

  66. 66.

    Malon, C. D. & Cosatto, E. Classification of mitotic figures with convolutional neural networks and seeded blob features. J. Pathol. Inform. 4, 9 (2013).

  67. 67.

    Whitney, J. et al. Quantitative nuclear histomorphometry predicts oncotype DX risk categories for early stage ER+ breast cancer. BMC Cancer 18, 610 (2018).

  68. 68.

    Simon, I., Pound, C. R., Partin, A. W., Clemens, J. Q. & Christens‐Barry, W. A. Automated image analysis system for detecting boundaries of live prostate cancer cells. Cytom. J. Int. Soc. Anal. Cytol. 31, 287–294 (1998).

  69. 69.

    Basavanhally, A. et al. Multi-field-of-view framework for distinguishing tumor grade in ER+ breast cancer from entire histopathology slides. IEEE Trans. Biomed. Eng. 60, 2089–2099 (2013).

  70. 70.

    Lewis, J. S., Ali, S., Luo, J., Thorstad, W. L. & Madabhushi, A. A quantitative histomorphometric classifier (QuHbIC) identifies aggressive versus indolent p16-positive oropharyngeal squamous cell carcinoma. Am. J. Surg. Pathol. 38, 128–137 (2014).

  71. 71.

    Barker, J., Hoogi, A., Depeursinge, A. & Rubin, D. L. Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles. Med. Image Anal. 30, 60–71 (2016).

  72. 72.

    Kong, J. et al. Machine-based morphologic analysis of glioblastoma using whole-slide pathology images uncovers clinically relevant molecular correlates. PLOS ONE 8, e81049 (2013).

  73. 73.

    Osareh, A. & Shadgar, B. in 2010 5th Int. Symp. Health Informat. Bioinformat. 114–120 (2010, IEEE).

  74. 74.

    Veta, M. et al. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Mod. Pathol. 25, 1559 (2012).

  75. 75.

    Lu, C. et al. An oral cavity squamous cell carcinoma quantitative histomorphometric-based image classifier of nuclear morphology can risk stratify patients for disease-specific survival. Mod. Pathol. 30, 1655–1665 (2017).

  76. 76.

    Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23, 181–193.e7 (2018).

  77. 77.

    Yuan, Y. Modelling the spatial heterogeneity and molecular correlates of lymphocytic infiltration in triple-negative breast cancer. J. R. Soc. Interface https://doi.org/10.1098/rsif.2014.1153 (2015).

  78. 78.

    Heindl, A., et al. Relevance of spatial heterogeneity of immune infiltration for predicting risk of recurrence after endocrine therapy of ER+ breast cancer. J. Natl Cancer. Inst. https://doi.org/10.1093/jnci/djx137 (2018).

  79. 79.

    Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl Med. 3, 108ra113 (2011).

  80. 80.

    Bhargava, H. K. et al. Computer-extracted stromal features of African-Americans versus Caucasians from H&E slides and impact on prognosis of biochemical recurrence. J. Clin. Oncol. 36(15_suppl), 12075–12075 (2018).

  81. 81.

    Vamathevan, J. et al. Applications of machine learning in drug discovery and development. Nat. Rev. Drug Discov. 18, 463–477 (2019).

  82. 82.

    Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J. Pathol. Inform. 7, 29 (2016).

  83. 83.

    Araújo, T. et al. Classification of breast cancer histology images using Convolutional Neural Networks. PLOS ONE 12, e0177544 (2017).

  84. 84.

    Ehteshami Bejnordi, B. et al. Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies. Mod. Pathol. 31, 1502–1512 (2018).

  85. 85.

    Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29, 1836–1842 (2018).

  86. 86.

    Sheikhzadeh F., Guillaud, M. & Ward, R. K. Automatic labeling of molecular biomarkers of whole slide immunohistochemistry images using fully convolutional networks. PLOS ONE 13, e0190783 (2018).

  87. 87.

    Basavanhally, A. N. et al. Computerized image-based detection and grading of lymphocytic infiltration in HER2+ breast cancer histopathology. IEEE Trans. Biomed. Eng. 57, 642–653 (2010).

  88. 88.

    Alzubaidi, L. et al. A robust deep learning approach to detect nuclei in histopathological images. Int. J. Innov. Res. Comp. Commun. Eng. 5, 4461–4467 (2007).

  89. 89.

    Höfener, H. et al. Deep learning nuclei detection: a simple approach can deliver state-of-the-art results. Comput. Med. Imaging Graph. 70, 43–52 (2018).

  90. 90.

    Xu, J., Luo, X., Wang, G., Gilmore, H. & Madabhushi, A. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191, 214–223 (2016).

  91. 91.

    Senaras, C., Niazi, M. K. K., Lozanski, G. & Gurcan, M. N. DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning. PLOS ONE 13, e0205387 (2018).

  92. 92.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  93. 93.

    Tschandl, P. et al. Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study. Lancet Oncol. 20, 938–947 (2019).

  94. 94.

    Couture, H. D. et al. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. npj Breast Cancer 4, 30 (2018).

  95. 95.

    Romo-Bucheli, D., Janowczyk, A., Gilmore, H., Romero, E. & Madabhushi, A. A deep learning based strategy for identifying and associating mitotic activity with gene expression derived risk categories in estrogen receptor positive breast cancers. Cytomtery A 91, 566–573 (2017).

  96. 96.

    Geessink, O. G. F., et al. Computer aided quantification of intratumoral stroma yields an independent prognosticator in rectal cancer. Cell Oncol. 42, 331–341 (2019).

  97. 97.

    Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLOS Med. 16, e1002730 (2019).

  98. 98.

    Rodner, E. et al. Fully convolutional networks in multimodal nonlinear microscopy images for automated detection of head and neck carcinoma: pilot study. Head Neck 41, 116–121 (2019).

  99. 99.

    Cruz-Roa, A. et al. High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: application to invasive breast cancer detection. PLOS ONE 13, e0196828 (2018).

  100. 100.

    Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

  101. 101.

    Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

  102. 102.

    Li, W., Gauci, M. & Gross, R. in Proc. 15th Ann. Conf. Genet. Evolut. Computat. 223–230. (ACM, 2013).

  103. 103.

    Hou, L., et al. Unsupervised histopathology image synthesis. ArXiv.org https://arxiv.org/abs/1712.05021 (2017).

  104. 104.

    Shaban, M. T., Baur, C., Navab, N. & Albarqouni, S. StainGAN: stain style transfer for digital histological images. IEEE https://doi.org/10.1109/ISBI.2019.8759152 (2018).

  105. 105.

    Xu, Z., Moro, C. F., Bozóky, B. & Zhang, Q. GAN-based virtual re-staining: a promising solution for whole slide image analysis. ArXiv.org https://arxiv.org/abs/1901.04059 (2019).

  106. 106.

    Gadermayr, M., Gupta, L., Klinkhammer, B. M., Boor, P. & Merhof, D. Unsupervisedly training GANs for segmenting digital pathology with automatically generated annotations. Proc. Machine Learn. 102, 175–184 (2018).

  107. 107.

    Kapil, A., et al. Deep semi supervised generative learning for automated PD-L1 tumor cell scoring on NSCLC tissue needle biopsies. Sci. Rep. 8,17343 (2018).

  108. 108.

    Boutros, P. C. The path to routine use of genomic biomarkers in the cancer clinic. Genome Res. 25, 1508–1513 (2015).

  109. 109.

    Chatterjee, S. K. & Zetter, B. R. Cancer biomarkers: knowing the present and predicting the future. Future Oncol. 1, 37–50 (2005).

  110. 110.

    Grigg, C. & Rizvi, N. A. PD-L1 biomarker testing for non-small cell lung cancer: truth or fiction? J. Immunother. Cancer 4, 48 (2016).

  111. 111.

    Mroz, E. A. & Rocco, J. W. Intra-tumor heterogeneity in head and neck cancer and its clinical implications. World J. Otorhinolaryngol. Head Neck Surg. 2, 60–67 (2016).

  112. 112.

    Morris, L. G. T. et al. Pan-cancer analysis of intratumor heterogeneity as a prognostic determinant of survival. Oncotarget 7, 10051–10063 (2016).

  113. 113.

    Sottoriva, A. et al. Intratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics. Proc. Natl Acad. Sci. USA 110, 4009–4014 (2013).

  114. 114.

    Verma, N. et al. Image-based risk score to predict recurrence of ER+ breast cancer in ECOG-ACRIN Cancer Research Group E2197. J. Clin. Oncol. 36 (15_suppl), 540 (2018).

  115. 115.

    Ginsburg, S. B., Lee, G., Ali, S. & Madabhushi, A. Feature importance in nonlinear embeddings (FINE): applications in digital pathology. IEEE Trans. Med. Imaging. 35, 76–88 (2016).

  116. 116.

    Cooper, L. A. D. et al. Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images. Lab. Investig. J. Tech. Methods Pathol. 95, 366–376 (2015).

  117. 117.

    Lloyd, M. C., Johnson, J. O., Kasprzak, A. & Bui, M. M. Image analysis of the tumor microenvironment. Adv. Exp. Med. Biol. 936, 1–10 (2016).

  118. 118.

    Hamilton, P. W. et al. Digital pathology and image analysis in tissue biomarker research. Methods 70, 59–73 (2014).

  119. 119.

    Barsoum, I., Tawedrous, E., Faragalla, H. & Yousef, G. M. Histo-genomics: digital pathology at the forefront of precision medicine. Diagnosis https://doi.org/10.1515/dx-2018-0064 (2018).

  120. 120.

    FDA. Developing a software precertification program: a working model. Fda.gov https://www.fda.gov/media/119722/download (2019).

  121. 121.

    Daniel, G., Silcox, C., Sharma, I. & Wright, M. Current state and near-term priorities for AI-enabled diagnostic support software in health care. Duke.edu https://healthpolicy.duke.edu/sites/default/files/atoms/files/dukemargolisaienableddxss.pdf (2019).

  122. 122.

    Pesapane, F., Volonté, C., Codari, M. & Sardanelli, F. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging 9, 745–753 (2018).

  123. 123.

    Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019).

  124. 124.

    Food and Drug Administration. FDA in brief: FDA proposes improvements to the de novo pathway for novel medical devices to advance safe, effective, and innovative treatments for patients. Fda.gov https://www.fda.gov/NewsEvents/Newsroom/FDAInBrief/ucm627522.htm (2018).

  125. 125.

    European Commission. MDCG 2018-2 Future EU medical device nomenclature – description of requirements. Europa.eu https://ec.europa.eu/docsroom/documents/28668 (2018).

  126. 126.

    European Economic Community. Council Directive 93/42/EEC of 14 June 1993 concerning medical devices. Europa.eu https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1993L0042:20071011:EN:PDF (1993).

  127. 127.

    European Parliament. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending directive 2001/83/EC, regulation (EC) no 178/2002 and regulation (EC) no 1223/2009 and repealing council directives 90/385/EEC and 93/42/EEC (text with EEA relevance). Europa.eu http://data.europa.eu/eli/reg/2017/745/oj/eng (2017).

  128. 128.

    European Parliament. Regulation (EU) 2017/746 of the European Parliament and of the council of 5 April 2017 on in vitro diagnostic medical devices and repealing directive 98/79/EC and commission decision 2010/227/EU (Text with EEA relevance) Europa.eu http://data.europa.eu/eli/reg/2017/746/oj/eng (2017).

  129. 129.

    García-Rojo, M., et al. New European Union regulations related to whole slide image scanners and image analysis software. J. Pathol. Inform. 10, https://doi.org/10.4103/jpi.jpi_33_18 (2019).

  130. 130.

    Arterys. Medical imaging cloud AI. Arterys.com https://www.arterys.com/ (2019).

  131. 131.

    Arterys Inc. Arterys receives first FDA clearance for broad oncology imaging suite with deep learning. Prnewswire.com https://www.prnewswire.com/news-releases/arterys-receives-first-fda-clearance-for-broad-oncology-imaging-suite-with-deep-learning-300599275.html (2018).

  132. 132.

    Food and Drug Administration. IntelliSite Pathology Solution (PIPS, Philips Medical Systems). Fda.gov https://www.fda.gov/drugs/resources-information-approved-drugs/intellisite-pathology-solution-pips-philips-medical-systems (2017).

  133. 133.

    PAIGE. PAIGE. Paige.ai https://paige.ai/ (2019).

  134. 134.

    Business Wire. FDA grants breakthrough designation to Paige.AI. Businesswire.com https://www.businesswire.com/news/home/20190307005205/en/FDA-Grants-Breakthrough-Designation-Paige.AI (2019).

  135. 135.

    Bueno, G., Fernández-Carrobles, M. M., Deniz, O. & García-Rojo, M. New trends of emerging technologies in digital pathology. Pathobiology 83, 61–69 (2016).

  136. 136.

    Food and Drug Administration. Radiological computer-assisted diagnostic (CADx) software for lesions suspicious for cancer. Fda.gov. https://www.accessdata.fda.gov/cdrh_docs/pdf17/DEN170022.pdf (2017).

  137. 137.

    Agendia. MammaPrint Test. Agendia.com https://www.agendia.com/our-tests/mammaprint/ (2019).

  138. 138.

    Food and Drug Administration. MammaPrint 510(k) premarket notification. Fda.gov https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=k070675 (2019).

  139. 139.

    OncotypeIQ. Smarter Cancer Care - Genomic Testing. Oncotypeiq.com https://www.oncotypeiq.com/en-US/ (2019).

  140. 140.

    Food and Drug Administration. FDA issues warning letter to genomics lab for illegally marketing genetic test that claims to predict patients’ responses to specific medications. Fda.gov https://www.fda.gov/news-events/press-announcements/fda-issues-warning-letter-genomics-lab-illegally-marketing-genetic-test-claims-predict-patients (2019).

  141. 141.

    Food and Drug Administration. Discussion paper on laboratory developed tests (LDTs) January 13, 2017. Fda.gov https://www.fda.gov/media/102367/download (2019).

  142. 142.

    Paxton, A. Laboratory-developed tests CAP suggests added oversight of homebrews. Captodayonline.com http://www.captodayonline.com/Archives/1009/1009d_laboratory_developed_tests.html (2009).

  143. 143.

    Higgins, C. Applications and challenges of digital pathology and whole slide imaging. Biotech. Histochem. 90, 341–347 (2015).

  144. 144.

    Tizhoosh, H. R. & Pantanowitz, L. Artificial intelligence and digital pathology: challenges and opportunities. J. Pathol. Inform. 9, 38 (2018).

  145. 145.

    Doyle, S., Feldman, M., Tomaszewski, J. & Madabhushi, A. A boosted Bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies. IEEE Trans. Biomed. Eng. 59, 1205–1218 (2012).

  146. 146.

    Huang, B., Bates, M. & Zhuang, X. Super resolution fluorescence microscopy. Annu. Rev. Biochem. 78, 993–1016 (2009).

  147. 147.

    Kleppe, A. et al. Chromatin organisation and cancer prognosis: a pan-cancer study. Lancet Oncol. 19, 356–369 (2018).

  148. 148.

    Gallas, B. D. NCIP Hub - Group: eeDAP studies ~ Wiki: High-throughput Truthing - Year 2. Ncihub.org https://nciphub.org/groups/eedapstudies/wiki/HighThroughputTruthing/HighThroughputTruthingYear2 (2018).

  149. 149.

    Grand Challenge. All Challenges. Grand-challenge.org https://grand-challenge.org/challenges/ (2019).

  150. 150.

    Ching, P. et al. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15, 20170387 (2018).

  151. 151.

    Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

  152. 152.

    Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Med. 15, e1002683 (2018).

  153. 153.

    Yagi, Y. & Gilbertson, J. R. Digital Imaging in Pathology: The Case for Standardization. (SAGE Publications, 2005).

  154. 154.

    Janowczyk, A., Basavanhally, A. & Madabhushi, A. Stain Normalization using Sparse AutoEncoders (StaNoSA): application to digital pathology. Comput. Med. Imaging Graph. 57, 50–61 (2017).

  155. 155.

    Khan, A. M., Rajpoot, N., Treanor, D. & Magee, D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans. Biomed. Eng. 61, 1729–1738 (2014).

  156. 156.

    Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging. 35, 1962–1971 (2016).

  157. 157.

    Janowczyk, A., Zuo, R., Gilmore, H., Feldman, M. & Madabhushi, A. HistoQC: an open-source quality control tool for digital pathology slides. JCO Clin. Cancer Inform. 3, 1–7 (2019).

  158. 158.

    choosehappy. HistoQC is an Open-source Quality Control Tool for Digital Pathology Slides: choosehappy/HistoQC. Github.com https://github.com/choosehappy/HistoQC (2019).

  159. 159.

    Leo, P. et al. Stable and discriminating features are predictive of cancer presence and Gleason grade in radical prostatectomy specimens: a multi-site study. Sci. Rep. 8, 14918 (2018).

  160. 160.

    Leo, P. et al. Evaluating stability of histomorphometric features across scanner and staining variations: prostate cancer diagnosis from whole slide images. J. Med. Imaging 3, 047502 (2016).

  161. 161.

    Lowe, A. et al. Validation of digital pathology in a healthcare environment. Digitalpathologyassociation.org https://digitalpathologyassociation.org/_data/files/DPA-Healthcare-White-Paper--FINAL_v1.0.pdf (2011).

  162. 162.

    Montalto, M. C. An industry perspective: an update on the adoption of whole slide imaging. J. Pathol. Inform. 7, 18 (2016).

  163. 163.

    Digital Pathology Association. Healthcare FAQs. Digitalpathologyassociation.org https://digitalpathologyassociation.org/healthcare-faqs (2019).

  164. 164.

    Simon, R. M., Paik, S. & Hayes, D. F. Use of archived specimens in evaluation of prognostic and predictive biomarkers. J. Natl Cancer Inst. 101, 1446–1452 (2009).

  165. 165.

    Hayes, D. F. et al. Tumor marker utility grading system: a framework to evaluate clinical utility of tumor markers. J. Natl Cancer Inst. 88, 1456–1466 (1996).

  166. 166.

    Goetz, M. P. et al. NCCN Guidelines insights: breast cancer, version 3.2018. J. Natl Compr. Canc. Netw. 17, 118–126 (2019).

  167. 167.

    Sparano, J. A. et al. Adjuvant chemotherapy guided by a 21-gene expression assay in breast cancer. N. Engl. J. Med. 379, 111–121 (2018).

  168. 168.

    Cardoso, F. et al. 70-gene signature as an aid to treatment decisions in early-stage breast cancer. N. Engl. J. Med. 375, 717–729 (2016).

  169. 169.

    Paik, S. et al. A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast cancer. N. Engl. J. Med. 351, 2817–2826 (2004).

  170. 170.

    Paik, S. et al. Gene expression and benefit of chemotherapy in women with node-negative, estrogen receptor-positive breast cancer. J. Clin. Oncol. 24, 3726–3734 (2006).

  171. 171.

    DecipherBio. GenomeDx Biosciences: Where to get decipher prostate cancer classifier. decipherbio.com https://genomedx.com/decipher-test/get-decipher/ (2019).

  172. 172.

    Erho, N. et al. Discovery and validation of a prostate cancer genomic classifier that predicts early metastasis following radical prostatectomy. PLOS ONE. 8, e66855 (2013).

  173. 173.

    Carroll, P. H. & Mohler, J. L. NCCN guidelines updates: prostate cancer and prostate cancer early detection. J. Natl Compr. Canc. Netw. 16(5S), 620–623 (2018).

  174. 174.

    Bhargava, R. & Madabhushi, A. Emerging themes in image informatics and molecular analysis for digital pathology. Annu. Rev. Biomed. Eng. 18, 387–412 (2016).

  175. 175.

    Madabhushi, A. Digital pathology image analysis: opportunities and challenges. Imaging Med. 1, 7–10 (2009).

  176. 176.

    Madabhushi, A., Agner, S., Basavanhally, A., Doyle, S. & Lee, G. Computer-aided prognosis: predicting patient and disease outcome via quantitative fusion of multi-scale, multi-modal data. Comput. Med. Imaging Graph. 35, 506–514 (2011).

  177. 177.

    Pantanowitz, L. et al. Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch. Pathol. Lab. Med. 137, 1710–1722 (2013).

  178. 178.

    Fuchs, T. J., Wild, P. J., Moch, H. & Buhmann, J. M. Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. Med. Image Comput. Comput. Assist. Interv. 11, 1–8 (2008).

  179. 179.

    Haybäck, J. Mechanisms of Molecular Carcinogenesis – Volume 2. (Springer, 2017).

  180. 180.

    Proscia. Proscia digital pathology. Proscia.com https://proscia.com (2019).

  181. 181.

    Deep Lens. Digital pathology cloud platform. Deeplens.ai https://www.deeplens.ai (2019).

  182. 182.

    PathAI. PathAI. Pathai.com https://www.pathai.com/ (2019).

  183. 183.

    Aifora. WebMicroscope. Big pictures. Deep Diagnosis. Aiforia.com https://www.aiforia.com/ (2019).

  184. 184.

    Glaser, A. K., et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nat. Biomed. Eng. 1, 0084 (2017).

  185. 185.

    Fereidouni, F. et al. Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nat. Biomed. Eng. 1, 957 (2017).

  186. 186.

    Fraggetta, F., Garozzo, S., Zannoni, G. F., Pantanowitz, L. & Rossi, E. D. Routine digital pathology workflow: the Catania experience. J. Pathol. Inform. 8, 51 (2017).

  187. 187.

    Cheng, C. L. et al. Enabling digital pathology in the diagnostic setting: navigating through the implementation journey in an academic medical centre. J. Clin. Pathol. 69, 784–792 (2016).

  188. 188.

    Tomczak, K., Czerwińska, P. & Wiznerowicz, M. The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. Contemp. Oncol. 19, A68–77 (2015).

  189. 189.

    Gutman, D. A. et al. Cancer Digital Slide Archive: an informatics resource to support integrated in silico analysis of TCGA pathology data. J. Am. Med. Inform. Assoc. 20, 1091–1098 (2013).

Download references

Acknowledgements

Research discussed in this publication was supported by the Department of Defence and Department of Veterans, the National Cancer Institute of the National Institutes of Health, the National Centre for Research Resources, the Ohio Third Frontier Technology Validation Fund, and the Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering and the Clinical and Translational Science Award Program (CTSA) at Case Western Reserve University. The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the institutions named.

Author information

K.B. and A.M. researched data for the article and discussed the article contents. All authors wrote, reviewed and edited the manuscript before submission.

Correspondence to Anant Madabhushi.

Ethics declarations

Competing interests

K.A.S. has received fees as a speaker for Merck and Takeda, is a consultant for Celgene, Moderna and Shattuck Labs, and receives research funding from Genoptix (Novartis), Onkaido, Pierre Fabre, Surface Oncology, Takeda, Tesaro and Tioma. D.L.R. is a consultant and research adviser to Agendia, Agilent, AstraZeneca, Biocept, BMS, Cell Signalling Technology, Cepheid, Merck, PAIGE, Perkin Elmer and Ultivue, owns equity in AstraZeneca, Cepheid, Lilly, Navigate–Novartis, NextCure, PixelGear and Ultivue, and receives research funding from Perkin Elmer. A.M. is a consultant and scientific adviser to Elucid Bioimaging and Inspirata, owns stock options in Elucid Bioimaging and Inspirata, and receives research funding from Philips. K.B. and V.V. declare no competing interests.

Additional information

Peer review information

Nature Reviews Clinical Oncology thanks J. Liu, A. Tsirigos and the other, anonymous, reviewer(s), for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark