Artificial intelligence (AI) algorithms, particularly deep learning, have demonstrated remarkable progress in image-recognition tasks. Methods ranging from convolutional neural networks to variational autoencoders have found myriad applications in the medical image analysis field, propelling it forward at a rapid pace. Historically, in radiology practice, trained physicians visually assessed medical images for the detection, characterization and monitoring of diseases. AI methods excel at automatically recognizing complex patterns in imaging data and providing quantitative, rather than qualitative, assessments of radiographic characteristics. In this Opinion article, we establish a general understanding of AI methods, particularly those pertaining to image-based tasks. We explore how these methods could impact multiple facets of radiology, with a general focus on applications in oncology, and demonstrate ways in which these methods are advancing the field. Finally, we discuss the challenges facing clinical implementation and provide our perspective on how the domain could be advanced.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Additional information

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


  1. 1.

    Editors, N. Auspicious machine learning. Nat. Biomed. Engineer. 1, 0036 (2017).

  2. 2.

    Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

  3. 3.

    Moravcík, M. et al. DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science 356, 508–513 (2017).

  4. 4.

    Xiong, W. et al. Toward human parity in conversational speech recognition. IEEE/ACM Trans. Audio Speech Language Process. 25, 2410–2423 (2017).

  5. 5.

    Pendleton, S. D. et al. Perception, planning, control, and coordination for autonomous vehicles. Machines 5, 6 (2017).

  6. 6.

    Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

  7. 7.

    Grace, K., Salvatier, J., Dafoe, A., Zhang, B. & Evans, O. When will AI exceed human performance? Evidence from AI experts. Preprint at arXiv, 1705.08807 (2017).

  8. 8.

    Rusk, N. Deep learning. Nat. Methods 13, 35–35 (2015).

  9. 9.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  10. 10.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

  11. 11.

    Aerts, H. J. W. L. The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol. 2, 1636–1642 (2016).

  12. 12.

    Kumar, V. et al. Radiomics: the process and the challenges. Magn. Reson. Imag. 30, 1234–1248 (2012).

  13. 13.

    Lambin, P. et al. Radiomics: extracting more information from medical images using advanced feature analysis. Eur. J. Cancer 48, 441–446 (2012).

  14. 14.

    Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).

  15. 15.

    Kolossváry, M., Kellermayer, M., Merkely, B. & Maurovich-Horvat, P. Cardiac computed tomography radiomics: a comprehensive review on radiomic techniques. J. Thorac. Imag. 33, 26–34 (2018).

  16. 16.

    Aerts, H. J. W. L. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 5, 4006 (2014).

  17. 17.

    Coroller, T. P. et al. CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma. Radiother. Oncol. 114, 345–350 (2015).

  18. 18.

    Wu, W. et al. Exploratory study to identify radiomics classifiers for lung cancer histology. Front. Oncol. 6, 71 (2016).

  19. 19.

    Huynh, E. et al. Associations of radiomic data extracted from static and respiratory-gated CT scans with disease recurrence in lung cancer patients treated with SBRT. PLoS ONE 12, e0169172 (2017).

  20. 20.

    Rios Velazquez, E. et al. Somatic mutations drive distinct imaging phenotypes in lung cancer. Cancer Res. 77, 3922–3930 (2017).

  21. 21.

    Grossmann, P. et al. Defining the biological basis of radiomic phenotypes in lung cancer. eLife 6, e23421 (2017).

  22. 22.

    Parmar, C., Grossmann, P., Bussink, J., Lambin, P. & Aerts, H. J. W. L. Machine learning methods for quantitative radiomic biomarkers. Sci. Rep. 5, 13087 (2015).

  23. 23.

    O’Connor, J. P. B. et al. Imaging biomarker roadmap for cancer studies. Nat. Rev. Clin. Oncol. 14, 169–186 (2017).

  24. 24.

    Boland, G. W. L., Guimaraes, A. S. & Mueller, P. R. The radiologist’s conundrum: benefits and costs of increasing CT capacity and utilization. Eur. Radiol. 19, 9–12 (2009).

  25. 25.

    McDonald, R. J. et al. The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload. Acad. Radiol. 22, 1191–1198 (2015).

  26. 26.

    Fitzgerald, R. Error in radiology. Clin. Radiol. 56, 938–946 (2001).

  27. 27.

    Ledley, R. S. & Lusted, L. B. Reasoning foundations of medical diagnosis; symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science 130, 9–21 (1959).

  28. 28.

    Lodwick, G. S., Keats, T. E. & Dorst, J. P. The coding of Roentgen images for computer analysis as applied to lung cancer. Radiology 81, 185–200 (1963).

  29. 29.

    Ambinder, E. P. A history of the shift toward full computerization of medicine. J. Oncol. Pract. 1, 54–56 (2005).

  30. 30.

    Haug, P. J. Uses of diagnostic expert systems in clinical care. Proc. Annu. Symp. Comput. Appl. Med. Care, 379–383 (1993).

  31. 31.

    Castellino, R. A. Computer aided detection (CAD): an overview. Cancer Imag. 5, 17–19 (2005).

  32. 32.

    Shen, D., Wu, G. & Suk, H.-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017).

  33. 33.

    Veeraraghavan, H. MO-A-207B-01: Radiomics: Segmentation & feature extraction techniques. Med. Phys. 43, 3694–3694 (2016).

  34. 34.

    Paul, R. et al. Deep feature transfer learning in combination with traditional features predicts survival among patients with lung adenocarcinoma. Tomography 2, 388–395 (2016).

  35. 35.

    Cheng, J.-Z. et al. Computer-aided diagnosis with deep learning architecture: applications to breast lesions in US images and pulmonary nodules in CT scans. Sci. Rep. 6, 24454 (2016).

  36. 36.

    Chen, H., Zheng, Y., Park, J.-H., Heng, P.-A. & Zhou, S. K. in Medical Image Computing and Computer-Assisted Intervention — MICCAI 2016 487–495 (Athens, Greece, 2016).

  37. 37.

    Ghafoorian, M. et al. Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Sci. Rep. 7, 5110 (2017).

  38. 38.

    Wang, H. et al. Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images. EJNMMI Res. 7, 11 (2017).

  39. 39.

    van Ginneken, B., Schaefer-Prokop, C. M. & Prokop, M. Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 261, 719–732 (2011).

  40. 40.

    Nagaraj, S., Rao, G. N. & Koteswararao, K. The role of pattern recognition in computer-aided diagnosis and computer-aided detection in medical imaging: a clinical validation. Int. J. Comput. Appl. 8, 18–22 (2010).

  41. 41.

    Cole, E. B. et al. Impact of computer-aided detection systems on radiologist accuracy with digital mammography. AJR Am. J. Roentgenol. 203, 909–916 (2014).

  42. 42.

    Lehman, C. D. et al. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern. Med. 175, 1828–1837 (2015).

  43. 43.

    Huang, X., Shan, J. & Vaidya, V. in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) 379–383 (Melbourne, Australia, 2017).

  44. 44.

    Tsehay, Y. K. et al. in Proceedings of SPIE https://doi.org/10.1117/12.2254423 (2017).

  45. 45.

    Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35, 303–312 (2017).

  46. 46.

    Sharma, N. & Aggarwal, L. M. Automated medical image segmentation techniques. J. Med. Phys. 35, 3–14 (2010).

  47. 47.

    Haralick, R. M. & Shapiro, L. G. Image segmentation techniques. Computer Vision Graph. Image Process. 29, 100–132 (1985).

  48. 48.

    Pham, D. L., Xu, C. & Prince, J. L. Current methods in medical image segmentation. Annu. Rev. Biomed. Eng. 2, 315–337 (2000).

  49. 49.

    Grau, V., Mewes, A. U. J., Alcañiz, M., Kikinis, R. & Warfield, S. K. Improved watershed transform for medical image segmentation using prior information. IEEE Trans. Med. Imag. 23, 447–458 (2004).

  50. 50.

    Parisot, S. et al. A probabilistic atlas of diffuse WHO grade II glioma locations in the brain. PLoS ONE 11, e0144200 (2016).

  51. 51.

    Ghose, S. et al. in 2012 19th IEEE International Conference on Image Processing 541–544 (Orlando, FL, USA, 2012).

  52. 52.

    Han, X. et al. Atlas-based auto-segmentation of head and neck CT images. Med. Image Comput. Comput. Assist. Interv. 11, 434–441 (2008).

  53. 53.

    Long, J., Shelhamer, E. & Darrell, T. in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 3431–3440 (Boston, MA, USA, 2015).

  54. 54.

    Ronneberger, O., Fischer, P. & Brox, T. U. in Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015 234–241 (Munich, Germany, 2015).

  55. 55.

    Moeskops, P. et al. in Medical Image Computing and Computer-Assisted Intervention — MICCAI 2016 478–486 (Athens, Greece, 2016).

  56. 56.

    de Brebisson, A. & Montana, G. in 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 20–28 (Boston, MA, USA, 2015).

  57. 57.

    Cioffi, U., Raveglia, F., De Simone, M. & Baisi, A. Ground-glass opacities: a curable disease but a big challenge for surgeons. J. Thorac. Cardiovasc. Surg. 154, 375–376 (2017).

  58. 58.

    Champaign, J. L. & Cederbom, G. J. Advances in breast cancer detection with screening mammography. Ochsner J. 2, 33–35 (2000).

  59. 59.

    Shiraishi, J., Li, Q., Appelbaum, D. & Doi, K. Computer-aided diagnosis and artificial intelligence in clinical imaging. Semin. Nucl. Med. 41, 449–462 (2011).

  60. 60.

    Ayer, T., Ayvaci, M. U., Liu, Z. X., Alagoz, O. & Burnside, E. S. Computer-aided diagnostic models in breast cancer screening. Imag. Med. 2, 313–323 (2010).

  61. 61.

    Zhang, J., Wang, Y., Yu, B., Shi, X. & Zhang, Y. Application of computer-aided diagnosis to the sonographic evaluation of cervical lymph nodes. Ultrason. Imag. 38, 159–171 (2016).

  62. 62.

    Giannini, V. et al. A fully automatic computer aided diagnosis system for peripheral zone prostate cancer detection using multi-parametric magnetic resonance imaging. Comput. Med. Imaging Graph. 46, 219–226 (2015).

  63. 63.

    El-Baz, A. et al. Computer-aided diagnosis systems for lung cancer: challenges and methodologies. Int. J. Biomed. Imag. 2013, 942353 (2013).

  64. 64.

    Edey, A. J. & Hansell, D. M. Incidentally detected small pulmonary nodules on CT. Clin. Radiol. 64, 872–884 (2009).

  65. 65.

    Mirsadraee, S., Oswal, D., Alizadeh, Y., Caulo, A. & van Beek, E. Jr. The7th lung cancer TNM classification and staging system: review of the changes and implications. World J. Radiol. 4, 128–134 (2012).

  66. 66.

    Sohn, K., Shang, W. & Lee, H. in Advances in Neural Information Processing Systems 27 (NIPS 2014) (eds Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 2141–2149 (Montreal, Canada, 2014).

  67. 67.

    Litjens, G. et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci. Rep. 6, 26286 (2016).

  68. 68.

    Cruz-Roa, A. et al. Accurate and reproducible invasive breast cancer detection in whole-slide images: A Deep Learning approach for quantifying tumor extent. Sci. Rep. 7, 46450 (2017).

  69. 69.

    Jaffe, C. C. Measures of response: RECIST, WHO, and new alternatives. J. Clin. Oncol. 24, 3245–3251 (2006).

  70. 70.

    Thiesse, P. et al. Response rate accuracy in oncology trials: reasons for interobserver variability. Groupe Français d’Immunothérapie of the Fédération Nationale des Centres de Lutte Contre le Cancer. J. Clin. Oncol. 15, 3507–3514 (1997).

  71. 71.

    Khorasani, R., Erickson, B. J. & Patriarche, J. New opportunities in computer-aided diagnosis: change detection and characterization. J. Am. Coll. Radiol. 3, 468–469 (2006).

  72. 72.

    Patriarche, J. W. & Erickson, B. J. Part 1. Automated change detection and characterization in serial MR studies of brain-tumor patients. J. Digit. Imag. 20, 203–222 (2007).

  73. 73.

    Pan, X., Sidky, E. Y. & Vannier, M. Why do commercial CT scanners still employ traditional, filtered back-projection for image reconstruction? Inverse Probl. 25, 1230009 (2009).

  74. 74.

    Pipatsrisawat, T., Gacic, A., Franchetti, F., Puschel, M. & Moura, J. M. F. in Proceedings. (ICASSP ‘05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005 v/153–v/156 (Philadelphia, PA, USA, 2005).

  75. 75.

    Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R. & Rosen, M. S. Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018).

  76. 76.

    Hammernik, K., Würfl, T., Pock, T. & Maier, A. A. in Bildverarbeitung für die Medizin 2017 (eds Maier-Hein, K., Deserno, T., Handels, H. & Tolxdorff, T.) 92–97 (Springer, Berlin, Heidelberg, 2017).

  77. 77.

    Gjesteby, L. et al. in Developments in X-Ray Tomography XI 10391-31 (San Diego, CA, USA, 2017).

  78. 78.

    El-Gamal, F. E.-Z. A., Elmogy, M. & Atwan, A. Current trends in medical image registration and fusion. Egypt. Informat. J. 17, 99–124 (2016).

  79. 79.

    Yang, X., Kwitt, R., Styner, M. & Niethammer, M. Quicksilver: fast predictive image registration — a deep learning approach. Neuroimage 158, 378–396 (2017).

  80. 80.

    Ngiam, J. et al. in Proceedings of the 28th International Conference on Machine Learning 689–696 (Bellevue, WA, USA, 2011).

  81. 81.

    Yankeelov, T. E., Abramson, R. G. & Quarles, C. C. Quantitative multimodality imaging in cancer research and therapy. Nat. Rev. Clin. Oncol. 11, 670–680 (2014).

  82. 82.

    Johnson, A. J., Chen, M. Y. M., Zapadka, M. E., Lyders, E. M. & Littenberg, B. Radiology report clarity: a cohort study of structured reporting compared with conventional dictation. J. Am. Coll. Radiol. 7, 501–506 (2010).

  83. 83.

    Levy, M. A. & Rubin, D. L. Tool support to enable evaluation of the clinical response to treatment. AMIA Annu. Symp. Proc. 2008, 399–403 (2008).

  84. 84.

    European Society of Radiology (ESR). Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR). Insights Imag. 2, 93–96 (2011).

  85. 85.

    Folio, L. R. et al. Quantitative radiology reporting in oncology: survey of oncologists and radiologists. AJR Am. J. Roentgenol. 205, W233–W243 (2015).

  86. 86.

    Karpathy, A. & Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 39, 664–676 (2017).

  87. 87.

    Shin, H.-C. et al. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2497–2506 (Las Vegas, NV, USA, 2016).

  88. 88.

    Lee, J.-G. et al. Deep learning in medical imaging: general overview. Kor. J. Radiol. 18, 570–584 (2017).

  89. 89.

    OECD. Computed tomography (CT) exams. https://doi.org/10.1787/3c994537-en (2018).

  90. 90.

    OECD. Magnetic resonance imaging (MRI) exams. https://doi.org/10.1787/1d89353f-en (2018).

  91. 91.

    Bryan, S. et al. Radiology report times: impact of picture archiving and communication systems. AJR Am. J. Roentgenol. 170, 1153–1159 (1998).

  92. 92.

    Mansoori, B., Erhard, K. K. & Sunshine, J. L. Picture Archiving and Communication System (PACS) implementation, integration and benefits in an integrated health system. Acad. Radiol. 19, 229–235 (2012).

  93. 93.

    Lemke, H. U. PACS developments in Europe. Comput. Med. Imag. Graph. 27, 111–120 (2003).

  94. 94.

    Mendel, J. B. & Schweitzer, A. L. PACS for the developing world. J. Global Radiol. 1, 5 (2015).

  95. 95.

    Goodfellow, I. et al. in Advances in Neural Information Processing Systems 27 (NIPS 2014) (eds Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 2672–2680 (Montreal, Canada, 2014).

  96. 96.

    Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. Preprint at arXiv, 1312.6114 (2013).

  97. 97.

    Kamnitsas, K. et al. in Information Processing in Medical Imaging 597–609 (Springer, Cham, 2017).

  98. 98.

    Kallenberg, M. et al. Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring. IEEE Trans. Med. Imag. 35, 1322–1331 (2016).

  99. 99.

    Zhang, P., Wang, F. & Zheng, Y. in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) 578–582 (Melbourne, Australia, 2017).

  100. 100.

    Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imag. 26, 1045–1057 (2013).

  101. 101.

    Wang, G. A. Perspective on deep imaging. IEEE Access 4, 8914–8924 (2016).

  102. 102.

    Ford, R. A., Price, W. & Nicholson, I. I. Privacy and accountability in black-box medicine. Mich. Telecomm. Tech. L. Rev. 23, 1 (2016).

  103. 103.

    Selbst, A. D. & Powles, J. Meaningful information and the right to explanation. Int. Data Privacy Law 7, 233–242 (2017).

  104. 104.

    Imming, P., Sinning, C. & Meyer, A. Drugs, their targets and the nature and number of drug targets. Nat. Rev. Drug Discov. 5, 821–834 (2006).

  105. 105.

    Mehlhorn, H. et al. in Encyclopedia of Parasitology 3rd edn (ed. Mehlhorn, H.) 400–402 (Springer, Berlin, Heidelberg, 2008).

  106. 106.

    Shokri, R. & Shmatikov, V. in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security 1310–1321 (Denver, CO, USA, 2015).

  107. 107.

    Phong, L. T., Aono, Y., Hayashi, T., Wang, L. & Moriai, S. in Applications and Techniques in Information Security. 8th International Conference, ATIS 2017 (eds Batten, L., Kim, D. S., Zhang, X. & Li, G.) 719, 100–110 (Auckland, New Zealand, 2017).

  108. 108.

    McMahan, H. B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) 1273–1282 (Fort Lauderdale, FL, USA, 2017).

  109. 109.

    Gilad-Bachrach, R. et al. in Proceedings of the 33rd International Conference on Machine Learning 201–210 (New York, NY, USA, 2016).

  110. 110.

    Cahan, A. & Cimino, J. J. A. Learning health care system using computer-aided diagnosis. J. Med. Internet Res. 19, e54 (2017).

  111. 111.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  112. 112.

    Miotto, R., Wang, F., Wang, S., Jiang, X. & Dudley, J. T. Deep learning for healthcare: review, opportunities and challenges. Brief. Bioinform. https://doi.org/10.1093/bib/bbx044 (2017).

  113. 113.

    Kevin Zhou, S., Greenspan, H. & Shen, D. Deep Learning for Medical Image Analysis. (Academic Press, 2017).

  114. 114.

    Shin, H.-C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imag. 35, 1285–1298 (2016).

  115. 115.

    Shin, Y. & Balasingham, I. in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3277–3280 (Jeju Island, Korea, 2017).

  116. 116.

    Orringer, D. A. et al. Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy. Nat. Biomed. Eng. 1, 0027 (2017).

  117. 117.

    Albarqouni, S. et al. AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35, 1313–1321 (2016).

  118. 118.

    Djuric, U., Zadeh, G., Aldape, K. & Diamandis, P. Precision histology: how deep learning is poised to revitalize histomorphology for personalized cancer care. Precision Oncol. 1, 22 (2017).

  119. 119.

    Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J. Pathol. Inform. 7, 29 (2016).

  120. 120.

    Bejnordi, B. E. et al. in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) 929–932 (Melbourne, Australia, 2017).

  121. 121.

    Yuan, Y. et al. DeepGene: an advanced cancer type classifier based on deep learning and somatic point mutations. BMC Bioinform. 17, 476 (2016).

  122. 122.

    Alipanahi, B., Delong, A., Weirauch, M. T. & Frey, B. J. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33, 831–838 (2015).

Download references


The authors acknowledge financial support from the US National Institutes of Health (NIH-USA U24CA194354 and NIH-USA U01CA190234).

Author information


  1. Department of Radiation Oncology, Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA, USA

    • Ahmed Hosny
    • , Chintan Parmar
    •  & Hugo J. W. L. Aerts
  2. Department of Biostatistics & Computational Biology, Dana-Farber Cancer Institute, Boston, MA, USA

    • John Quackenbush
  3. Department of Cancer Biology, Dana-Farber Cancer Institute, Boston, MA, USA

    • John Quackenbush
  4. Department of Radiology, Columbia University College of Physicians and Surgeons, New York, NY, USA

    • Lawrence H. Schwartz
  5. Department of Radiology, New York Presbyterian Hospital, New York, NY, USA

    • Lawrence H. Schwartz
  6. Department of Radiology, Dana-Farber Cancer Institute, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

    • Hugo J. W. L. Aerts


  1. Search for Ahmed Hosny in:

  2. Search for Chintan Parmar in:

  3. Search for John Quackenbush in:

  4. Search for Lawrence H. Schwartz in:

  5. Search for Hugo J. W. L. Aerts in:


A.H., C.P. and H.J.W.L.A. performed the literature survey, curated the content and general direction and wrote the manuscript. J.Q. and L.H.S. provided substantial contributions to discussions of the content. All authors contributed to reviewing and editing the manuscript before submission.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to Hugo J. W. L. Aerts.


Area under receiver operating characteristic curve

(AUC). A sensitivity versus specificity metric for measuring the performance of binary classifiers that can be extended to multi-class problems. The area under the curve is equal to the probability that a randomly chosen positive sample ranks above a randomly chosen negative one or is regarded to have a higher probability of being positive.

Artificial intelligence

(AI). A branch of computer science involved with the development of machines that are able to perform cognitive tasks that would normally require human intelligence.

Caption generation

The often automated generation of qualitative text describing an illustration or image and its contents.

Ground-glass opacity

(GGO). A visual feature of some subsolid pulmonary nodules that is characterized by focal areas of slightly increased attenuation on computed tomography. Underlying bronchial structures and vessels are often visually preserved (being even more recognizable owing to increased contrast), thus making the detection and diagnosis of such nodules somewhat challenging.

Health Insurance Portability and Accountability Act

(HIPAA). A US act that sets provisions for protecting and securing sensitive patient medical data.

Image registration

A process that involves aligning medical images either in terms of spatial or temporal characteristics, mostly intramodality and occasionally intermodality.

Imaging modalities

A multitude of imaging methods that are used to non-invasively generate visualizations of the human anatomy. Examples of these include computed tomography (CT), computed tomography angiography (CTA), magnetic resonance imaging (MRI), mammography, ultrasonography (echocardiography) and positron emission tomography (PET).


Within optimization problems, constantly adjusted parameters during run time need to be initialized to some value before the start of the process. Good initialization techniques aid models in converging faster and hence speed up the iteration process.

Machine learning

A branch of artificial intelligence and computer science that enables computers to learn without being explicitly programmed.

Multiparametric imaging

Medical imaging in which two or more parameters are used to visualize differences between healthy and diseased tissue. In multiparametric magnetic resonance imaging (MRI), these parameters include T2-weighted MRI, diffusion-weighted MRI and dynamic contrast-enhanced MRI, among others.

Predefined engineered features

A set of context-based human-crafted features designed to represent knowledge regarding a specific data space.

Probabilistic atlas

A single composite image formed by combining and registering pre-segmented images of multiple patients that thus contains knowledge on population variability.


A data-centric field investigating the clinical relevance of radiographic tissue characteristics automatically quantified from medical images.

Report generation

The communication of assessments and findings in both image and text formats among medical professionals.


The partitioning of images to produce boundary delineations of objects of interest. Such a boundary is defined by pixels and voxels (3D pixels) when performed in 2D and 3D, respectively.

Self-supervised learning

A type of supervised learning where labels are determined by the input data as opposed to being explicitly provided.

Supervised learning

A type of machine learning where functions are inferred from labelled training data. Example data pairs consist of the input together with its desired output or label.

Unsupervised learning

A type of machine learning where functions are inferred from training data without corresponding labels.


A collective term describing health-monitoring devices, smartwatches and fitness trackers that have recently been integrated into the health-care ecosystem as a means to remotely track vitals and adhere to treatment plans.

About this article

Publication history