Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Predicting cancer outcomes with radiomics and artificial intelligence in radiology

Abstract

The successful use of artificial intelligence (AI) for diagnostic purposes has prompted the application of AI-based cancer imaging analysis to address other, more complex, clinical needs. In this Perspective, we discuss the next generation of challenges in clinical decision-making that AI tools can solve using radiology images, such as prognostication of outcome across multiple cancers, prediction of response to various treatment modalities, discrimination of benign treatment confounders from true progression, identification of unusual response patterns and prediction of the mutational and molecular profile of tumours. We describe the evolution of and opportunities for AI in oncology imaging, focusing on hand-crafted radiomic approaches and deep learning-derived representations, with examples of their application for decision support. We also address the challenges faced on the path to clinical adoption, including data curation and annotation, interpretability, and regulatory and reimbursement issues. We hope to demystify AI in radiology for clinicians by helping them to understand its limitations and challenges, as well as the opportunities it provides as a decision-support tool in cancer management.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Workflow for AI-enabled biomarkers in radiology.
Fig. 2: Examples of the types of radiomic feature used in oncology.
Fig. 3: Building blocks and types of neural network commonly applied to medical imaging data.
Fig. 4: Different levels of annotation detail in radiomics and deep learning studies.

Similar content being viewed by others

References

  1. Giger, M. L., Chan, H.-P. & Boone, J. Anniversary Paper: History and status of CAD and quantitative image analysis: the role of medical physics and AAPM. Med. Phys. 35, 5799–5820 (2008).

    PubMed  PubMed Central  Google Scholar 

  2. Giger, M. L., Doi, K. & MacMahon, H. Computerized detection of lung nodules in digital chest radiographs. Med. Imaging Proc. 767, 384–387 (1987).

    Google Scholar 

  3. Carmody, D. P., Nodine, C. F. & Kundel, H. L. An analysis of perceptual and cognitive factors in radiographic interpretation. Perception 9, 339–344 (1980).

    CAS  PubMed  Google Scholar 

  4. Kundel, H. L. & Hendee, W. R. The perception of radiologic image information. Report of an NCI workshop on April 15-16, 1985. Invest. Radiol. 20, 874–877 (1985).

    CAS  PubMed  Google Scholar 

  5. Rao, V. M. et al. How widely Is computer-aided detection used in screening and diagnostic mammography? J. Am. Coll. Radiol. 7, 802–805 (2010).

    PubMed  Google Scholar 

  6. McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020).

    CAS  Google Scholar 

  7. Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

    Google Scholar 

  8. Frelaut, M., Le Tourneau, C. & Borcoman, E. Hyperprogression under immunotherapy. Int. J. Mol. Sci. 20, 2674 (2019).

    CAS  PubMed Central  Google Scholar 

  9. Frelaut, M., du Rusquec, P., de Moura, A., Le Tourneau, C. & Borcoman, E. Pseudoprogression and hyperprogression as new forms of response to immunotherapy. BioDrugs 34, 463–476 (2020).

    PubMed  Google Scholar 

  10. Cruz, L. C. H., da, Rodriguez, I., Domingues, R. C., Gasparetto, E. L. & Sorensen, A. G. Pseudoprogression and pseudoresponse: imaging challenges in the assessment of posttreatment glioma. Am. J. Neuroradiol. 32, 1978–1985 (2011).

    Google Scholar 

  11. FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource (FDA, 2016).

  12. Griethuysen, J. J. M. V. et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77, e104–e107 (2017).

    PubMed  PubMed Central  Google Scholar 

  13. Nicolini, A., Ferrari, P. & Duffy, M. J. Prognostic and predictive biomarkers in breast cancer: past, present and future. Semin. Cancer Biol. 52, 56–73 (2018).

    CAS  PubMed  Google Scholar 

  14. Cucchiara, V. et al. Genomic markers in prostate cancer decision making. Eur. Urol. 73, 572–582 (2018).

    PubMed  Google Scholar 

  15. LI, S. G. & LI, L. Targeted therapy in HER2-positive breast cancer. Biomed. Rep. 1, 499–505 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  16. Chan, B. A. & Hughes, B. G. Targeted therapy for non-small cell lung cancer: current standards and the promise of the future. Transl. Lung Cancer Res. 4, 36 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Sparano, J. A. et al. Adjuvant chemotherapy guided by a 21-gene expression assay in breast cancer. N. Engl. J. Med. 379, 111–121 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  18. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    CAS  PubMed  Google Scholar 

  19. Pfaehler, E., Zwanenburg, A., Jong, J. Rde & Boellaard, R. RaCaT: an open source and easy to use radiomics calculator tool. PLoS ONE 14, e0212223 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  20. Verma, V. et al. The rise of radiomics and implications for oncologic management. J. Natl Cancer Inst. 109, djx055 (2017).

    Google Scholar 

  21. Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V. & Madabhushi, A. Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16, 703–715 (2019).

    PubMed  PubMed Central  Google Scholar 

  22. Wan, T. et al. A radio-genomics approach for identifying high risk estrogen receptor-positive breast cancers on DCE-MRI: preliminary results in predicting oncotypeDX risk scores. Sci. Rep. 6, 21394 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  23. Li, H. et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of mammaprint, oncotype DX, and PAM50 gene assays. Radiology 281, 382–391 (2016).

    PubMed  Google Scholar 

  24. Cyll, K. et al. Tumour heterogeneity poses a significant challenge to cancer biomarker research. Br. J. Cancer 117, 367–375 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  25. Crowley, E., Di Nicolantonio, F., Loupakis, F. & Bardelli, A. Liquid biopsy: monitoring cancer-genetics in the blood. Nat. Rev. Clin. Oncol. 10, 472–484 (2013).

    CAS  PubMed  Google Scholar 

  26. Lim, Z.-F. & Ma, P. C. Emerging insights of tumor heterogeneity and drug resistance mechanisms in lung cancer targeted therapy. J. Hematol. Oncol. 12, 134 (2019).

    PubMed  PubMed Central  Google Scholar 

  27. Mazurowski, M. A. Radiogenomics: what it is and why it is important. J. Am. Coll. Radiol. 12, 862–866 (2015).

    PubMed  Google Scholar 

  28. Bodalal, Z., Trebeschi, S., Nguyen-Kim, T. D. L., Schats, W. & Beets-Tan, R. Radiogenomics: bridging imaging and genomics. Abdom. Radiol. 44, 1960–1984 (2019).

    Google Scholar 

  29. Eben, J., Braman, N. & Madabhushi, A. in Medical Image Computing and Computer Assisted Intervention Vol. 11767 (eds Shen, D. et al.) 602–610 (Springer, 2019).

  30. Bizzego, A. et al. Integrating deep and radiomics features in cancer bioimaging. IEEE Conf. Comput. Intell. Bioinform. Comput. Biol. https://doi.org/10.1109/CIBCB.2019.8791473 (2019).

    Article  Google Scholar 

  31. Lao, J. et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci. Rep. 7, 10353 (2017).

    PubMed  PubMed Central  Google Scholar 

  32. ‘Student’. The probable error of a mean. Biometrika 6, 1–25 (1908).

    Google Scholar 

  33. Wilcoxon, F. Individual comparisons by ranking methods. Biometrics Bull. 1, 80–83 (1945).

    Google Scholar 

  34. Ding, C. & Peng, H. Minimum redundancy feature selection from microarray gene expression data. J. Bioinform. Comput. Biol. 3, 185–205 (2005).

    CAS  PubMed  Google Scholar 

  35. Chirra, P. et al. Multisite evaluation of radiomic feature reproducibility and discriminability for identifying peripheral zone prostate tumors on MRI. J. Med. Imaging 6, 024502 (2019).

    Google Scholar 

  36. Lee, J. W. & et al. Prognostic significance of CT-attenuation of tumor-adjacent breast adipose tissue in breast cancer patients with surgical resection. Cancers 11, 1135 (2019).

    CAS  PubMed Central  Google Scholar 

  37. Eguchi, T. et al. Tumor size and computed tomography attenuation of pulmonary pure ground-glass nodules are useful for predicting pathological invasiveness. PLoS ONE 9, e97867 (2014).

    PubMed  PubMed Central  Google Scholar 

  38. Kinahan, P. E. & Fletcher, J. W. PET/CT standardized uptake values (SUVs) in clinical practice and assessing response to therapy. Semin. Ultrasound CT MR 31, 496–505 (2010).

    PubMed  PubMed Central  Google Scholar 

  39. Kubota, K. From tumor biology to clinical PET: a review of positron emission tomography (PET) in oncology. Ann. Nucl. Med. 15, 471–486 (2001).

    CAS  PubMed  Google Scholar 

  40. Sheikhbahaei, S. et al. The value of FDG PET/CT in treatment response assessment, follow-up, and surveillance of lung cancer. Am. J. Roentgenol. 208, 420–433 (2016).

    Google Scholar 

  41. Eckstein, J. M. et al. Primary vs nodal site PET/CT response as a prognostic marker in oropharyngeal squamous cell carcinoma treated with intensity-modulated radiation therapy. Head Neck 42, 2405–2413 (2020).

    PubMed  Google Scholar 

  42. Lin, C. et al. Early 18F-FDG PET for prediction of prognosis in patients with diffuse large B-cell lymphoma: SUV-based assessment versus visual analysis. J. Nucl. Med. 48, 1626–1632 (2007).

    PubMed  Google Scholar 

  43. Prasanna, P., Tiwari, P. & Madabhushi, A. Co-occurrence of local anisotropic gradient orientations (CoLlAGe): a new radiomics descriptor. Sci. Rep. 6, 37241 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Haralick, R. M., Shanmugam, K. & Dinstein, I. Textural features for image classification. IEEE Trans. Syst. Man Cybern. SMC-3, 610–621 (1973).

    Google Scholar 

  45. Laws, K. I. Rapid texture identification. SPIE Proc. https://doi.org/10.1117/12.959169 (1980).

    Article  Google Scholar 

  46. Eisenhauer, E. A. et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45, 228–247 (2009).

    CAS  PubMed  Google Scholar 

  47. Kuhl, C. K. et al. Validity of RECIST version 1.1 for response assessment in metastatic cancer: a prospective, multireader study. Radiology 290, 349–356 (2018).

    PubMed  Google Scholar 

  48. Nishino, M. Tumor response assessment for precision cancer therapy: response evaluation criteria in solid tumors and beyond. Am. Soc. Clin. Oncol. Educ. Book 38, 1019–1029 (2018).

    PubMed  Google Scholar 

  49. Hylton, N. M. et al. Locally advanced breast cancer: MR imaging for prediction of response to neoadjuvant chemotherapy–results from ACRIN 6657/I-SPY TRIAL. Radiology 263, 663–672 (2012).

    PubMed  PubMed Central  Google Scholar 

  50. Xiao, J. et al. Tumor volume reduction rate is superior to RECIST for predicting the pathological response of rectal cancer treated with neoadjuvant chemoradiation: results from a prospective study. Oncol. Lett. 9, 2680–2686 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  51. Decazes, P. et al. Tumor fragmentation estimated by the volume surface ratio of tumors measured on FDG PET/CT is an independent prognostic factor of diffuse large B-cell lymphoma. J. Nucl. Med. 59, 1416–1416 (2018).

    Google Scholar 

  52. Jang, K., Russo, C. & Di Ieva, A. Radiomics in gliomas: clinical implications of computational modeling and fractal-based analysis. Neuroradiology 62, 771–790 (2020).

    PubMed  Google Scholar 

  53. Ismail, M. et al. Shape features of the lesion habitat to differentiate brain tumor progression from pseudoprogression on routine multiparametric MRI: a multisite study. Am. J. Neuroradiol. 39, 2187–2193 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  54. Ghose, S. et al. Prostate shapes on pre-treatment MRI between prostate cancer patients who do and do not undergo biochemical recurrence are different: preliminary findings. Sci. Rep. 7, 1–8 (2017).

    CAS  Google Scholar 

  55. Grove, O. et al. Quantitative computed tomographic descriptors associate tumor shape complexity and intratumor heterogeneity with prognosis in lung adenocarcinoma. PLoS ONE 10, e0118261 (2015).

    PubMed  PubMed Central  Google Scholar 

  56. Prasanna, P. et al. Mass effect deformation heterogeneity (MEDH) on gadolinium-contrast T1-weighted MRI is associated with decreased survival in patients with right cerebral hemisphere glioblastoma: a feasibility study. Sci. Rep. 9, 1–13 (2019).

    Google Scholar 

  57. Antunes, J. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 11767 (eds Shen, D. et al.) 611–619 (Springer, 2019).

  58. Braman, N. M. et al. Intratumoral and peritumoral radiomics for the pretreatment prediction of pathological complete response to neoadjuvant chemotherapy based on breast DCE-MRI. Breast Cancer Res. 19, 57 (2017).

    PubMed  PubMed Central  Google Scholar 

  59. Braman, N. et al. Association of peritumoral radiomics with tumor biology and pathologic response to preoperative targeted therapy for HER2 (ERBB2)–positive breast cancer. JAMA Netw. Open. 2, e192561 (2019).

    PubMed  PubMed Central  Google Scholar 

  60. Jones, E. F. et al. MRI enhancement in stromal tissue surrounding breast tumors: association with recurrence free survival following neoadjuvant chemotherapy. PLoS ONE 8, e61969 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  61. Khorrami, M. et al. Combination of peri- and intratumoral radiomic features on baseline CT scans predicts response to chemotherapy in lung adenocarcinoma. Radiol. Artif. Intell. 1, 180012 (2019).

    PubMed Central  Google Scholar 

  62. Khorrami, M. et al. Predicting pathologic response to neoadjuvant chemoradiation in resectable stage III non-small cell lung cancer patients using computed tomography radiomic features. Lung Cancer 135, 1–9 (2019).

    PubMed  Google Scholar 

  63. Khorrami, M. et al. Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non–small cell lung cancer. Cancer Immunol. Res. 8, 108–119 (2020).

    CAS  PubMed  Google Scholar 

  64. Vaidya, P. et al. CT derived radiomic score for predicting the added benefit of adjuvant chemotherapy following surgery in stage I, II resectable non-small cell lung cancer: a retrospective multicohort study for outcome prediction. Lancet Digital Health 2, e116–e128 (2020).

    PubMed  Google Scholar 

  65. Vaidya, P. et al. Novel, non-invasive imaging approach to identify patients with advanced non-small cell lung cancer at risk of hyperprogressive disease with immune checkpoint blockade. J. Immunother. Cancer 8, e001343 (2020).

    PubMed  PubMed Central  Google Scholar 

  66. Akinci D’Antonoli, T. et al. CT radiomics signature of tumor and peritumoral lung parenchyma to predict nonsmall cell lung cancer postsurgical recurrence risk. Acad. Radiol. 27, 497–507 (2020).

    PubMed  Google Scholar 

  67. Beig, N. et al. Radiogenomic-based survival risk stratification of tumor habitat on Gd-T1w MRI is associated with biological processes in glioblastoma. Clin. Cancer Res. 26, 1866–1876 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  68. Prasanna, P., Patel, J., Partovi, S., Madabhushi, A. & Tiwari, P. Radiomic features from the peritumoral brain parenchyma on treatment-naïve multi-parametric MR imaging predict long versus short-term survival in glioblastoma multiforme: preliminary findings. Eur. Radiol. 27, 4188–4197 (2017).

    PubMed  Google Scholar 

  69. Hu, Y. et al. Assessment of intratumoral and peritumoral computed tomography radiomics for predicting pathological complete response to neoadjuvant chemoradiation in patients with esophageal squamous cell carcinoma. JAMA Netw. Open 3, e2015927 (2020).

    PubMed  PubMed Central  Google Scholar 

  70. Li, J. et al. Intratumoral and peritumoral radiomics of contrast-enhanced CT for prediction of disease-free survival and chemotherapy response in stage II/III gastric cancer. Front. Oncol. 10, 552270 (2020).

    PubMed  PubMed Central  Google Scholar 

  71. Jiang, Y. et al. Noninvasive imaging evaluation of tumor immune microenvironment to predict outcomes in gastric cancer. Ann. Oncol. 31, 760–768 (2020).

    CAS  PubMed  Google Scholar 

  72. Algohary, A. et al. Combination of peri-tumoral and intra-tumoral radiomic features on Bi-Parametric MRI accurately stratifies prostate cancer risk: a multi-site study. Cancers 12, 2200 (2020).

    PubMed Central  Google Scholar 

  73. Keek, S. et al. Computed tomography-derived radiomic signature of head and neck squamous cell carcinoma (peri)tumoral tissue for the prediction of locoregional recurrence and distant metastasis after concurrent chemo-radiotherapy. PLoS ONE 15, e0232639 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  74. Shan, Q. et al. CT-based peritumoral radiomics signatures to predict early recurrence in hepatocellular carcinoma after curative tumor resection or ablation. Cancer Imaging 19, 11 (2019).

    PubMed  PubMed Central  Google Scholar 

  75. Ding, J. et al. Optimizing the peritumoral region size in radiomics analysis for sentinel lymph node status prediction in breast cancer. Acad. Radiol. https://doi.org/10.1016/j.acra.2020.10.015 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  76. Dou, T. H., Coroller, T. P., Griethuysen, J. J. M. V., Mak, R. H. & Aerts, H. J. W. L. Peritumoral radiomics features predict distant metastasis in locally advanced NSCLC. PLoS ONE 13, e0206108 (2018).

    PubMed  PubMed Central  Google Scholar 

  77. Chen, S. et al. Pretreatment prediction of immunoscore in hepatocellular cancer: a radiomics-based clinical model based on Gd-EOB-DTPA-enhanced MRI imaging. Eur. Radiol. 29, 4177–4187 (2019).

    PubMed  Google Scholar 

  78. Braman, N., Prasanna, P., Alilou, M., Beig, N. & Madabhushi, A. in Medical Image Computing and Computer Assisted Intervention Vol. 11071 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola- López, C.& Fichtinger, G.) 803–811 (Springer, 2018).

  79. Bullitt, E. et al. Blood vessel morphologic changes depicted with MR angiography during treatment of brain metastases: a feasibility study. Radiology 245, 824–830 (2007).

    PubMed  Google Scholar 

  80. Cheplygina, V., Bruijne, M. D. & Pluim, J. P. W. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Med. Image Anal. 54, 280–296 (2019).

    PubMed  Google Scholar 

  81. Chartrand, G. et al. Deep learning: a primer for radiologists. RadioGraphics 37, 2113–2131 (2017).

    PubMed  Google Scholar 

  82. Miotto, R., Wang, F., Wang, S., Jiang, X. & Dudley, J. T. Deep learning for healthcare: review, opportunities and challenges. Brief. Bioinform. 19, 1236–1246 (2018).

    PubMed  Google Scholar 

  83. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Google Scholar 

  84. Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 15, e1002686 (2017).

    Google Scholar 

  85. Ardila, D. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25, 954–961 (2019).

    CAS  PubMed  Google Scholar 

  86. Wu, N. et al. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39, 1184–1194 (2020).

    PubMed  Google Scholar 

  87. Zhou, T. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 12262 (eds Martel A. L. et al.) 221–231 (Springer, 2020).

  88. Braman, N. et al. Deep learning-based prediction of response to HER2-targeted neoadjuvant chemotherapy from pre-treatment dynamic breast MRI: a multi-institutional validation study. Preprint at arXiv https://arxiv.org/abs/2001.08570 (2020).

  89. Shelhamer, E., Long, J. & Darrell, T. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 640–651 (2017).

    PubMed  Google Scholar 

  90. Zhou, Z.-H. A brief introduction to weakly supervised learning. Natl Sci. Rev. 5, 44–53 (2018).

    Google Scholar 

  91. Huang, Y. et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non — small cell lung cancer. Radiology 281, 947–957 (2016).

    PubMed  Google Scholar 

  92. Kamran, S. C. et al. The impact of quantitative CT-based tumor volumetric features on the outcomes of patients with limited stage small cell lung cancer. Radiat. Oncol. 15, 14 (2020).

    PubMed  PubMed Central  Google Scholar 

  93. Pavic, M. et al. FDG PET versus CT radiomics to predict outcome in malignant pleural mesothelioma patients. EJNMMI Res. 10, 81 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  94. Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018).

    PubMed  PubMed Central  Google Scholar 

  95. Park, H. et al. Radiomics signature on magnetic resonance imaging: association with disease-free survival in patients with invasive breast cancer. Clin. Cancer Res. 24, 4705–4714 (2018).

    PubMed  Google Scholar 

  96. Wu, J. et al. Intratumoral spatial heterogeneity at perfusion MR imaging predicts recurrence-free survival in locally advanced breast cancer treated with neoadjuvant chemotherapy. Radiology 288, 26–35 (2018).

    PubMed  Google Scholar 

  97. Yu, Y. et al. Development and validation of a preoperative magnetic resonance imaging radiomics-based signature to predict axillary lymph node metastasis and disease-free survival in patients with early-stage breast cancer. JAMA Netw. Open 3, e2028086 (2020).

    PubMed  PubMed Central  Google Scholar 

  98. Chitalia, R. D. et al. Imaging phenotypes of breast cancer heterogeneity in preoperative breast dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) scans predict 10-year recurrence. Clin. Cancer Res. 26, 862–869 (2020).

    PubMed  Google Scholar 

  99. Drukker, K., Edwards, A., Papaioannou, J. & Giger, M. Deep learning predicts breast cancer recurrence in analysis of consecutive MRIs acquired during the course of neoadjuvant chemotherapy. Proc. SPIE 11314, 1131410 (2020).

  100. Kickingereder, P. et al. Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology 280, 880–889 (2016).

    PubMed  Google Scholar 

  101. Kickingereder, P. et al. Automated quantitative tumour response assessment of MRI in neuro-oncology with artificial neural networks: a multicentre, retrospective study. Lancet Oncol. 20, 728–740 (2019).

    PubMed  Google Scholar 

  102. Shiradkar, R. et al. Radiomic features from pretreatment biparametric MRI predict prostate cancer biochemical recurrence: preliminary findings. J. Magn. Reson. Imaging 48, 1626–1636 (2018).

    PubMed  PubMed Central  Google Scholar 

  103. Zhang, Y.-D. et al. An imaging-based approach predicts clinical outcomes in prostate cancer through a novel support vector machine classification. Oncotarget 7, 78140–78151 (2016).

    PubMed  PubMed Central  Google Scholar 

  104. Zhong, X. et al. Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI. Abdom. Radiol. 44, 2030–2039 (2019).

    Google Scholar 

  105. Wang, S. et al. Deep learning provides a new computed tomography-based prognostic biomarker for recurrence prediction in high-grade serous ovarian cancer. Radiother. Oncol. 132, 171–177 (2019).

    PubMed  Google Scholar 

  106. Parmar, C. et al. Radiomic feature clusters and Prognostic Signatures specific for Lung and Head & Neck cancer. Sci. Rep. 5, 11044 (2015).

    PubMed  PubMed Central  Google Scholar 

  107. Zheng, B.-H. et al. Radiomics score: a potential prognostic imaging feature for postoperative survival of solitary HCC patients. BMC Cancer 18, 1148 (2018).

    PubMed  PubMed Central  Google Scholar 

  108. Negreros-Osuna, A. A. et al. Radiomics texture features in advanced colorectal cancer: correlation with BRAF mutation and 5-year overall survival. Radiology 2, e190084 (2020).

    PubMed  PubMed Central  Google Scholar 

  109. Creasy, J. M. et al. Differences in liver parenchyma are measurable with CT radiomics at initial colon resection in patients that develop hepatic metastases from stage II/III colon cancer. Ann. Surg. Oncol. 28, 1982–1989 (2021).

    PubMed  Google Scholar 

  110. Langley, R. R. & Fidler, I. J. The seed and soil hypothesis revisited–the role of tumor–stroma interactions in metastasis to different organs. Int. J. Cancer 128, 2527–2535 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  111. Peng, H. et al. Prognostic value of deep learning PET/CT-based radiomics: potential role for future individual induction chemotherapy in advanced nasopharyngeal carcinoma. Clin. Cancer Res. 25, 4271–4279 (2019).

    CAS  PubMed  Google Scholar 

  112. Zhang, Y. et al. Improving prognostic performance in resectable pancreatic ductal adenocarcinoma using radiomics and deep learning features fusion in CT images. Sci. Rep. 11, 1378 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  113. Coroller, T. P. et al. Radiomic-based pathological response prediction from primary tumors and lymph nodes in NSCLC. J. Thorac. Oncol. 12, 467–476 (2017).

    PubMed  Google Scholar 

  114. Wei, H. et al. Application of computed tomography‑based radiomics signature analysis in the prediction of the response of small cell lung cancer patients to first‑line chemotherapy. Exp. Ther. Med. 17, 3621–3629 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  115. Xu, Y. et al. Deep learning predicts lung cancer treatment response from serial medical imaging. Clin. Cancer Res. 25, 3266–3275 (2019).

    PubMed  PubMed Central  Google Scholar 

  116. Granzier, R. W. Y., van Nijnatten, T. J. A., Woodruff, H. C., Smidt, M. L. & Lobbes, M. B. I. Exploring breast cancer response prediction to neoadjuvant systemic therapy using MRI-based radiomics: a systematic review. Eur. J. Radiol. 121, 108736 (2019).

    CAS  PubMed  Google Scholar 

  117. Liu, Z. et al. Radiomics of multi-parametric MRI for pretreatment prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer: a multicenter study. Clin. Cancer Res. 25, 3538–3547 (2019).

    CAS  PubMed  Google Scholar 

  118. Mazurowski, M. A. et al. Association of distant recurrence-free survival with algorithmically extracted MRI characteristics in breast cancer. J. Med. Reson. Imaging 49, e231–e240 (2019).

    Google Scholar 

  119. Cain, E. H. et al. Multivariate machine learning models for prediction of pathologic response to neoadjuvant therapy in breast cancer using MRI features: a study using an independent validation set. Breast Cancer Res. Treat. 173, 455–463 (2019).

    CAS  PubMed  Google Scholar 

  120. Tadayyon, H. et al. A priori prediction of breast tumour response to chemotherapy using quantitative ultrasound imaging and artificial neural networks. Oncotarget 10, 3910–3923 (2019).

    PubMed  PubMed Central  Google Scholar 

  121. Ha, R. et al. Prior to initiation of chemotherapy, can we predict breast tumor response? Deep learning convolutional neural networks approach using a breast MRI tumor dataset. J. Digit. Imaging 32, 693–701 (2019).

    PubMed  Google Scholar 

  122. Houssami, N., Macaskill, P., von Minckwitz, G., Marinovich, M. L. & Mamounas, E. Meta-analysis of the association of breast cancer subtype and pathologic complete response to neoadjuvant chemotherapy. Eur. J. Cancer 48, 3342–3354 (2012).

    CAS  PubMed  Google Scholar 

  123. Nie, K. et al. Rectal cancer: assessment of neoadjuvant chemoradiation outcome based on radiomics of multiparametric MRI. Clin. Cancer Res. 22, 5256–5264 (2016).

    PubMed  Google Scholar 

  124. Antunes, J. T. et al. Radiomic features of primary rectal cancers on baseline T2-weighted MRI are associated with pathologic complete response to neoadjuvant chemoradiation: a multisite study. J. Magn. Reson. Imaging 52, 1531–1541 (2020).

    PubMed  PubMed Central  Google Scholar 

  125. Cha, K. H. et al. Bladder cancer treatment response assessment in CT using radiomics with deep-learning. Sci. Rep. 7, 1–12 (2017).

    Google Scholar 

  126. Fang, M. et al. Multi-habitat based radiomics for the prediction of treatment response to concurrent chemotherapy and radiation therapy in locally advanced cervical cancer. Front. Oncol. 10, 563 (2020).

    PubMed  PubMed Central  Google Scholar 

  127. Jiang, Y. et al. Development and validation of a deep learning CT signature to predict survival and chemotherapy benefit in gastric cancer: a multicenter, retrospective study. Ann. Surg. https://doi.org/10.1097/SLA.0000000000003778 (2020).

    Article  PubMed  Google Scholar 

  128. Mehta, S. et al. Radiogenomics monitoring in breast cancer identifies metabolism and immune checkpoints as early actionable mechanisms of resistance to anti-angiogenic treatment. EBioMedicine 10, 109–116 (2016).

    PubMed  PubMed Central  Google Scholar 

  129. Kunte, S. et al. Radiomics risk score (RRS) on CT to predict survival and response to CDK 4/6 inhibitors in hormone receptor (HR) positive metastatic breast cancer (MBC). J. Clin. Oncol. 38, e13041–e13041 (2020).

    Google Scholar 

  130. Aerts, H. J. W. L. et al. Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC. Sci. Rep. 6, 33860 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  131. Sun, R. et al. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol. 19, 1180–1191 (2018).

    CAS  PubMed  Google Scholar 

  132. Trebeschi, S. et al. Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers. Ann. Oncol. 30, 998–1004 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  133. Yang, J. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 12262 (eds Martel, A. L. et al.) 211–220 (2020).

  134. Tunali, I. et al. Novel clinical and radiomic predictors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report. Lung Cancer 129, 75–79 (2019).

    PubMed  Google Scholar 

  135. Wu, J. et al. Unsupervised Clustering of quantitative image phenotypes reveals breast cancer subtypes with distinct prognoses and molecular pathways. Clin. Cancer Res. 23, 3334–3342 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  136. Wu, J. et al. Magnetic resonance imaging and molecular features associated with tumor-infiltrating lymphocytes in breast cancer. Breast Cancer Res. 20, 101 (2018).

    PubMed  PubMed Central  Google Scholar 

  137. Loi, S. et al. Tumor-infiltrating lymphocytes and prognosis: a pooled individual patient analysis of early-stage triple-negative breast cancers. J. Clin. Oncol. 37, 559–569 (2019).

    PubMed  PubMed Central  Google Scholar 

  138. Rao, A. et al. A combinatorial radiographic phenotype may stratify patient survival and be associated with invasion and proliferation characteristics in glioblastoma. J. Neurosurg. 124, 1008–1017 (2016).

    CAS  PubMed  Google Scholar 

  139. Wang, S. et al. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur. Respir. J. 53, 1800986 (2019).

    PubMed  PubMed Central  Google Scholar 

  140. Mu, W. et al. Non-invasive decision support for NSCLC treatment using PET/CT radiomics. Nat. Commun. 11, 5228 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  141. Yang, L. et al. Can CT-based radiomics signature predict KRAS/NRAS/BRAF mutations in colorectal cancer? Eur. Radiol. 28, 2058–2067 (2018).

    PubMed  Google Scholar 

  142. Golia Pernicka, J. S. et al. Radiomic-based prediction of microsatellite instability in colorectal cancer at initial computed tomography evaluation. Abdom. Radiol. 44, 3755–3763 (2019).

    Google Scholar 

  143. Liu, S. et al. CT textural analysis of gastric cancer: correlations with immunohistochemical biomarkers. Sci. Rep. 8, 11844 (2018).

    PubMed  PubMed Central  Google Scholar 

  144. Park, S. H. & Han, K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286, 800–809 (2018).

    PubMed  Google Scholar 

  145. Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013).

    PubMed  PubMed Central  Google Scholar 

  146. Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 12598 (2020).

    PubMed  PubMed Central  Google Scholar 

  147. Traverso, A., Wee, L., Dekker, A. & Gillies, R. Repeatability and reproducibility of radiomic features: a systematic review. Int. J. Radiat. Oncol. Biol. Phys. 102, 1143–1158 (2018).

    PubMed  PubMed Central  Google Scholar 

  148. Lambin, P. et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol. 14, 749–762 (2017).

    PubMed  Google Scholar 

  149. Park, J. E. et al. Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement. Eur. Radiol. 30, 523–536 (2020).

    PubMed  Google Scholar 

  150. Khorrami, M. et al. Stable and discriminating radiomic predictor of recurrence in early stage non-small cell lung cancer: multi-site study. Lung Cancer 142, 90–97 (2020).

    PubMed  Google Scholar 

  151. Johnson, W. E., Li, C. & Rabinovic, A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8, 118–127 (2007).

    PubMed  Google Scholar 

  152. Orlhac, F., Frouin, F., Nioche, C., Ayache, N. & Buvat, I. Validation of a method to compensate multicenter effects affecting CT radiomics. Radiology 291, 53–59 (2019).

    PubMed  Google Scholar 

  153. Kumar, V. et al. Radiomics: the process and the challenges. Magn. Reson. Imaging 30, 1234–1248 (2012).

    PubMed  PubMed Central  Google Scholar 

  154. Zwanenburg, A. et al. The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 295, 328–338 (2020).

    PubMed  Google Scholar 

  155. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206 (2019).

    Google Scholar 

  156. London, A. J. Artificial intelligence and black-box medical decisions: accuracy versus explainability. Hastings Cent. Rep. 49, 15–21 (2019).

    PubMed  Google Scholar 

  157. Holzinger, A., Langs, G., Denk, H., Zatloukal, K. & Müller, H. Causability and explainability of artificial intelligence in medicine. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 9, e1312 (2019).

    PubMed  PubMed Central  Google Scholar 

  158. US Food and Drug Administration. MammaPrint 510(k) premarket notification. FDA https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=k070675 (2007).

  159. US Food and Drug Administration. FDA releases artificial intelligence/machine learning action plan. FDA https://www.fda.gov/news-events/press-announcements/fda-releases-artificial-intelligencemachine-learning-action-plan (2021).

  160. Institute of Medicine. Policy Issues in the Development of Personalized Medicine in Oncology: Workshop Summary (National Academies, 2010).

  161. US Food and Drug Administration. Discussion paper on laboratory developed tests (LDTs) (FDA, 2017).

  162. Nakasu, S., Onishi, T., Kitahara, S., Oowaki, H. & Matsumura, K. CT Hounsfield unit is a good predictor of growth in meningiomas. Neurol. Med. Chir. 59, 54–62 (2019).

    Google Scholar 

  163. Urata, M. et al. Computed tomography Hounsfield units can predict breast cancer metastasis to axillary lymph nodes. BMC Cancer 14, 54 (2014).

    Google Scholar 

  164. Galloway, M. M. Texture analysis using gray level run lengths. Comput. Graph. Image Process. 4, 172–179 (1975).

    Google Scholar 

  165. Wang, L. & He, D.-C. Texture classification using texture spectrum. Pattern Recognit. 23, 905–910 (1990).

    Google Scholar 

  166. Fogel, I. & Sagi, D. Gabor filters as texture discriminator. Biol. Cybern. 61, 103–113 (1989).

    Google Scholar 

  167. Chen, S. S., Keller, J. M. & Crownover, R. M. On the calculation of fractal features from images. IEEE Trans. Pattern Anal. Mach. Intell. 15, 1087–1090 (1993).

    Google Scholar 

  168. Kontos, D. et al. Radiomic phenotypes of mammographic parenchymal complexity: toward augmenting breast density in breast cancer risk assessment. Radiology 290, 41–49 (2019).

    PubMed  Google Scholar 

  169. Yang, J. et al. Integrating tumor and nodal radiomics to predict lymph node metastasis in gastric cancer. Radiother. Oncol. 150, 89–96 (2020).

    PubMed  Google Scholar 

  170. Bullitt, E. et al. Abnormal vessel tortuosity as a marker of treatment response of malignant gliomas: preliminary report. Technol. Cancer Res. Treat. 3, 577–584 (2004).

    PubMed  Google Scholar 

  171. Alilou, M. et al. Quantitative vessel tortuosity: a potential CT imaging biomarker for distinguishing lung granulomas from adenocarcinomas. Sci. Rep. 8, 1–16 (2018).

    CAS  Google Scholar 

  172. Wu, C., Pineda, F., Hormuth, D. A., Karczmar, G. S. & Yankeelov, T. E. Quantitative analysis of vascular properties derived from ultrafast DCE-MRI to discriminate malignant and benign breast tumors. Magn. Reson. Med. 81, 2147–2160 (2019).

    PubMed  Google Scholar 

Download references

Acknowledgements

Research reported in this publication was supported by the Clinical and Translational Science Collaborative of Cleveland (UL1TR0002548) from the National Center for Advancing Translational Sciences (NCATS) component of the NIH and NIH roadmap for Medical Research; the Kidney Precision Medicine Project (KPMP) Glue Grant; the National Cancer Institute (award numbers 1F31CA221383-01A1, 1U24CA199374-01, R01CA249992-01A1, R01CA202752-01A1, R01CA208236-01A1, R01CA216579-01A1, R01CA220581-01A1, R01CA257612-01A1, 1U01CA239055-01, 1U01CA248226-01 and 1U54CA254566-01); the National Center for Research Resources (award number 1 C06 RR12463-01); the National Heart, Lung and Blood Institute (1R01HL15127701A1 and R01HL15807101A1); the National Institute of Biomedical Imaging and Bioengineering (1R43EB028736-01 and T32EB007509); the Office of the Assistant Secretary of Defense for Health Affairs, through the Breast Cancer Research Program (W81XWH-19-1-0668), the Lung Cancer Research Program (W81XWH-18-1-0440, W81XWH-20-1-0595), the Peer Reviewed Cancer Research Program (W81XWH-18-1-0404, W81XWH-21-1-0345) and the Prostate Cancer Research Program (W81XWH-15-1-0558, W81XWH-20-1-0851); the Ohio Third Frontier Technology Validation Fund; the VA Merit Review Award IBX004121A from the United States Department of Veterans Affairs Biomedical Laboratory Research and Development Service; and The Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering at Case Western Reserve University; and through sponsored research agreements from AstraZeneca, Boehringer Ingelheim and Bristol Myers Squibb. The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the institutions named.

Author information

Authors and Affiliations

Authors

Contributions

K.B., N.B. and A.M. researched data for this manuscript. All authors contributed to all other aspects of preparation of this manuscript.

Corresponding author

Correspondence to Anant Madabhushi.

Ethics declarations

Competing interests

N.B. is a current employee of Tempus Labs and a former employee of IBM Research, with both of which he is an inventor on several pending patents pertaining to medical image analysis. He additionally holds equity in Tempus Labs. V.V. is a consultant for Alkermes, AstraZeneca, Bristol Myers Squibb, Celgene, Foundation Medicine, Genentech, Merck, Nektar Therapeutics and Takeda, has current or pending grants from Alkermes, AstraZeneca, Bristol Myers Squibb, Genentech and Merck, is on the speakers’ bureaus of Bristol Myers Squibb, Celgene, Foundation Medicine and Novartis, and has received payment for the development of educational presentations from Bristol Myers Squibb and Foundation Medicine. A.M. holds equity in Elucid Bioimaging and Inspirata, has been or is a scientific advisory board member for Aiforia, AstraZeneca, Bristol Myers Squibb, Inspirata and Merck, serves as a consultant for Caris, Inc. and Roche Diagnostics, has sponsored research agreements with AstraZeneca, Boehringer Ingelheim, Bristol Myers Squibb and Philips, has developed a technology relating to cardiovascular imaging that has been licensed to Elucid Bioimaging, and is involved in an NIH U24 grant with PathCore and three different NIH R01 grants with Inspirata. The other authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Clinical Oncology thanks D. Kontos, R. Mak and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

Genomic Data Commons Data Portal: https://portal.gdc.cancer.gov/

The Cancer Imaging Archive: https://www.cancerimagingarchive.net/

Supplementary information

Glossary

Cross-validation

A method of analysing model validity without an independent validation set on a limited data sample by dividing the training data into subsets for training and assessing the performance on the complementary subset of data. Several methods of cross-validation include holdout, k-fold or leave-one-out.

Elastic net survival model

Type of Cox proportional hazard model that is used to calculate hazard ratios, which are a way of evaluating the strength of the association of a variable (for example, survival outcomes) with a time point. An elastic net has the added advantage over a standard Cox model of adjusting for high dimensional data and covariates that might be correlated with each other, while making survival estimations.

Grey-level co-occurrence matrix features

Class of commonly used radiomic features, also known as Haralick features, which rely on higher-order statistics to describe the spatial arrangement and apparent position of the different grey levels present throughout the analysed image.

Kurtosis

Statistical measure to indicate the shape of a probability distribution in terms of its ‘tailedness’. High kurtosis means high deviation from the mean.

Laws’ energy measures

Eponymously named after K. I. Laws, this radiomic feature focuses on measuring variations of energy within a fixed window size, to calculate a combined texture energy of the pixels analysed.

Long short-term memory

Type of recurrent neural network that has been supplemented by the addition of recurrent or ‘forget’ gates, which enables the network to learn by looking back at propagated errors.

Skewness

Statistical measure to indicate the apparent distance between the mean and mode of a distribution. Skewness = (mean – mode)/standard deviation.

Support vector machine

Supervised machine learning model used to classify data by constructing hyperplanes and choosing the hyperplane that has the largest separation between the two classes of interest.

Tumour-infiltrating lymphocytes

(TILs). Lymphocytes that have invaded the tumour tissue from the bloodstream. In the past few years, studies have found TILs to be prognostic of survival and predictive of treatment benefit in several solid tumour types, including breast and lung tumours.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bera, K., Braman, N., Gupta, A. et al. Predicting cancer outcomes with radiomics and artificial intelligence in radiology. Nat Rev Clin Oncol 19, 132–146 (2022). https://doi.org/10.1038/s41571-021-00560-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41571-021-00560-7

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing