The successful use of artificial intelligence (AI) for diagnostic purposes has prompted the application of AI-based cancer imaging analysis to address other, more complex, clinical needs. In this Perspective, we discuss the next generation of challenges in clinical decision-making that AI tools can solve using radiology images, such as prognostication of outcome across multiple cancers, prediction of response to various treatment modalities, discrimination of benign treatment confounders from true progression, identification of unusual response patterns and prediction of the mutational and molecular profile of tumours. We describe the evolution of and opportunities for AI in oncology imaging, focusing on hand-crafted radiomic approaches and deep learning-derived representations, with examples of their application for decision support. We also address the challenges faced on the path to clinical adoption, including data curation and annotation, interpretability, and regulatory and reimbursement issues. We hope to demystify AI in radiology for clinicians by helping them to understand its limitations and challenges, as well as the opportunities it provides as a decision-support tool in cancer management.
This is a preview of subscription content
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Giger, M. L., Chan, H.-P. & Boone, J. Anniversary Paper: History and status of CAD and quantitative image analysis: the role of medical physics and AAPM. Med. Phys. 35, 5799–5820 (2008).
Giger, M. L., Doi, K. & MacMahon, H. Computerized detection of lung nodules in digital chest radiographs. Med. Imaging Proc. 767, 384–387 (1987).
Carmody, D. P., Nodine, C. F. & Kundel, H. L. An analysis of perceptual and cognitive factors in radiographic interpretation. Perception 9, 339–344 (1980).
Kundel, H. L. & Hendee, W. R. The perception of radiologic image information. Report of an NCI workshop on April 15-16, 1985. Invest. Radiol. 20, 874–877 (1985).
Rao, V. M. et al. How widely Is computer-aided detection used in screening and diagnostic mammography? J. Am. Coll. Radiol. 7, 802–805 (2010).
McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020).
Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).
Frelaut, M., Le Tourneau, C. & Borcoman, E. Hyperprogression under immunotherapy. Int. J. Mol. Sci. 20, 2674 (2019).
Frelaut, M., du Rusquec, P., de Moura, A., Le Tourneau, C. & Borcoman, E. Pseudoprogression and hyperprogression as new forms of response to immunotherapy. BioDrugs 34, 463–476 (2020).
Cruz, L. C. H., da, Rodriguez, I., Domingues, R. C., Gasparetto, E. L. & Sorensen, A. G. Pseudoprogression and pseudoresponse: imaging challenges in the assessment of posttreatment glioma. Am. J. Neuroradiol. 32, 1978–1985 (2011).
FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource (FDA, 2016).
Griethuysen, J. J. M. V. et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77, e104–e107 (2017).
Nicolini, A., Ferrari, P. & Duffy, M. J. Prognostic and predictive biomarkers in breast cancer: past, present and future. Semin. Cancer Biol. 52, 56–73 (2018).
Cucchiara, V. et al. Genomic markers in prostate cancer decision making. Eur. Urol. 73, 572–582 (2018).
LI, S. G. & LI, L. Targeted therapy in HER2-positive breast cancer. Biomed. Rep. 1, 499–505 (2013).
Chan, B. A. & Hughes, B. G. Targeted therapy for non-small cell lung cancer: current standards and the promise of the future. Transl. Lung Cancer Res. 4, 36 (2015).
Sparano, J. A. et al. Adjuvant chemotherapy guided by a 21-gene expression assay in breast cancer. N. Engl. J. Med. 379, 111–121 (2018).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Pfaehler, E., Zwanenburg, A., Jong, J. Rde & Boellaard, R. RaCaT: an open source and easy to use radiomics calculator tool. PLoS ONE 14, e0212223 (2019).
Verma, V. et al. The rise of radiomics and implications for oncologic management. J. Natl Cancer Inst. 109, djx055 (2017).
Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V. & Madabhushi, A. Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16, 703–715 (2019).
Wan, T. et al. A radio-genomics approach for identifying high risk estrogen receptor-positive breast cancers on DCE-MRI: preliminary results in predicting oncotypeDX risk scores. Sci. Rep. 6, 21394 (2016).
Li, H. et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of mammaprint, oncotype DX, and PAM50 gene assays. Radiology 281, 382–391 (2016).
Cyll, K. et al. Tumour heterogeneity poses a significant challenge to cancer biomarker research. Br. J. Cancer 117, 367–375 (2017).
Crowley, E., Di Nicolantonio, F., Loupakis, F. & Bardelli, A. Liquid biopsy: monitoring cancer-genetics in the blood. Nat. Rev. Clin. Oncol. 10, 472–484 (2013).
Lim, Z.-F. & Ma, P. C. Emerging insights of tumor heterogeneity and drug resistance mechanisms in lung cancer targeted therapy. J. Hematol. Oncol. 12, 134 (2019).
Mazurowski, M. A. Radiogenomics: what it is and why it is important. J. Am. Coll. Radiol. 12, 862–866 (2015).
Bodalal, Z., Trebeschi, S., Nguyen-Kim, T. D. L., Schats, W. & Beets-Tan, R. Radiogenomics: bridging imaging and genomics. Abdom. Radiol. 44, 1960–1984 (2019).
Eben, J., Braman, N. & Madabhushi, A. in Medical Image Computing and Computer Assisted Intervention Vol. 11767 (eds Shen, D. et al.) 602–610 (Springer, 2019).
Bizzego, A. et al. Integrating deep and radiomics features in cancer bioimaging. IEEE Conf. Comput. Intell. Bioinform. Comput. Biol. https://doi.org/10.1109/CIBCB.2019.8791473 (2019).
Lao, J. et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci. Rep. 7, 10353 (2017).
‘Student’. The probable error of a mean. Biometrika 6, 1–25 (1908).
Wilcoxon, F. Individual comparisons by ranking methods. Biometrics Bull. 1, 80–83 (1945).
Ding, C. & Peng, H. Minimum redundancy feature selection from microarray gene expression data. J. Bioinform. Comput. Biol. 3, 185–205 (2005).
Chirra, P. et al. Multisite evaluation of radiomic feature reproducibility and discriminability for identifying peripheral zone prostate tumors on MRI. J. Med. Imaging 6, 024502 (2019).
Lee, J. W. & et al. Prognostic significance of CT-attenuation of tumor-adjacent breast adipose tissue in breast cancer patients with surgical resection. Cancers 11, 1135 (2019).
Eguchi, T. et al. Tumor size and computed tomography attenuation of pulmonary pure ground-glass nodules are useful for predicting pathological invasiveness. PLoS ONE 9, e97867 (2014).
Kinahan, P. E. & Fletcher, J. W. PET/CT standardized uptake values (SUVs) in clinical practice and assessing response to therapy. Semin. Ultrasound CT MR 31, 496–505 (2010).
Kubota, K. From tumor biology to clinical PET: a review of positron emission tomography (PET) in oncology. Ann. Nucl. Med. 15, 471–486 (2001).
Sheikhbahaei, S. et al. The value of FDG PET/CT in treatment response assessment, follow-up, and surveillance of lung cancer. Am. J. Roentgenol. 208, 420–433 (2016).
Eckstein, J. M. et al. Primary vs nodal site PET/CT response as a prognostic marker in oropharyngeal squamous cell carcinoma treated with intensity-modulated radiation therapy. Head Neck 42, 2405–2413 (2020).
Lin, C. et al. Early 18F-FDG PET for prediction of prognosis in patients with diffuse large B-cell lymphoma: SUV-based assessment versus visual analysis. J. Nucl. Med. 48, 1626–1632 (2007).
Prasanna, P., Tiwari, P. & Madabhushi, A. Co-occurrence of local anisotropic gradient orientations (CoLlAGe): a new radiomics descriptor. Sci. Rep. 6, 37241 (2016).
Haralick, R. M., Shanmugam, K. & Dinstein, I. Textural features for image classification. IEEE Trans. Syst. Man Cybern. SMC-3, 610–621 (1973).
Laws, K. I. Rapid texture identification. SPIE Proc. https://doi.org/10.1117/12.959169 (1980).
Eisenhauer, E. A. et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45, 228–247 (2009).
Kuhl, C. K. et al. Validity of RECIST version 1.1 for response assessment in metastatic cancer: a prospective, multireader study. Radiology 290, 349–356 (2018).
Nishino, M. Tumor response assessment for precision cancer therapy: response evaluation criteria in solid tumors and beyond. Am. Soc. Clin. Oncol. Educ. Book 38, 1019–1029 (2018).
Hylton, N. M. et al. Locally advanced breast cancer: MR imaging for prediction of response to neoadjuvant chemotherapy–results from ACRIN 6657/I-SPY TRIAL. Radiology 263, 663–672 (2012).
Xiao, J. et al. Tumor volume reduction rate is superior to RECIST for predicting the pathological response of rectal cancer treated with neoadjuvant chemoradiation: results from a prospective study. Oncol. Lett. 9, 2680–2686 (2015).
Decazes, P. et al. Tumor fragmentation estimated by the volume surface ratio of tumors measured on FDG PET/CT is an independent prognostic factor of diffuse large B-cell lymphoma. J. Nucl. Med. 59, 1416–1416 (2018).
Jang, K., Russo, C. & Di Ieva, A. Radiomics in gliomas: clinical implications of computational modeling and fractal-based analysis. Neuroradiology 62, 771–790 (2020).
Ismail, M. et al. Shape features of the lesion habitat to differentiate brain tumor progression from pseudoprogression on routine multiparametric MRI: a multisite study. Am. J. Neuroradiol. 39, 2187–2193 (2018).
Ghose, S. et al. Prostate shapes on pre-treatment MRI between prostate cancer patients who do and do not undergo biochemical recurrence are different: preliminary findings. Sci. Rep. 7, 1–8 (2017).
Grove, O. et al. Quantitative computed tomographic descriptors associate tumor shape complexity and intratumor heterogeneity with prognosis in lung adenocarcinoma. PLoS ONE 10, e0118261 (2015).
Prasanna, P. et al. Mass effect deformation heterogeneity (MEDH) on gadolinium-contrast T1-weighted MRI is associated with decreased survival in patients with right cerebral hemisphere glioblastoma: a feasibility study. Sci. Rep. 9, 1–13 (2019).
Antunes, J. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 11767 (eds Shen, D. et al.) 611–619 (Springer, 2019).
Braman, N. M. et al. Intratumoral and peritumoral radiomics for the pretreatment prediction of pathological complete response to neoadjuvant chemotherapy based on breast DCE-MRI. Breast Cancer Res. 19, 57 (2017).
Braman, N. et al. Association of peritumoral radiomics with tumor biology and pathologic response to preoperative targeted therapy for HER2 (ERBB2)–positive breast cancer. JAMA Netw. Open. 2, e192561 (2019).
Jones, E. F. et al. MRI enhancement in stromal tissue surrounding breast tumors: association with recurrence free survival following neoadjuvant chemotherapy. PLoS ONE 8, e61969 (2013).
Khorrami, M. et al. Combination of peri- and intratumoral radiomic features on baseline CT scans predicts response to chemotherapy in lung adenocarcinoma. Radiol. Artif. Intell. 1, 180012 (2019).
Khorrami, M. et al. Predicting pathologic response to neoadjuvant chemoradiation in resectable stage III non-small cell lung cancer patients using computed tomography radiomic features. Lung Cancer 135, 1–9 (2019).
Khorrami, M. et al. Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non–small cell lung cancer. Cancer Immunol. Res. 8, 108–119 (2020).
Vaidya, P. et al. CT derived radiomic score for predicting the added benefit of adjuvant chemotherapy following surgery in stage I, II resectable non-small cell lung cancer: a retrospective multicohort study for outcome prediction. Lancet Digital Health 2, e116–e128 (2020).
Vaidya, P. et al. Novel, non-invasive imaging approach to identify patients with advanced non-small cell lung cancer at risk of hyperprogressive disease with immune checkpoint blockade. J. Immunother. Cancer 8, e001343 (2020).
Akinci D’Antonoli, T. et al. CT radiomics signature of tumor and peritumoral lung parenchyma to predict nonsmall cell lung cancer postsurgical recurrence risk. Acad. Radiol. 27, 497–507 (2020).
Beig, N. et al. Radiogenomic-based survival risk stratification of tumor habitat on Gd-T1w MRI is associated with biological processes in glioblastoma. Clin. Cancer Res. 26, 1866–1876 (2020).
Prasanna, P., Patel, J., Partovi, S., Madabhushi, A. & Tiwari, P. Radiomic features from the peritumoral brain parenchyma on treatment-naïve multi-parametric MR imaging predict long versus short-term survival in glioblastoma multiforme: preliminary findings. Eur. Radiol. 27, 4188–4197 (2017).
Hu, Y. et al. Assessment of intratumoral and peritumoral computed tomography radiomics for predicting pathological complete response to neoadjuvant chemoradiation in patients with esophageal squamous cell carcinoma. JAMA Netw. Open 3, e2015927 (2020).
Li, J. et al. Intratumoral and peritumoral radiomics of contrast-enhanced CT for prediction of disease-free survival and chemotherapy response in stage II/III gastric cancer. Front. Oncol. 10, 552270 (2020).
Jiang, Y. et al. Noninvasive imaging evaluation of tumor immune microenvironment to predict outcomes in gastric cancer. Ann. Oncol. 31, 760–768 (2020).
Algohary, A. et al. Combination of peri-tumoral and intra-tumoral radiomic features on Bi-Parametric MRI accurately stratifies prostate cancer risk: a multi-site study. Cancers 12, 2200 (2020).
Keek, S. et al. Computed tomography-derived radiomic signature of head and neck squamous cell carcinoma (peri)tumoral tissue for the prediction of locoregional recurrence and distant metastasis after concurrent chemo-radiotherapy. PLoS ONE 15, e0232639 (2020).
Shan, Q. et al. CT-based peritumoral radiomics signatures to predict early recurrence in hepatocellular carcinoma after curative tumor resection or ablation. Cancer Imaging 19, 11 (2019).
Ding, J. et al. Optimizing the peritumoral region size in radiomics analysis for sentinel lymph node status prediction in breast cancer. Acad. Radiol. https://doi.org/10.1016/j.acra.2020.10.015 (2020).
Dou, T. H., Coroller, T. P., Griethuysen, J. J. M. V., Mak, R. H. & Aerts, H. J. W. L. Peritumoral radiomics features predict distant metastasis in locally advanced NSCLC. PLoS ONE 13, e0206108 (2018).
Chen, S. et al. Pretreatment prediction of immunoscore in hepatocellular cancer: a radiomics-based clinical model based on Gd-EOB-DTPA-enhanced MRI imaging. Eur. Radiol. 29, 4177–4187 (2019).
Braman, N., Prasanna, P., Alilou, M., Beig, N. & Madabhushi, A. in Medical Image Computing and Computer Assisted Intervention Vol. 11071 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola- López, C.& Fichtinger, G.) 803–811 (Springer, 2018).
Bullitt, E. et al. Blood vessel morphologic changes depicted with MR angiography during treatment of brain metastases: a feasibility study. Radiology 245, 824–830 (2007).
Cheplygina, V., Bruijne, M. D. & Pluim, J. P. W. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Med. Image Anal. 54, 280–296 (2019).
Chartrand, G. et al. Deep learning: a primer for radiologists. RadioGraphics 37, 2113–2131 (2017).
Miotto, R., Wang, F., Wang, S., Jiang, X. & Dudley, J. T. Deep learning for healthcare: review, opportunities and challenges. Brief. Bioinform. 19, 1236–1246 (2018).
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).
Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 15, e1002686 (2017).
Ardila, D. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25, 954–961 (2019).
Wu, N. et al. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39, 1184–1194 (2020).
Zhou, T. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 12262 (eds Martel A. L. et al.) 221–231 (Springer, 2020).
Braman, N. et al. Deep learning-based prediction of response to HER2-targeted neoadjuvant chemotherapy from pre-treatment dynamic breast MRI: a multi-institutional validation study. Preprint at arXiv https://arxiv.org/abs/2001.08570 (2020).
Shelhamer, E., Long, J. & Darrell, T. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 640–651 (2017).
Zhou, Z.-H. A brief introduction to weakly supervised learning. Natl Sci. Rev. 5, 44–53 (2018).
Huang, Y. et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non — small cell lung cancer. Radiology 281, 947–957 (2016).
Kamran, S. C. et al. The impact of quantitative CT-based tumor volumetric features on the outcomes of patients with limited stage small cell lung cancer. Radiat. Oncol. 15, 14 (2020).
Pavic, M. et al. FDG PET versus CT radiomics to predict outcome in malignant pleural mesothelioma patients. EJNMMI Res. 10, 81 (2020).
Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018).
Park, H. et al. Radiomics signature on magnetic resonance imaging: association with disease-free survival in patients with invasive breast cancer. Clin. Cancer Res. 24, 4705–4714 (2018).
Wu, J. et al. Intratumoral spatial heterogeneity at perfusion MR imaging predicts recurrence-free survival in locally advanced breast cancer treated with neoadjuvant chemotherapy. Radiology 288, 26–35 (2018).
Yu, Y. et al. Development and validation of a preoperative magnetic resonance imaging radiomics-based signature to predict axillary lymph node metastasis and disease-free survival in patients with early-stage breast cancer. JAMA Netw. Open 3, e2028086 (2020).
Chitalia, R. D. et al. Imaging phenotypes of breast cancer heterogeneity in preoperative breast dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) scans predict 10-year recurrence. Clin. Cancer Res. 26, 862–869 (2020).
Drukker, K., Edwards, A., Papaioannou, J. & Giger, M. Deep learning predicts breast cancer recurrence in analysis of consecutive MRIs acquired during the course of neoadjuvant chemotherapy. Proc. SPIE 11314, 1131410 (2020).
Kickingereder, P. et al. Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology 280, 880–889 (2016).
Kickingereder, P. et al. Automated quantitative tumour response assessment of MRI in neuro-oncology with artificial neural networks: a multicentre, retrospective study. Lancet Oncol. 20, 728–740 (2019).
Shiradkar, R. et al. Radiomic features from pretreatment biparametric MRI predict prostate cancer biochemical recurrence: preliminary findings. J. Magn. Reson. Imaging 48, 1626–1636 (2018).
Zhang, Y.-D. et al. An imaging-based approach predicts clinical outcomes in prostate cancer through a novel support vector machine classification. Oncotarget 7, 78140–78151 (2016).
Zhong, X. et al. Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI. Abdom. Radiol. 44, 2030–2039 (2019).
Wang, S. et al. Deep learning provides a new computed tomography-based prognostic biomarker for recurrence prediction in high-grade serous ovarian cancer. Radiother. Oncol. 132, 171–177 (2019).
Parmar, C. et al. Radiomic feature clusters and Prognostic Signatures specific for Lung and Head & Neck cancer. Sci. Rep. 5, 11044 (2015).
Zheng, B.-H. et al. Radiomics score: a potential prognostic imaging feature for postoperative survival of solitary HCC patients. BMC Cancer 18, 1148 (2018).
Negreros-Osuna, A. A. et al. Radiomics texture features in advanced colorectal cancer: correlation with BRAF mutation and 5-year overall survival. Radiology 2, e190084 (2020).
Creasy, J. M. et al. Differences in liver parenchyma are measurable with CT radiomics at initial colon resection in patients that develop hepatic metastases from stage II/III colon cancer. Ann. Surg. Oncol. 28, 1982–1989 (2021).
Langley, R. R. & Fidler, I. J. The seed and soil hypothesis revisited–the role of tumor–stroma interactions in metastasis to different organs. Int. J. Cancer 128, 2527–2535 (2011).
Peng, H. et al. Prognostic value of deep learning PET/CT-based radiomics: potential role for future individual induction chemotherapy in advanced nasopharyngeal carcinoma. Clin. Cancer Res. 25, 4271–4279 (2019).
Zhang, Y. et al. Improving prognostic performance in resectable pancreatic ductal adenocarcinoma using radiomics and deep learning features fusion in CT images. Sci. Rep. 11, 1378 (2021).
Coroller, T. P. et al. Radiomic-based pathological response prediction from primary tumors and lymph nodes in NSCLC. J. Thorac. Oncol. 12, 467–476 (2017).
Wei, H. et al. Application of computed tomography‑based radiomics signature analysis in the prediction of the response of small cell lung cancer patients to first‑line chemotherapy. Exp. Ther. Med. 17, 3621–3629 (2019).
Xu, Y. et al. Deep learning predicts lung cancer treatment response from serial medical imaging. Clin. Cancer Res. 25, 3266–3275 (2019).
Granzier, R. W. Y., van Nijnatten, T. J. A., Woodruff, H. C., Smidt, M. L. & Lobbes, M. B. I. Exploring breast cancer response prediction to neoadjuvant systemic therapy using MRI-based radiomics: a systematic review. Eur. J. Radiol. 121, 108736 (2019).
Liu, Z. et al. Radiomics of multi-parametric MRI for pretreatment prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer: a multicenter study. Clin. Cancer Res. 25, 3538–3547 (2019).
Mazurowski, M. A. et al. Association of distant recurrence-free survival with algorithmically extracted MRI characteristics in breast cancer. J. Med. Reson. Imaging 49, e231–e240 (2019).
Cain, E. H. et al. Multivariate machine learning models for prediction of pathologic response to neoadjuvant therapy in breast cancer using MRI features: a study using an independent validation set. Breast Cancer Res. Treat. 173, 455–463 (2019).
Tadayyon, H. et al. A priori prediction of breast tumour response to chemotherapy using quantitative ultrasound imaging and artificial neural networks. Oncotarget 10, 3910–3923 (2019).
Ha, R. et al. Prior to initiation of chemotherapy, can we predict breast tumor response? Deep learning convolutional neural networks approach using a breast MRI tumor dataset. J. Digit. Imaging 32, 693–701 (2019).
Houssami, N., Macaskill, P., von Minckwitz, G., Marinovich, M. L. & Mamounas, E. Meta-analysis of the association of breast cancer subtype and pathologic complete response to neoadjuvant chemotherapy. Eur. J. Cancer 48, 3342–3354 (2012).
Nie, K. et al. Rectal cancer: assessment of neoadjuvant chemoradiation outcome based on radiomics of multiparametric MRI. Clin. Cancer Res. 22, 5256–5264 (2016).
Antunes, J. T. et al. Radiomic features of primary rectal cancers on baseline T2-weighted MRI are associated with pathologic complete response to neoadjuvant chemoradiation: a multisite study. J. Magn. Reson. Imaging 52, 1531–1541 (2020).
Cha, K. H. et al. Bladder cancer treatment response assessment in CT using radiomics with deep-learning. Sci. Rep. 7, 1–12 (2017).
Fang, M. et al. Multi-habitat based radiomics for the prediction of treatment response to concurrent chemotherapy and radiation therapy in locally advanced cervical cancer. Front. Oncol. 10, 563 (2020).
Jiang, Y. et al. Development and validation of a deep learning CT signature to predict survival and chemotherapy benefit in gastric cancer: a multicenter, retrospective study. Ann. Surg. https://doi.org/10.1097/SLA.0000000000003778 (2020).
Mehta, S. et al. Radiogenomics monitoring in breast cancer identifies metabolism and immune checkpoints as early actionable mechanisms of resistance to anti-angiogenic treatment. EBioMedicine 10, 109–116 (2016).
Kunte, S. et al. Radiomics risk score (RRS) on CT to predict survival and response to CDK 4/6 inhibitors in hormone receptor (HR) positive metastatic breast cancer (MBC). J. Clin. Oncol. 38, e13041–e13041 (2020).
Aerts, H. J. W. L. et al. Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC. Sci. Rep. 6, 33860 (2016).
Sun, R. et al. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol. 19, 1180–1191 (2018).
Trebeschi, S. et al. Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers. Ann. Oncol. 30, 998–1004 (2019).
Yang, J. et al. in Medical Image Computing and Computer Assisted Intervention Vol. 12262 (eds Martel, A. L. et al.) 211–220 (2020).
Tunali, I. et al. Novel clinical and radiomic predictors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report. Lung Cancer 129, 75–79 (2019).
Wu, J. et al. Unsupervised Clustering of quantitative image phenotypes reveals breast cancer subtypes with distinct prognoses and molecular pathways. Clin. Cancer Res. 23, 3334–3342 (2017).
Wu, J. et al. Magnetic resonance imaging and molecular features associated with tumor-infiltrating lymphocytes in breast cancer. Breast Cancer Res. 20, 101 (2018).
Loi, S. et al. Tumor-infiltrating lymphocytes and prognosis: a pooled individual patient analysis of early-stage triple-negative breast cancers. J. Clin. Oncol. 37, 559–569 (2019).
Rao, A. et al. A combinatorial radiographic phenotype may stratify patient survival and be associated with invasion and proliferation characteristics in glioblastoma. J. Neurosurg. 124, 1008–1017 (2016).
Wang, S. et al. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur. Respir. J. 53, 1800986 (2019).
Mu, W. et al. Non-invasive decision support for NSCLC treatment using PET/CT radiomics. Nat. Commun. 11, 5228 (2020).
Yang, L. et al. Can CT-based radiomics signature predict KRAS/NRAS/BRAF mutations in colorectal cancer? Eur. Radiol. 28, 2058–2067 (2018).
Golia Pernicka, J. S. et al. Radiomic-based prediction of microsatellite instability in colorectal cancer at initial computed tomography evaluation. Abdom. Radiol. 44, 3755–3763 (2019).
Liu, S. et al. CT textural analysis of gastric cancer: correlations with immunohistochemical biomarkers. Sci. Rep. 8, 11844 (2018).
Park, S. H. & Han, K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286, 800–809 (2018).
Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013).
Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 12598 (2020).
Traverso, A., Wee, L., Dekker, A. & Gillies, R. Repeatability and reproducibility of radiomic features: a systematic review. Int. J. Radiat. Oncol. Biol. Phys. 102, 1143–1158 (2018).
Lambin, P. et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol. 14, 749–762 (2017).
Park, J. E. et al. Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement. Eur. Radiol. 30, 523–536 (2020).
Khorrami, M. et al. Stable and discriminating radiomic predictor of recurrence in early stage non-small cell lung cancer: multi-site study. Lung Cancer 142, 90–97 (2020).
Johnson, W. E., Li, C. & Rabinovic, A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8, 118–127 (2007).
Orlhac, F., Frouin, F., Nioche, C., Ayache, N. & Buvat, I. Validation of a method to compensate multicenter effects affecting CT radiomics. Radiology 291, 53–59 (2019).
Kumar, V. et al. Radiomics: the process and the challenges. Magn. Reson. Imaging 30, 1234–1248 (2012).
Zwanenburg, A. et al. The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 295, 328–338 (2020).
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206 (2019).
London, A. J. Artificial intelligence and black-box medical decisions: accuracy versus explainability. Hastings Cent. Rep. 49, 15–21 (2019).
Holzinger, A., Langs, G., Denk, H., Zatloukal, K. & Müller, H. Causability and explainability of artificial intelligence in medicine. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 9, e1312 (2019).
US Food and Drug Administration. MammaPrint 510(k) premarket notification. FDA https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=k070675 (2007).
US Food and Drug Administration. FDA releases artificial intelligence/machine learning action plan. FDA https://www.fda.gov/news-events/press-announcements/fda-releases-artificial-intelligencemachine-learning-action-plan (2021).
Institute of Medicine. Policy Issues in the Development of Personalized Medicine in Oncology: Workshop Summary (National Academies, 2010).
US Food and Drug Administration. Discussion paper on laboratory developed tests (LDTs) (FDA, 2017).
Nakasu, S., Onishi, T., Kitahara, S., Oowaki, H. & Matsumura, K. CT Hounsfield unit is a good predictor of growth in meningiomas. Neurol. Med. Chir. 59, 54–62 (2019).
Urata, M. et al. Computed tomography Hounsfield units can predict breast cancer metastasis to axillary lymph nodes. BMC Cancer 14, 54 (2014).
Galloway, M. M. Texture analysis using gray level run lengths. Comput. Graph. Image Process. 4, 172–179 (1975).
Wang, L. & He, D.-C. Texture classification using texture spectrum. Pattern Recognit. 23, 905–910 (1990).
Fogel, I. & Sagi, D. Gabor filters as texture discriminator. Biol. Cybern. 61, 103–113 (1989).
Chen, S. S., Keller, J. M. & Crownover, R. M. On the calculation of fractal features from images. IEEE Trans. Pattern Anal. Mach. Intell. 15, 1087–1090 (1993).
Kontos, D. et al. Radiomic phenotypes of mammographic parenchymal complexity: toward augmenting breast density in breast cancer risk assessment. Radiology 290, 41–49 (2019).
Yang, J. et al. Integrating tumor and nodal radiomics to predict lymph node metastasis in gastric cancer. Radiother. Oncol. 150, 89–96 (2020).
Bullitt, E. et al. Abnormal vessel tortuosity as a marker of treatment response of malignant gliomas: preliminary report. Technol. Cancer Res. Treat. 3, 577–584 (2004).
Alilou, M. et al. Quantitative vessel tortuosity: a potential CT imaging biomarker for distinguishing lung granulomas from adenocarcinomas. Sci. Rep. 8, 1–16 (2018).
Wu, C., Pineda, F., Hormuth, D. A., Karczmar, G. S. & Yankeelov, T. E. Quantitative analysis of vascular properties derived from ultrafast DCE-MRI to discriminate malignant and benign breast tumors. Magn. Reson. Med. 81, 2147–2160 (2019).
Research reported in this publication was supported by the Clinical and Translational Science Collaborative of Cleveland (UL1TR0002548) from the National Center for Advancing Translational Sciences (NCATS) component of the NIH and NIH roadmap for Medical Research; the Kidney Precision Medicine Project (KPMP) Glue Grant; the National Cancer Institute (award numbers 1F31CA221383-01A1, 1U24CA199374-01, R01CA249992-01A1, R01CA202752-01A1, R01CA208236-01A1, R01CA216579-01A1, R01CA220581-01A1, R01CA257612-01A1, 1U01CA239055-01, 1U01CA248226-01 and 1U54CA254566-01); the National Center for Research Resources (award number 1 C06 RR12463-01); the National Heart, Lung and Blood Institute (1R01HL15127701A1 and R01HL15807101A1); the National Institute of Biomedical Imaging and Bioengineering (1R43EB028736-01 and T32EB007509); the Office of the Assistant Secretary of Defense for Health Affairs, through the Breast Cancer Research Program (W81XWH-19-1-0668), the Lung Cancer Research Program (W81XWH-18-1-0440, W81XWH-20-1-0595), the Peer Reviewed Cancer Research Program (W81XWH-18-1-0404, W81XWH-21-1-0345) and the Prostate Cancer Research Program (W81XWH-15-1-0558, W81XWH-20-1-0851); the Ohio Third Frontier Technology Validation Fund; the VA Merit Review Award IBX004121A from the United States Department of Veterans Affairs Biomedical Laboratory Research and Development Service; and The Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering at Case Western Reserve University; and through sponsored research agreements from AstraZeneca, Boehringer Ingelheim and Bristol Myers Squibb. The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the institutions named.
N.B. is a current employee of Tempus Labs and a former employee of IBM Research, with both of which he is an inventor on several pending patents pertaining to medical image analysis. He additionally holds equity in Tempus Labs. V.V. is a consultant for Alkermes, AstraZeneca, Bristol Myers Squibb, Celgene, Foundation Medicine, Genentech, Merck, Nektar Therapeutics and Takeda, has current or pending grants from Alkermes, AstraZeneca, Bristol Myers Squibb, Genentech and Merck, is on the speakers’ bureaus of Bristol Myers Squibb, Celgene, Foundation Medicine and Novartis, and has received payment for the development of educational presentations from Bristol Myers Squibb and Foundation Medicine. A.M. holds equity in Elucid Bioimaging and Inspirata, has been or is a scientific advisory board member for Aiforia, AstraZeneca, Bristol Myers Squibb, Inspirata and Merck, serves as a consultant for Caris, Inc. and Roche Diagnostics, has sponsored research agreements with AstraZeneca, Boehringer Ingelheim, Bristol Myers Squibb and Philips, has developed a technology relating to cardiovascular imaging that has been licensed to Elucid Bioimaging, and is involved in an NIH U24 grant with PathCore and three different NIH R01 grants with Inspirata. The other authors declare no competing interests.
Peer review information
Nature Reviews Clinical Oncology thanks D. Kontos, R. Mak and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Genomic Data Commons Data Portal: https://portal.gdc.cancer.gov/
The Cancer Imaging Archive: https://www.cancerimagingarchive.net/
A method of analysing model validity without an independent validation set on a limited data sample by dividing the training data into subsets for training and assessing the performance on the complementary subset of data. Several methods of cross-validation include holdout, k-fold or leave-one-out.
- Elastic net survival model
Type of Cox proportional hazard model that is used to calculate hazard ratios, which are a way of evaluating the strength of the association of a variable (for example, survival outcomes) with a time point. An elastic net has the added advantage over a standard Cox model of adjusting for high dimensional data and covariates that might be correlated with each other, while making survival estimations.
- Grey-level co-occurrence matrix features
Class of commonly used radiomic features, also known as Haralick features, which rely on higher-order statistics to describe the spatial arrangement and apparent position of the different grey levels present throughout the analysed image.
Statistical measure to indicate the shape of a probability distribution in terms of its ‘tailedness’. High kurtosis means high deviation from the mean.
- Laws’ energy measures
Eponymously named after K. I. Laws, this radiomic feature focuses on measuring variations of energy within a fixed window size, to calculate a combined texture energy of the pixels analysed.
- Long short-term memory
Type of recurrent neural network that has been supplemented by the addition of recurrent or ‘forget’ gates, which enables the network to learn by looking back at propagated errors.
Statistical measure to indicate the apparent distance between the mean and mode of a distribution. Skewness = (mean – mode)/standard deviation.
- Support vector machine
Supervised machine learning model used to classify data by constructing hyperplanes and choosing the hyperplane that has the largest separation between the two classes of interest.
- Tumour-infiltrating lymphocytes
(TILs). Lymphocytes that have invaded the tumour tissue from the bloodstream. In the past few years, studies have found TILs to be prognostic of survival and predictive of treatment benefit in several solid tumour types, including breast and lung tumours.
About this article
Cite this article
Bera, K., Braman, N., Gupta, A. et al. Predicting cancer outcomes with radiomics and artificial intelligence in radiology. Nat Rev Clin Oncol (2021). https://doi.org/10.1038/s41571-021-00560-7