Abstract
In oncology, intratumoural heterogeneity is closely linked with the efficacy of therapy, and can be partially characterized via tumour biopsies. Here we show that intratumoural heterogeneity can be characterized spatially via phenotype-specific, multi-view learning classifiers trained with data from dynamic positron emission tomography (PET) and multiparametric magnetic resonance imaging (MRI). Classifiers trained with PET–MRI data from mice with subcutaneous colon cancer quantified phenotypic changes resulting from an apoptosis-inducing targeted therapeutic and provided biologically relevant probability maps of tumour-tissue subtypes. When applied to retrospective PET–MRI data of patients with liver metastases from colorectal cancer, the trained classifiers characterized intratumoural tissue subregions in agreement with tumour histology. The spatial characterization of intratumoural heterogeneity in mice and patients via multimodal, multiparametric imaging aided by machine-learning may facilitate applications in precision oncology.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$99.00 per year
only $8.25 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Data availability
The main data supporting the results in this study are available within the paper and its Supplementary Information. The preclinical data are available from the corresponding author on reasonable request. Clinical data cannot be shared owing to patient-confidentiality obligations.
Code availability
Wherever applicable, default methods from MATLAB were used for the machine-learning analyses. Custom code for visualization and data processing is available from the corresponding author on request.
References
Eisenhauer, E. A. et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45, 228–247 (2009).
O’Connor, J. P. B. et al. Imaging biomarker roadmap for cancer studies. Nat. Rev. Clin. Oncol. 14, 169–186 (2016).
Judenhofer, M. S. et al. Simultaneous PET–MRI: a new approach for functional and morphological imaging. Nat. Med. 14, 459–465 (2008).
Disselhorst, J. A., Bezrukov, I., Kolb, A., Parl, C. & Pichler, B. J. Principles of PET/MR imaging. J. Nucl. Med. 55, 2S–10S (2014).
Bailey, D. L. et al. Combined PET/MR: the real work has just started. Summary report of the Third International Workshop on PET/MR Imaging; February 17–21, 2014, Tübingen, Germany. Mol. Imaging Biol. 17, 297–312 (2015).
Gillies, R. J. & Beyer, T. PET and MRI: is the whole greater than the sum of its parts? Cancer Res. 76, 6163–6166 (2016).
Schmitz, J. et al. Decoding intratumoral heterogeneity of breast cancer by multiparametric in vivo imaging: a translational study. Cancer Res. 76, 5512–5522 (2016).
O’Connor, J. P. B. et al. Imaging intratumor heterogeneity: role in therapy response, resistance, and clinical outcome. Clin. Cancer Res. 21, 249–257 (2015).
Napel, S., Mu, W., Jardim-Perassi, B. V., Aerts, H. J. W. L. & Gillies, R. J. Quantitative imaging of cancer in the postgenomic era: radio(geno)mics, deep learning, and habitats. Cancer 124, 4633–4649 (2018).
Heinzmann, K., Carter, L. M., Lewis, J. S. & Aboagye, E. O. Multiplexed imaging for diagnosis and therapy. Nat. Biomed. Eng 1, 697–713 (2017).
Junttila, M. R. & de Sauvage, F. J. Influence of tumour micro-environment heterogeneity on therapeutic response. Nature 501, 346–354 (2013).
Schmidt, H. et al. Correlation of simultaneously acquired diffusion-weighted imaging and 2-deoxy-[18F] fluoro-2-D-glucose positron emission tomography of pulmonary lesions in a dedicated whole-body magnetic resonance/positron emission tomography system. Invest. Radiol. 48, 247–255 (2013).
Divine, M. R. et al. A population-based gaussian mixture model incorporating 18F-FDG PET and diffusion-weighted MRI quantifies tumor tissue classes. J. Nucl. Med. 57, 473–479 (2016).
Kim, J., Ryu, S. Y., Lee, S. H., Lee, H. Y. & Park, H. Clustering approach to identify intratumour heterogeneity combining FDG PET and diffusion-weighted MRI in lung adenocarcinoma. Eur. Radiol. 29, 468–475 (2019).
Stoyanova, R. et al. Association of multiparametric MRI quantitative imaging features with prostate cancer gene expression in MRI-targeted prostate biopsies. Oncotarget 7, 53362–53376 (2016).
Katiyar, P. et al. A novel unsupervised segmentation approach quantifies tumor tissue populations using multiparametric MRI: first results with histological validation. Mol. Imaging Biol. 19, 391–397 (2017).
Katiyar, P. et al. Spectral clustering predicts tumor tissue heterogeneity using dynamic 18 F-FDG PET: a complement to the standard compartmental modeling approach. J. Nucl. Med. 58, 651–657 (2017).
Carano, R. A. D. et al. Quantification of tumor tissue populations by multispectral analysis. Magn. Reson. Med. 51, 542–551 (2004).
Berry, L. R. et al. Quantification of viable tumor microvascular characteristics by multispectral analysis. Magn. Reson. Med. 60, 64–72 (2008).
Barck, K. H. et al. Viable tumor tissue detection in murine metastatic breast cancer by whole-body MRI and multispectral analysis. Magn. Reson. Med. 62, 1423–1430 (2009).
Henning, E. C., Azuma, C., Sotak, C. H. & Helmer, K. G. Multispectral quantification of tissue types in a RIF-1 tumor model with histological validation. Part I. Magn. Reson. Med. 57, 501–512 (2007).
Schölkopf, B. Artificial intelligence: learning to see and act. Nature 518, 486–487 (2015).
de Vries, E. G. E. et al. Integrating molecular nuclear imaging in clinical research to improve anticancer therapy. Nat. Rev. Clin. Oncol. 16, 241–255 (2019).
Siegemund, M. et al. Superior antitumoral activity of dimerized targeted single-chain TRAIL fusion proteins under retention of tumor selectivity. Cell Death Dis. 3, e295 (2012).
Gillies, R. J., Kinahan, P. E. & Hricak, H. Radiomics: images are more than pictures, they are data. Radiology 278, 563–577 (2016).
Aerts, H. J. W. L. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 5, 4006 (2014).
Kumar, V. et al. Radiomics: the process and the challenges. Magn. Reson. Imaging 30, 1234–1248 (2012).
Stewart, G. D. et al. Sunitinib treatment exacerbates intratumoral heterogeneity in metastatic renal cancer. Clin. Cancer Res. 21, 4212–4223 (2015).
Lee, B. S. et al. Induced phenotype targeted therapy: radiation-induced apoptosis-targeted chemotherapy. J. Natl Cancer Inst. 107, dju403 (2015).
Disselhorst, J. A. et al. Linking imaging to omics utilizing image-guided tissue extraction. Proc. Natl Acad. Sci. USA 115, E2980–E2987 (2018).
Jaffray, D. A. Image-guided radiotherapy: from current concept to future perspectives. Nat. Rev. Clin. Oncol. 9, 688–699 (2012).
Hynynen, K. MRIgHIFU: a tool for image-guided therapeutics. J. Magn. Reson. Imaging 34, 482–493 (2011).
Reinke, A. et al. Common limitations of image processing metrics: a picture story. Preprint at https://arxiv.org/abs/2104.05642 (2021).
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
Sequist, L. V. et al. Genotypic and histological evolution of lung cancers acquiring resistance to EGFR inhibitors. Sci. Transl. Med. 3, 75ra26 (2011).
Hodi, F. S. et al. Evaluation of immune-related response criteria and RECIST v1.1 in patients with advanced melanoma treated with pembrolizumab. J. Clin. Oncol. 34, 1510–1517 (2016).
Veuthey, T. V., Herrera, G. & Dodero, V. I. Dyes and stains: from molecular structure to histological application. Front. Biosci. 19, 91–112 (2014).
Gown, A. M. & Willingham, M. C. Improved detection of apoptotic cells in archival paraffin sections: immunohistochemistry using antibodies to cleaved caspase 3. J. Histochem. Cytochem. 50, 449–454 (2002).
Austyn, J. M. & Gordon, S. F4/80, a monoclonal antibody directed specifically against the mouse macrophage. Eur. J. Immunol. 11, 805–815 (1981).
Dubuisson, M.-P. & Jain, A. K. A modified Hausdorff distance for object matching. In Proc. 12th International Conference on Pattern Recognition 566–568 (IEEE, 1994).
Dice, L. R. Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945).
von Luxburg, U. A tutorial on spectral clustering. Stat. Comput. 17, 395–416 (2007).
Kumar, A. & Daumé, H. A co-training approach for multi-view spectral clustering. In Proc. 28th International Conference on Machine Learning (ICML-11) 393–400 (ACM, 2011).
Strobl, C., Boulesteix, A.-L., Zeileis, A. & Hothorn, T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 8, 25 (2007).
Acknowledgements
We thank S. Castaneda Vega from the Werner Siemens Imaging Center for help with image analysis and M. Siegemund from the University of Stuttgart for providing support with Db-scTRAIL therapy. B.J.P. received support for the research described in this study from the European Union Seventh Framework Programme (FP7/2007–2013) under the European Research Council grant agreement number 323196 (ImageLink) and Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy (EXC-2180390900677). BS is a member of the excellence cluster “Machine Learning in the Sciences” funded by the Deutsche Forschungsgemeinschaft under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645.
Author information
Authors and Affiliations
Contributions
P.K., J.A.D., M.R.D. and B.J.P. designed the preclinical imaging and ex vivo experiments. J.A.D. and M.R.D. collected the preclinical imaging data. J.S., S.G. and C.l.F. conceptualized and performed the clinical imaging work. L.F., U.K. and L.Q.-M. performed the histopathology work and wrote the pathology findings. R.K. provided the Db-scTRAIL therapy. A.K. performed the surgery. P.K., J.A.D., B.S. and B.J.P. designed the machine-learning algorithmic pipeline. P.K. performed the machine-learning experiments and prepared all figures. P.K., V.A. and J.S. performed the data analysis. P.K., J.S., L.F., U.K., L.Q.-M., J.A.D and B.J.P. wrote the paper. All the authors contributed to the final version of the paper.
Corresponding author
Ethics declarations
Competing interests
P.K. is working at Novartis Institutes for BioMedical Research on artificial intelligence and machine-learning. J.A.D. is working at Siemens Healthineers. B.S. has a spin-off company and is working with several companies on artificial intelligence and machine-learning. B.J.P. has several research collaborations with Siemens, Bruker and larger pharmaceutical companies. However, none of these affiliations are directly linked to this work. The other authors declare no competing interests. R.K. is a named inventor on patents covering the TRAIL technology and a consultant to Immatics, Roche, SunRock and Oncomatryx.
Peer review
Peer review information
Nature Biomedical Engineering thanks Nathaniel Braman, Faisal Mahmood and Natarajan Raghunand for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 A-Casp-3, F4-80, MT and H&E stainings of a Th-24 tumour.
a, Left to right: A-Casp-3 and F4-80 stainings of an exemplary Th-24 tumour. The images in the inset show macrophages present at the border and in the fibrous area of the tumour. b, The MT insets show two non-apoptotic regions of the Th-24 tumour that majorly consist of loose fibrous tissue. c, The H&E inset additionally shows a small necrotic region present at the top of the tumour.
Extended Data Fig. 2 Phenotype-wise probability maps of CRLM predicted by the PET and MRI classifiers.
Left to right, pairs of columns: Viable, NF and apoptosis probability maps of the CRLM shown in Fig. 6, predicted by the PET and MRI classifiers. For both the models, the probability values of each voxel across the maps of all phenotypes sum to one.
Extended Data Fig. 3 Tissue fraction correlation plots.
The Pearson’s correlation plot illustrating the relationship between the viable and NF tissue fractions predicted by the a, PET/MRI, b, MRI, and c, PET Random Forest classifiers and the ground truth evaluations made by the pathologist using tumour histology. The correlation coefficients are indicated by the symbol ρ. For each histology slide, the classifier predicted viable, apoptosis and NF tissue fractions sum to 1.0.
Extended Data Fig. 4 Comparison of the PET/MRI and PET/MRITOP classifiers on the preclinical test data.
Column-wise, left to right: H&E, A-Casp-3 and MT histology of six representative test set tumours, and the corresponding phenotypic maps predicted by the PET/MRI and PET/MRITOP classifiers. The PET/MRITOP model was trained using the combined training data of the eight most relevant PET/MRI features, shown in Fig. 4g. The top, middle and bottom pairs of the tumours belong to the test Control, Th-24 and Th-72 groups, respectively. The phenotypic maps are joint probability maps that were colour-coded based on the tumour tissue class colour map shown in the key. Thus, the trained classifiers probabilistically assigned each voxel in the phenotypic maps to either one of the three phenotypic classes. The colour-coded arrows in the histology indicate the respective phenotypic class. The image in the first inset shows a patch of the Control tumour containing viable tumour tissue and necrosis. The second inset illustrates bundles of viable tumour cells distributed within a largely apoptotic region of the Th-24 tumour. Lastly, the image in the third inset shows the highly complex microenvironment of the Th-72 tumour.
Extended Data Fig. 5 Comparison of the PETTOP, MRITOP, PET/MRITOP and PET/MRI classifiers on the clinical test data.
Column-wise, left to right: H&E histology of four CRLM each resected from a different patient, the corresponding phenotypic maps predicted by the PETTOP, MRITOP, PET/MRITOP and PET/MRI classifiers, and the factored probability maps of each phenotype for the PET/MRITOP model. The PET/MRITOP model was trained using the combined training data of the eight most relevant PET/MRI features, shown in Fig. 4g. Whereas the PETTOP and MRITOP models were trained using the relevant features only from the respective modalities. The solid lines partition the presented examples patient-wise and the blue contours in the histology isolate the tumour from the liver tissue. The metastasis in the third row was too large to be processed on a single histology slide, and therefore had to be sectioned into three separate pieces. The dashed black lines in the stitched H&E image show the borders of the separated tumour units. The phenotypic maps are joint probability maps that were colour-coded based on the tumour tissue class colour map shown in the key. Thus, the trained classifiers probabilistically assigned each voxel in the phenotypic maps to either one of the three phenotypic classes. The colour-coded arrows in the histology indicate the respective phenotypic class, whereas the white arrow points towards the tumour region that was falsely classified by all models. All four insets show viable tumour regions within the largely necrotic CRLM. The factored probability maps of the PET/MRI classifier are shown in Fig. 6.
Extended Data Fig. 6 MSC workflow.
The MSC workflow broadly comprised three analysis stages. The first and second stages utilized spectral clustering to segment the multiparametric MRI and dynamic 18F-FDG PET data, respectively. In the third stage, complementary information from both the previous steps was combined to obtain the PET/MRI maps of intratumoural heterogeneity. WMRI and WPET denote the affinity matrices that yielded the optimal clustering results for the associated modalities. The PET/MRI affinity matrix, WPET/MRI, was obtained using a convex combination of the matrices from the two views, where the parameter α regulated the influence of each view. The optimal parameters in each step were selected based on a grid search to maximize the DSC between the histology and imaging clustering maps.
Supplementary information
Supplementary Information
Supplementary Methods and references.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Katiyar, P., Schwenck, J., Frauenfeld, L. et al. Quantification of intratumoural heterogeneity in mice and patients via machine-learning models trained on PET–MRI data. Nat. Biomed. Eng 7, 1014–1027 (2023). https://doi.org/10.1038/s41551-023-01047-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41551-023-01047-9
