Abstract
The complex diagnostic criteria for gliomas pose great challenges for making accurate diagnoses with computational pathology methods. There are no in-depth analyses of the accuracy, reliability and auxiliary capability of present approaches from a clinical perspective. Previous studies have overlooked the exploration of molecular and morphological correlations. To overcome these limitations, we propose ROAM, a multiple-instance learning model based on large regions of interest and a pyramid transformer. ROAM enlarges regions of interest to facilitate the consideration of tissue contexts. It utilizes the pyramid transformer to model both intrascale and interscale correlations of morphological features and leverages class-specific multiple-instance learning based on attention to extract slide-level visual representations that can be used to diagnose gliomas. Through comprehensive experiments on both in-house and external glioma datasets, we demonstrate that ROAM can automatically capture key morphological features consistent with the experience of pathologists and thus provide accurate, reliable and adaptable clinical-grade diagnoses of gliomas. Moreover, ROAM has clinical value for auxiliary diagnoses and could pave the way for the study of molecular and morphological correlations.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The TCGA public glioma WSI dataset is available from the National Institutes of Health’s genomic data commons (https://portal.gdc.cancer.gov). Detailed information about the dataset, including sample IDs and labels, can be obtained from Zenodo via https://doi.org/10.5281/zenodo.11469546 (ref. 43). The Xiangya glioma WSI dataset is not publicly available in accordance with institutional requirements governing the protection of the privacy of human subjects. The in-house Xiangya dataset is not publicly available due to privacy concerns regarding patient information.
Code availability
The ROAM project, including detailed documents and instructions, is available on GitHub (https://github.com/whiteyunjie/ROAM)44. The source code is also available on Zenodo via https://doi.org/10.5281/zenodo.11469423 (ref. 45).
References
Ostrom, Q. T. et al. The epidemiology of glioma in adults: a ‘state of the science’ review. Neuro-oncology 16, 896–913 (2014).
Weller, M. et al. Glioma. Nat. Rev. Dis. Primers 1, 15017 (2015).
Chen, R., Smith-Cohn, M., Cohen, A. L. & Colman, H. Glioma subclassifications and their clinical significance. Neurotherapeutics 14, 284–297 (2017).
Louis, D. N. et al. The 2021 WHO classification of tumors of the central nervous system: a summary. Neuro-oncology 23, 1231–1251 (2021).
Horbinski, C. et al. NCCN guidelines® insights: central nervous system cancers, version 2.2022: featured updates to the NCCN guidelines. J. Natl Compr. Cancer Netw. 21, 12–20 (2023).
Van Den Bent, M. J. Interobserver variation of the histopathological diagnosis in clinical trials on glioma: a clinician’s perspective. Acta Neuropathol. 120, 297–304 (2010).
Ker, J., Bai, Y. Q., Lee, H. Y., Rao, J. & Wang, L. P. Automated brain histology classification using machine learning. J. Clin. Neurosci. 66, 239–245 (2019).
Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V. & Madabhushi, A. Artificial intelligence in digital pathology – new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16, 703–715 (2019).
Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N. Digital pathology and artificial intelligence. Lancet Oncol. 20, e253–e261 (2019).
Echle, A. et al. Deep learning in cancer pathology: a new generation of clinical biomarkers. Br. J. Cancer 124, 686–696 (2021).
Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).
Ertosun, M. G. & Rubin, D. L. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu. Symp. Proc. Arch. 2015, 1899–1908 (2015).
Jin, L. et al. Artificial intelligence neuropathologist for glioma classification using deep learning on hematoxylin and eosin stained slide images and molecular markers. Neuro-oncology 23, 44–52 (2021).
Im, S. et al. Classification of diffuse glioma subtype from clinical-grade pathological images using deep transfer learning. Sensors 21, 3500 (2021).
Ocampo, P. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. J. Thorac. Oncol. 13, S562 (2018).
Hsu, W. W. et al. A weakly supervised deep learninÿg-based method for glioma subtype classification using WSI and mpMRIs. Sci. Rep. 12, 6111 (2022).
Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).
Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021).
Shao, Z. C. et al. TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In Proc. 35th Conference on Neural Information Processing Systems (eds Ranzato, M. et al.) 2136–2147 (Curran Associates, 2021).
Li, B., Li, Y. & Eliceiri, K. W. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 14318–14323 (IEEE, 2021).
Zhang, H. R. et al. DTFD-MIL: double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 18802–18812 (IEEE, 2022).
Zhang, J. et al. Attention multiple instance learning with transformer aggregation for breast cancer whole slide image classification. In Proc. 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1804–1809 (IEEE, 2022).
Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 16144–16155 (IEEE, 2022).
Hou, W. et al. H2-MIL: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proc. AAAI Conference on Artificial Intelligence 933–941 (AAAI Press, 2022).
Zheng, Y. et al. A graph-transformer for whole slide image classification. IEEE Trans. Med. Imaging 41, 3003–3015 (2022).
Lee, Y. et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-022-00923-0 (2022).
Chefer, H., Gur, S. & Wolf, L. Transformer interpretability beyond attention visualization. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 782–791 (IEEE, 2021).
Dosovitskiy, A. et al. An image is worth 16×16 words: transformers for image recognition at scale. Preprint at https://arxiv.org/abs/2010.11929 (2020).
Vaswani, A. et al. Attention is all you need. In Proc. 30th Conference on Neural Information Processing Systems 5998–6008 (eds Guyon, I. et al.) (Curran Associates, 2017).
Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 10012–10022 (IEEE, 2021).
Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2127–2136 (PMLR, 2018).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).
Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Louis, D. N. et al. The 2016 World Health Organization classification of tumors of the central nervous system: a summary. Acta Neuropathol. 131, 803–820 (2016).
Wang, X. et al. RetCCL: clustering-guided contrastive learning for whole-slide image retrieval. Med. Image Anal. 83, 102645 (2023).
Yang, P. et al. CS-CO: a hybrid self-supervised visual representation learning method for H&E-stained histopathological images. Med. Image Anal. 81, 102539 (2022).
Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med. Image Anal. 81, 102559 (2022).
Ciga, O., Xu, T. & Martel, A. L. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022).
Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10, e0130140 (2015).
Selvaraju, R. R. et al. Grad-cam: visual explanations from deep networks via gradient-based localization. In Proc. IEEE Conference on Computer Vision 618–626 (IEEE, 2017).
Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging 35, 1962–1971 (2016).
Carpenter, J. & Bithell, J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat. Med. 19, 1141–1164 (2000).
Yin, X. TCGA glioma WSI dataset used for evaluating ROAM. Zenodo https://doi.org/10.5281/zenodo.11469546 (2024).
Yin, X. ROAM. GitHub https://github.com/whiteyunjie/ROAM (2024).
Yin, X. A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular state revelation of gliomas (v1.0.0). Zenodo https://doi.org/10.5281/zenodo.11469423 (2024).
Acknowledgements
This work was supported by the National Key Research and Development Program of China (Grant Nos. 2023YFF1204802 and 2021YFF1200902 to R.J. and 2022YFF1202403 to H.L.) and the National Natural Science Foundation of China (Grant Nos. 62273194 to R.J. and 62250005 to X.Z.).
Author information
Authors and Affiliations
Contributions
R.J., X.Z., Z.H. and H.L. conceived the study and supervised the research. R.J., X.Y. and P.Y. designed, implemented and validated the ROAM project. Y.W., J.H., J.Y., Z.H. and L.C. collected, curated and annotated the data and helped with analysing the results. F.C. provided technical support for the online platform. X.F., L.S., L.L., W.L., Z.H., L.C., Y.W. and H.Z. participated in the clinical-grade assessment of ROAM and the auxiliary diagnosis experiments. Z.H. and L.C. participated in the summarization and discovery of molecular and morphological biomarkers. R.J., X.Y., P.Y. and H.L. wrote the paper. L.C., R.J., X.Z., Z.H. and H.L. provided valuable comments on the paper.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Machine Intelligence thanks the anonymous reviewers for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Workflow of the Pyramid Transformer and ROI-level supervision.
a-b, the implementation process of the multi-scale self-attention module that incorporates visual features of ROI at 10× with features of ROI at 5×. Patch tokens with the same numerical ID represent that their corresponding tissue regions locate at the identity location within ROI, differing only in their magnification scales. c, the process of instance-level supervision. The top k ROIs with the highest attention scores are assigned the same label as their corresponding slide. Classification is performed using an ROI-level classifier, and the instance-level loss is combined with the slice-level loss to train the entire model.
Extended Data Fig. 2 Overview of the glioma data and the cascade diagnostic system.
a, Detailed categories and crucial molecular features of glioma. Glioma can be broadly classified into normal, gliosis and tumor. The subtypes of glioma include astrocytoma, oligodendroglioma, and ependymoma, each having various grades. IDH mutation, and MGMT promoter methylation are two of the most critical molecular features associated with glioma diagnosis. b, Information of the in-house Xiangya glioma dataset. The dataset consists of a total of 1109 slides and supports 7 distinct classification tasks based on the provided labels. c, Details of the external TCGA glioma dataset. The dataset consists of 618 slides and supports 4 external classification tasks. d, The cascade diagnostic system of glioma based on ROAM. For instance, in the case of astrocytoma, the final diagnosis involves a sequential process utilizing 3 diagnostic models: model 1, model 2 and model 3.
Extended Data Fig. 3 Cross-validation of ROAM and baseline methods in glioma diagnosis without the use of the ensemble strategy.
Tasks of glioma diagnosis include glioma detection (a,f,k, n = 222), glioma subtyping (b,g,l, n = 193), astrocytoma grading (c,h,m, n = 106), oligodendroglioma grading (d,i,n, n = 45), and ependymoma grading (e,j,o, n = 43). a-o, Results are derived from 5-fold cross-validation experiments on the Xiangya dataset. Models are trained without the use of the ensemble strategy. a-e, Receiver operating characteristic (ROC) curves and the corresponding area under the curves (AUC±s.d). The confidence bond shows ± 1 s.d for a curve. Inserts: zoomed-in view of the curves. f-j, Averaged Accuracy, macro recall, macro precision, and macro F1-score. The metrics are plotted using box plot and each box ranges from the upper to lower quartiles with the median as the horizontal line, whiskers extend to 1.5 times the interquartile range, and diamond points represent outliers. All the legends are consistent with the legend in f. k-o, Mean normalized confusion matrices of ROAM.
Extended Data Fig. 4 Performance of ROAM and baseline methods in the prediction of molecular features.
Tasks include the prediction of IDH mutation (a, c, e, n = 216) and that of MGMT promoter methylation (b, d, f, n = 218). a-f, Results are derived from five ensemble models that are trained using the in-house training dataset and tested on the in-house test dataset, both split from the Xiangya dataset at the ratio of 2:1. a-b, Receiver operating characteristic (ROC) curves and the corresponding area under the curves (AUC±s.d). The confidence bond shows ± 1 s.d for a curve. Inserts: zoomed-in view of the curves. c-d, Accuracy, macro recall, macro precision, and macro F1-score. The metrics are plotted using box plot and each box ranges from the upper to lower quartiles with the median as the horizontal line, whiskers extend to 1.5 times the interquartile range, and diamond points represent outliers. e-f, Mean normalized confusion matrices of ROAM.
Extended Data Fig. 5 Clinical-grade performance in the overall cascade diagnostic task.
a, Performance of the cascade diagnostic system and the five pathologists for the overall glioma diagnosis task. Macro-averaged one-versus-rest operating points are plotted. There is no ROC curve because the prediction for the task is integrated based on the outcomes of multiple sub-tasks. b, Macro-averaged F1-score and accuracy of ROAM and pathologists (before and after AI assistance) for the overall glioma diagnosis task. The F1-score of mid-career 2 not being displayed implies that this pathologist made completely incorrect predictions for at least one category during the diagnosis.
Supplementary information
Supplementary Information
Supplementary Figs. 1–22, Tables 1–5 and Notes 1–6.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jiang, R., Yin, X., Yang, P. et al. A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas. Nat Mach Intell 6, 876–891 (2024). https://doi.org/10.1038/s42256-024-00868-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-024-00868-w
