Abstract
Malignant mesothelioma (MM) is an aggressive cancer primarily diagnosed on the basis of histological criteria1. The 2015 World Health Organization classification subdivides mesothelioma tumors into three histological types: epithelioid, biphasic and sarcomatoid MM. MM is a highly complex and heterogeneous disease, rendering its diagnosis and histological typing difficult and leading to suboptimal patient care and decisions regarding treatment modalities2. Here we have developed a new approach—based on deep convolutional neural networks—called MesoNet to accurately predict the overall survival of mesothelioma patients from whole-slide digitized images, without any pathologist-provided locally annotated regions. We validated MesoNet on both an internal validation cohort from the French MESOBANK and an independent cohort from The Cancer Genome Atlas (TCGA). We also demonstrated that the model was more accurate in predicting patient survival than using current pathology practices. Furthermore, unlike classical black-box deep learning methods, MesoNet identified regions contributing to patient outcome prediction. Strikingly, we found that these regions are mainly located in the stroma and are histological features associated with inflammation, cellular diversity and vacuolization. These findings suggest that deep learning models can identify new features predictive of patient survival and potentially lead to new biomarker discoveries.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The external validation TCGA dataset is publicly available at the TCGA portal (https://portal.gdc.cancer.gov). We provide a manifest linking to the sample IDs considered in the study (Supplementary Table 4). The MESOBANK/MESOPATH dataset that supports the findings of this study is available from the Centre Léon Bérard, but restrictions apply to its availability (used with permission for the current study), and so it is not publicly available. The data, or a test subset, may be available from the Centre Léon Bérard subject to ethical approvals.
Code availability
The code used for training the models has a large number of dependencies on internal tooling and its release is therefore not feasible. However, all experiments and implementation details are described thoroughly in the Methods so that it can be independently replicated with non-proprietary libraries.
References
Galateau-Sallé, F., Churg, A., Roggli, V. & Travis, W. D. The 2015 World Health Organization classification of tumors of the pleura: advances since the 2004 classification. J. Thorac. Oncol. 11, 142–154 (2016).
Galateau-Sallé, F. et al. New insights on diagnostic reproducibility of biphasic mesotheliomas: a multi-institutional evaluation by the International Mesothelioma Panel from the MESOPATH reference center. J. Thorac. Oncol. 13, 1189–1203 (2018).
Noonan, C. W. Environmental asbestos exposure and mesothelioma. Ann. Transl. Med. 5, 234 (2017).
Lacourt, A. et al. Dose–time-response association between occupational asbestos exposure and pleural mesothelioma. Occup. Environ. Med. 74, 691–697 (2017).
Robinson, B. W. S. & Lake, R. A. Advances in malignant mesothelioma. N. Engl. J. Med. 353, 1591–1603 (2005).
Yap, T. A., Aerts, J. G., Popat, S. & Fennell, D. A. Novel insights into mesothelioma biology and implications for therapy. Nat. Rev. Cancer 17, 475–488 (2017).
Opitz, I. et al. A new prognostic score supporting treatment allocation for multimodality therapy for malignant pleural mesothelioma: a review of 12 years’ experience. J. Thorac. Oncol. 10, 1634–1641 (2015).
Kindler, H. L. et al. Treatment of malignant pleural mesothelioma: American Society of Clinical Oncology clinical practice guideline. J. Clin. Oncol. 36, 1343–1373 (2018).
Brcic, L., Vlacic, G., Quehenberger, F. & Kern, I. Reproducibility of malignant pleural mesothelioma histopathologic subtyping. Arch. Pathol. Lab. Med. 142, 747–752 (2018).
Hmeljak, J. et al. Integrative molecular characterization of malignant pleural mesothelioma. Cancer Discov. 8, 1548–1565 (2018).
Shrestha, R. et al. BAP1 haploinsufficiency predicts a distinct immunogenic class of malignant peritoneal mesothelioma. Genom. Med. 11, 8 (2019).
Yu, K. H. et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 7, 12474 (2016).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1090–1098 (2012).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Hou, L. et al. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of 2016 IEEE Conference Computer Vision and Pattern Recognitition (IEEE, 2016); https://doi.org/10.1109/CVPR.2016.266
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, 2970–2979 (2018).
Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).
Schaumberg, A. J. et al. Large-scale annotation of histopathology images from social media. Preprint at https://doi.org/10.1101/396663 (2018).
Nagpal, K. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. Npj Digit. Med. 2, 48 (2019).
Courtiol, P., Tramel, E. W., Sanselme, M. & Wainrib, G. Classification and disease localization in histopathology using only global labels: a weakly-supervised approach. Preprint at https://arxiv.org/abs/1802.02212 (2018).
Zarella, M. D. et al. A practical guide to whole slide imaging. Arch. Pathol. Lab. Med. 143, 222–234 (2019).
Mukhopadhyay, S. et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology. Am. J. Surg. Pathol. 42, 1 (2018).
Galateau-sallé, F. et al. [The French mesothelioma network from 1998 to 2013]. Ann. Pathol. Elsevier Masson 34, 51–63 (2014).
Baas, P. et al. Malignant pleural mesothelioma: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann. Oncol. 21, 126–169 (2015).
Kadota, K. et al. Pleomorphic epithelioid diffuse malignant pleural mesothelioma: a clinicopathological review and conceptual proposal to reclassify as biphasic or sarcomatoid mesothelioma. J. Thorac. Oncol. 6, 896–904 (2011).
Junttila, M. R. & De Sauvage, F. J. Influence of tumour micro-environment heterogeneity on therapeutic response. Nature 501, 346–354 (2013).
Dacic, S. et al. Prognostic significance of p16/cdkn2a loss in pleural malignant mesotheliomas. Virchows Arch. 453, 627–635 (2008).
Pillai, K., Pourgholami, M. H., Chua, T. C. & Morris, D. L. Prognostic significance of Ki67 expression in malignant peritoneal mesothelioma. Am. J. Clin. Oncol. Cancer Clin. Trials 38, 388–394 (2015).
Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).
Ujiie, H. et al. The tumoral and stromal immune microenvironment in malignant pleural mesothelioma: a comprehensive analysis reveals prognostic immune markers. Oncoimmunology 4, 1–9 (2015).
Rosen, L. E. et al. Nuclear grade and necrosis predict prognosis in malignant epithelioid pleural mesothelioma: a multi-institutional study. Mod. Pathol. 31, 598–606 (2018).
Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference 234–241 (Springer, 2015); https://doi.org/10.1007/978-3-319-24574-4_28
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2016); https://doi.org/10.1109/CVPR.2016.90
Wang, D., Khosla, A., Gargeya, R., Irsha, H. & Beck, A.H. Deep learning for identifying metastatic breast cancer. Preprint at https://arxiv.org/abs/1606.05718 (2016).
Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B. & Wei, L. J. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat. Med. 30, 1105–1117 (2011).
Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (ACM Press, 2016); https://doi.org/10.1145/2939672.2939785
Acknowledgements
We thank the Owkin Lab for interesting discussions and the Centre Léon Bérard for providing access to the MESOBANK/MESOPATH dataset. We thank the MESOPATH consortium—I. Abd-Alsamad, H. Begueret, E. Brambilla, F. Capron, M.-C. Copin, C. Danel, A.-Y. de Lajartre, E. Foulet-Roge, F. Galateau-Sallé, L. Garbe, O. Groussard, S. Giusiano, V. Hofman, S. Lantejuoul, J.-M. Piquenot, I. Rouquette, S. Sagan, F. Thivolet-Bejui and J.-M. Vignaud—for the histological typing of the mesothelioma samples. We also thank the TCGA network for providing the external validation set (http://cancergenome.nih.gov/).
Author information
Authors and Affiliations
Contributions
P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., N.L.S., N.G., O.E., A.G.N., J.-Y.B., F.G.-S., G.W. and T.C. conceived the idea for this paper. P.C., C.M. and M.M. implemented the analysis. A.G.N. and F.G.-S. reviewed the tiles of interest. P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., N.L.S., N.G., O.E., A.G.N., J.-Y.B., F.G.-S., G.W. and T.C. contributed to the writing of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare the following competing interests. Employment: P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., T.C. and G.W. are employed by Owkin. Advisory: O.E. is on the scientific advisory board of Owkin and equity holder in the company. Financial: Owkin has submitted a patent application entitled ‘Systems and methods for mesothelioma feature detection and enhanced prognosis or response to treatment’ listing J.-Y.B., F.G.-S., T.C., G.W., P.C. and C.M. as the co-inventors.
Additional information
Peer Review Information Javier Carmona was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 MesoNet layout.
Our prognosis prediction is composed of five steps. (1) The ‘tiling’ process divides the whole slide image into small tiles of 224 × 224 pixels. (2) Features are then extracted from each tile using ResNet-50. (3) Features are then auto-encoded to reduce the dimension to 512 features per tile. (4) We use a convolutional 1D layer to score each tile. (5) Tiles associated with the largest and lowest scores are then retained to train a multilayer perceptron (MLP) to predict overall survival of the patient.
Extended Data Fig. 2 Comparison of the performance between MesoNet and models including additional non-pathology variables such as age and sex to predict MM patient overall survival.
The c index distribution for different predictive models. For the box plots, whiskers represent the minima and maxima. The middle line within the box represents the median. The upper and lower boundaries of the whiskers represent the 25th and 75th percentiles, respectively. A two-sided t-test was performed to determine significance.
Extended Data Fig. 3 Robustness of MesoNet to tissue sampling.
Distributions of variability of the MesoNet prediction for the 167 patients where multiple biopsies were available. The boxplot on the left side represents the distribution of maximal intra-individual variability, that is, the absolute difference between the minimum score and maximum score predicted for all the slides from a single patient for all patients with multiple biopsies (n = 167 patients). The boxplot on the right side represents the inter-individual variability, that is, the absolute difference between the mean of the scores for all the slides of a given patient to the mean of the score for all patients with multiple biopsies (n = 167 patients). For the two boxplots, whiskers represent the minima and maxima. The middle line within the box represents the median. The upper and lower boundaries of the whiskers represent the 25th and 75th percentiles, respectively. Significance was calculated using a two-sided t-test (P = 9.66 × 10−20).
Extended Data Fig. 4 Histological review of tiles of interest.
a, Schematic of the reviewing process. b, Repartition of features of interests in low survival tiles (n = 42) and high survival tiles (n = 42). c, Tiles of low survival with a transitional pattern. d, Tiles of unknown significance, TUS. Scale bars, 112 µm.
Extended Data Fig. 5 Illustration of predictive and non-predictive tiles selection.
Similarity of predictive and non-predictive tiles to a given predictive tile of interest, calculated based on the vector of coefficients obtained with ResNet-50. Similar predictive and non-predictive tiles are then reviewed manually for each extremal tile by pathologists. Scale bars, 112 µm.
Extended Data Fig. 6 Comparative histological analysis of predictive and non-predictive tiles.
Histogram of histological features associated with either predictive or non-predictive tiles that are similar to extremal tiles of high and low survival (n = 42). The review was performed independently by two mesothelioma pathologists.
Extended Data Fig. 7 Biological correlates.
Correlation between MesoNet risk score and the Ki67 expression (n = 54 samples), the ploidy level (n = 36 samples), the EMT score (n = 36 samples) and the leukocyte fraction (n = 36 samples) available for the TCGA dataset.
Supplementary information
Supplementary Tables
Supplementary Tables 1–4
Rights and permissions
About this article
Cite this article
Courtiol, P., Maussion, C., Moarii, M. et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat Med 25, 1519–1525 (2019). https://doi.org/10.1038/s41591-019-0583-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41591-019-0583-3
This article is cited by
-
Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview
Journal of Translational Medicine (2024)
-
Oral epithelial dysplasia detection and grading in oral leukoplakia using deep learning
BMC Oral Health (2024)
-
Regression-based Deep-Learning predicts molecular biomarkers from pathology slides
Nature Communications (2024)
-
Demographic bias in misdiagnosis by computational pathology models
Nature Medicine (2024)
-
A visual-language foundation model for computational pathology
Nature Medicine (2024)