Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Predicting tumour mutational burden from histopathological images using multiscale deep learning

Nature Machine Intelligence volume 2, pages 356–362 (2020)Cite this article

Subjects

Abstract

Tumour mutational burden (TMB) is an important biomarker for predicting the response to immunotherapy in patients with cancer. Gold-standard measurement of TMB is performed using whole exome sequencing (WES), which is not available at most hospitals because of its high cost, operational complexity and long turnover times. We have developed a machine learning algorithm, Image2TMB, which can predict TMB from readily available lung adenocarcinoma histopathological images. Image2TMB integrates the predictions of three deep learning models that operate at different resolution scales (×5, ×10 and ×20 magnification) to determine if the TMB of a cancer is high or low. On a held-out set of patients, Image2TMB achieves an area under the precision recall curve of 0.92, an average precision of 0.89, and has the predictive power of a targeted sequencing panel of ~100 genes. This study demonstrates that it is possible to infer genomic features from histopathology images, and potentially opens avenues for exploring genotype–phenotype relationships.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of Image2TMB and dataset.
Fig. 2: Performance of Image2TMB.
Fig. 3: Stratification study of TMB prediction.
Fig. 4: Spatial heterogeneity of predictions.

Similar content being viewed by others

Data availability

All data used in this study are publicly available via the TCGA Research Network (https://www.cancer.gov/tcga).

Code availability

Our full approach, including data download, preprocessing and Image2TMB, is publicly available from Github (https://github.com/msj3/Image2TMB).

References

  1. Garon, E. et al. Pembrolizumab for the treatment of non-small cell lung cancer. N. Engl. J. Med. 372, 2018–2028 (2015).

    Article  Google Scholar 

  2. Pardoll, D. The blockade of immune checkpoints in cancer immunotherapy. Nat. Rev. Cancer 12, 252–264 (2012).

    Article  Google Scholar 

  3. Rizvi, N. et al. Activity and safety of nivolumab and an anti-PD-1 immune checkpoint inhibitor and for patients with advanced and refractory squamous non-small-cell lung cancer (CheckMate 063): a phase 2, single-arm trial. Lancet Oncol. 16, 257–265 (2015).

    Article  Google Scholar 

  4. Chan, T. et al. Development of tumor mutation burden as an immunotherapy biomarker: utility for the oncology clinic. Ann. Oncol. 30, 44–56 (2018).

    Article  Google Scholar 

  5. Gandara, D. et al. Blood-based tumor mutational burden as a predictor of clinical benefit in non-small-cell lung cancer patients treated with atezolizumab. Nat. Med. 24, 1441–1448 (2018).

    Article  Google Scholar 

  6. Goodman, M. et al. Tumor mutational burden as an independent predictor of response to immunotherapy in diverse cancers. Mol. Cancer Ther. 16, 2598–2608 (2017).

    Article  Google Scholar 

  7. Samstein, R. et al. Tumor mutational load predicts survival after immunotherapy across multiple cancer types. Nat. Genet. 51, 202–206 (2019).

    Article  Google Scholar 

  8. Hellmann, M. et al. Nivolumab plus ipilimumab in lung cancer with a high tumor mutational burden. N. Engl. J. Med. 378, 2093–2104 (2018).

    Article  Google Scholar 

  9. Steuer, C. & Ramalingam, S. Tumor mutation burden: leading immunotherapy to the era of precision medicine? J. Clin. Oncol. 36, 631–632 (2018).

    Article  Google Scholar 

  10. Chalmers, Z. et al. Analysis of 100,000 human cancer genomes reveals the landscape of tumor mutational burden. Genome Med. 9, 34 (2017).

    Article  Google Scholar 

  11. Buchhalter, I. et al. Size matters: dissecting key parameters for panel-based tumor mutational burden analysis. Int. J. Cancer 144, 848–858 (2019).

    Article  Google Scholar 

  12. Carpenter, A. et al. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100 (2006).

    Article  Google Scholar 

  13. McQuin, C. et al. Cellprofiler 3.0: next-generation image processing for biology. Genome Biol. 16, e2005970 (2018).

    Google Scholar 

  14. Yu, K. et al. Association of omics features with histopathology patterns in lung adenocarcinoma. Cell Syst. 5, 620–627 (2017).

    Article  Google Scholar 

  15. Yu, K. et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 7, 12474 (2016).

    Article  Google Scholar 

  16. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  Google Scholar 

  17. Ehteshami, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Genome Biol. 318, 2199–2210 (2017).

    Google Scholar 

  18. Araújo, T. et al. Classification of breast cancer histology images using convolutional neural networks. PLoS ONE 12, e0177544 (2017).

    Article  Google Scholar 

  19. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

    Article  Google Scholar 

  20. Khosravi, P., Kazemi, E., Imielinski, M., Elemento, O. & Hajirasouliha, I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine 27, 317–328 (2018).

    Article  Google Scholar 

  21. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016); preprint at https://arxiv.org/abs/1512.00567.

  22. Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).

    Article  MathSciNet  Google Scholar 

  23. Hellmann, M. et al. Genomic features of response to combination immunotherapy in patients with advanced non-small-cell lung cancer. Cancer Cell 33, 843–852 (2018).

    Article  Google Scholar 

  24. Kim, J., Kim, H., Kim, B., Lee, J. & Jang, H. Prognostic value of c-Met overexpression in pancreatic adenocarcinoma: a metaanalysis. Oncotarget 8, 73098–73104 (2017).

    Article  Google Scholar 

  25. Offin, M. et al. Tumor mutation burden and efficacy of EGFR-tyrosine kinase inhibitors in patients with EGFR-mutant lung cancers. Clin. Cancer Res. 25, 1063–1069 (2019).

    Article  Google Scholar 

  26. Koboldt, D. et al. Varscan 2: somatic mutation and copy number alteration discovery in cancer by exome sequencing. Genome Res. 22, 568–576 (2012).

    Article  Google Scholar 

Download references

Acknowledgements

The presented results are based on data generated by the TCGA Research Network (https://www.cancer.gov/tcga). We thank R. Joshi, N. Neishaboori and E. Nohr for feedback and comments on the manuscript.

Author information

Authors and Affiliations

  1. Department of Physics, Stanford University, Stanford, CA, USA

    Mika S. Jain

  2. Department of Computer Science, Stanford University, Stanford, CA, USA

    Mika S. Jain

  3. Molecular Imaging Program at Stanford, Stanford University, Stanford, CA, USA

    Tarik F. Massoud

Authors
  1. Mika S. Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tarik F. Massoud
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.S.J. conceived and performed the study and experiments. T.F.M. supervised the experiments. M.S.J. and T.F.M. wrote the manuscript. Both authors discussed the results and commented on the manuscript.

Corresponding authors

Correspondence to Mika S. Jain or Tarik F. Massoud.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–3, Tables 1–2, methods and references.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jain, M.S., Massoud, T.F. Predicting tumour mutational burden from histopathological images using multiscale deep learning. Nat Mach Intell 2, 356–362 (2020). https://doi.org/10.1038/s42256-020-0190-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-020-0190-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer