Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens

Nature Biomedical Engineering volume 6, pages 1435–1448 (2022)Cite this article

Subjects

Abstract

Multiplexed immunofluorescence imaging allows the multidimensional molecular profiling of cellular environments at subcellular resolution. However, identifying and characterizing disease-relevant microenvironments from these rich datasets is challenging. Here we show that a graph neural network that leverages spatial protein profiles in tissue specimens to model tumour microenvironments as local subgraphs captures distinctive cellular interactions associated with differential clinical outcomes. We applied this spatial cellular-graph strategy to specimens of human head-and-neck and colorectal cancers assayed with 40-plex immunofluorescence imaging to identify spatial motifs associated with cancer recurrence and with patient survival after treatment. The graph deep learning model was substantially more accurate in predicting patient outcomes than deep learning approaches that model spatial data on the basis of the local composition of cell types, and it generated insights into the effect of the spatial compartmentalization of tumour cells and granulocytes on patient prognosis. Local graphs may also aid in the analysis of disease-relevant motifs in histology samples characterized via spatial transcriptomics and other -omics techniques.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Geometric deep learning on multiplexed immunofluorescence imaging.
Fig. 2: CODEX samples and cell characteristics.
Fig. 3: Model predictions of primary outcome and survival length in HNC.
Fig. 4: Clustering of SPACE-GM embeddings identifies disease-relevant microenvironments.
Fig. 5: Permutation of nodes in microenvironments helps identify cell–cell interactions that affect predictions.

Similar content being viewed by others

Data availability

Data supporting the results in this study involve patient data and are available from the corresponding author on reasonable request.

Code availability

Commercial software (Akoya Biosciences, Enable Medicine) was used to preprocess images and to classify cells using methods based on published algorithms. Deepcell (https://github.com/vanvalenlab/deepcell-tf) was used to segment cells from multiplexed immunofluorescence images. The codes used for the construction of spatial cellular graphs and for the following analyses are available at https://gitlab.com/enable-medicine-public/space-gm.

References

  1. Luca, B. A. et al. Atlas of clinically distinct cell states and ecosystems across human solid tumors. Cell 184, 5482–5496 (2021).

    Article  CAS  Google Scholar 

  2. Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 53, 1334–1347 (2021).

    Article  CAS  Google Scholar 

  3. Bejarano, L., Jordāo, M. J. C. & Joyce, J. A. Therapeutic targeting of the tumor microenvironment. Cancer Discov. 11, 933–959 (2021).

    Article  CAS  Google Scholar 

  4. Lomakin, A. et al. Spatial genomics maps the structure, character and evolution of cancer clones. Preprint at bioRxiv https://doi.org/10.1101/2021.04.16.439912 (2021).

  5. Rodriques, S. G. et al. Slide-seq: a scalable technology for measuring genome-wide expression at high spatial resolution. Science 363, 1463–1467 (2019).

    Article  CAS  Google Scholar 

  6. Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science 361, eaat5691 (2018).

    Article  Google Scholar 

  7. Moffitt, J. R. et al. High-throughput single-cell gene-expression profiling with multiplexed error-robust fluorescence in situ hybridization. Proc. Natl Acad. Sci. USA 113, 11046–11051 (2016).

    Article  CAS  Google Scholar 

  8. Goltsev, Y. et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell 174, 968–981 (2018).

    Article  CAS  Google Scholar 

  9. Angelo, M. et al. Multiplexed ion beam imaging of human breast tumors. Nat. Med. 20, 436–442 (2014).

    Article  CAS  Google Scholar 

  10. Lin, J.-R., Fallahi-Sichani, M., Chen, J.-Y. & Sorger, P. K. Cyclic immunofluorescence (CycIF), a highly multiplexed method for single-cell imaging. Curr. Protoc. Chem. Biol. 8, 251–264 (2016).

    Article  Google Scholar 

  11. Ali, H. R. et al. Imaging mass cytometry and multiplatform genomics define the phenogenomic landscape of breast cancer. Nat. Cancer 1, 163–175 (2020).

    Article  CAS  Google Scholar 

  12. Schürch, C. M. et al. Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell 182, 1341–1359 (2020); erratum 183, 838 (2020).

    Article  Google Scholar 

  13. Bhate, S. S., Barlow, G. L., Schürch, C. M. & Nolan, G. P. Tissue schematics map the specialization of immune tissue motifs and their appropriation by tumors. Cell Syst. 13, 109–130 (2022).

    Article  CAS  Google Scholar 

  14. Zhou, Y. et al. Cgc-net: cell graph convolutional network for grading of colorectal cancer histology images. In Proc. IEEE/CVF International Conference on Computer Vision Workshops 388–398 (IEEE, 2019).

  15. Lu, W., Graham, S., Bilal, M., Rajpoot, N. & Minhas, F. Capturing cellular topology in multi-gigapixel pathology images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops 260–261 (IEEE, 2020).

  16. Anand, D., Gadiya, S. & Sethi, A. Histographs: graphs in histopathology. In Medical Imaging 2020: Digital Pathology Vol. 11320 (eds Tomaszewski, J. E. & Ward, A. D.) 150–155 (SPIE, 2020).

  17. Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. In Proc. 34th International Conference on Machine Learning Vol. 70 (eds Precup, D. & Teh, Y. W.) 1263–1272 (JMLR.org, 2017).

  18. Wu, Z. et al. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2021).

    Article  Google Scholar 

  19. Brbić, M. et al. Annotation of spatially resolved single-cell data with STELLAR. Preprint at bioRxiv https://doi.org/10.1101/2021.11.24.469947 (2021).

  20. Innocenti, C. et al. An unsupervised graph embeddings approach to multiplex immunofluorescence image exploration. Preprint at bioRxiv https://doi.org/10.1101/2021.06.09.447654 (2021).

  21. Fischer, D. S., Schaar, A. C. & Theis, F. J. Learning cell communication from spatial graphs of cells. Preprint at bioRxiv https://doi.org/10.1101/2021.07.11.451750 (2021).

  22. Kim, J. et al. Unsupervised discovery of tissue architecture in multiplexed imaging. Preprint at bioRxiv https://doi.org/10.1101/2022.03.15.484534 (2022).

  23. Xu, K., Hu, W., Leskovec, J. & Jegelka, S. How powerful are graph neural networks? In Proc.7th International Conference on Learning Representations (OpenReview, 2019); https://openreview.net/forum?id=ryGs6iA5Km

  24. Argiris, A., Karamouzis, M. V., Raben, D. & Ferris, R. L. Head and neck cancer. Lancet 371, 1695–1709 (2008).

    Article  CAS  Google Scholar 

  25. Dalerba, P. et al. CDX2 as a prognostic biomarker in stage II and stage III colon cancer. N. Engl. J. Med. 374, 211–222 (2016).

    Article  CAS  Google Scholar 

  26. Uppaluri, R. et al. Neoadjuvant and adjuvant pembrolizumab in resectable locally advanced, human papillomavirus–unrelated head and neck cancer: a multicenter, phase II trial. Clin. Cancer Res. 26, 5140–5152 (2020).

    Article  CAS  Google Scholar 

  27. McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at https://doi.org/10.48550/arXiv.1802.03426 (2018).

  28. Blise, K. E., Sivagnanam, S., Banik, G. L., Coussens, L. M. & Goecks, J. Single-cell spatial architectures associated with clinical outcome in head and neck squamous cell carcinoma. NPJ Precis. Oncol. 6, 10 (2022).

    Article  CAS  Google Scholar 

  29. Jackson, H. W. et al. The single-cell pathology landscape of breast cancer. Nature 578, 615–620 (2020).

    Article  CAS  Google Scholar 

  30. Trellakis, S. et al. Polymorphonuclear granulocytes in human head and neck cancer: enhanced inflammatory activity, modulation by cancer cells and expansion in advanced disease. Int. J. Cancer 129, 2183–2193 (2011).

    Article  CAS  Google Scholar 

  31. Lonardi, S. et al. Tumor-associated neutrophils (TANs) in human carcinoma-draining lymph nodes: a novel TAN compartment. Clin. Transl. Immunol. 10, e1252 (2021).

    Article  CAS  Google Scholar 

  32. Coffelt, S. B., Wellenstein, M. D. & de Visser, K. E. Neutrophils in cancer: neutral no more. Nat. Rev. Cancer 16, 431–446 (2016).

    Article  CAS  Google Scholar 

  33. Shojaei, F. et al. Bv8 regulates myeloid-cell-dependent tumour angiogenesis. Nature 450, 825–831 (2007).

    Article  CAS  Google Scholar 

  34. Di Mitri, D. et al. Tumour-infiltrating Gr-1+ myeloid cells antagonize senescence in cancer. Nature 515, 134–137 (2014).

    Article  Google Scholar 

  35. Coffelt, S. B. et al. IL-17-producing γδ T cells and neutrophils conspire to promote breast cancer metastasis. Nature 522, 345–348 (2015).

    Article  CAS  Google Scholar 

  36. Wculek, S. K. & Malanchi, I. Neutrophils support lung colonization of metastasis-initiating breast cancer cells. Nature 528, 413–417 (2015).

    Article  CAS  Google Scholar 

  37. Regev, A. et al. The Human Cell Atlas. eLife 6, e27041 (2017).

    Article  Google Scholar 

  38. HuBMAP Consortium. The human body at cellular resolution: the NIH Human Biomolecular Atlas Program. Nature 574, 187–192 (2019).

    Article  CAS  Google Scholar 

  39. Greenwald, N. F. et al. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40, 555–565 (2021).

    Article  Google Scholar 

  40. Hickey, J. W., Tan, Y., Nolan, G. P. & Goltsev, Y. Strategies for accurate cell type identification in CODEX multiplexed imaging data. Front. Immunol. 12, 727626 (2021).

    Article  CAS  Google Scholar 

  41. Blondel, V. D., Guillaume, J.-L., Lambiotte, R. & Lefebvre, E. Fast unfolding of communities in large networks. J. Stat. Mech. Theory Exp. 2008, P10008 (2008).

    Article  Google Scholar 

  42. Kvamme, H., Borgan, Ø. & Scheel, I. Time-to-event prediction with neural networks and Cox regression. J. Mach. Learn. Res. 20, 1–30 (2019).

    Google Scholar 

  43. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (ICLR, 2015); https://www.iclr.cc/archive/www/2015.html

  44. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

    Google Scholar 

  45. Paszke, A. et al. PyTorch: An imperative style, high-performance deep learning library. In Proc. 33rd International Conference on Neural Information Processing Systems 8026–8037 (Curran Associates, 2019).

  46. Fey, M. & Lenssen, J. E. Fast graph representation learning with PyTorch geometric. Preprint at https://doi.org/10.48550/arXiv.1903.02428 (2019).

  47. Kipf, T. N., & Welling, M. Semi-supervised classification with graph convolutional networks. In Proc. 5th International Conference on Learning Representations (OpenReview, 2017); https://openreview.net/forum?id=SJU4ayYgl

  48. Veličković, P. et al. Graph attention networks. In Proc. 6th International Conference on Learning Representations (OpenReview, 2018); https://openreview.net/forum?id=rJXMpikCZ

  49. Hamilton, W., Ying, Z., & Leskovec, J. Inductive representation learning on large graphs. In Proc. 31st International Conference on Neural Information Processing Systems 1025–1035 (Curran Associates, 2017).

  50. Li, Y., Tarlow, D., Brockschmidt, M. & Zemel, R. Gated graph sequence neural networks. In Proc. 4th International Conference on Learning Representations (OpenReview, 2016); https://openreview.net/forum?id=HSgW989Kp-q

  51. Vinyals, O., Bengio, S. & Kudlur, M. Order matters: sequence to sequence for sets. In Proc. 4th International Conference on Learning Representations (ICLR, 2016); https://www.iclr.cc/archive/www/2016.html

Download references

Acknowledgements

J.Z. discloses support for the research described in this study from NSF CAREER (grant no. 1942926). K.S. discloses support for the research described in this study from a Knight-Hennessy Fellowship.

Author information

Author notes

  1. These authors contributed equally: Zhenqin Wu, Alexandro E. Trevino.

Authors and Affiliations

  1. Enable Medicine, Menlo Park, CA, USA

    Zhenqin Wu, Alexandro E. Trevino, Eric Wu, Honesty J. Kim, H. Blaize D’Angio, Ryan Preska, Aaron T. Mayer & James Zou

  2. Department of Chemistry, Stanford University, Stanford, CA, USA

    Zhenqin Wu

  3. Department of Electrical Engineering, Stanford University, Stanford, CA, USA

    Eric Wu & James Zou

  4. Department of Computer Science, Stanford University, Stanford, CA, USA

    Kyle Swanson & James Zou

  5. Department of Pathology, Stanford University, Stanford, CA, USA

    Gregory W. Charville

  6. Department of Pathology and Cell Biology, Columbia University, New York, NY, USA

    Piero D. Dalerba

  7. Department of Surgery, Brigham and Women’s Hospital, Boston, MA, USA

    Ann Marie Egloff & Ravindra Uppaluri

  8. Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA, USA

    Ann Marie Egloff & Ravindra Uppaluri

  9. Department of Otolaryngology, University of Pittsburgh, Pittsburgh, PA, USA

    Umamaheswar Duvvuri

  10. Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

    James Zou

Authors
  1. Zhenqin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandro E. Trevino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kyle Swanson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Honesty J. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. H. Blaize D’Angio
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ryan Preska
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gregory W. Charville
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Piero D. Dalerba
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ann Marie Egloff
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ravindra Uppaluri
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Umamaheswar Duvvuri
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Aaron T. Mayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. James Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Z.W. and A.E.T. designed the model and computational experiments in consultation with E.W. and K.S. Z.W. and A.E.T. analysed the data. Z.W. and A.E.T. wrote the manuscript with input from all authors. G.W.C., P.D.D., A.M.E., R.U. and U.D. provided samples for the experiments. H.J.K., H.B.D. and R.P. performed the experiments and data preprocessing. J.Z. and A.T.M. were responsible for the overall direction and planning of the project.

Corresponding authors

Correspondence to Alexandro E. Trevino, Aaron T. Mayer or James Zou.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Biomedical Engineering thanks Tae Hyun Hwang, Jonathan Nowak and Jianyu Rao for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Comparison of composition-based clusters and microenvironment clusters.

A. and B. We generate composition-based clusters with cell type compositions of 3-hop subgraphs (microenvironments) following the same procedure. Note that the average predictions of microenvironment clusters are much more polarized. C. The composition cluster highly enriched with granulocytes has neutral average predictions/labels, it could be further dissected into multiple sub-clusters that belong to different microenvironment clusters. We see a pair of granulocyte/tumor microenvironments that have opposite outcome labels, see Results for further discussion. D. Similarly, the composition cluster enriched with vessel/lymph vessel cells could be dissected into multiple sub-clusters, from which we notice two microenvironment clusters that are both enriched with lymph vessel cells but have different composition and outcome predictions, see Supplementary Note 7 for further discussion. E. Comparison of the two microenvironments enriched in lymph vessel cells: the left column shows a microenvironment with more lymphocytes and has overall positive outcomes; the right column shows a contrasting group with more tumor cells and much worse outcome predictions. Observation of tumor cells in close vicinity of lymph vessels indicates potential lymphovascular invasion and will lead to worse prognosis, which aligns with model predictions.

Supplementary information

Supplementary Information

Supplementary Notes 1–7, Tables 1–7 and Figs.1–10.

Reporting Summary.

Peer Review File.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Z., Trevino, A.E., Wu, E. et al. Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens. Nat. Biomed. Eng 6, 1435–1448 (2022). https://doi.org/10.1038/s41551-022-00951-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-022-00951-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer