Multiplexed tissue imaging facilitates the diagnosis and understanding of complex disease traits. However, the analysis of such digital images heavily relies on the experience of anatomical pathologists for the review, annotation and description of tissue features. In addition, the wider use of data from tissue atlases in basic and translational research and in classrooms would benefit from software that facilitates the easy visualization and sharing of the images and the results of their analyses. In this Perspective, we describe the ecosystem of software available for the analysis of tissue images and discuss the need for interactive online guides that help histopathologists make complex images comprehensible to non-specialists. We illustrate this idea via a software interface (Minerva), accessible via web browsers, that integrates multi-omic and tissue-atlas features. We argue that such interactive narrative guides can effectively disseminate digital histology data and aid their interpretation.
This is a preview of subscription content, access via your institution
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $6.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Giesen, C. et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11, 417–422 (2014).
Angelo, M. et al. Multiplexed ion beam imaging of human breast tumors. Nat. Med. 20, 436–442 (2014).
Gerdes, M. J. et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc. Natl Acad. Sci. USA 110, 11982–11987 (2013).
Lin, J.-R. et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. eLife https://doi.org/10.7554/eLife.31657 (2018).
Bodenmiller, B. Multiplexed epitope-based tissue imaging for discovery and healthcare applications. Cell Syst. 2, 225–238 (2016).
Coy, S. et al. Multiplexed immunofluorescence reveals potential PD-1/PD-L1 pathway vulnerabilities in craniopharyngioma. Neuro Oncol. 20, 1101–1112 (2018).
Goltsev, Y. et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell 174, 968–981.e15 (2018).
Stack, E. C., Wang, C., Roman, K. A. & Hoyt, C. C. Multiplexed immunohistochemistry, imaging, and quantitation: a review, with an assessment of tyramide signal amplification, multispectral imaging and multiplex analysis. Methods 70, 46–58 (2014).
Akturk, G., Sweeney, R., Remark, R., Merad, M. & Gnjatic, S. Multiplexed immunohistochemical consecutive staining on single slide (MICSSS): multiplexed chromogenic IHC assay for high-dimensional tissue analysis. Methods Mol. Biol. 2055, 497–519 (2020).
Saka, S. K. et al. Immuno-SABER enables highly multiplexed and amplified protein imaging in tissues. Nat. Biotechnol. 37, 1080–1090 (2019).
Li, W., Germain, R. N. & Gerner, M. Y. Multiplex, quantitative cellular analysis in large tissue volumes with clearing-enhanced 3D microscopy (Ce3D). Proc. Natl Acad. Sci. USA 114, E7321–E7330 (2017).
Regev, A. et al. The Human Cell Atlas. eLife https://doi.org/10.7554/eLife.27041 (2017).
HuBMAP Consortium. The human body at cellular resolution: the NIH Human Biomolecular Atlas Program. Nature 574, 187–192 (2019).
Rozenblatt-Rosen, O. et al. The Human Tumor Atlas Network: charting tumor transitions across space and time at single-cell resolution. Cell 181, 236–249 (2020).
The Cancer Genome Atlas Research Network et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet. 45, 1113–1120 (2013).
Shin, D. et al. PathEdEx—uncovering high-explanatory visual diagnostics heuristics using digital pathology and multiscale gaze data. J. Pathol. Inf. 8, 29 (2017).
Wagner, J. et al. A single-cell atlas of the tumor and immune ecosystem of human breast cancer. Cell 177, 1330–1345.e18 (2019).
Bui, M. M. et al. Digital and computational pathology: bring the future into focus. J. Pathol. Inform. 10, 10 (2019).
Brown, M. & Wittwer, C. Flow cytometry: principles and clinical applications in hematology. Clin. Chem. 46, 1221–1229 (2000).
Coons, A. H., Creech, H. J., Jones, R. N. & Berliner, E. The demonstration of pneumococcal antigen in tissues by the use of fluorescent antibody. J. Immunol. 45, 159–170 (1942).
Pantanowitz, L. et al. Twenty years of digital pathology: an overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J. Pathol. Inf. 9, 40 (2018).
Gurcan, M. N. et al. Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009).
Technical Performance Assessment of Digital Pathology Whole Slide Imaging Devices (US Food and Drug Administration, 2016); http://www.fda.gov/regulatory-information/search-fda-guidance-documents/technical-performance-assessment-digital-pathology-whole-slide-imaging-devices
Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).
Greenspan, H., van Ginneken, B. & Summers, R. M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35, 1153–1159 (2016).
Abels, E. et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association. J. Pathol. 249, 286–294 (2019).
Schapiro, D. et al. MITI Minimum Information guidelines for highly multiplexed tissue images. Preprint at https://arxiv.org/abs/2108.09499 (2021).
Goldberg, I. G. et al. The Open Microscopy Environment (OME) data model and XML file: open tools for informatics and quantitative analysis in biological imaging. Genome Biol. 6, R47 (2005).
Hill, E. Announcing the JCB DataViewer, a browser-based application for viewing original image files. J. Cell Biol. 183, 969–970 (2008).
Swedlow, J. R., Goldberg, I., Brauner, E. & Sorger, P. K. Informatics and quantitative analysis in biological imaging. Science 300, 100–102 (2003).
Singh, J. FigShare. J. Pharm. Pharmacother. 2, 138–139 (2011).
Allan, C. et al. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods 9, 245–253 (2012).
Williams, E. et al. The Image Data Resource: a bioimage data integration and publication platform. Nat. Methods 14, 775–781 (2017).
Levit, L. A. et al. Ethical framework for including research biopsies in oncology clinical trials: American Society of Clinical Oncology research statement. J. Clin. Oncol. 37, 2368–2377 (2019).
Kaye, J., Heeney, C., Hawkins, N., de Vries, J. & Boddington, P. Data sharing in genomics—re-shaping scientific practice. Nat. Rev. Genet. 10, 331–335 (2009).
Reardon, J. et al. Bermuda 2.0: reflections from Santa Cruz. Gigascience 5, 1–4 (2016).
Gutman, D. A. et al. The Digital Slide Archive: a software platform for management, integration, and analysis of histology for cancer research. Cancer Res. 77, e75–e78 (2017).
Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).
Robinson, J. T. et al. Integrative genomics viewer. Nat. Biotechnol. 29, 24–26 (2011).
Stephens, G. J., Silbert, L. J. & Hasson, U. Speaker–listener neural coupling underlies successful communication. Proc. Natl Aacd. Sci. USA 107, 14425–14430 (2010).
Rieger, K. L. et al. Digital storytelling as a method in health research: a systematic review protocol. Syst. Rev. 7, 41 (2018).
Wilson, D. K., Hutson, S. P. & Wyatt, T. H. Exploring the role of digital storytelling in pediatric oncology patients’ perspectives regarding diagnosis: a literature review. SAGE Open https://doi.org/10.1177/2158244015572099 (2015).
De Vecchi, N., Kenny, A., Dickson-Swift, V. & Kidd, S. How digital storytelling is used in mental health: a scoping review. Int J. Ment. Health Nurs. 25, 183–193 (2016).
Lee, H., Fawcett, J. & DeMarco, R. Storytelling/narrative theory to address health communication with minority populations. Appl. Nurs. Res 30, 58–60 (2016).
Botsis, T., Fairman, J. E., Moran, M. B. & Anagnostou, V. Visual storytelling enhances knowledge dissemination in biomedical science. J. Biomed. Inf. 107, 103458 (2020).
ElShafie, S. J. Making science meaningful for broad audiences through stories. Integr. Comp. Biol. 58, 1213–1223 (2018).
OpenSeadragon v.2.4.2 (OpenSeadragon contributors, 2013); https://openseadragon.github.io/
Jianu, R. & Laidlaw, D. H. What Google Maps can do for biomedical data dissemination: examples and a design study. BMC Res. Notes 6, 179 (2013).
Hoffer, J. et al. Minerva: a light-weight, narrative image browser for multiplexed tissue images. J. Open Source Softw. 5, 2579 (2020).
Jekyll v.4.2.0 (The Jekyll Team; 2021); https://jekyllrb.com/
Aeffner, F. et al. Introduction to digital image analysis in whole-slide imaging: a white paper from the Digital Pathology Association. J. Pathol. Inf. 10, 9 (2019).
Hiner, M. C., Rueden, C. T. & Eliceiri, K. W. SCIFIO: an extensible framework to support scientific image formats. BMC Bioinformatics 17, 521 (2016).
Rashid, R. et al. Highly multiplexed immunofluorescence images and single-cell data of immune markers in tonsil and lung cancer. Sci. Data 6, 323 (2019).
Rebhan, M., Chalifa-Caspi, V., Prilusky, J. & Lancet, D. GeneCards: integrating information about genes, proteins and diseases. Trends Genet. 13, 163 (1997).
Schapiro, D. et al. histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat. Methods 14, 873–876 (2017).
Krueger, R. et al. Facetto: combining unsupervised and supervised learning for hierarchical phenotype analysis in multi-channel image data. IEEE Trans. Vis. Comput. Graph. 26, 227–237 (2020).
García, M., Victory, N., Navarro-Sempere, A. & Segovia, Y. Students’ views on difficulties in learning histology. Anat. Sci. Educ. 12, 541–549 (2019).
Mione, S., Valcke, M. & Cornelissen, M. Remote histology learning from static versus dynamic microscopic images. Anat. Sci. Educ. 9, 222–230 (2016).
Schapiro, D. et al. MCMICRO: a scalable, modular image-processing pipeline for multiplexed tissue imaging. Preprint at bioRxiv https://doi.org/10.1101/2021.03.15.435473 (2021).
O’Connor, B. D. et al. The Dockstore: enabling modular, community-focused sharing of Docker-based genomics tools and workflows. F1000Res 6, 52 (2017).
Di Tommaso, P. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol. 35, 316–319 (2017).
Siepel, A. Challenges in funding and developing genomic software: roots and remedies. Genome Biol. 20, 147 (2019).
Macaulay, I. C., Ponting, C. P. & Voet, T. Single-cell multiomics: multiple measurements from single cells. Trends Genet. 33, 155–168 (2017).
Trapnell, C. Defining cell types and states with single-cell genomics. Genome Res. 25, 1491–1498 (2015).
Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).
Schulz, D. et al. Simultaneous multiplexed imaging of mRNA and proteins with subcellular resolution in breast cancer tissue samples by mass cytometry. Cell Syst. 6, 25–36.e5 (2018).
Van de Plas, R., Yang, J., Spraggins, J. & Caprioli, R. M. Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping. Nat. Methods 12, 366–372 (2015).
Liu, X. et al. Molecular imaging of drug transit through the blood–brain barrier with MALDI mass spectrometry imaging. Sci. Rep. 3, 2859 (2013).
Moncada, R. et al. Integrating microarray-based spatial transcriptomics and single-cell RNA-seq reveals tissue architecture in pancreatic ductal adenocarcinomas. Nat. Biotechnol. 38, 333–342 (2020).
Cerami, E. et al. The cBio Cancer Genomics Portal: an open platform for exploring multidimensional cancer genomics data. Cancer Discov. 2, 401–404 (2012).
Mildenberger, P., Eichelberg, M. & Martin, E. Introduction to the DICOM standard. Eur. Radio. 12, 920–927 (2002).
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3, 210–229 (1959).
Inoué, S. & Spring, K. Video Microscopy: The Fundamentals (Springer US, 1997).
Bacus, J. V. & Bacus, J. W. Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope. US patent 6,101,265 (2000).
HANDEL-P (Pancreatlas, 2021); https://pancreatlas.org/
Rubin, D. L., Greenspan, H. & Brinkley, J. F. in Biomedical Informatics: Computer Applications in Health Care and Biomedicine (eds Shortliffe, E. H. & Cimino, J. J.) 285–327 (Springer, 2014).
caMicroscope (GitHub, 2021); https://github.com/camicroscope
Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013).
PathPresenter (Singh, R. et al., 2021); https://public.pathpresenter.net/#/login
Olson, A. H. Image Analysis Using the Aperio ScanScope (Quorum Technologies, 2006).
McQuin, C. et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).
Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).
Stritt, M., Stalder, A. K. & Vezzali, E. Orbit Image Analysis: an open-source whole slide image analysis tool. PLoS Comput. Biol. 16, e1007313 (2020).
Mantis Viewer (Parker Institute for Cancer Immunotherapy, 2021); https://mantis.parkerici.org/
ASAP—Automated Slide Analysis Platform (Computation Pathology Group—Radboud University Medical Center, 2021); https://computationalpathologygroup.github.io/ASAP/#home
Maaten, Lvander & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at https://arxiv.org/abs/1802.03426 (2018).
Arnol, D., Schapiro, D., Bodenmiller, B., Saez-Rodriguez, J. & Stegle, O. Modeling cell–cell interactions from spatial molecular data with spatial variance component analysis. Cell Rep. 29, 202–211.e6 (2019).
Berg, S. et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).
Chen, J. et al. The Allen Cell Structure Segmenter: a new open source toolkit for segmenting 3D intracellular structures in fluorescence microscopy images. Preprint at bioRxiv https://doi.org/10.1101/491035 (2018).
Sofroniew, N. et al. napari/napari: 0.3.6rc2. Zenodo https://doi.org/10.5281/zenodo.3951241 (2020).
Schindelin, J., Rueden, C. T., Hiner, M. C. & Eliceiri, K. W. The ImageJ ecosystem: an open platform for biomedical image analysis. Mol. Reprod. Dev. 82, 518–529 (2015).
This work was funded by NIH grants U54-CA225088 to P.K.S. and S.S., and by the Ludwig Center at Harvard. The Dana-Farber/Harvard Cancer Center is supported in part by NCI Cancer Center Support Grant P30-CA06516.
P.K.S. is a member of the SAB and BOD member of Applied Biomath, RareCyte Inc., and Glencoe Software, which distributes a commercial version of the OMERO database. P.K.S. is also a member of the NanoString SAB. In the past 5 years, the Sorger Laboratory has received research funding from Novartis and Merck. P.K.S. declares that none of these relationships have influenced the content of this manuscript. S.S. is a consultant for RareCyte Inc. The remaining authors declare no competing interests.
Peer review information Nature Biomedical Engineering thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rashid, R., Chen, YA., Hoffer, J. et al. Narrative online guides for the interpretation of digital-pathology images and tissue-atlas data. Nat. Biomed. Eng 6, 515–526 (2022). https://doi.org/10.1038/s41551-021-00789-8
This article is cited by
Nature Cell Biology (2022)
Nature Methods (2022)