Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Narrative online guides for the interpretation of digital-pathology images and tissue-atlas data

Abstract

Multiplexed tissue imaging facilitates the diagnosis and understanding of complex disease traits. However, the analysis of such digital images heavily relies on the experience of anatomical pathologists for the review, annotation and description of tissue features. In addition, the wider use of data from tissue atlases in basic and translational research and in classrooms would benefit from software that facilitates the easy visualization and sharing of the images and the results of their analyses. In this Perspective, we describe the ecosystem of software available for the analysis of tissue images and discuss the need for interactive online guides that help histopathologists make complex images comprehensible to non-specialists. We illustrate this idea via a software interface (Minerva), accessible via web browsers, that integrates multi-omic and tissue-atlas features. We argue that such interactive narrative guides can effectively disseminate digital histology data and aid their interpretation.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Milestones in the development of histopathology, image processing and microscopy.
Fig. 2: Software used to visualize, analyse, manage and share tissue images.
Fig. 3: A system for generating and viewing online narrative guides for histopathology tissue images.
Fig. 4: The key features of the user interface of Minerva Story.
Fig. 5: Minerva Story for medical education.

References

  1. 1.

    Giesen, C. et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11, 417–422 (2014).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  2. 2.

    Angelo, M. et al. Multiplexed ion beam imaging of human breast tumors. Nat. Med. 20, 436–442 (2014).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  3. 3.

    Gerdes, M. J. et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc. Natl Acad. Sci. USA 110, 11982–11987 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  4. 4.

    Lin, J.-R. et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. eLife https://doi.org/10.7554/eLife.31657 (2018).

  5. 5.

    Bodenmiller, B. Multiplexed epitope-based tissue imaging for discovery and healthcare applications. Cell Syst. 2, 225–238 (2016).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  6. 6.

    Coy, S. et al. Multiplexed immunofluorescence reveals potential PD-1/PD-L1 pathway vulnerabilities in craniopharyngioma. Neuro Oncol. 20, 1101–1112 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  7. 7.

    Goltsev, Y. et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell 174, 968–981.e15 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  8. 8.

    Stack, E. C., Wang, C., Roman, K. A. & Hoyt, C. C. Multiplexed immunohistochemistry, imaging, and quantitation: a review, with an assessment of tyramide signal amplification, multispectral imaging and multiplex analysis. Methods 70, 46–58 (2014).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  9. 9.

    Akturk, G., Sweeney, R., Remark, R., Merad, M. & Gnjatic, S. Multiplexed immunohistochemical consecutive staining on single slide (MICSSS): multiplexed chromogenic IHC assay for high-dimensional tissue analysis. Methods Mol. Biol. 2055, 497–519 (2020).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  10. 10.

    Saka, S. K. et al. Immuno-SABER enables highly multiplexed and amplified protein imaging in tissues. Nat. Biotechnol. 37, 1080–1090 (2019).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  11. 11.

    Li, W., Germain, R. N. & Gerner, M. Y. Multiplex, quantitative cellular analysis in large tissue volumes with clearing-enhanced 3D microscopy (Ce3D). Proc. Natl Acad. Sci. USA 114, E7321–E7330 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  12. 12.

    Regev, A. et al. The Human Cell Atlas. eLife https://doi.org/10.7554/eLife.27041 (2017).

  13. 13.

    HuBMAP Consortium. The human body at cellular resolution: the NIH Human Biomolecular Atlas Program. Nature 574, 187–192 (2019).

    CAS  Article  Google Scholar 

  14. 14.

    Rozenblatt-Rosen, O. et al. The Human Tumor Atlas Network: charting tumor transitions across space and time at single-cell resolution. Cell 181, 236–249 (2020).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  15. 15.

    The Cancer Genome Atlas Research Network et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet. 45, 1113–1120 (2013).

    Article  CAS  Google Scholar 

  16. 16.

    Shin, D. et al. PathEdEx—uncovering high-explanatory visual diagnostics heuristics using digital pathology and multiscale gaze data. J. Pathol. Inf. 8, 29 (2017).

    Article  Google Scholar 

  17. 17.

    Wagner, J. et al. A single-cell atlas of the tumor and immune ecosystem of human breast cancer. Cell 177, 1330–1345.e18 (2019).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  18. 18.

    Bui, M. M. et al. Digital and computational pathology: bring the future into focus. J. Pathol. Inform. 10, 10 (2019).

    PubMed Central  Google Scholar 

  19. 19.

    Brown, M. & Wittwer, C. Flow cytometry: principles and clinical applications in hematology. Clin. Chem. 46, 1221–1229 (2000).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  20. 20.

    Coons, A. H., Creech, H. J., Jones, R. N. & Berliner, E. The demonstration of pneumococcal antigen in tissues by the use of fluorescent antibody. J. Immunol. 45, 159–170 (1942).

    CAS  Google Scholar 

  21. 21.

    Pantanowitz, L. et al. Twenty years of digital pathology: an overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J. Pathol. Inf. 9, 40 (2018).

    Article  Google Scholar 

  22. 22.

    Gurcan, M. N. et al. Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009).

    PubMed  PubMed Central  Article  Google Scholar 

  23. 23.

    Technical Performance Assessment of Digital Pathology Whole Slide Imaging Devices (US Food and Drug Administration, 2016); http://www.fda.gov/regulatory-information/search-fda-guidance-documents/technical-performance-assessment-digital-pathology-whole-slide-imaging-devices

  24. 24.

    Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  25. 25.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

    PubMed  Article  PubMed Central  Google Scholar 

  26. 26.

    Greenspan, H., van Ginneken, B. & Summers, R. M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35, 1153–1159 (2016).

    Article  Google Scholar 

  27. 27.

    Abels, E. et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association. J. Pathol. 249, 286–294 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  28. 28.

    Schapiro, D. et al. MITI Minimum Information guidelines for highly multiplexed tissue images. Preprint at https://arxiv.org/abs/2108.09499 (2021).

  29. 29.

    Goldberg, I. G. et al. The Open Microscopy Environment (OME) data model and XML file: open tools for informatics and quantitative analysis in biological imaging. Genome Biol. 6, R47 (2005).

    PubMed  PubMed Central  Article  Google Scholar 

  30. 30.

    Hill, E. Announcing the JCB DataViewer, a browser-based application for viewing original image files. J. Cell Biol. 183, 969–970 (2008).

    CAS  PubMed Central  Article  Google Scholar 

  31. 31.

    Swedlow, J. R., Goldberg, I., Brauner, E. & Sorger, P. K. Informatics and quantitative analysis in biological imaging. Science 300, 100–102 (2003).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  32. 32.

    Singh, J. FigShare. J. Pharm. Pharmacother. 2, 138–139 (2011).

    Article  Google Scholar 

  33. 33.

    Allan, C. et al. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods 9, 245–253 (2012).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  34. 34.

    Williams, E. et al. The Image Data Resource: a bioimage data integration and publication platform. Nat. Methods 14, 775–781 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  35. 35.

    Levit, L. A. et al. Ethical framework for including research biopsies in oncology clinical trials: American Society of Clinical Oncology research statement. J. Clin. Oncol. 37, 2368–2377 (2019).

    PubMed  Article  PubMed Central  Google Scholar 

  36. 36.

    Kaye, J., Heeney, C., Hawkins, N., de Vries, J. & Boddington, P. Data sharing in genomics—re-shaping scientific practice. Nat. Rev. Genet. 10, 331–335 (2009).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  37. 37.

    Reardon, J. et al. Bermuda 2.0: reflections from Santa Cruz. Gigascience 5, 1–4 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  38. 38.

    Gutman, D. A. et al. The Digital Slide Archive: a software platform for management, integration, and analysis of histology for cancer research. Cancer Res. 77, e75–e78 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  39. 39.

    Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  40. 40.

    Robinson, J. T. et al. Integrative genomics viewer. Nat. Biotechnol. 29, 24–26 (2011).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  41. 41.

    Stephens, G. J., Silbert, L. J. & Hasson, U. Speaker–listener neural coupling underlies successful communication. Proc. Natl Aacd. Sci. USA 107, 14425–14430 (2010).

    CAS  Article  Google Scholar 

  42. 42.

    Rieger, K. L. et al. Digital storytelling as a method in health research: a systematic review protocol. Syst. Rev. 7, 41 (2018).

    PubMed  PubMed Central  Article  Google Scholar 

  43. 43.

    Wilson, D. K., Hutson, S. P. & Wyatt, T. H. Exploring the role of digital storytelling in pediatric oncology patients’ perspectives regarding diagnosis: a literature review. SAGE Open https://doi.org/10.1177/2158244015572099 (2015).

  44. 44.

    De Vecchi, N., Kenny, A., Dickson-Swift, V. & Kidd, S. How digital storytelling is used in mental health: a scoping review. Int J. Ment. Health Nurs. 25, 183–193 (2016).

    PubMed  Article  PubMed Central  Google Scholar 

  45. 45.

    Lee, H., Fawcett, J. & DeMarco, R. Storytelling/narrative theory to address health communication with minority populations. Appl. Nurs. Res 30, 58–60 (2016).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  46. 46.

    Botsis, T., Fairman, J. E., Moran, M. B. & Anagnostou, V. Visual storytelling enhances knowledge dissemination in biomedical science. J. Biomed. Inf. 107, 103458 (2020).

    Article  Google Scholar 

  47. 47.

    ElShafie, S. J. Making science meaningful for broad audiences through stories. Integr. Comp. Biol. 58, 1213–1223 (2018).

    PubMed  Article  PubMed Central  Google Scholar 

  48. 48.

    OpenSeadragon v.2.4.2 (OpenSeadragon contributors, 2013); https://openseadragon.github.io/

  49. 49.

    Jianu, R. & Laidlaw, D. H. What Google Maps can do for biomedical data dissemination: examples and a design study. BMC Res. Notes 6, 179 (2013).

    PubMed  PubMed Central  Article  Google Scholar 

  50. 50.

    Hoffer, J. et al. Minerva: a light-weight, narrative image browser for multiplexed tissue images. J. Open Source Softw. 5, 2579 (2020).

    PubMed  PubMed Central  Article  Google Scholar 

  51. 51.

    Jekyll v.4.2.0 (The Jekyll Team; 2021); https://jekyllrb.com/

  52. 52.

    Aeffner, F. et al. Introduction to digital image analysis in whole-slide imaging: a white paper from the Digital Pathology Association. J. Pathol. Inf. 10, 9 (2019).

    Article  Google Scholar 

  53. 53.

    Hiner, M. C., Rueden, C. T. & Eliceiri, K. W. SCIFIO: an extensible framework to support scientific image formats. BMC Bioinformatics 17, 521 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  54. 54.

    Rashid, R. et al. Highly multiplexed immunofluorescence images and single-cell data of immune markers in tonsil and lung cancer. Sci. Data 6, 323 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  55. 55.

    Rebhan, M., Chalifa-Caspi, V., Prilusky, J. & Lancet, D. GeneCards: integrating information about genes, proteins and diseases. Trends Genet. 13, 163 (1997).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  56. 56.

    Schapiro, D. et al. histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat. Methods 14, 873–876 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  57. 57.

    Krueger, R. et al. Facetto: combining unsupervised and supervised learning for hierarchical phenotype analysis in multi-channel image data. IEEE Trans. Vis. Comput. Graph. 26, 227–237 (2020).

    PubMed  Article  PubMed Central  Google Scholar 

  58. 58.

    García, M., Victory, N., Navarro-Sempere, A. & Segovia, Y. Students’ views on difficulties in learning histology. Anat. Sci. Educ. 12, 541–549 (2019).

    PubMed  Article  PubMed Central  Google Scholar 

  59. 59.

    Mione, S., Valcke, M. & Cornelissen, M. Remote histology learning from static versus dynamic microscopic images. Anat. Sci. Educ. 9, 222–230 (2016).

    PubMed  Article  PubMed Central  Google Scholar 

  60. 60.

    Schapiro, D. et al. MCMICRO: a scalable, modular image-processing pipeline for multiplexed tissue imaging. Preprint at bioRxiv https://doi.org/10.1101/2021.03.15.435473 (2021).

  61. 61.

    O’Connor, B. D. et al. The Dockstore: enabling modular, community-focused sharing of Docker-based genomics tools and workflows. F1000Res 6, 52 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  62. 62.

    Di Tommaso, P. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol. 35, 316–319 (2017).

    PubMed  Article  CAS  PubMed Central  Google Scholar 

  63. 63.

    Siepel, A. Challenges in funding and developing genomic software: roots and remedies. Genome Biol. 20, 147 (2019).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  64. 64.

    Macaulay, I. C., Ponting, C. P. & Voet, T. Single-cell multiomics: multiple measurements from single cells. Trends Genet. 33, 155–168 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  65. 65.

    Trapnell, C. Defining cell types and states with single-cell genomics. Genome Res. 25, 1491–1498 (2015).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  66. 66.

    Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).

    PubMed  Article  CAS  PubMed Central  Google Scholar 

  67. 67.

    Schulz, D. et al. Simultaneous multiplexed imaging of mRNA and proteins with subcellular resolution in breast cancer tissue samples by mass cytometry. Cell Syst. 6, 25–36.e5 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  68. 68.

    Van de Plas, R., Yang, J., Spraggins, J. & Caprioli, R. M. Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping. Nat. Methods 12, 366–372 (2015).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  69. 69.

    Liu, X. et al. Molecular imaging of drug transit through the blood–brain barrier with MALDI mass spectrometry imaging. Sci. Rep. 3, 2859 (2013).

    PubMed  PubMed Central  Article  Google Scholar 

  70. 70.

    Moncada, R. et al. Integrating microarray-based spatial transcriptomics and single-cell RNA-seq reveals tissue architecture in pancreatic ductal adenocarcinomas. Nat. Biotechnol. 38, 333–342 (2020).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  71. 71.

    Cerami, E. et al. The cBio Cancer Genomics Portal: an open platform for exploring multidimensional cancer genomics data. Cancer Discov. 2, 401–404 (2012).

    PubMed  Article  PubMed Central  Google Scholar 

  72. 72.

    Mildenberger, P., Eichelberg, M. & Martin, E. Introduction to the DICOM standard. Eur. Radio. 12, 920–927 (2002).

    Article  Google Scholar 

  73. 73.

    Samuel, A. L. Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3, 210–229 (1959).

    Article  Google Scholar 

  74. 74.

    Inoué, S. & Spring, K. Video Microscopy: The Fundamentals (Springer US, 1997).

  75. 75.

    Bacus, J. V. & Bacus, J. W. Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope. US patent 6,101,265 (2000).

  76. 76.

    HANDEL-P (Pancreatlas, 2021); https://pancreatlas.org/

  77. 77.

    Rubin, D. L., Greenspan, H. & Brinkley, J. F. in Biomedical Informatics: Computer Applications in Health Care and Biomedicine (eds Shortliffe, E. H. & Cimino, J. J.) 285–327 (Springer, 2014).

  78. 78.

    caMicroscope (GitHub, 2021); https://github.com/camicroscope

  79. 79.

    Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013).

    PubMed  PubMed Central  Article  Google Scholar 

  80. 80.

    PathPresenter (Singh, R. et al., 2021); https://public.pathpresenter.net/#/login

  81. 81.

    Olson, A. H. Image Analysis Using the Aperio ScanScope (Quorum Technologies, 2006).

  82. 82.

    McQuin, C. et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  83. 83.

    Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  84. 84.

    Stritt, M., Stalder, A. K. & Vezzali, E. Orbit Image Analysis: an open-source whole slide image analysis tool. PLoS Comput. Biol. 16, e1007313 (2020).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  85. 85.

    Mantis Viewer (Parker Institute for Cancer Immunotherapy, 2021); https://mantis.parkerici.org/

  86. 86.

    ASAP—Automated Slide Analysis Platform (Computation Pathology Group—Radboud University Medical Center, 2021); https://computationalpathologygroup.github.io/ASAP/#home

  87. 87.

    Maaten, Lvander & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    Google Scholar 

  88. 88.

    McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at https://arxiv.org/abs/1802.03426 (2018).

  89. 89.

    Arnol, D., Schapiro, D., Bodenmiller, B., Saez-Rodriguez, J. & Stegle, O. Modeling cell–cell interactions from spatial molecular data with spatial variance component analysis. Cell Rep. 29, 202–211.e6 (2019).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  90. 90.

    Berg, S. et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  91. 91.

    Chen, J. et al. The Allen Cell Structure Segmenter: a new open source toolkit for segmenting 3D intracellular structures in fluorescence microscopy images. Preprint at bioRxiv https://doi.org/10.1101/491035 (2018).

  92. 92.

    Sofroniew, N. et al. napari/napari: 0.3.6rc2. Zenodo https://doi.org/10.5281/zenodo.3951241 (2020).

  93. 93.

    Schindelin, J., Rueden, C. T., Hiner, M. C. & Eliceiri, K. W. The ImageJ ecosystem: an open platform for biomedical image analysis. Mol. Reprod. Dev. 82, 518–529 (2015).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

Download references

Acknowledgements

This work was funded by NIH grants U54-CA225088 to P.K.S. and S.S., and by the Ludwig Center at Harvard. The Dana-Farber/Harvard Cancer Center is supported in part by NCI Cancer Center Support Grant P30-CA06516.

Author information

Affiliations

Authors

Contributions

R.R., Y.-A.C., J.H., J.L.M. and R.K. contributed to researching information for the writing of this article, to the development of the Minerva software and to curating data. R.M. contributed to the content and discussion. H.P. contributed to data visualization. S.S. and P.K.S. contributed to all aspects of the article. All authors contributed to the writing of the manuscript.

Corresponding authors

Correspondence to Sandro Santagata or Peter K. Sorger.

Ethics declarations

Competing interests

P.K.S. is a member of the SAB and BOD member of Applied Biomath, RareCyte Inc., and Glencoe Software, which distributes a commercial version of the OMERO database. P.K.S. is also a member of the NanoString SAB. In the past 5 years, the Sorger Laboratory has received research funding from Novartis and Merck. P.K.S. declares that none of these relationships have influenced the content of this manuscript. S.S. is a consultant for RareCyte Inc. The remaining authors declare no competing interests.

Additional information

Peer review information Nature Biomedical Engineering thanks the anonymous reviewers for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rashid, R., Chen, YA., Hoffer, J. et al. Narrative online guides for the interpretation of digital-pathology images and tissue-atlas data. Nat Biomed Eng (2021). https://doi.org/10.1038/s41551-021-00789-8

Download citation

Search

Quick links