Automated molecular-image cytometry and analysis in modern oncology

Abstract

Diagnostic methods for initial diagnosis and patient stratification for treatment are key to modern oncology, but many challenges remain. In developed countries, advances in early diagnosis and therapeutics have led to challenges in the sampling of sub-centimetre lesions, with repeat biopsies straining accuracy and throughput of pathological assessment. Conversely, low-income and middle-income countries face extremely limited pathology and imaging resources, large caseloads, convoluted and inefficient workflows, and a lack of specialists. Advances in material sciences, chemistry, engineering and artificial intelligence have led to the introduction of a new class of affordable image cytometers that enable automated cell phenotyping, with ongoing clinical testing indicating that these systems can alleviate existing bottlenecks and improve diagnostic efficiency. Ultimately, these diagnostic methods are likely to surpass current pathology approaches on the basis of the richness of molecular measurements and the fact that they require only scant cellular material, rather than tissue sections. As these methods can be miniaturized and are low-power, they can also be used in point-of-care settings. In this Review, we focus on new devices and approaches for the integrated analysis of scant cancer samples, particularly those obtained by fine-needle aspiration.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Overview of automated molecular-image cytometry.
Fig. 2: Cyclic labelling technologies for multiplexed cancer-marker and host-cell-marker assessment.
Fig. 3: LDIH.
Fig. 4: FPM.
Fig. 5: Miniaturized fluorescent cytometers.
Fig. 6: Machine learning in imaging analyses.

References

  1. 1.

    Weissleder, R., Schwaiger, M. C., Gambhir, S. S. & Hricak, H. Imaging approaches to optimize molecular therapies. Sci. Transl Med. 8, 355ps16 (2016).

  2. 2.

    Frenk, N. E. et al. High-content biopsies facilitate molecular analyses and do not increase complication rates in patients with advanced solid tumors. JCO Precis. Oncol. 1, 1–9 (2017).

  3. 3.

    Capitanio, A., Dina, R. E. & Treanor, D. Digital cytology: A short review of technical and methodological approaches and applications. Cytopathology 29, 317–325 (2018).

  4. 4.

    Asthana, V. et al. An inexpensive, customizable microscopy system for the automated quantification and characterization of multiple adherent cell types. PeerJ 6, e4937 (2018).

  5. 5.

    Balsam, J., Bruck, H. A. & Rasooly, A. Mobile flow cytometer for mHealth. Methods Mol. Biol. 1256, 139–153 (2015).

  6. 6.

    Molnár, B. et al. Circulating cell-free nucleic acids as biomarkers in colorectal cancer screening and diagnosis — an update. Expert Rev. Mol. Diagn. 19, 477–498 (2019).

  7. 7.

    Shao, H. et al. New technologies for analysis of extracellular vesicles. Chem. Rev. 118, 1917–1950 (2018).

  8. 8.

    Sandlin, R. D. et al. Ultra-fast vitrification of patient-derived circulating tumor cell lines. PLOS ONE 13, e0192734 (2018).

  9. 9.

    Sarioglu, A. F. et al. A microfluidic device for label-free, physical capture of circulating tumor cell clusters. Nat. Methods 12, 685–691 (2015).

  10. 10.

    Cristofanilli, M. et al. The clinical use of circulating tumor cells (CTCs) enumeration for staging of metastatic breast cancer (MBC): International expert consensus paper. Crit. Rev. Oncol. Hematol. 134, 39–45 (2019).

  11. 11.

    No authors listed. First comprehensive companion diagnostic OK’d. Cancer Discov. 8, OF4 (2018).

  12. 12.

    Van Hoeck, A., Tjoonk, N. H., van Boxtel, R. & Cuppen, E. Portrait of a cancer: mutational signature analyses for cancer diagnostics. BMC Cancer 19, 457 (2019).

  13. 13.

    Hannouf, M. B. et al. Cost-effectiveness analysis of multigene expression profiling assays to guide adjuvant therapy decisions in women with invasive early-stage breast cancer. Pharmacogenomics J. 20, 27–46 (2020).

  14. 14.

    Saha, M., Mukherjee, R. & Chakraborty, C. Computer-aided diagnosis of breast cancer using cytological images: a systematic review. Tissue Cell 48, 461–474 (2016).

  15. 15.

    Filipczuk, P., Fevens, T., Krzyzak, A. & Monczak, R. Computer-aided breast cancer diagnosis based on the analysis of cytological images of fine needle biopsies. IEEE Trans. Med. Imaging 32, 2169–2178 (2013).

  16. 16.

    Dey, P., Logasundaram, R. & Joshi, K. Artificial neural network in diagnosis of lobular carcinoma of breast in fine-needle aspiration cytology. Diagn. Cytopathol. 41, 102–106 (2013).

  17. 17.

    Landau, M. A. & Pantanowitz, L. Artificial intelligence in cytopathology: a review of the literature and overview of commercial landscape. J. Am. Soc. Cytopathol. 8, 230–241 (2019).

  18. 18.

    Stephens, L., Bevins, N. J., Bengtsson, H. I. & Broome, H. E. Comparison of different small clinical hematology laboratory configurations with focus on remote smear imaging. Arch. Pathol. Lab. Med. 143, 1234–1245 (2019).

  19. 19.

    Andrade, A. R. et al. Recent computational methods for white blood cell nuclei segmentation: a comparative study. Comput. Methods Prog. Biomed. 173, 1–14 (2019).

  20. 20.

    Prieto, S. P., Powless, A. J., Boice, J. W., Sharma, S. G. & Muldoon, T. J. Proflavine hemisulfate as a fluorescent contrast agent for point-of-care cytology. PLOS ONE 10, e0125598 (2015).

  21. 21.

    Lin, J. R., Fallahi-Sichani, M. & Sorger, P. K. Highly multiplexed imaging of single cells using a high-throughput cyclic immunofluorescence method. Nat. Commun. 6, 8390 (2015).

  22. 22.

    Gerdes, M. J. et al. Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc. Natl Acad. Sci. USA 110, 11982–11987 (2013).

  23. 23.

    Schubert, W. et al. Analyzing proteome topology and function by automated multidimensional fluorescence microscopy. Nat. Biotechnol. 24, 1270–1278 (2006).

  24. 24.

    Giedt, R. J. et al. Single-cell barcode analysis provides a rapid readout of cellular signaling pathways in clinical specimens. Nat. Commun. 9, 4550 (2018).

  25. 25.

    Ullal, A. V. et al. Cancer cell profiling by barcoding allows multiplexed protein analysis in fine-needle aspirates. Sci. Transl Med. 6, 219ra9 (2014).

  26. 26.

    Agasti, S. S., Liong, M., Peterson, V. M., Lee, H. & Weissleder, R. Photocleavable DNA barcode–antibody conjugates allow sensitive and multiplexed protein analysis in single cells. J. Am. Chem. Soc. 134, 18499–18502 (2012).

  27. 27.

    Kishi, J. Y. et al. SABER amplifies FISH: enhanced multiplexed imaging of RNA and DNA in cells and tissues. Nat. Methods 16, 533–544 (2019).

  28. 28.

    De Wit, S. et al. Classification of cells in CTC-enriched samples by advanced image analysis. Cancers 10, 377 (2018).

  29. 29.

    Haun, J. B. et al. Micro-NMR for rapid molecular analysis of human tumor samples. Sci. Transl Med. 3, 71ra16 (2011).

  30. 30.

    Yang, K. S. et al. Multiparametric plasma EV profiling facilitates diagnosis of pancreatic malignancy. Sci. Transl Med. 9, eaal3226 (2017).

  31. 31.

    Min, J. et al. Computational optics enables breast cancer profiling in point-of-care settings. ACS Nano 12, 9081–9090 (2018).

  32. 32.

    Im, H. et al. Design and clinical validation of a point-of-care device for the diagnosis of lymphoma via contrast-enhanced microholography and machine learning. Nat. Biomed. Eng. 2, 666–674 (2018).

  33. 33.

    Ghazani, A. A. et al. Molecular characterization of scant lung tumor cells using iron-oxide nanoparticles and micro-nuclear magnetic resonance. Nanomedicine 10, 661–668 (2014).

  34. 34.

    Peterson, V. M. et al. Ascites analysis by a microfluidic chip allows tumor-cell profiling. Proc. Natl Acad. Sci. USA 110, E4978–E4986 (2013).

  35. 35.

    Marquard, A. N., Carlson, J. C. T. & Weissleder, R. Glass chemistry to analyze human cells under adverse conditions. ACS Omega 4, 11515–11521 (2019).

  36. 36.

    Lohmann, A. W., Dorsch, R. G., Mendlovic, D., Ferreira, C. & Zalevsky, Z. Space–bandwidth product of optical signals and systems. J. Opt. Soc. Am. A 13, 470–473 (1996).

  37. 37.

    Van Es, S. L. et al. Constant quest for quality: digital cytopathology. J. Pathol. Inform. 9, 13 (2018).

  38. 38.

    Hu, B., Bolus, D. & Brown, J. Q. Improved contrast in inverted selective plane illumination microscopy of thick tissues using confocal detection and structured illumination. Biomed. Opt. Express 8, 5546–5559 (2017).

  39. 39.

    Wang, M. et al. Gigapixel surface imaging of radical prostatectomy specimens for comprehensive detection of cancer-positive surgical margins using structured illumination microscopy. Sci. Rep. 6, 27419 (2016).

  40. 40.

    Garcia-Sucerquia, J. et al. Digital in-line holographic microscopy. Appl. Opt. 45, 836–850 (2006).

  41. 41.

    Gurkan, U. A. et al. Miniaturized lensless imaging systems for cell and microorganism visualization in point-of-care testing. Biotechnol. J. 6, 138–149 (2011).

  42. 42.

    Zheng, G., Lee, S. A., Antebi, Y., Elowitz, M. B. & Yang, C. The ePetri dish, an on-chip cell imaging platform based on subpixel perspective sweeping microscopy (SPSM). Proc. Natl Acad. Sci. USA 108, 16889–16894 (2011).

  43. 43.

    Kim, S. B. et al. A cell-based biosensor for real-time detection of cardiotoxicity using lensfree imaging. Lab. Chip 11, 1801–1807 (2011).

  44. 44.

    Greenbaum, A. et al. Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy. Nat. Methods 9, 889–895 (2012).

  45. 45.

    Im, H. et al. Digital diffraction analysis enables low-cost molecular diagnostics on a smartphone. Proc. Natl Acad. Sci. USA 112, 5613–5618 (2015).

  46. 46.

    Rostykus, M., Soulez, F., Unser, M. & Moser, C. Compact in-line lensfree digital holographic microscope. Methods 136, 17–23 (2018).

  47. 47.

    Rappaz, B. et al. Comparative study of human erythrocytes by digital holographic microscopy, confocal microscopy, and impedance volume analyzer. Cytometry A 73, 895–903 (2008).

  48. 48.

    Seo, S. et al. High-throughput lens-free blood analysis on a chip. Anal. Chem. 82, 4621–4627 (2010).

  49. 49.

    Mudanyali, O., Bishara, W. & Ozcan, A. Lensfree super-resolution holographic microscopy using wetting films on a chip. Opt. Express 19, 17378–17389 (2011).

  50. 50.

    Lee, S. A. et al. Color capable sub-pixel resolving optofluidic microscope and its application to blood cell imaging for malaria diagnosis. PLOS ONE 6, e26127 (2011).

  51. 51.

    Kim, S. J. et al. Deep transfer learning-based hologram classification for molecular diagnostics. Sci. Rep. 8, 17003 (2018).

  52. 52.

    Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).

  53. 53.

    Greenbaum, A. & Ozcan, A. Maskless imaging of dense samples using pixel super-resolution based multi-height lensfree on-chip microscopy. Opt. Express 20, 3129–3143 (2012).

  54. 54.

    Luo, W., Zhang, Y., Feizi, A., Göröcs, Z. & Ozcan, A. Pixel super-resolution using wavelength scanning. Light Sci. Appl. 5, e16060 (2016).

  55. 55.

    Bao, P., Situ, G., Pedrini, G. & Osten, W. Lensless phase microscopy using phase retrieval with multiple illumination wavelengths. Appl. Opt. 51, 5486–5494 (2012).

  56. 56.

    Ou, X., Horstmeyer, R., Zheng, G. & Yang, C. High numerical aperture Fourier ptychography: principle, implementation and characterization. Opt. Express 23, 3472–3491 (2015).

  57. 57.

    Tian, L. et al. Computational illumination for high-speed in vitro Fourier ptychographic microscopy. Optica 2, 904–911 (2015).

  58. 58.

    Nguyen, T., Xue, Y., Li, Y., Tian, L. & Nehmetallah, G. Deep learning approach for Fourier ptychography microscopy. Opt. Express 26, 26470–26484 (2018).

  59. 59.

    Zheng, G., Horstmeyer, R. & Yang, C. Wide-field, high-resolution Fourier ptychographic microscopy. Nat. Photonics 7, 739–745 (2013).

  60. 60.

    Horstmeyer, R., Chung, J., Ou, X., Zheng, G. & Yang, C. Diffraction tomography with Fourier ptychography. Optica 3, 827–835 (2016).

  61. 61.

    Dong, S. et al. Aperture-scanning Fourier ptychography for 3D refocusing and super-resolution macroscopic imaging. Opt. Express 22, 13586–13599 (2014).

  62. 62.

    Ghosh, K. K. et al. Miniaturized integration of a fluorescence microscope. Nat. Methods 8, 871–878 (2011).

  63. 63.

    Liberti, W. A., Perkins, L. N., Leman, D. P. & Gardner, T. J. An open source, wireless capable miniature microscope system. J. Neural Eng. 14, 045001 (2017).

  64. 64.

    Jacob, A. D. et al. A compact head-mounted endoscope for in vivo calcium imaging in freely behaving mice. Curr. Protoc. Neurosci. 84, e51 (2018).

  65. 65.

    Aharoni, D. & Hoogland, T. M. Circuit investigations with open-source miniaturized microscopes: past, present and future. Front. Cell. Neurosci. 13, 141 (2019).

  66. 66.

    Helmchen, F., Fee, M. S., Tank, D. W. & Denk, W. A miniature head-mounted two-photon microscope. High-resolution brain imaging in freely moving animals. Neuron 31, 903–912 (2001).

  67. 67.

    Skocek, O. et al. High-speed volumetric imaging of neuronal activity in freely moving rodents. Nat. Methods 15, 429–432 (2018).

  68. 68.

    Adams, J. K. et al. Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope. Sci. Adv. 3, e1701548 (2017).

  69. 69.

    Ah Lee, S., Ou, X., Lee, J. E. & Yang, C. Chip-scale fluorescence microscope based on a silo-filter complementary metal-oxide semiconductor image sensor. Opt. Lett. 38, 1817–1819 (2013).

  70. 70.

    Almada, P. et al. Automating multimodal microscopy with NanoJ-Fluidics. Nat. Commun. 10, 1223 (2019).

  71. 71.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  72. 72.

    LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

  73. 73.

    Cao, C. et al. Deep learning and its applications in biomedicine. Genom. Proteom. Bioinform. 16, 17–32 (2018).

  74. 74.

    Abadi, M. et al. TensorFlow: a system for large-scale machine learning. Preprint at arXiv https://arxiv.org/abs/1605.08695 (2016).

  75. 75.

    Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. Med. Image Comput. Comput. Assist. Interv. 9351, 234–241 (2015).

  76. 76.

    Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2018).

  77. 77.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1, 1097–1105 (2012).

  78. 78.

    He, K., Zhang, X., Ren, S. & Sun, J. in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 770–778 (IEEE, 2016).

  79. 79.

    Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at arXiv https://arxiv.org/abs/1409.1556 (2014).

  80. 80.

    Szegedy, C. et al. in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1–9 (IEEE, 2015).

  81. 81.

    Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

  82. 82.

    Kamentsky, L. et al. Improved structure, function and compatibility for CellProfiler: modular high-throughput image analysis software. Bioinformatics 27, 1179–1180 (2011).

  83. 83.

    Held, M. et al. CellCognition: time-resolved phenotype annotation in high-throughput live cell imaging. Nat. Methods 7, 747–754 (2010).

  84. 84.

    Murphy, R. F. CellOrganizer: image-derived models of subcellular organization and protein distribution. Methods Cell Biol. 110, 179–193 (2012).

  85. 85.

    Sommer, C., Straehle, C., Koethe, U. & Hamprecht, F. A. in Proc. IEEE Int. Symp. Biomed. Imaging 230–233 (IEEE, 2011).

  86. 86.

    Kvilekval, K., Fedorov, D., Obara, B., Singh, A. & Manjunath, B. S. Bisque: a platform for bioimage analysis and management. Bioinformatics 26, 544–552 (2010).

  87. 87.

    Thrall, M. J. Automated screening of Papanicolaou tests: a review of the literature. Diagn. Cytopathol. 47, 20–27 (2019).

  88. 88.

    Kashyap, A., Jain, M., Shukla, S. & Andley, M. Study of nuclear morphometry on cytology specimens of benign and malignant breast lesions: a study of 122 cases. J. Cytol. 34, 10–15 (2017).

  89. 89.

    Subbaiah, R. M., Dey, P. & Nijhawan, R. Artificial neural network in breast lesions from fine-needle aspiration cytology smear. Diagn. Cytopathol. 42, 218–224 (2014).

  90. 90.

    Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

  91. 91.

    Buggenthin, F. et al. Prospective identification of hematopoietic lineage choice by deep learning. Nat. Methods 14, 403–406 (2017).

  92. 92.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  93. 93.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

  94. 94.

    Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLOS Comput. Biol. 12, e1005177 (2016).

  95. 95.

    Heo, Y. J., Lee, D., Kang, J., Lee, K. & Chung, W. K. Real-time image processing for microscopy-based label-free imaging flow cytometry in a microfluidic chip. Sci. Rep. 7, 11651 (2017).

  96. 96.

    Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803.e19 (2018).

  97. 97.

    Wang, C. et al. Deconvolution of subcellular protrusion heterogeneity and the underlying actin regulator dynamics from live cell imaging. Nat. Commun. 9, 1688 (2018).

  98. 98.

    Varmus, H. & Kumar, H. S. Addressing the growing international challenge of cancer: a multinational perspective. Sci. Transl Med. 5, 175cm2 (2013).

  99. 99.

    Livingston, J. Cancer in the shadow of the AIDS epidemic in southern Africa. Oncologist 18, 783–786 (2013).

  100. 100.

    Chabner, B. A. et al. Cancer in Botswana: the second wave of AIDS in Sub-Saharan Africa. Oncologist 18, 777–778 (2013).

  101. 101.

    Naresh, K. N. et al. Lymphomas in sub-Saharan Africa – what can we learn and how can we help in improving diagnosis, managing patients and fostering translational research. Br. J. Haematol. 154, 696–703 (2011).

  102. 102.

    Mwamba, P. M. et al. AIDS-related non-Hodgkin’s lymphoma in sub-Saharan Africa: current status and realities of therapeutic approach. Lymphoma https://doi.org/10.1155/2012/904367 (2012).

  103. 103.

    Pai, S. I. & Westra, W. H. Molecular pathology of head and neck cancer: implications for diagnosis, prognosis, and treatment. Annu. Rev. Pathol. 4, 49–70 (2009).

  104. 104.

    Pai, S. I. et al. Comparative analysis of the phase III clinical trials of anti-PD1 monotherapy in head and neck squamous cell carcinoma patients (CheckMate 141 and KEYNOTE 040). J. Immunother. Cancer 7, 96 (2019).

  105. 105.

    Carney, B. et al. Target engagement imaging of PARP inhibitors in small-cell lung cancer. Nat. Commun. 9, 176 (2018).

  106. 106.

    Kossatz, S. et al. Detection and delineation of oral cancer with a PARP1 targeted optical imaging agent. Sci. Rep. 6, 21371 (2016).

  107. 107.

    Kossatz, S., Weber, W. & Reiner, T. Detection and delineation of oral cancer with a PARP1-targeted optical imaging agent. Mol. Imaging 16, 1536012117723786 (2017).

  108. 108.

    Kossatz, S. et al. PARP1 as a biomarker for early detection and intraoperative tumor delineation in epithelial cancers–first-in-human results. Preprint at bioRxiv https://doi.org/10.1101/663385 (2019).

  109. 109.

    Couzin-Frankel, J. Medicine contends with how to use artificial intelligence. Science 364, 1119–1120 (2019).

  110. 110.

    Goodsaid, F. M. The labyrinth of product development and regulatory approvals in liquid biopsy diagnostics. Clin. Transl. Sci. 12, 431–439 (2019).

  111. 111.

    Pantanowitz, L. et al. Twenty years of digital pathology: an overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. J. Pathol. Inform. 9, 40 (2018).

  112. 112.

    Kozma, P., Kehl, F., Ehrentreich-Förster, E., Stamm, C. & Bier, F. F. Integrated planar optical waveguide interferometer biosensors: a comparative review. Biosens. Bioelectron. 58, 287–307 (2014).

  113. 113.

    He, Z., Lee, Y. H., Chanda, D. & Wu, S. T. Adaptive liquid crystal microlens array enabled by two-photon polymerization. Opt. Express 26, 21184–21193 (2018).

  114. 114.

    Kuznetsov, A. I. et al. Laser-induced transfer of metallic nanodroplets for plasmonics and metamaterial applications. J. Opt. Soc. Am. B 26, B130–B138 (2009).

  115. 115.

    Lin, R. J. et al. Achromatic metalens array for full-colour light-field imaging. Nat. Nanotechnol. 14, 227–231 (2019).

  116. 116.

    Wolfer, T., Bollgruen, P., Mager, D., Overmeyer, L. & Korvink, J. G. Printing and preparation of integrated optical waveguides for optronic sensor networks. Mechatronics 34, 119–127 (2016).

  117. 117.

    Hui, H. Y. L. et al. “Immuno-flowFISH” for the assessment of cytogenetic abnormalities in chronic lymphocytic leukemia. Cytometry A 95, 521–533 (2019).

  118. 118.

    Mazzini, G. & Danova, M. Fluorochromes for DNA staining and quantitation. Methods Mol. Biol. 1560, 239–259 (2017).

  119. 119.

    Ng, B. L., Fu, B., Graham, J., Hall, C. & Thompson, S. Chromosome analysis using benchtop flow analysers and high speed cell sorters. Cytometry A 95, 323–331 (2019).

  120. 120.

    Smith, P. J., Darzynkiewicz, Z. & Errington, R. J. Nuclear cytometry and chromatin organization. Cytometry A 93, 771–784 (2018).

  121. 121.

    Gupta, A. et al. Deep learning in image cytometry: a review. Cytometry A 95, 366–380 (2019).

  122. 122.

    Jarrett, K., Kavukcuoglu, K., Ranzato, M. A. & LeCun, Y. in IEEE 12th Int. Conf. Comput. Vis. 2146–2153 (IEEE, 2009).

  123. 123.

    Nair, V. & Hinton, G. E. in Proc. 27th Int. Conf. Mach. Learn. 807–814 (2010).

  124. 124.

    Bridle, J. S. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Adv. Neural Inf. Process. Syst. 2, 211–217 (1990).

  125. 125.

    Zhou, Y. T. & Chellappa, R. in IEEE Int. Conf. Neural Netw. 71–78 (IEEE, 1988).

  126. 126.

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).

  127. 127.

    Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? Adv. Neural Inf. Process. Syst. 27, 3320–3328 (2014).

Download references

Acknowledgements

The authors acknowledge extensive discussions with L.K. Chin, J. Min, J. Oh, H. Im, T. Rainer and J. Carlson, and thank C. Landeros for discussions on machine learning and K. Joyes for editing the manuscript. The authors are indebted to J. Higgins and C. Vinegoni for the critical review of the manuscript. The authors are supported by the following grants: NIH-UH3 CA202637, NIH-U01CA206997, NIH-R01CA204019 and NIH-R01CA206890 (R.W.); NIH-R01CA229777, NIH-U01CA233360, DoD-W81XWH1910199, DoD-W81XWH1910194 and MGH Scholar Fund (H.L.).

Author information

Both authors contributed equally to the preparation of this review.

Correspondence to Ralph Weissleder.

Ethics declarations

Competing interests

R.W. declares that he has received consultancy payments from Tarveda Pharmaceuticals, ModeRNA, Alivio Therapeutics and Accure Health, and that he is a shareholder of T2 Biosystems, Lumicell, Accure Health and Aikili Biosystems. H.L. declares that he has received consultancy payments from Exosome Diagnostics and Accure Health, and that he is a shareholder of Accure Health and Aikili Biosystems. Patents: all patents associated with R.W. and H.L. have been assigned to and handled by Massachusetts General Hospital.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Weissleder, R., Lee, H. Automated molecular-image cytometry and analysis in modern oncology. Nat Rev Mater (2020). https://doi.org/10.1038/s41578-020-0180-6

Download citation