Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Deepometry, a framework for applying supervised and weakly supervised deep learning to imaging cytometry

Nature Protocols volume 16, pages 3572–3595 (2021)Cite this article

Subjects

Abstract

Deep learning offers the potential to extract more than meets the eye from images captured by imaging flow cytometry. This protocol describes the application of deep learning to single-cell images to perform supervised cell classification and weakly supervised learning, using example data from an experiment exploring red blood cell morphology. We describe how to acquire and transform suitable input data as well as the steps required for deep learning training and inference using an open-source web-based application. All steps of the protocol are provided as open-source Python as well as MATLAB runtime scripts, through both command-line and graphic user interfaces. The protocol enables a flexible and friendly environment for morphological phenotyping using supervised and weakly supervised learning and the subsequent exploration of the deep learning features using multi-dimensional visualization tools. The protocol requires 40 h when training from scratch and 1 h when using a pre-trained model.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A comparison of the Deepometry protocol with traditional machine learning approaches.
Fig. 2: The overall workflow of the Deepometry procedure.
Fig. 3: Screenshot of the Deepometry GUI (Python version).
Fig. 4: Screenshot of the Deepometry GUI (MATLAB version).
Fig. 5: Example data structure for use in Deepometry.
Fig. 6: A typical montage image generated by Deepometry.
Fig. 7: Typical outputs of the Deepometry protocol.

Data availability

The full dataset for annotated images of different phenotypes of red blood cells is publicly available at https://figshare.com/articles/URL7_Annotated_Data/12432506 (deposited 6 May 2020). The smaller subset for testing Deepometry functionality, containing annotated images of red blood cells, is publicly available at https://figshare.com/articles/software/Expert_Annotated_RBC/13053968 (deposited 9 October 2020).

Code availability

The codebase for Deepometry (Python) is publicly accessible at https://github.com/broadinstitute/deepometry, under BSD 3-Clause License, Broad Institute. The codebase for Deepometry (MATLAB) is publicly accessible at https://github.com/ClaireBarnes197/Deepometry_MATLAB_GUI, under BSD 3-Clause License, Broad Institute. The standalone MATLAB app is publicly available at https://doi.org/10.6084/m9.figshare.13082231 (deposited 18 December 2020).

References

  1. Basiji, D. A., Ortyn, W. E., Liang, L., Venkatachalam, V. & Morrissey, P. Cellular image analysis and imaging by flow cytometry. Clin. Lab. Med. 27, 653–670 (2007).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Chang, S., Serena, K., Karen, S. & Gyongyi, S. Impaired expression and function of toll-like receptor 7 in hepatitis C virus infection in human hepatoma cells. Hepatology 51, 35–42 (2010).

    Article  CAS  PubMed  Google Scholar 

  3. Maguire, O., Collins, C., O’Loughlin, K., Miecznikowski, H. & Minderman, H. Quantifying nuclear p65 as a parameter for NF-κB activation: correlation between ImageStream cytometry, microscopy, and Western blot. Cytometry A 79, 461–469 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Bourton, E. C. et al. Multispectral imaging flow cytometry reveals distinct frequencies of γ-H2AX foci induction in DNA double strand break repair defective human cell lines. Cytometry A 81, 130–137 (2012).

    Article  PubMed  Google Scholar 

  5. Begum, J. et al. A method for evaluating the use of fluorescent dyes to track proliferation in cell lines by dye dilution. Cytometry A 83, 1085–1095 (2013).

    Article  PubMed  Google Scholar 

  6. de la Calle, C., Joubert, P.-E., Law, H. K. W., Hasan, M. & Albert, M. L. Simultaneous assessment of autophagy and apoptosis using multispectral imaging cytometry. Autophagy 7, 1045–1051 (2011).

    Article  PubMed  Google Scholar 

  7. Filby, A. et al. An imaging flow cytometric method for measuring cell division history and molecular symmetry during mitosis. Cytometry A 79, 496–506 (2011).

    Article  PubMed  Google Scholar 

  8. Riordon, J. et al. Deep learning with microfluidics for biotechnology. Trends Biotechnol. 37, 310–324 (2019).

    Article  CAS  PubMed  Google Scholar 

  9. Isozaki, A. et al. AI on a chip. Lab Chip 17, 3074–3090 (2020).

    Article  Google Scholar 

  10. Heath, J. R., Ribas, A. & Mischel, P. S. Single-cell analysis tools for drug discovery and development. Nat. Rev. Drug Discov. 15, 204–216 (2016).

    Article  CAS  PubMed  Google Scholar 

  11. Sommer, C. & Gerlich, D. W. Machine learning in cell biology—teaching computers to recognize phenotypes. J. Cell Sci. 126, 1–11 (2013).

    Google Scholar 

  12. Caicedo, J. et al. Data-analysis strategies for image-based cell profiling. Nat. Methods 14, 849–863 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Lee, K. C. M. et al. Quantitative phase imaging flow cytometry for ultra-large-scale single-cell biophysical phenotyping. Cytometry A 95, 510–520 (2019).

    Article  PubMed  Google Scholar 

  14. Kumamoto, Y. et al. High-throughput cell imaging and classification by narrowband and low-spectral-resolution Raman microscopy. J. Phys. Chem. B. 123, 2654–2661 (2019).

    Article  CAS  PubMed  Google Scholar 

  15. Hennig, H. et al. An open-source solution for advanced imaging flow cytometry data analysis using machine learning. Methods 1, 201–210 (2017).

    Article  Google Scholar 

  16. Jones, T. R. et al. Cell Profiler Analyst: data exploration and analysis software for complex image-based screens. BMC Bioinformatics 9, 482 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Blasi, T. et al. Label-free cell cycle analysis for high-throughput imaging flow cytometry. Nat. Commun. 7, 10256 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Eulenberg, P. et al. Reconstructing cell cycle and disease progression using deep learning. Nat. Commun. 8, 463 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  19. van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    Google Scholar 

  20. Haghverdi, L., Buettner, F. & Theis, F. J. Diffusion maps for high-dimensional single-cell analysis of differentiation data. Bioinformatics 15, 2989–2998 (2015).

    Article  Google Scholar 

  21. Becht, E. et al. Dimensionality reduction for visualizing single-cell data using UMAP. Nat. Biotechnol. 37, 38–44 (2019).

    Article  CAS  Google Scholar 

  22. Doan, M. et al. Objective assessment of stored blood quality by deep learning. Proc. Natl Acad. Sci. USA 117, 21381–21390 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Hennig, H. et al. An open-source solution for advanced imaging flow cytometry data analysis using machine learning. Methods 112, 201–210 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Szegedy, C. et al. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1–9 (IEEE, 2015).

  25. Doan, M. et al. Label-free leukemia monitoring by computer vision. Cytometry A 97, 407–414 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  27. Nitta, N. et al. Intelligent image-activated cell sorting. Cell 175, 266–276 (2018).

    Article  CAS  PubMed  Google Scholar 

  28. Ota, S. et al. Ghost cytometry. Science 360, 1246–1251 (2018).

    Article  CAS  PubMed  Google Scholar 

  29. Mikami, H. et al. Virtual-freezing fluorescence imaging flow cytometry. Nat. Commun 11, 1162 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Dang, V. Q. et al. Methods for segmentation and classification of digital microscopy tissue images. Front. Bioeng. Biotechnol. 7, 53 (2019).

    Article  Google Scholar 

  31. Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).

    Article  CAS  PubMed  Google Scholar 

  32. Nassar, M. et al. Label-free identification of white blood cells using machine learning. Cytometry A 95, 836–842 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  33. Lippeveld, M. et al. Classification of human white blood cells using machine learning for stain-free imaging flow cytometry. Cytometry A 97, 308–319 (2020).

    Article  PubMed  Google Scholar 

  34. Dickson, M. D. S. et al. Deep-learning-assisted biophysical imaging cytometry at massive throughput delineates cell population heterogeneity. Lab Chip 20, 3696–3708 (2020).

    Article  Google Scholar 

  35. Nissim, N., Dudale, M., Barnea, I. and Shaked, N. T. Real-time stain-free classification of cancer cells and blood cells using interferometric phase microscopy and machine learning. Cytometry https://doi.org/10.1002/cyto.a.24227 (2021).

  36. Ugele, M. et al. Label-free high-throughput leukemia detection by holographic microscopy. Adv. Sci. (Weinh) 5, 1800761 (2018).

    Article  Google Scholar 

  37. Karandikar, S. H. et al. Reagent-free and rapid assessment of T cell activation state using diffraction phase microscopy and deep learning. Anal. Chem 91, 3405–3411 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Ayyappan, V. et al. Classical machine learning based on manual feature extraction: identification and staging of B-cell acute lymphoblastic leukemia using quantitative phase imaging and machine learning. ACS Sens 5, 3281–3289 (2020).

    Article  CAS  PubMed  Google Scholar 

  39. Kräter, M. et al. AIDeveloper: deep learning image classification in life science and beyond. Adv. Sci. https://doi.org/10.1002/advs.202003743 (2021).

  40. Teschendorff, A. E. et al. Avoiding common pitfalls in machine learning omic data science. Nat. Mater. 18, 422–427 (2019).

    Article  CAS  PubMed  Google Scholar 

  41. Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C. & Kording, K. P. The need to approximate the use-case in clinical machine learning. Gigascience 6, 1–9 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Linkert, M. et al. Metadata matters: access to image data in the real world. J. Cell Biol. 189, 777–782 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Meehan, C., Ebrahimian, J., Moore, W. and Meehan, S. Uniform Manifold Approximation and Projection (UMAP). MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/71902-uniform-manifold-approximation-and-projection-umap (2021).

  44. Moon, K. R. et al. Visualizing structure and transitions in high-dimensional biological data. Nat. Biotechnol. 37, 1482–1492 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was supported by the BBSRC (BB/P026818/1 to P.R.), the National Science Foundation (DBI 1458626 to A.E.C.) and the National Institutes of Health (R35 GM122547 to A.E.C.). We thank P. Ryder for testing and confirming the functionality of the protocol.

Author information

Author notes

  1. These authors contributed equally: Minh Doan, Claire Barnes.

  2. These authors jointly supervised this work: Anne E. Carpenter, Paul Rees.

Authors and Affiliations

  1. Imaging Platform, Broad Institute of Harvard and MIT, Cambridge, MA, USA

    Minh Doan, Claire McQuin, Juan C. Caicedo, Allen Goodman, Anne E. Carpenter & Paul Rees

  2. Bioimaging Analytics, GlaxoSmithKline, Collegeville, PA, USA

    Minh Doan

  3. College of Engineering, Swansea University, Bay Campus, Swansea, UK

    Claire Barnes & Paul Rees

Authors
  1. Minh Doan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Claire Barnes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claire McQuin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan C. Caicedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Allen Goodman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anne E. Carpenter
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Paul Rees
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.D., C.B., and P.R. contributed to software development, design and interpretation of results and manuscript writing. J.C.C. contributed to software development and design and interpretation of results. C.M. and A.G. contributed to software development. A.E.C. contributed to design and interpretation of results and manuscript writing.

Corresponding authors

Correspondence to Minh Doan, Anne E. Carpenter or Paul Rees.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Protocols thanks the anonymous reviewers for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

Key references using this protocol

Doan, M. et al. Proc. Natl Acad. Sci. USA 117, 21381–21390 (2020): https://doi.org/10.1073/pnas.2001227117

Doan, M. et al. Cytometry A 97, 407–414 (2020): https://doi.org/10.1002/cyto.a.23987

Supplementary information

Supplementary Information

Supplementary Fig. 1 and Supplementary Notes 1–4.

Reporting Summary

Supplementary Video 1

Video guide to using MATLAB UMAP with Deepometry

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Doan, M., Barnes, C., McQuin, C. et al. Deepometry, a framework for applying supervised and weakly supervised deep learning to imaging cytometry. Nat Protoc 16, 3572–3595 (2021). https://doi.org/10.1038/s41596-021-00549-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41596-021-00549-7

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing