Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Technical Report
  • Published:

Label-free multiplexed microtomography of endogenous subcellular dynamics using generalizable deep learning

Nature Cell Biology volume 23, pages 1329–1337 (2021)Cite this article

Subjects

Abstract

Simultaneous imaging of various facets of intact biological systems across multiple spatiotemporal scales is a long-standing goal in biology and medicine, for which progress is hindered by limits of conventional imaging modalities. Here we propose using the refractive index (RI), an intrinsic quantity governing light–matter interaction, as a means for such measurement. We show that major endogenous subcellular structures, which are conventionally accessed via exogenous fluorescence labelling, are encoded in three-dimensional (3D) RI tomograms. We decode this information in a data-driven manner, with a deep learning-based model that infers multiple 3D fluorescence tomograms from RI measurements of the corresponding subcellular targets, thereby achieving multiplexed microtomography. This approach, called RI2FL for refractive index to fluorescence, inherits the advantages of both high-specificity fluorescence imaging and label-free RI imaging. Importantly, full 3D modelling of absolute and unbiased RI improves generalization, such that the approach is applicable to a broad range of new samples without retraining to facilitate immediate applicability. The performance, reliability and scalability of this technology are extensively characterized, and its various applications within single-cell profiling at unprecedented scales (which can generate new experimentally testable hypotheses) are demonstrated.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Data-driven microtomography of multiple subcellular structures.
Fig. 2: Performance characterization in comparison to conventional approaches.
Fig. 3: Uncertainty quantification and scalability in space and time.
Fig. 4: Photobleaching-free imaging over time.
Fig. 5: Application to single-cell profiling at unprecedented scales.
Fig. 6: Application to a new dataset without model retraining.

Similar content being viewed by others

Data availability

RI2FL datasets are available at https://github.com/NySunShine/ri2fl. Source data are provided with this paper. All other data supporting the findings of this study are available from the corresponding author on reasonable request.

Code availability

RI2FL source code, example datasets and step-by-step interactive tutorials are available at https://github.com/NySunShine/ri2fl.

References

  1. Mertz, J. Introduction to Optical Microscopy (Cambridge Univ. Press, 2019).

  2. Lichtman, J. W. & Conchello, J.-A. Fluorescence microscopy. Nat. Methods 2, 910–919 (2005).

    Article  CAS  Google Scholar 

  3. Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803.e19 (2018).

    Article  CAS  Google Scholar 

  4. Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).

    Article  CAS  Google Scholar 

  5. Guo, S.-M. et al. Revealing architectural order with quantitative label-free imaging and deep learning. eLife 9, e55502 (2020).

    Article  CAS  Google Scholar 

  6. Cheng, S. et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci. Adv. 7, eabe0431 (2021).

    Article  CAS  Google Scholar 

  7. Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light Sci. Appl. 8, 23 (2019).

    Article  Google Scholar 

  8. Nygate, Y. N. et al. Holographic virtual staining of individual biological cells. Proc. Natl Acad. Sci. USA 117, 9223–9231 (2020).

    Article  CAS  Google Scholar 

  9. Kandel, M. E. et al. Phase imaging with computational specificity (PICS) for measuring dry mass changes in sub-cellular compartments. Nat. Commun. 11, 6256 (2020).

    Article  CAS  Google Scholar 

  10. Kandel, M. E. et al. Multiscale assay of unlabeled neurite dynamics using phase imaging with computational specificity. ACS Sens. 6, 1864–1874 (2021).

    Article  CAS  Google Scholar 

  11. Park, Y., Depeursinge, C. & Popescu, G. Quantitative phase imaging in biomedicine. Nat. Photonics 12, 578–589 (2018).

    Article  CAS  Google Scholar 

  12. Kim, K. et al. Correlative three-dimensional fluorescence and refractive index tomography: bridging the gap between molecular specificity and quantitative bioimaging. Biomed. Opt. Express 8, 5688–5697 (2017).

    Article  CAS  Google Scholar 

  13. Lee, J. et al. Deep-learning-based label-free segmentation of cell nuclei in time-lapse refractive index tomograms. IEEE Access 7, 83449–83460 (2019).

    Article  Google Scholar 

  14. Park, S. et al. Label-free tomographic imaging of lipid droplets in foam cells for machine-learning-assisted therapeutic evaluation of targeted nanodrugs. ACS Nano 14, 1856–1865 (2020).

    Article  CAS  Google Scholar 

  15. Yoon, J. et al. Label-free identification of lymphocyte subtypes using three-dimensional quantitative phase imaging and machine learning. J. Vis. Exp. https://doi.org/10.3791/58305 (2018).

  16. Gal, Y. & Ghahramani, Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. Proc. Int. Conf. Mach. Learn. 48, 1050–1059 (2016).

    Google Scholar 

  17. Wang, G. et al. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338, 34–45 (2019).

    Article  Google Scholar 

  18. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  Google Scholar 

  19. Xue, Y., Cheng, S., Li, Y. & Tian, L. Reliable deep-learning-based phase imaging with uncertainty quantification. Optica 6, 618–629 (2019).

    Article  Google Scholar 

  20. Kendall, A. & Gal, Y. What uncertainties do we need in Bayesian deep learning for computer vision? In Proc. 31st International Conference on Neural Information Processing Systems (eds von Luxburg, U. et al.) 5574–5584 (Curran Associates, 2017).

  21. Zheng, G., Horstmeyer, R. & Yang, C. Wide-field, high-resolution Fourier ptychographic microscopy. Nat. Photonics 7, 739–745 (2013).

    Article  CAS  Google Scholar 

  22. Baek, Y., Lee, K., Shin, S. & Park, Y. Kramers–Kronig holographic imaging for high-space-bandwidth product. Optica 6, 45–51 (2019).

    Article  Google Scholar 

  23. Bray, M.-A. et al. Cell Painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nat. Protoc. 11, 1757–1774 (2016).

    Article  CAS  Google Scholar 

  24. Barer, R. Determination of dry mass, thickness, solid and water concentration in living cells. Nature 172, 1097–1098 (1953).

    Article  CAS  Google Scholar 

  25. Cooper, K. L. et al. Multiple phases of chondrocyte enlargement underlie differences in skeletal proportions. Nature 495, 375–378 (2013).

    Article  CAS  Google Scholar 

  26. Mir, M. et al. Optical measurement of cycle-dependent cell growth. Proc. Natl Acad. Sci. USA 108, 13124–13129 (2011).

    Article  CAS  Google Scholar 

  27. McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at https://arxiv.org/abs/1802.03426 (2018).

  28. Martin, K., Vilela, M., Jeon, N. L., Danuser, G. & Pertz, O. A growth factor-induced, spatially organizing cytoskeletal module enables rapid and persistent fibroblast migration. Dev. Cell 30, 701–716 (2014).

    Article  CAS  Google Scholar 

  29. Kim, K., Kim, K. S., Park, H., Ye, J. C. & Park, Y. Real-time visualization of 3-D dynamic microscopic objects using optical diffraction tomography. Opt. Express 21, 32269–32278 (2013).

    Article  Google Scholar 

  30. Inoue, T., Do Heo, W., Grimley, J. S., Wandless, T. J. & Meyer, T. An inducible translocation strategy to rapidly activate and inhibit small GTPase signaling pathways. Nat. Methods 2, 415–418 (2005).

    Article  CAS  Google Scholar 

  31. Liu, P. et al. Rab-regulated interaction of early endosomes with lipid droplets. Biochim. Biophys. Acta 1773, 784–793 (2007).

    Article  CAS  Google Scholar 

  32. Kounakis, K., Chaniotakis, M., Markaki, M. & Tavernarakis, N. Emerging roles of lipophagy in health and disease. Front. Cell Dev. Biol. 7, 185 (2019).

    Article  Google Scholar 

  33. Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–815 (2018).

    Article  CAS  Google Scholar 

  34. Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).

    Article  CAS  Google Scholar 

  35. Jo, Y. et al. Quantitative phase imaging and artificial intelligence: a review. IEEE J. Sel. Top. Quantum Electron. 25, 1–14 (2019).

    Article  Google Scholar 

  36. Chen, M., Ren, D., Liu, H.-Y., Chowdhury, S. & Waller, L. Multi-layer Born multiple-scattering model for 3D phase microscopy. Optica 7, 394–403 (2020).

    Article  CAS  Google Scholar 

  37. Lim, J., Ayoub, A. B., Antoine, E. E. & Psaltis, D. High-fidelity optical diffraction tomography of multiple scattering samples. Light Sci. Appl. 8, 82 (2019).

    Article  Google Scholar 

  38. Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at https://arxiv.org/abs/1702.08608 (2017).

  39. Sussillo, D. & Barak, O. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25, 626–649 (2013).

    Article  Google Scholar 

  40. Zaritsky, A. et al. Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma. Cell Syst. 12, 733–747.e6 (2021).

    Article  CAS  Google Scholar 

  41. Lim, J. et al. Comparative study of iterative reconstruction algorithms for missing cone problems in optical diffraction tomography. Opt. Express 23, 16933–16948 (2015).

    Article  CAS  Google Scholar 

  42. Jo, Y., Park, W. S. & Park, Y. Holotomographic imaging of eukaryotic cells. Protoc. Exch. https://doi.org/10.21203/rs.3.pex-1667/v1 (2021).

  43. Kim, S. et al. in International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Shen, D. et al.) 220-228 (Springer, 2019).

  44. Tan, M. & Le, Q. Efficientnet: rethinking model scaling for convolutional neural networks. Proc. Int. Conf. Mach. Learn. 97, 6105–6114 (2019).

    Google Scholar 

  45. Berg, S. et al. Ilastik: interactive machine learning for (bio) image analysis. Nat. Methods 16, 1226–1232 (2019).

    Article  CAS  Google Scholar 

  46. Kandel, M. E. et al. Reproductive outcomes predicted by phase imaging with computational specificity of spermatozoon ultrastructure. Proc. Natl Acad. Sci. USA 117, 18302–18309 (2020).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank the members of KAIST Biomedical Optics Laboratory for helpful discussions. This work was supported by the KAIST Up program, the BK21+ program, Tomocube, the National Research Foundation of Korea (2015R1A3A2066550), an Institute of Information & Communications Technology Planning & Evaluation (IITP; 2021-0-00745) grant and the Commercialization Promotion Agency for R&D Outcomes (COMPA; 055586) funded by the Korean government (MSIT) to Y.P.; the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Korea (HI21C0977) to H.-s.M.; and a KAIST Presidential Fellowship and Asan Foundation Biomedical Science Scholarship to Y.J.

Author information

Author notes

  1. YoungJu Jo

    Present address: Departments of Applied Physics and of Biology, Stanford University, Stanford, CA, USA

  2. These authors contributed equally: YoungJu Jo, Hyungjoo Cho, Wei Sun Park.

  3. These authors jointly supervised this work: Hyun-seok Min, Won Do Heo, YongKeun Park.

Authors and Affiliations

  1. Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

    YoungJu Jo, Wei Sun Park, Geon Kim, DongHun Ryu, Young Seo Kim, Moosung Lee & YongKeun Park

  2. KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea

    YoungJu Jo, Wei Sun Park, Geon Kim, DongHun Ryu, Young Seo Kim, Moosung Lee, Mahn Jae Lee & YongKeun Park

  3. Tomocube, Daejeon, Republic of Korea

    YoungJu Jo, Hyungjoo Cho, Hosung Joo, HangHun Jo, Sumin Lee, Hyun-seok Min & YongKeun Park

  4. Graduate School of Medial Science and Engineering, KAIST, Daejeon, Republic of Korea

    Young Seo Kim & Mahn Jae Lee

  5. Gwangju Center, Korea Basic Science Institute (KBSI), Gwangju, Republic of Korea

    Sangwoo Park & Seongsoo Lee

  6. Department of Biological Sciences, KAIST, Daejeon, Republic of Korea

    Won Do Heo

  7. KAIST Institute for the BioCentury, KAIST, Daejeon, Republic of Korea

    Won Do Heo

Authors
  1. YoungJu Jo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyungjoo Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Sun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Geon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. DongHun Ryu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Young Seo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Moosung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sangwoo Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mahn Jae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Hosung Joo
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. HangHun Jo
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Seongsoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Sumin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Hyun-seok Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Won Do Heo
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. YongKeun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.J. and Y.P. conceived the idea. Y.J. coordinated the project and carried out all analyses. Y.J., H.C. and H.-s.M. designed the deep-learning pipeline, and H.C. implemented and optimized the pipeline. Y.J., W.S.P. and W.D.H. designed the main experiments, and W.S.P. collected the data. W.S.P. and W.D.H. established the molecular biology and imaging protocols. G.K., D.R., Y.S.K. and M.L. contributed to processing the data. H. Joo, H. Jo and Sumin Lee collected and processed the large FOV data. S.P., M.J.L. and Seongsoo Lee contributed new RI tomogram datasets for testing the applicability of the technique. Y.J., H.C., W.S.P. and Y.P. wrote the manuscript with input from all co-authors. Y.P. supervised all aspects of the work.

Corresponding authors

Correspondence to Hyun-seok Min, Won Do Heo or YongKeun Park.

Ethics declarations

Competing interests

H.C., M.L., H. Jo, Sumin Lee, H.-s.M. and Y.P. have financial interests in Tomocube, a company that commercializes ODT and QPI instruments and is one of the sponsors of the work. The remaining authors declare no competing interests.

Additional information

Peer review information Nature Cell Biology thanks Gaudenz Danuser, and the other, anonymous reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Network architecture and patch-based processing.

(a) We used a single encoder-decoder network, systematically discovered by SCNAS, for all subcellular targets. (b) An example inference from an RI tomogram to the corresponding actin tomogram is illustrated as a flow chart, highlighting the patch-based processing for GPU memory management. All images represent MIPs of 3D data.

Extended Data Fig. 2 Discrete structure-based characterization of RI2FL performance.

Discrete subcellular structures, specifically lipid droplets for acceptable segmentation, were detected and matched in the inferred and ground-truth FL data in the held-out dataset. (a) Schematic summarizing the segmentation procedure (see Methods for details). (b) Matched versus unmatched lipid droplets could be well-defined based on inter-centroid distance and overlap fraction. (c) Distribution of inter-centroid distance in the matched lipid droplets. (d) Correlation between measured and inferred intensity averaged over the lipid droplet segments. The black line represents the least-squares linear regression fit of the data. N = 282 lipid droplets. r, Pearson’s correlation coefficient (calculated using the definition). Numerical source data is provided in source data.

Extended Data Fig. 3 Comparison between pre- and post-staining data.

In order to validate the inference of endogenous subcellular targets, we imaged identical cells before and after staining. The qualitative correspondence between the pre- and post-staining data, despite the irreducible discrepancy due to the temporal difference (from minutes to an hour) and fixation, further validates the successful operation of RI2FL in unlabeled cells.

Extended Data Fig. 4 Iterative tomographic reconstruction and RI2FL performance.

(a) RI reconstruction accuracy over iterations of non-negativity-constrained missing cone recovery algorithm in tomographic reconstruction. Note that the first few iterations are essential for accurate RI reconstruction but around the chosen condition (40 iterations) RI is minimally sensitive to the number of iterations. (b) FL prediction accuracy as a function of RI reconstruction accuracy. Note the strong positive correlation between RI reconstruction accuracy and FL prediction accuracy. Interestingly, this error sensitivity seems to be dependent on the target structure: for example, lipid droplets inference (yellow dots) are least sensitive to RI error. N = 41 tomograms in the test set. Numerical source data is provided in source data.

Extended Data Fig. 5 Uncertainty quantification schemes and visualizations.

(a, b) Quantifications of (a) data uncertainty and (b) model uncertainty were conducted by test-time augmentation and Monte Carlo dropout, respectively. All images in (a) and (b) represent MIPs of 3D data. (c, d) Quantitative visualizations of data uncertainty (c) on a 2D cross-section of Fig. 3a, t = 120 min and (d) along the 1D dashed line shown in (c).

Extended Data Fig. 6 Validation of RI2FL-inferred features.

Total 65 features were extracted with the inferred or ground-truth FL data in the held-out dataset, for one FL channel at a time (but note that most features are defined based on multiple channels; Fig. 3a). (a, b) Example features with inferred or ground-truth actin. (c) Distribution of r values for all features over the six FL channels. r = 0.97 across all features and channels. The least accurate features were actin-mitochondria correlation, mitochondria contrast, RI-lipid droplets correlation, cytoplasmic RI entropy, nuclear sphericity, and RI-nucleoli correlation, for actin, mitochondria, lipid droplets, plasma membranes, nuclei, and nucleoli channels, respectively. N = 102 cells. Numerical source data is provided in source data.

Extended Data Fig. 7 Intracellular distribution of inferred lipid droplets and measured endosomes.

Unexpectedly, the intracellular distribution of the inferred lipid droplets was starkly different from that of the ground-truth endosomes (see main text). While the lipid droplets were strongly correlated with high RI, this was not the case for endosomes.

Extended Data Fig. 8 Deep feature visualization.

In order to facilitate interpreting the operation of the trained networks, feature map activations for a single input tomogram were visualized. (a) Average feature map at each layer. (b, c) Individual feature maps at the last layers of (b) encoder and (c) decoder parts of the network inferring plasma membranes. All images represent MIPs of 3D data.

Supplementary information

Supplementary Information

Supplementary Note 1

Reporting Summary

Supplementary Tables

Table 1: Comparison of the related cross-modality inference methods. Table 2: Dataset summary. Table 3: Performance metrics. Table 4: Single-cell feature statistics.

Supplementary Video 1

RI2FL across cell types and subcellular targets.

Supplementary Video 2

Dynamics of cell division.

Supplementary Video 3

Large-scale imaging.

Supplementary Video 4

Growth factor stimulation.

Supplementary Video 5

Chemogenetic stimulation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jo, Y., Cho, H., Park, W.S. et al. Label-free multiplexed microtomography of endogenous subcellular dynamics using generalizable deep learning. Nat Cell Biol 23, 1329–1337 (2021). https://doi.org/10.1038/s41556-021-00802-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41556-021-00802-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing