Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Nature Methods volume 18, pages 551–556 (2021)Cite this article

Subjects

Abstract

Light-field microscopy has emerged as a technique of choice for high-speed volumetric imaging of fast biological processes. However, artifacts, nonuniform resolution and a slow reconstruction speed have limited its full capabilities for in toto extraction of dynamic spatiotemporal patterns in samples. Here, we combined a view-channel-depth (VCD) neural network with light-field microscopy to mitigate these limitations, yielding artifact-free three-dimensional image sequences with uniform spatial resolution and high-video-rate reconstruction throughput. We imaged neuronal activities across moving Caenorhabditis elegans and blood flow in a beating zebrafish heart at single-cell resolution with volumetric imaging rates up to 200 Hz.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: VCD-LFM and its performance.
Fig. 2: Whole-animal Ca2+ imaging of moving C. elegans using VCD-LFM.
Fig. 3: Imaging of various cardiac dynamics in beating zebrafish heart using VCD-LFM.

Similar content being viewed by others

Data availability

The datasets generated and analyzed in this study are available at Zenodo at https://doi.org/10.5281/zenodo.4390067. The supplementary information is also available from the corresponding authors upon request. The example model checkpoints and data are packaged with the code. Source data for Figs. 1–3 are provided with this paper.

Code availability

Codes for VCD-Net are available at https://github.com/feilab-hust/VCD-Net. Codes for data analysis and synthetic data generation used in current study are available from the corresponding authors upon request.

References

  1. Keller, P. J., Schmidt, A. D., Wittbrodt, J. & Stelzer, E. H. K. Reconstruction of zebrafish early embryonic development by scanned light sheet microscopy. Science 322, 1065–1069 (2008).

    Article  CAS  Google Scholar 

  2. Planchon, T. A. et al. Rapid three-dimensional isotropic imaging of living cells using Bessel beam plane illumination. Nat. Methods 8, 417–423 (2011).

    Article  CAS  Google Scholar 

  3. Conchello, J.-A. & Lichtman, J. W. Optical sectioning microscopy. Nat. Methods 2, 920–931 (2005).

    Article  CAS  Google Scholar 

  4. Huisken, J., Swoger, J., Del, B. F., Wittbrodt, J. & Stelzer, E. H. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305, 1007–1009 (2004).

    Article  CAS  Google Scholar 

  5. Yang, W. & Yuste, R. In vivo imaging of neural activity. Nat. Methods 14, 349–359 (2017).

    Article  CAS  Google Scholar 

  6. Levoy, M., Ng, R., Adams, A., Footer, M. & Horowitz, M. Light field microscopy. ACM Trans. Graph. 25, 924–934 (2006).

    Article  Google Scholar 

  7. Prevedel, R. et al. Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy. Nat. Methods 11, 727–730 (2014).

    Article  CAS  Google Scholar 

  8. Pégard, N. C. et al. Compressive light-field microscopy for 3D neural activity recording. Optica 3, 517–524 (2016).

    Article  Google Scholar 

  9. Cong, L. et al. Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (Danio rerio). eLife 6, e28158 (2017).

    Article  Google Scholar 

  10. Nobauer, T. et al. Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (SID) microscopy. Nat. Methods 14, 811–818 (2017).

    Article  CAS  Google Scholar 

  11. Wagner, N. et al. Instantaneous isotropic volumetric imaging of fast biological processes. Nat. Methods 16, 497–500 (2019).

    Article  CAS  Google Scholar 

  12. Cohen, N. et al. Enhancing the performance of the light field microscope using wavefront coding. Opt. Express 22, 24817–24839 (2014).

    Article  Google Scholar 

  13. Broxton, M. et al. Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Express 21, 25418–25439 (2013).

    Article  Google Scholar 

  14. Guo, C., Liu, W., Hua, X., Li, H. & Jia, S. Fourier light-field microscopy. Opt. Express 27, 25573–25594 (2019).

    Article  Google Scholar 

  15. Gao, S. et al. Excitatory motor neurons are local oscillators for backward locomotion. eLife 7, e29915 (2018).

    Article  Google Scholar 

  16. Wen, Q., Gao, S. & Zhen, M. Caenorhabditis elegans excitatory ventral cord motor neurons derive rhythm for body undulation. Philos. Trans. R. Soc. B. 373, 20170370 (2018).

    Article  Google Scholar 

  17. Xu, T. et al. Descending pathway facilitates undulatory wave propagation in Caenorhabditis elegans through gap junctions. Proc. Natl Acad. Sci. USA 115, E4493–E4502 (2018).

    Article  CAS  Google Scholar 

  18. Nguyen, J. P. et al. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans. Proc. Natl Acad. Sci. USA 113, E1074–E1081 (2016).

    Article  CAS  Google Scholar 

  19. Venkatachalam, V. et al. Pan-neuronal imaging in roaming Caenorhabditis elegans. Proc. Natl Acad. Sci. USA 113, E1082–E1088 (2016).

    Article  CAS  Google Scholar 

  20. Kawano, T. et al. An imbalancing act: gap junctions reduce the backward motor circuit activity to bias C. elegans for forward locomotion. Neuron 72, 572–586 (2011).

    Article  CAS  Google Scholar 

  21. Truong, T. V. et al. High-contrast, synchronous volumetric imaging with selective volume illumination microscopy. Commun. Biol. 3, 1–8 (2020).

    Article  Google Scholar 

  22. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  Google Scholar 

  23. Mickoleit, M. et al. High-resolution reconstruction of the beating zebrafish heart. Nat. Methods 11, 919–922 (2014).

    Article  CAS  Google Scholar 

  24. Voleti, V. et al. Real-time volumetric microscopy of in vivo dynamics and large-scale samples with SCAPE 2.0. Nat. Methods 16, 1054–1062 (2019).

    Article  CAS  Google Scholar 

  25. Ng, R. et al. Light Field Photography with a Hand-Held Plenoptic Camera. Dissertation, Stanford Univ. (2005).

  26. Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).

    Article  CAS  Google Scholar 

  27. Tinevez, J. Y. et al. TrackMate: an open and extensible platform for single-particle tracking. Methods 115, 80–90 (2017).

    Article  CAS  Google Scholar 

  28. Restif, C. et al. CeleST: computer vision software for quantitative analysis of C. elegans swim behavior reveals novel features of locomotion. PLoS Comput. Biol. 10, e1003702 (2014).

    Article  Google Scholar 

  29. Lee, J. et al. 4-Dimensional light-sheet microscopy to elucidate shear stress modulation of cardiac trabeculation. J. Clin. Investig. 126, 1679–1690 (2016).

    Article  Google Scholar 

  30. Liebling, M., Forouhar, A. S., Gharib, M., Fraser, S. E. & Dickinson, M. E. Four-dimensional cardiac imaging in living embryos via postacquisition synchronization of nongated slice sequences. J. Biomed. Opt. 10, 054001 (2005).

    Article  Google Scholar 

  31. Descloux, A. C., Grussmayer, K. S. & Radenovic, A. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods 16, 918–924 (2019).

    Article  CAS  Google Scholar 

  32. Ramachandran, P. & Varoquaux, G. Mayavi: 3D visualization of scientific data. Comput. Sci. Eng. 13, 40–51 (2011).

    Article  Google Scholar 

Download references

Acknowledgements

We thank J.-N. Chen and Y. Dong at UCLA for their help with zebrafish experiments. We thank Y. Huang at Peking University and J. Wang at Tsinghua University for their discussions and comments on the manuscript. We thank L. Xiao for his help on the code implementation. We acknowledge the David Geffen School of Medicine at UCLA, for providing the Zebrafish Core Facilities and laboratory space. This work was supported by the following grants: National Natural Science Foundation of China (grant nos. 21874052 and 21927802 to P.F., 31671052 and 32020103007 to S.G.), National Key R&D program of China (grant no. 2017YFA0700501 to P.F.), Innovation Fund of WNLO (P.F.), National Science Foundation of Hubei (grant no. 2018CFA039 to S.G.), National Institutes of Health (NIH) (grant nos. R01 HL149808, R01 HL118650, R01 HL129727 and R01 HL111437 to T.K.H. and K99 HL148493 to Y.D.), Department of Veterans Affairs (grant no. I01 BX004356 to T.K.H.), the Fundamental Research Funds for the Central Universities (grant no. 2020JYCXJJ076 to L.Z.), and the Junior Thousand Talents Program of China (P.F. and S.G.). The confocal experiment was done at the Advanced Light Microscopy/Spectroscopy Laboratory and the Leica Microsystems Center of Excellence at the California Nano Systems Institute at UCLA.

Author information

Author notes

  1. These authors contributed equally: Zhaoqiang Wang, Lanxin Zhu, Hao Zhang.

Authors and Affiliations

  1. School of Optical and Electronic Information-Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China

    Zhaoqiang Wang, Lanxin Zhu, Hao Zhang, Guo Li, Chengqiang Yi, Yi Li, Yicong Yang & Peng Fei

  2. Department of Bioengineering, Henry Samueli School of Engineering and Applied Science, University of California, Los Angeles, Los Angeles, CA, USA

    Zhaoqiang Wang & Tzung K. Hsiai

  3. Division of Cardiology, Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA

    Zhaoqiang Wang, Yichen Ding & Tzung K. Hsiai

  4. College of Life Science and Technology, Huazhong University of Science and Technology, Wuhan, China

    Yi Li & Shangbang Gao

  5. Lunenfeld-Tanenbaum Research Institute, Mount Sinai Hospital, University of Toronto, Toronto, Ontario, Canada

    Mei Zhen

  6. Division of Cardiology, Department of Medicine, Greater Los Angeles VA Healthcare System, Los Angeles, CA, USA

    Tzung K. Hsiai

Authors
  1. Zhaoqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lanxin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chengqiang Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yicong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yichen Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mei Zhen
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Shangbang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Tzung K. Hsiai
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Peng Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.F., Z.W. and H.Z. conceived the idea. P.F., S.G. and T.K.H. oversaw the project. Z.W., L.Z., G.L. and Y.Y. developed the optical setups. H.Z., Z.W. and L.Z. developed the programs. Z.W., L.Z., G.L., Y.D. and Y.L. conducted the experiments and processed the images. Z.W., L.Z., H.Z., M.Z., S.G., T.K.H. and P.F. analyzed the data and wrote the paper.

Corresponding authors

Correspondence to Shangbang Gao, Tzung K. Hsiai or Peng Fei.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nina Vogt was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–20, Notes 1–5 and Tables 1 and 2.

Reporting Summary

Supplementary Video 1

VCD-Net reconstruction of pan-neuronally labeled C. elegans over 10 s. MIPs are shown in xy (top left), xz (bottom left) and yz (top right) views. Scale bar, 30 μm.

Supplementary Video 2

VCD-Net reconstruction of motor neurons of L4 stage C. elegans over 60 s. MIPs are shown in xy (top left), xz (bottom left) and yz (top right) views. Scale bar, 40 μm.

Supplementary Video 3

Dual-color imaging of neuronal activity in freely moving C. elegans using VCD-LFM over 60 s. Left, the tracked neurons of the moving worm are identified and labeled. Top right, the GCaMP6(f)/wCherry fluorescence signals of eight example neurons. Bottom right, body curvature and velocity characterize the behavior of the worm.

Supplementary Video 4

The comparisons between the LFD and VCD-Net reconstructions of the flowing blood cells in a beating zebrafish heart. MIPs of the reconstructions are shown. Magnified region marked in white boxes are shown in corners. Scale bar, 30 μm.

Supplementary Video 5

The comparisons between the LFD and VCD-Net reconstructions of cardiomyocyte nuclei in a beating zebrafish heart. MIPs of the reconstructions are shown. Scale bar, 30 μm.

Supplementary Video 6

Comparisons between LFD and VCD-Net reconstructions of the myocardium in a beating zebrafish heart, shown in 3D.

Supplementary Video 7

Flowing blood cells in the tail vessels of embryonic zebrafish, acquired by VCD-LFM and light-sheet fluorescence microscopy (LSFM). The red LFM channel captures the RBCs while green LSFM channel captures the static vessels. Reconstructed results have been registered to reveal how the RBCs flow through the intersegmental vessels. Numbers on the white box denote the size of the ROI.

Supplementary Video 8

Trajectories of flowing blood cells in embryonic zebrafish tail vessels, which are automatically tracked using Imaris. Red balls denote the tracked cells and the colors on the trajectories represent the maximum traveling speeds at that position (color bar below). Scale bar, 50 μm.

Supplementary Video 9

Dual-color imaging of the myocardium (green) and blood cells (red) in the beating embryonic zebrafish heart. The blood cells are captured by VCD-LFM while the myocardium is reconstructed by LSFM via retrospective gating. The two image sequences have been registered and rendered in 3D in xy and yz views, showing how the RBCs flow through the beating myocardium. Scale bar, 50 μm.

Source data

Source Data Fig. 1

Fig. 1g, Averaged axial and lateral FWHM of the beads across the volumes reconstructed by LFDM, anisotropic and isotropic VCD-LFM, respectively. The standard deviation indicated by error bar in Fig. 1g is included. Fig. 1i,j, The signal fluctuation of two synthetic adjacent neurons extracted from the reconstruction of a light-field video by VCD-Net and LFD.

Source Data Fig. 2

Fig. 2e, The raw intensity of GCaMP signals and RFP signals of the neurons over 60 s in Fig. 2e. Fig. 2f, The curvature of the freely moving C. elegans during 60 s. Fig. 2g, The velocity of the freely moving C. elegans for 60 s.

Source Data Fig. 3

Fig. 3j, The volume of myocardium and ventricle chamber over 400 ms.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Zhu, L., Zhang, H. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat Methods 18, 551–556 (2021). https://doi.org/10.1038/s41592-021-01058-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-021-01058-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing