Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Deep learning-enhanced light-field imaging with continuous validation

Abstract

Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence–enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning–based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Schematic principle and network architecture of the HyLFM approach.
Fig. 2: Evaluation of HyLFM-Net’s imaging performance on subdiffraction-sized, fluorescent beads.
Fig. 3: Experimental demonstration of HyLFM on medaka heart.
Fig. 4: Online network validation and refinement in HyLFM.

Data availability

The datasets generated and/or analyzed during the current study are available at https://doi.org/10.5281/zenodo.4020352, https://doi.org/10.5281/zenodo.4020404 and https://doi.org/10.5281/zenodo.4019246. Links to additional datasets are provided at https://github.com/kreshuklab/hylfm-net. Source data are provided with this paper.

Code availability

The neural network code with routines for training and inference are available at https://doi.org/10.5281/zenodo.4647764.

References

  1. 1.

    Winter, P. W. & Shroff, H. Faster fluorescence microscopy: advances in high speed biological imaging. Curr. Opin. Chem. Biol. 20, 46–53 (2014).

    CAS  Article  Google Scholar 

  2. 2.

    Katona, G. et al. Fast two-photon in vivo imaging with three-dimensional random-access scanning in large tissue volumes. Nat. Methods 9, 201–208 (2012).

    CAS  Article  Google Scholar 

  3. 3.

    Duemani, R. G., Kelleher, K., Fink, R. & Saggau, P. Three-dimensional random access multiphoton microscopy for functional imaging of neuronal activity. Nat. Neurosci. 11, 713–720 (2008).

    Article  Google Scholar 

  4. 4.

    Kong, L. et al. Continuous volumetric imaging via an optical phase-locked ultrasound lens. Nat. Methods 12, 759–762 (2015).

    CAS  Article  Google Scholar 

  5. 5.

    Kazemipour, A. et al. Kilohertz frame-rate two-photon tomography. Nat. Methods 16, 778–786 (2019).

    CAS  Article  Google Scholar 

  6. 6.

    Huisken, J., Swoger, J., Del Bene, F., Wittbrodt, J. & Stelzer, E. H. K. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305, 1007–1009 (2004).

    CAS  Article  Google Scholar 

  7. 7.

    Bouchard, M. B. et al. Swept confocally-aligned planar excitation (SCAPE) microscopy for high speed volumetric imaging of behaving organisms. Nat. Photonics 9, 113–119 (2015).

    CAS  Article  Google Scholar 

  8. 8.

    Lu, R. et al. Video-rate volumetric functional imaging of the brain at synaptic resolution. Nat. Neurosci. 20, 620–628 (2017).

    CAS  Article  Google Scholar 

  9. 9.

    Quirin, S., Peterka, D. S. & Yuste, R. Instantaneous three-dimensional sensing using spatial light modulator illumination with extended depth of field imaging. Opt. Express 21, 16007–16021 (2013).

    Article  Google Scholar 

  10. 10.

    Levoy, M., Ng, R., Adams, A., Footer, M. & Horowitz, M. Light field microscopy. ACM Trans. Graph. 25, 924 (2006).

    Article  Google Scholar 

  11. 11.

    Broxton, M. et al. Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Express 21, 25418 (2013).

    Article  Google Scholar 

  12. 12.

    Prevedel, R. et al. Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy. Nat. Methods 11, 727–730 (2014).

    CAS  Article  Google Scholar 

  13. 13.

    Aimon, S. et al. Fast near-whole–brain imaging in adult Drosophila during responses to stimuli and behavior. PLoS Biol. 17, e2006732 (2019).

    Article  Google Scholar 

  14. 14.

    Nöbauer, T. et al. Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (SID) microscopy. Nat. Methods 14, 811–818 (2017).

    Article  Google Scholar 

  15. 15.

    Lin, Q. et al. Cerebellar neurodynamics predict decision timing and outcome on the single-trial level. Cell 180, 536–551 (2020).

    CAS  Article  Google Scholar 

  16. 16.

    Wagner, N. et al. Instantaneous isotropic volumetric imaging of fast biological processes. Nat. Methods 16, 497–500 (2019).

    CAS  Article  Google Scholar 

  17. 17.

    Truong, T. V. et al. High-contrast, synchronous volumetric imaging with selective volume illumination microscopy. Commun. Biol. 3, 74 (2020).

    Article  Google Scholar 

  18. 18.

    Taylor, M. A., Nöbauer, T., Pernia-Andrade, A., Schlumm, F. & Vaziri, A. Brain-wide 3D light-field imaging of neuronal activity with speckle-enhanced resolution. Optica 5, 345 (2018).

    Article  Google Scholar 

  19. 19.

    Cohen, N. et al. Enhancing the performance of the light field microscope using wavefront coding. Opt. Express 22, 24817–24839 (2014).

    Article  Google Scholar 

  20. 20.

    Li, H. et al. Fast, volumetric live-cell imaging using high-resolution light-field microscopy. Biomed. Opt. Express 10, 29 (2019).

    CAS  Article  Google Scholar 

  21. 21.

    Stefanoiu, A., Page, J., Symvoulidis, P., Westmeyer, G. G. & Lasser, T. Artifact-free deconvolution in light field microscopy. Opt. Express 27, 31644 (2019).

    Article  Google Scholar 

  22. 22.

    Scrofani, G. et al. FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples. Biomed. Opt. Express 9, 335 (2018).

    CAS  Article  Google Scholar 

  23. 23.

    Guo, C., Liu, W., Hua, X., Li, H. & Jia, S. Fourier light-field microscopy. Opt. Express 27, 25573 (2019).

    Article  Google Scholar 

  24. 24.

    Guo, M. et al. Rapid image deconvolution and multiview fusion for optical microscopy. Nat. Biotechnol. 38, 1337–1346 (2020).

  25. 25.

    Schuler, C. J., Hirsch, M., Harmeling, S. & Scholkopf, B. Learning to Deblur. IEEE Trans. Pattern Anal. Mach. Intell. 38, 1439–1451 (2016).

    Article  Google Scholar 

  26. 26.

    Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    CAS  Article  Google Scholar 

  27. 27.

    Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    CAS  Article  Google Scholar 

  28. 28.

    Hoffman, D. P., Slavitt, I. & Fitzpatrick, C. A. The promise and peril of deep learning in microscopy. Nat. Methods 18, 131–132 (2021).

    CAS  Article  Google Scholar 

  29. 29.

    Page, J., Saltarin, F., Belyaev, Y., Lyck, R. & Favaro, P. Learning to reconstruct confocal microscopy stacks from single light field images. Preprint at https://arxiv.org/abs/2003.11004 (2020).

  30. 30.

    Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods https://doi.org/10.1038/s41592-021-01058-x (2021).

  31. 31.

    Fahrbach, F. O., Voigt, F. F., Schmid, B., Helmchen, F. & Huisken, J. Rapid 3D light-sheet microscopy with a tunable lens. Opt. Express 21, 21010 (2013).

    Article  Google Scholar 

  32. 32.

    Li, X. et al. DeepLFM: deep learning-based 3D reconstruction for light field microscopy. In Proc. Biophotonics Congress: Optics in the Life Sciences Congress 2019 (BODA,BRAIN,NTM,OMA,OMP) NM3C.2 (Optical Society of America, 2019); https://doi.org/10.1364/NTM.2019.NM3C.2

  33. 33.

    Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void—learning denoising from single noisy images. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2124–2132 (IEEE, 2019); https://doi.org/10.1109/CVPR.2019.00223

  34. 34.

    Kobayashi, H., Solak, A. C., Batson, J. & Royer, L. A. Image deconvolution via noise-tolerant self-supervised inversion. Preprint at https://arxiv.org/abs/2006.06156 (2020).

  35. 35.

    Preibisch, S. et al. Efficient Bayesian-based multiview deconvolution. Nat. Methods 11, 645–648 (2014).

    CAS  Article  Google Scholar 

  36. 36.

    Biewald, L. Experiment tracking with Weights and Biases. https://www.wandb.com/ (2021).

  37. 37.

    Shivanandan, A., Radenovic, A. & Sbalzarini, I. F. MosaicIA: an ImageJ/Fiji plugin for spatial pattern and interaction analysis. BMC Bioinf. 14, 349 (2013).

    Article  Google Scholar 

  38. 38.

    Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. In Proceedings of the Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 1398–1402 (IEEE, 2003); https://doi.org/10.1109/ACSSC.2003.1292216

  39. 39.

    Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

    CAS  Article  Google Scholar 

  40. 40.

    Koster, R., Stick, R., Loosli, F. & Wittbrodt, J. Medaka spalt acts as a target gene of hedgehog signaling. Development 124, 3147–3156 (1997).

    CAS  Article  Google Scholar 

  41. 41.

    Rembold, M., Lahiri, K., Foulkes, N. S. & Wittbrodt, J. Transgenesis in fish: efficient selection of transgenic fish by co-injection with a fluorescent reporter construct. Nat. Protoc. 1, 1133–1139 (2006).

    CAS  Article  Google Scholar 

  42. 42.

    Pnevmatikakis, E. A. & Giovannucci, A. NoRMCorre: an online algorithm for piecewise rigid motion correction of calcium imaging data. J. Neurosci. Methods 291, 83–94 (2017).

    CAS  Article  Google Scholar 

Download references

Acknowledgements

We thank the European Molecular Biology Laboratory (EMBL) Heidelberg mechanical and electronic workshop for help as well as the IT Services Department for high-performance computing cluster support and C. Tischer from CBA for his help with volume registration. We thank B. Balázs, Luxendo GmbH, for help with software and electronics, R. Singh and D. Kromm for general support and C. Pape for help with benchmarking the algorithms. We also thank M. Majewsky, E. Leist and A. Saraceno for fish husbandry, and K. Slanchev and H. Baier (MPI Martinsried) as well as M. Hoffmann and B. Judkewitz (Charite Berlin) for providing calcium reporter zebrafish lines. N.W. was supported by the Helmholtz Association under the joint research school Munich School for Data Science (MUDS). J.G. was supported by a Research Center for Molecular Medicine (HRCMM) Career Development Fellowship, the MD/PhD program of the Medical Faculty Heidelberg, the Deutsche Herzstiftung e.V. (S/02/17), and by an Add-On Fellowship for Interdisciplinary Science of the Joachim Herz Stiftung and is grateful to M. Gorenflo for supervision and guidance. N.N acknowledges support from Åke Wiberg foundation, Ingabritt and Arne Lundberg foundation, and Sten K Johnson Foundation. J.C.B. acknowledges supporting fellowships from the EMBL Interdisciplinary Postdoctoral Programme under Marie Skłodowska Curie Cofund Actions MSCACOFUND-FP (664726). L.H. thanks Luxendo GmbH for help with microscope software and equipment support. This work was supported by the European Molecular Biology Laboratory (F.B., N.W., N.N., J.C.B., L.H., A.K. and R.P.).

Author information

Affiliations

Authors

Contributions

A.K., L.H. and R.P. conceived the project. N.W. and N.N. built the imaging system and performed experiments with the help of J.G. J.G. generated transgenic animals under guidance of J.W. F.B., A.K. and M.W. conceived the CNN architecture. F.B. and N.W. implemented the CNN and other image processing parts of the computational pipeline and evaluated its performance. J.C.B. performed Ca2+ data analysis. A.K. and R.P. led the project and wrote the paper with input from all authors. Author order for equal contributions was determined by coin toss.

Corresponding authors

Correspondence to Robert Prevedel or Anna Kreshuk.

Ethics declarations

Competing interests

L.H. is scientific cofounder and employee of Luxendo GmbH (part of Bruker), which makes light-sheet-based microscopes commercially available, and a shareholder of Suricube GmbH, which makes high-performance electronics and software for instrumentation control available. The other authors declare no competing interests.

Additional information

Peer review information Nature Methods thanks the anonymous reviewers for their contribution to the peer review of this work. Nina Vogt was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Network architecture.

This architecture was used for beads and neural activity volumes. For the medaka heart, slightly different layer depth was used with the same overall structure (see Supplementary Table 1). Res2/3d: residual blocks with 2d or 3d convolutions with kernel size (3×)3 × 3. Residual blocks contain an additional projection layer (1 × 1 or 1 × 1 × 1 convolution) if the number of input channels is different from the number of output channels. Up2/3d: transposed convolution layers with kernel size (3×)2 × 2 and stride (1×)2 × 2. Proj2d/3d: projection layers (1 × 1 or 1 × 1 × 1 convolutions). The numbers always correspond to the number of channels. With 19 × 19 pixel lenslets (nnum = 19) the rearranged light field input image has 192 = 361 channels. The affine transformation layer at the end is only part of the network when training on dynamic, single plane targets; otherwise, in inference mode it might be used in post-processing to yield a SPIM aligned prediction, or the inverse affine transformation is applied to the SPIM target for static samples to avoid unnecessary computations.

Extended Data Fig. 2 LFM-SPIM optical setup.

Schematic 2D drawing of the LFM-SPIM setup showing the main opto-mechanical components. The sample is illuminated through a single illumination objective with two excitation beam paths (ocra, light sheet illumination and blue, light field selective volume illumination) combined by a dichroic mirror (D1). The fluorescence is detected by an orthogonally oriented detection objective and optically separated onto two detection arms with a dichroic mirror (D2). Bandpass filters (BP1 and BP2) are placed in front of a tube lens (TL3,TL4) for the respective detection path. For the light field detection path (green), the tube lens (TL4) focuses on the microlens array (ML) and the image plane (shown in magenta) displaced by one microlens focal length is relayed by a 1-1 relay lens system (RL6) to an image plane coinciding with the camera sensor (shown in magenta). For the light sheet detection path, a combination of several relay lenses (RL1 to RL4), a 1:1 macro lens (RL5) together with a lens pair consisting of an offset lens (OL) and an electrically tunable lens (ETL) is used to image two axially displaced objective focal planes (shown in magenta, dotted and solid) to a common image plane at the sensor. The refocusing is achieved by applying different currents on the ETL. The mirror M1 is placed at a Fourier plane, such that the FOV of the light sheet path can be laterally aligned to fit the light field detection FOV. For single color imaging, the dichroic mirrors D1 and D2 are replaced by beamsplitters. See Methods for details.

Extended Data Fig. 3 HyLFM-Net provides artifact-free deconvolution of LFM data.

a, Ground truth single light sheet image. b, Subdiffraction beads reconstructed by LFM-Net and (c) iterative light field deconvolution (LFD). Scale bar is 10 µm.

Extended Data Fig. 4 Precision and recall for sub-diffraction beads.

Precision recall measurements (a-d) and curve (e) for HyLFM-Net-beads, LFD and LFD + CARE. In (a-d) each point represents the average for an individual axial plane. In (e) precision and recall were averaged over all volumes, such that each point represents a threshold. All reconstructions were scaled to the SPIM ground truth to minimize L2 distance; beads were found independently using the Difference of Gaussian (DoG) method with varying thresholds and associated with beads found in SPIM (with threshold 0.1) by Hungarian matching. N = 6716 samples were used in each panel. Note that panel (e) is identical to Fig. 2i.

Source data

Extended Data Fig. 5 Cross-application of trained deep neural networks can reveal bias to training data.

We created two kinds of samples, one with small (0.1 μm) and one with medium-sized (4 μm) beads suspended in agarose. In (a), HyLFM-Net was trained on small beads and applied to small beads. FWHM of the beads in the reconstructed volume is shown (6025 beads measured). b, HyLFM-Net was trained on large beads and applied to large beads (682 beads measured). In (c), HyLFM-Net was trained on small beads and used to reconstruct a volume with large beads (525 beads measured). Similarly, in (d), HyLFM-Net trained on large beads and used to reconstruct a volume with small beads (2185 beads measured). e, SPIM image of 0.1 μm beads, (f) reconstructions of HyLFM-Net from (a), trained on small beads, (g) reconstructions from HyLFM-Net from (d), trained on large beads. h, SPIM image of 4 μm beads, i, reconstructions of HyLFM-Net from (b), trained on large beads, (j) reconstructions of HyLFM-Net from (c), trained on small beads. Line profile is shown to highlight a reconstruction error (red arrows), where the network reconstructs very small beads (as found in the training data) and produces an additional erroneous peak where none is present in the ground truth SPIM volume. Shadows in (a–d) denote standard deviation. Scale bar 2 μm in (e–g), and 10 μm in (h–j).

Source data

Extended Data Fig. 6 Network fine-tuning for different domain gaps.

Refinement of three differently pre-trained HyLFM networks on dynamically acquired medaka heart images. Column (a) shows SPIM ground truth plane at axial position z = -19µm. Columns (b-d) depict corresponding slices after increasingly many refinement iterations of different pre-trained HyLFM networks. b, A statically trained HyLFM-Net (such as the one in Fig. 3b), c, HyLFM-Net trained on LFD reconstructions of light-field images acquired on another microscope setup (Wagner, Norlin et al, Nat. Meth. 16, 497–500, 2019) and (d) HyLFM-Net trained on medium-sized beads (see Fig. 2). e, Respective MS-SSIM image quality metrics for each network and stage of refinement. Standard deviation shown as shadows inferred from N = 756 individual time-points. Note that depending on the domain gap, the refinement converges at different speeds, but high fidelity results can be obtained for all pre-trained networks.

Source data

Extended Data Fig. 7 Ca2+-imaging in zebrafish larvae brain.

a-c, Representative image plane acquired with SPIM, HyLFM-Net and LFD, respectively (standard deviation projection over time). d, Selected Ca2+-traces extracted from regions indicated in (a-c), 36 traces with 150 time points analyzed. e, Comparison of Pearson correlation coefficients (R) of Ca2+-traces extracted by SPIM, HyLFM-Net and LFD. Note that the difference between LFD and HyLFM-Net performance is not statistically significant (p = 0.053, Dunn-Sidak). Scale bar in (a) is 50 µm. Results representative of n = 5 individual image planes in the volume.

Source data

Supplementary information

Supplementary Information

Supplementary Tables 1 and 2.

Reporting Summary

Supplementary Video 1

Volumetric HyLFM reconstruction of a beating medaka heart at 40 Hz. Volumetric reconstructions (LFD, HyLFM-Net-stat, and HyLFM-Net-stat refined) of the medaka heart at 40 Hz image acquisition speed shown in Figs. 3 and 4. The cyan plane corresponds to the sweeping SPIM image plane. The panels from left to right show the overlay of the light sheet plane with the respective plane from the HyLFM-Net/LFD volume, a projection of the HyLFM-Net/LFD volume rotated by 45° around the y axis, and a maximum projection of prediction/reconstruction volume along the z, y and x axes, respectively. Scale bar 30 µm.

41592_2021_1136_MOESM4_ESM.mp4

Supplementary Video 2 Single-plane HyLFM reconstruction of a beating medaka heart at 56 Hz. Single-plane comparison of SPIM ground truth to the corresponding plane of the prediction volume of HyLFM-Net and reconstruction volume of LFD at indicated axial positions of the medaka heart at 56 Hz image acquisition speed (Fig. 3f,k,m). Scale bar 30 µm.

41592_2021_1136_MOESM5_ESM.mp4

Supplementary Video 3 Single-plane HyLFM reconstruction of a beating medaka heart at 100 Hz. Single-plane comparison of SPIM ground truth to the corresponding plane of the prediction volume of HyLFM-Net at indicated axial positions of the medaka heart at 100 Hz image acquisition speed. Scale bar 30 µm.

41592_2021_1136_MOESM6_ESM.mp4

Supplementary Video 4 Demonstration of continuous validation and network refinement. Refinement of three differently pretrained HyLFM networks on dynamically acquired medaka heart images at 40 Hz image acquisition speed. Video of refinement experiments in Supplementary Fig. 6 (see figure caption for details).

Source data

Source Data Fig. 2

Statistical source data.

Source Data Fig. 3

Statistical source data.

Source Data Fig. 4

Statistical source data.

Source Data Extended Data Fig. 4

Precision recall measurements of subdiffraction beads for HyLFM-Net-beads, LFD and LFD + CARE. Source data for Extended Data Fig. 4.

Source Data Extended Data Fig. 5

Cross-application of trained deep neural networks and the resulting FWHM of the beads in the reconstructed volume. Source data for Extended Data Fig. 5a–d.

Source Data Extended Data Fig. 6

MS-SSIM image quality metrics for each network and stage of refinement. Source data for Extended Data Fig. 6e.

Source Data Extended Data Fig. 7

Selected Ca2+ traces extracted from HyLFM, LFD and SPIM and comparison of Pearson correlation coefficients. Source data for Extended Data Fig. 7b,c.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wagner, N., Beuttenmueller, F., Norlin, N. et al. Deep learning-enhanced light-field imaging with continuous validation. Nat Methods 18, 557–563 (2021). https://doi.org/10.1038/s41592-021-01136-0

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing