Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence–enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning–based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Nature Communications Open Access 02 September 2022
Neuroscience Bulletin Open Access 08 August 2022
BioChip Journal Open Access 27 July 2022
Subscribe to Nature+
Get immediate online access to the entire Nature family of 50+ journals
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The datasets generated and/or analyzed during the current study are available at https://doi.org/10.5281/zenodo.4020352, https://doi.org/10.5281/zenodo.4020404 and https://doi.org/10.5281/zenodo.4019246. Links to additional datasets are provided at https://github.com/kreshuklab/hylfm-net. Source data are provided with this paper.
The neural network code with routines for training and inference are available at https://doi.org/10.5281/zenodo.4647764.
Winter, P. W. & Shroff, H. Faster fluorescence microscopy: advances in high speed biological imaging. Curr. Opin. Chem. Biol. 20, 46–53 (2014).
Katona, G. et al. Fast two-photon in vivo imaging with three-dimensional random-access scanning in large tissue volumes. Nat. Methods 9, 201–208 (2012).
Duemani, R. G., Kelleher, K., Fink, R. & Saggau, P. Three-dimensional random access multiphoton microscopy for functional imaging of neuronal activity. Nat. Neurosci. 11, 713–720 (2008).
Kong, L. et al. Continuous volumetric imaging via an optical phase-locked ultrasound lens. Nat. Methods 12, 759–762 (2015).
Kazemipour, A. et al. Kilohertz frame-rate two-photon tomography. Nat. Methods 16, 778–786 (2019).
Huisken, J., Swoger, J., Del Bene, F., Wittbrodt, J. & Stelzer, E. H. K. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305, 1007–1009 (2004).
Bouchard, M. B. et al. Swept confocally-aligned planar excitation (SCAPE) microscopy for high speed volumetric imaging of behaving organisms. Nat. Photonics 9, 113–119 (2015).
Lu, R. et al. Video-rate volumetric functional imaging of the brain at synaptic resolution. Nat. Neurosci. 20, 620–628 (2017).
Quirin, S., Peterka, D. S. & Yuste, R. Instantaneous three-dimensional sensing using spatial light modulator illumination with extended depth of field imaging. Opt. Express 21, 16007–16021 (2013).
Levoy, M., Ng, R., Adams, A., Footer, M. & Horowitz, M. Light field microscopy. ACM Trans. Graph. 25, 924 (2006).
Broxton, M. et al. Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Express 21, 25418 (2013).
Prevedel, R. et al. Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy. Nat. Methods 11, 727–730 (2014).
Aimon, S. et al. Fast near-whole–brain imaging in adult Drosophila during responses to stimuli and behavior. PLoS Biol. 17, e2006732 (2019).
Nöbauer, T. et al. Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (SID) microscopy. Nat. Methods 14, 811–818 (2017).
Lin, Q. et al. Cerebellar neurodynamics predict decision timing and outcome on the single-trial level. Cell 180, 536–551 (2020).
Wagner, N. et al. Instantaneous isotropic volumetric imaging of fast biological processes. Nat. Methods 16, 497–500 (2019).
Truong, T. V. et al. High-contrast, synchronous volumetric imaging with selective volume illumination microscopy. Commun. Biol. 3, 74 (2020).
Taylor, M. A., Nöbauer, T., Pernia-Andrade, A., Schlumm, F. & Vaziri, A. Brain-wide 3D light-field imaging of neuronal activity with speckle-enhanced resolution. Optica 5, 345 (2018).
Cohen, N. et al. Enhancing the performance of the light field microscope using wavefront coding. Opt. Express 22, 24817–24839 (2014).
Li, H. et al. Fast, volumetric live-cell imaging using high-resolution light-field microscopy. Biomed. Opt. Express 10, 29 (2019).
Stefanoiu, A., Page, J., Symvoulidis, P., Westmeyer, G. G. & Lasser, T. Artifact-free deconvolution in light field microscopy. Opt. Express 27, 31644 (2019).
Scrofani, G. et al. FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples. Biomed. Opt. Express 9, 335 (2018).
Guo, C., Liu, W., Hua, X., Li, H. & Jia, S. Fourier light-field microscopy. Opt. Express 27, 25573 (2019).
Guo, M. et al. Rapid image deconvolution and multiview fusion for optical microscopy. Nat. Biotechnol. 38, 1337–1346 (2020).
Schuler, C. J., Hirsch, M., Harmeling, S. & Scholkopf, B. Learning to Deblur. IEEE Trans. Pattern Anal. Mach. Intell. 38, 1439–1451 (2016).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).
Hoffman, D. P., Slavitt, I. & Fitzpatrick, C. A. The promise and peril of deep learning in microscopy. Nat. Methods 18, 131–132 (2021).
Page, J., Saltarin, F., Belyaev, Y., Lyck, R. & Favaro, P. Learning to reconstruct confocal microscopy stacks from single light field images. Preprint at https://arxiv.org/abs/2003.11004 (2020).
Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods https://doi.org/10.1038/s41592-021-01058-x (2021).
Fahrbach, F. O., Voigt, F. F., Schmid, B., Helmchen, F. & Huisken, J. Rapid 3D light-sheet microscopy with a tunable lens. Opt. Express 21, 21010 (2013).
Li, X. et al. DeepLFM: deep learning-based 3D reconstruction for light field microscopy. In Proc. Biophotonics Congress: Optics in the Life Sciences Congress 2019 (BODA,BRAIN,NTM,OMA,OMP) NM3C.2 (Optical Society of America, 2019); https://doi.org/10.1364/NTM.2019.NM3C.2
Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void—learning denoising from single noisy images. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2124–2132 (IEEE, 2019); https://doi.org/10.1109/CVPR.2019.00223
Kobayashi, H., Solak, A. C., Batson, J. & Royer, L. A. Image deconvolution via noise-tolerant self-supervised inversion. Preprint at https://arxiv.org/abs/2006.06156 (2020).
Preibisch, S. et al. Efficient Bayesian-based multiview deconvolution. Nat. Methods 11, 645–648 (2014).
Biewald, L. Experiment tracking with Weights and Biases. https://www.wandb.com/ (2021).
Shivanandan, A., Radenovic, A. & Sbalzarini, I. F. MosaicIA: an ImageJ/Fiji plugin for spatial pattern and interaction analysis. BMC Bioinf. 14, 349 (2013).
Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. In Proceedings of the Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 1398–1402 (IEEE, 2003); https://doi.org/10.1109/ACSSC.2003.1292216
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
Koster, R., Stick, R., Loosli, F. & Wittbrodt, J. Medaka spalt acts as a target gene of hedgehog signaling. Development 124, 3147–3156 (1997).
Rembold, M., Lahiri, K., Foulkes, N. S. & Wittbrodt, J. Transgenesis in fish: efficient selection of transgenic fish by co-injection with a fluorescent reporter construct. Nat. Protoc. 1, 1133–1139 (2006).
Pnevmatikakis, E. A. & Giovannucci, A. NoRMCorre: an online algorithm for piecewise rigid motion correction of calcium imaging data. J. Neurosci. Methods 291, 83–94 (2017).
We thank the European Molecular Biology Laboratory (EMBL) Heidelberg mechanical and electronic workshop for help as well as the IT Services Department for high-performance computing cluster support and C. Tischer from CBA for his help with volume registration. We thank B. Balázs, Luxendo GmbH, for help with software and electronics, R. Singh and D. Kromm for general support and C. Pape for help with benchmarking the algorithms. We also thank M. Majewsky, E. Leist and A. Saraceno for fish husbandry, and K. Slanchev and H. Baier (MPI Martinsried) as well as M. Hoffmann and B. Judkewitz (Charite Berlin) for providing calcium reporter zebrafish lines. N.W. was supported by the Helmholtz Association under the joint research school Munich School for Data Science (MUDS). J.G. was supported by a Research Center for Molecular Medicine (HRCMM) Career Development Fellowship, the MD/PhD program of the Medical Faculty Heidelberg, the Deutsche Herzstiftung e.V. (S/02/17), and by an Add-On Fellowship for Interdisciplinary Science of the Joachim Herz Stiftung and is grateful to M. Gorenflo for supervision and guidance. N.N acknowledges support from Åke Wiberg foundation, Ingabritt and Arne Lundberg foundation, and Sten K Johnson Foundation. J.C.B. acknowledges supporting fellowships from the EMBL Interdisciplinary Postdoctoral Programme under Marie Skłodowska Curie Cofund Actions MSCACOFUND-FP (664726). L.H. thanks Luxendo GmbH for help with microscope software and equipment support. This work was supported by the European Molecular Biology Laboratory (F.B., N.W., N.N., J.C.B., L.H., A.K. and R.P.).
L.H. is scientific cofounder and employee of Luxendo GmbH (part of Bruker), which makes light-sheet-based microscopes commercially available, and a shareholder of Suricube GmbH, which makes high-performance electronics and software for instrumentation control available. The other authors declare no competing interests.
Peer review information Nature Methods thanks the anonymous reviewers for their contribution to the peer review of this work. Nina Vogt was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This architecture was used for beads and neural activity volumes. For the medaka heart, slightly different layer depth was used with the same overall structure (see Supplementary Table 1). Res2/3d: residual blocks with 2d or 3d convolutions with kernel size (3×)3 × 3. Residual blocks contain an additional projection layer (1 × 1 or 1 × 1 × 1 convolution) if the number of input channels is different from the number of output channels. Up2/3d: transposed convolution layers with kernel size (3×)2 × 2 and stride (1×)2 × 2. Proj2d/3d: projection layers (1 × 1 or 1 × 1 × 1 convolutions). The numbers always correspond to the number of channels. With 19 × 19 pixel lenslets (nnum = 19) the rearranged light field input image has 192 = 361 channels. The affine transformation layer at the end is only part of the network when training on dynamic, single plane targets; otherwise, in inference mode it might be used in post-processing to yield a SPIM aligned prediction, or the inverse affine transformation is applied to the SPIM target for static samples to avoid unnecessary computations.
Schematic 2D drawing of the LFM-SPIM setup showing the main opto-mechanical components. The sample is illuminated through a single illumination objective with two excitation beam paths (ocra, light sheet illumination and blue, light field selective volume illumination) combined by a dichroic mirror (D1). The fluorescence is detected by an orthogonally oriented detection objective and optically separated onto two detection arms with a dichroic mirror (D2). Bandpass filters (BP1 and BP2) are placed in front of a tube lens (TL3,TL4) for the respective detection path. For the light field detection path (green), the tube lens (TL4) focuses on the microlens array (ML) and the image plane (shown in magenta) displaced by one microlens focal length is relayed by a 1-1 relay lens system (RL6) to an image plane coinciding with the camera sensor (shown in magenta). For the light sheet detection path, a combination of several relay lenses (RL1 to RL4), a 1:1 macro lens (RL5) together with a lens pair consisting of an offset lens (OL) and an electrically tunable lens (ETL) is used to image two axially displaced objective focal planes (shown in magenta, dotted and solid) to a common image plane at the sensor. The refocusing is achieved by applying different currents on the ETL. The mirror M1 is placed at a Fourier plane, such that the FOV of the light sheet path can be laterally aligned to fit the light field detection FOV. For single color imaging, the dichroic mirrors D1 and D2 are replaced by beamsplitters. See Methods for details.
a, Ground truth single light sheet image. b, Subdiffraction beads reconstructed by LFM-Net and (c) iterative light field deconvolution (LFD). Scale bar is 10 µm.
Precision recall measurements (a-d) and curve (e) for HyLFM-Net-beads, LFD and LFD + CARE. In (a-d) each point represents the average for an individual axial plane. In (e) precision and recall were averaged over all volumes, such that each point represents a threshold. All reconstructions were scaled to the SPIM ground truth to minimize L2 distance; beads were found independently using the Difference of Gaussian (DoG) method with varying thresholds and associated with beads found in SPIM (with threshold 0.1) by Hungarian matching. N = 6716 samples were used in each panel. Note that panel (e) is identical to Fig. 2i.
Extended Data Fig. 5 Cross-application of trained deep neural networks can reveal bias to training data.
We created two kinds of samples, one with small (0.1 μm) and one with medium-sized (4 μm) beads suspended in agarose. In (a), HyLFM-Net was trained on small beads and applied to small beads. FWHM of the beads in the reconstructed volume is shown (6025 beads measured). b, HyLFM-Net was trained on large beads and applied to large beads (682 beads measured). In (c), HyLFM-Net was trained on small beads and used to reconstruct a volume with large beads (525 beads measured). Similarly, in (d), HyLFM-Net trained on large beads and used to reconstruct a volume with small beads (2185 beads measured). e, SPIM image of 0.1 μm beads, (f) reconstructions of HyLFM-Net from (a), trained on small beads, (g) reconstructions from HyLFM-Net from (d), trained on large beads. h, SPIM image of 4 μm beads, i, reconstructions of HyLFM-Net from (b), trained on large beads, (j) reconstructions of HyLFM-Net from (c), trained on small beads. Line profile is shown to highlight a reconstruction error (red arrows), where the network reconstructs very small beads (as found in the training data) and produces an additional erroneous peak where none is present in the ground truth SPIM volume. Shadows in (a–d) denote standard deviation. Scale bar 2 μm in (e–g), and 10 μm in (h–j).
Refinement of three differently pre-trained HyLFM networks on dynamically acquired medaka heart images. Column (a) shows SPIM ground truth plane at axial position z = -19µm. Columns (b-d) depict corresponding slices after increasingly many refinement iterations of different pre-trained HyLFM networks. b, A statically trained HyLFM-Net (such as the one in Fig. 3b), c, HyLFM-Net trained on LFD reconstructions of light-field images acquired on another microscope setup (Wagner, Norlin et al, Nat. Meth. 16, 497–500, 2019) and (d) HyLFM-Net trained on medium-sized beads (see Fig. 2). e, Respective MS-SSIM image quality metrics for each network and stage of refinement. Standard deviation shown as shadows inferred from N = 756 individual time-points. Note that depending on the domain gap, the refinement converges at different speeds, but high fidelity results can be obtained for all pre-trained networks.
a-c, Representative image plane acquired with SPIM, HyLFM-Net and LFD, respectively (standard deviation projection over time). d, Selected Ca2+-traces extracted from regions indicated in (a-c), 36 traces with 150 time points analyzed. e, Comparison of Pearson correlation coefficients (R) of Ca2+-traces extracted by SPIM, HyLFM-Net and LFD. Note that the difference between LFD and HyLFM-Net performance is not statistically significant (p = 0.053, Dunn-Sidak). Scale bar in (a) is 50 µm. Results representative of n = 5 individual image planes in the volume.
Supplementary Tables 1 and 2.
Volumetric HyLFM reconstruction of a beating medaka heart at 40 Hz. Volumetric reconstructions (LFD, HyLFM-Net-stat, and HyLFM-Net-stat refined) of the medaka heart at 40 Hz image acquisition speed shown in Figs. 3 and 4. The cyan plane corresponds to the sweeping SPIM image plane. The panels from left to right show the overlay of the light sheet plane with the respective plane from the HyLFM-Net/LFD volume, a projection of the HyLFM-Net/LFD volume rotated by 45° around the y axis, and a maximum projection of prediction/reconstruction volume along the z, y and x axes, respectively. Scale bar 30 µm.
Supplementary Video 2 Single-plane HyLFM reconstruction of a beating medaka heart at 56 Hz. Single-plane comparison of SPIM ground truth to the corresponding plane of the prediction volume of HyLFM-Net and reconstruction volume of LFD at indicated axial positions of the medaka heart at 56 Hz image acquisition speed (Fig. 3f,k,m). Scale bar 30 µm.
Supplementary Video 3 Single-plane HyLFM reconstruction of a beating medaka heart at 100 Hz. Single-plane comparison of SPIM ground truth to the corresponding plane of the prediction volume of HyLFM-Net at indicated axial positions of the medaka heart at 100 Hz image acquisition speed. Scale bar 30 µm.
Supplementary Video 4 Demonstration of continuous validation and network refinement. Refinement of three differently pretrained HyLFM networks on dynamically acquired medaka heart images at 40 Hz image acquisition speed. Video of refinement experiments in Supplementary Fig. 6 (see figure caption for details).
Statistical source data.
Statistical source data.
Statistical source data.
Precision recall measurements of subdiffraction beads for HyLFM-Net-beads, LFD and LFD + CARE. Source data for Extended Data Fig. 4.
Cross-application of trained deep neural networks and the resulting FWHM of the beads in the reconstructed volume. Source data for Extended Data Fig. 5a–d.
MS-SSIM image quality metrics for each network and stage of refinement. Source data for Extended Data Fig. 6e.
Selected Ca2+ traces extracted from HyLFM, LFD and SPIM and comparison of Pearson correlation coefficients. Source data for Extended Data Fig. 7b,c.
About this article
Cite this article
Wagner, N., Beuttenmueller, F., Norlin, N. et al. Deep learning-enhanced light-field imaging with continuous validation. Nat Methods 18, 557–563 (2021). https://doi.org/10.1038/s41592-021-01136-0
Nature Communications (2022)
Nature Protocols (2022)
Light: Science & Applications (2022)
Biophysical Reviews (2022)
Neuroscience Bulletin (2022)