Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes

Abstract

The goal when imaging bioprocesses with optical microscopy is to acquire the most spatiotemporal information with the least invasiveness. Deep neural networks have substantially improved optical microscopy, including image super-resolution and restoration, but still have substantial potential for artifacts. In this study, we developed rationalized deep learning (rDL) for structured illumination microscopy and lattice light sheet microscopy (LLSM) by incorporating prior knowledge of illumination patterns and, thereby, rationally guiding the network to denoise raw images. Here we demonstrate that rDL structured illumination microscopy eliminates spectral bias-induced resolution degradation and reduces model uncertainty by five-fold, improving the super-resolution information by more than ten-fold over other computational approaches. Moreover, rDL applied to LLSM enables self-supervised training by using the spatial or temporal continuity of noisy data itself, yielding results similar to those of supervised methods. We demonstrate the utility of rDL by imaging the rapid kinetics of motile cilia, nucleolar protein condensation during light-sensitive mitosis and long-term interactions between membranous and membrane-less organelles.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: rDL SIM method.
Fig. 2: Comparison of rDL SIM with state-of-the-art SIM methods.
Fig. 3: rDL SIM imaging of motile cilia and organelle interactions.
Fig. 4: Volumetric long-term SR imaging via rDL 3D-SIM and rDL LLS-SIM.
Fig. 5: Volumetric, high-speed TiS-rDL LLSM imaging of SiT-containing transport intermediates.
Fig. 6: Inheritance and interactions of membrane organelles in mitosis via SiS-rDL LLSM.

Similar content being viewed by others

Data availability

The published BioSR dataset was used to train the rDL models for the biological structures of ER, MT, actin and CCPs. The training data for the rDL models of other biological structures were also uploaded into the figshare repository of the BioSR dataset, which is publicly accessible via https://doi.org/10.6084/m9.figshare.13264793. Source data are provided with this paper.

Code availability

The source codes of rDL SIM, several representative pre-trained models as well as some example images for testing are publicly accessible via https://github.com/qc17-THU/rDL-SIM.

References

  1. Moore, A. S. et al. Actin cables and comet tails organize mitochondrial networks in mitosis. Nature 591, 659–664 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Liao, Y.-C. et al. RNA granules hitchhike on lysosomes for long-distance transport, using annexin A11 as a molecular tether. Cell 179, 147–164 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Phillips, M. J. & Voeltz, G. K. Structure and function of ER membrane contact sites with other organelles. Nat. Rev. Mol. Cell Biol. 17, 69–82 (2016).

    Article  CAS  PubMed  Google Scholar 

  4. Prinz, W. A., Toulmay, A. & Balla, T. The functional universe of membrane contact sites. Nat. Rev. Mol. Cell Biol. 21, 7–24 (2020).

    Article  CAS  PubMed  Google Scholar 

  5. Schermelleh, L. et al. Super-resolution microscopy demystified. Nat. Cell Biol. 21, 72–84 (2019).

    Article  CAS  PubMed  Google Scholar 

  6. Wu, Y. & Shroff, H. Faster, sharper, and deeper: structured illumination microscopy for biological imaging. Nat. Methods 15, 1011–1019 (2018).

    Article  CAS  PubMed  Google Scholar 

  7. Huang, X. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451–459 (2018).

    Article  CAS  PubMed  Google Scholar 

  8. Zhao, W. et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat. Biotechnol. 40, 606–617 (2022).

    Article  CAS  PubMed  Google Scholar 

  9. de Haan, K., Rivenson, Y., Wu, Y. & Ozcan, A. Deep-learning-based image reconstruction and enhancement in optical microscopy. Proc. IEEE 108, 30–50 (2019).

    Article  Google Scholar 

  10. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    Article  CAS  PubMed  Google Scholar 

  11. Rahaman, N. et al. On the spectral bias of neural networks. Proc. of the 36th International Conference on Machine Learning. 5301–5310 (PMLR, 2019).

  12. Guo, Y. et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175, 1430–1442 (2018).

    Article  CAS  PubMed  Google Scholar 

  13. Shah, Z. H. et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images. Photonics Res. 9, B168–B181 (2021).

    Article  Google Scholar 

  14. Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095 (2007).

    Article  PubMed  Google Scholar 

  15. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  PubMed  Google Scholar 

  16. Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).

    Article  CAS  PubMed  Google Scholar 

  17. Kiepas, A., Voorand, E., Mubaid, F., Siegel, P. M. & Brown, C. M. Optimizing live-cell fluorescence imaging conditions to minimize phototoxicity. J. Cell Sci. 133, jcs242834 (2020).

    Article  CAS  PubMed  Google Scholar 

  18. Li, D. et al. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 349, aab3500 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  19. Ma, Y., Li, D., Smith, Z. J., Li, D. & Chu, K. Structured illumination microscopy with interleaved reconstruction (SIMILR). J. Biophotonics 11, e201700090 (2018).

    Article  Google Scholar 

  20. Oltean, A., Schaffer, A. J., Bayly, P. V. & Brody, S. L. Quantifying ciliary dynamics during assembly reveals stepwise waveform maturation in airway cells. Am. J. Respir. Cell Mol. Biol. 59, 511–522 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Jing, J. C., Chen, J. J., Chou, L., Wong, B. J. F. & Chen, Z. Visualization and detection of ciliary beating pattern and frequency in the upper airway using phase resolved doppler optical coherence tomography. Sci Rep. 7, 8522 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Chien, A. et al. Dynamics of the IFT machinery at the ciliary tip. eLife 6, e28606 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Ott, C. & Lippincott‐Schwartz, J. Visualization of live primary cilia dynamics using fluorescence microscopy. Curr. Protoc. Cell Biol. Chapter 4, Unit 4.26 (2012).

  24. Du, M. & Chen, Z. J. DNA-induced liquid phase condensation of cGAS activates innate immune signaling. Science 361, 704–709 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Lehtinen, J. et al. Noise2noise: learning image restoration without clean data. Preprint at https://arxiv.org/abs/1803.04189 (2018).

  26. Sengupta, P. et al. ER trapping reveals Golgi enzymes continually revisit the ER through a recycling pathway that controls Golgi organization. Proc. Natl Acad. Sci. USA 112, E6752–E6761 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Chen, B. C. et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Lu, L., Ladinsky, M. S. & Kirchhausen, T. Cisternal organization of the endoplasmic reticulum during mitosis. Mol. Biol. Cell 20, 3471–3480 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Kendall, A. & Gal, Y. What uncertainties do we need in Bayesian deep learning for computer vision? in Advances in Neural Information Processing Systems 30. https://papers.nips.cc/paper/2017/hash/2650d6089a6d640c5e85b2b88265dc2b-Abstract.html (NIPS, 2017).

  30. Liu, T.-L. et al. Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms. Science 360, eaaq1392 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  31. Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).

    Article  CAS  PubMed  Google Scholar 

  32. Valm, A. M. et al. Applying systems-level spectral imaging and analysis to reveal the organelle interactome. Nature 546, 162–167 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Lubeck, E. & Cai, L. Single-cell systems biology by super-resolution imaging and combinatorial labeling. Nat. Methods 9, 743–748 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Gustafsson, M. G. et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J. 94, 4957–4970 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Muller, M., Monkemoller, V., Hennig, S., Hubner, W. & Huser, T. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat. Commun. 7, 10980 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  36. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778 (2016).

  37. Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. https://ojs.aaai.org/index.php/AAAI/article/view/11231. Thirty-First AAAI Conference on Artificial Intelligence (2017).

  38. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition. 7132–7141 (2018).

  39. Zhang, Y. et al. Image super-resolution using very deep residual channel attention networks. Proc. of the European Conference on Computer Vision (ECCV) 286–301 (2018).

  40. Liu, S. et al. sCMOS noise-correction algorithm for microscopy images. Nat. Methods 14, 760–761 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Descloux, A., Grussmayer, K. S. & Radenovic, A. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods 16, 918–924 (2019).

    Article  CAS  PubMed  Google Scholar 

  42. Cleveland, W. S. LOWESS: a program for smoothing scatterplots by robust locally weighted regression. Am. Stat. 35, 54 (1981).

    Article  Google Scholar 

  43. Qian, H., Sheetz, M. P. & Elson, E. L. Single particle tracking. Analysis of diffusion and flow in two-dimensional systems. Biophys. J. 60, 910–921 (1991).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Ba, Q., Raghavan, G., Kiselyov, K. & Yang, G. Whole-cell scale dynamic organization of lysosomes revealed by spatial statistical analysis. Cell Rep. 23, 3591–3606 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Grimm, J. B. et al. A general method to improve fluorophores for live-cell and single-molecule microscopy. Nat. Methods 12, 244–250 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Zhao, H. et al. Fibrogranular materials function as organizers to ensure the fidelity of multiciliary assembly. Nat. Commun. 12, 1273 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Delgehyr, N. et al. Ependymal cell differentiation, from monociliated to multiciliated cells. Methods Cell. Biol. 127, 19–35 (2015).

    Article  CAS  PubMed  Google Scholar 

  48. Sanjana, N. E. et al. A transcription activator-like effector toolbox for genome engineering. Nat. Protoc. 7, 171–192 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors thank T. Kirchhausen for the donor plasmids used for genome editing and help in generating the genome-edited cell lines. This work was supported by grants from the National Natural Science Foundation of China (31827802, 32125024, 31970659 and 62088102); the Ministry of Science and Technology (2021YFA1300303 and 2020AA0105500); the Chinese Academy of Sciences (ZDBS-LY-SM004 and XDA16021401); the Collaborative Research Fund of the Chinese Institute for Brain Research, Beijing (2021-NKX-XM-03); the Tencent Foundation through the XPLORER PRIZE; the Shuimu Tsinghua Scholar Program; and the Youth Innovation Promotion Association of the Chinese Academy of Sciences (2020094).

Author information

Authors and Affiliations

Authors

Contributions

Dong Li and C.Q. conceived the idea. Dong Li and Q.D. supervised the research. Dong Li, C.Q. and Di Li designed the experiments. Di Li, Y.L. and S.Z. built and improved the microscope systems at the Institute of Biophysics, under the supervision of Dong Li. Di Li, Y.L., C.L., Y.G., T.J., C.F. and N.L. prepared samples and performed experiments. C.Q., Di Li, Y.L., K.L. and Y.Z. analyzed the data, with conceptual advice from Dong Li, K.H., X.Z., and J.L.-S. C.Q., Di Li and Y.L. composed the figures and videos, under the supervision of Dong Li. Dong Li, C.Q., Q.D. and J.L.-S. wrote the manuscript, with input from all authors. All authors discussed the results and commented on the manuscript.

Corresponding authors

Correspondence to Jennifer Lippincott-Schwartz, Qionghai Dai or Dong Li.

Ethics declarations

Competing interests

Dong Li and C.Q. have pending patent applications on the presented frameworks. The other authors declare no competing interests.

Peer review

Peer review information

Nature Biotechnology thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Illustration of the spectral bias effect in deep learning super-resolution (DLSR) task.

a, Concepts (upper row) and typical results (lower row) of wide-field, DFCAN-SIM, rDL SIM and conventional SIM imaging. b, Typical DFCAN-SIM images of F-actin output at 4000, 24000, 52000, 100000, 200000, and 400,000 training iterations, respectively. c, The evolutions of the training loss and the resolution of the intermediate outputs during the training process of the DFCAN-SIM model of F-actin dataset (averaged from n = 20 test images). The curves shows that there exists a significant resolution gap between DFCAN-SIM (red dashed line) and GT-SIM (black dashed line) images, which implies DLSR models suffer from severe spectral bias. d, The corresponding GT-SIM image of the same ROI with (b) for reference. e, Typical rDL denoised images (top left) and rDL SIM images (bottom right) of F-actin output at 4000, 24000, 52000, 100000, 200000, and 400,000 training iterations, respectively. f, The evolutions of the training loss and the resolution of the intermediate outputs during the training process of the rDL denoising model of F-actin dataset (averaged from n = 180 test images). Compared with DLSR task in b and c, the rDL denoising of raw SIM images is hardly impaired by the spectral bias problem, preventing the resolution degradation of rDL SIM image relative to the GT-SIM image, as illustrated in (a). g, The corresponding ground truth raw SIM image of the same ROI with (e) for reference. Scale bar, 3 μm (a, b, d, e, g), 0.5 μm (zoom-in regions of a, b, d, e, g). Gamma value, 0.7 for DFCAN-SIM images, rDL SIM images and GT-SIM images.

Source data

Extended Data Fig. 2 Generalization and Robustness validation of rDL SIM.

a, Typical SR images of F-actin, MTs and CCPs reconstructed by rDL SIM (top right corner) and CARE-SIM (bottom left corner) trained with only F-actin data. Zoom-in regions show the comparisons of raw SIM images, GT-SIM images and SR images reconstructed by conventional SIM (Conv. SIM), CARE-SIM, rDL SIM trained with F-actin data (orange) and rDL SIM trained with corresponding specimens (red). These results illustrate the strong generalization capability of rDL SIM. b, PSNR quantifications of CARE-SIM and rDL-SIM in terms of F-actin, MTs and CCPs, where both models are trained with only F-actin data. The PSNRs of Conv. SIM are shown for reference, n = 145. Center line, medians; limits, 75% and 25%; whiskers, the larger value between the largest data point and the 75th percentiles plus 1.5× the interquartile range (IQR), and the smaller value between the smallest data point and the 25th percentiles minus 1.5× the IQR; outliers, data points larger than the upper whisker or smaller than the lower whisker. c, Plots of amplitudes (left) and orientations (right) of the pattern wave vector for the 45,000-time-point ER data shown in Fig. 3h and Supplementary Video 8. The pattern parameters are estimated from the single time point of raw SIM images (gray lines), averaged 20 consecutive time points of raw SIM images (black lines), single time point of CARE denoised images (blue lines) and rDL denoised images (orange lines). d, Representative rDL SIM images of F-actin with the pattern initial phase shifted by -0.2π, -0.1π, 0, 0.1π and 0.2π, respectively, relative to that of GT-SIM. Conv. SIM and GT-SIM images are shown for reference. The PSNRs are labelled at the top right of all Conv. SIM images and rDL SIM images. e, PSNR quantifications of the rDL SIM images at different phase errors under different fluorescence intensity levels in the cases of F-actin, n = 50. f, Representative rDL SIM images of F-actin with spherical or coma aberration distorted PSF in MPE branch. Conv. SIM and GT-SIM images are shown for reference. Scale bar, 1μm (a), 0.5 μm (zoom-in regions in a), 1.5 μm (d and f), and 0.5 μm (zoom-in regions in d and f). Gamma value: 0.7 for all SIM images of F-actin.

Source data

Extended Data Fig. 3 Uncertainty comparison of DFCAN-SIM, DFGAN-SIM and rDL SIM models.

a, Representative SR images of F-actin produced by the three models of DFCAN-SIM, DFGAN-SIM and rDL SIM that are independently trained with the same dataset (upper), and the corresponding standard deviation (SD) maps (lower). b, The magnified regions of the white box in (a). c, The raw SIM image, Conv. SIM and GT-SIM images shown for reference. The arrows indicate the fine structures of F-actin that exhibit large reconstruction variation by both DFCAN- and DFGAN-SIM models, whereas that are stably reconstructed by rDL SIM with high consistency to the GT-SIM image. Scale bar, 1 μm (a), 0.3 μm (b, c). Gamma value, 0.7 for all SIM images of F-actin.

Extended Data Fig. 4 rDL SIM resolves the ring structure of CCPs and visualizes multiple IFT trains’ unusual behaviors in ependymal motile cilia.

a, Representative SR images of CCPs reconstructed by conventional (Conv.) SIM, CARE-SIM and rDL SIM, whose raw images were acquired at 18-fold lower fluorescence than those of GT-SIM. It shows that rDL-SIM provides high-fidelity ring-like structure and prevents the reconstruction artifacts. (Supplementary Video 4). b-d, Kymographs (left) and time-lapse images (right) show that, (b) IFT trains stopped halfway or paused for a while before reaching the ciliary base; (c) IFT trains reversed their direction of motion in the midway of the microtubule (MT) tracks; (d) Two IFT trains moving along the same MT track collided with each other, which remodeled them into three independent IFT trains. As described in Fig. 3d, the IFT trains are resolved into three groups that transported along the microtubule tracks residing at the left (green), middle (magenta), and right (yellow) sides of the ciliary axoneme. Scale bar, 5 μm (a) and 0.5 μm (zoom-in regions of a, b-d).

Extended Data Fig. 5 Interactions between endoplasmic reticulum (ER) and peroxisomes (POs) as seen by rDL GI-SIM.

a, Comparisons of single raw SIM image (left), rDL denoised raw image (middle) and rDL GI-SIM image (right) in the case of two-color live-cell imaging of ER and POs. Scale bar, 2μm. b, Time-lapse two-color images of ER and POs, and the corresponding dynamic characteristics of Mander’s overlap coefficient (MOC) for POs that are stably contacted with ER over time (top and middle rows) or that is freely moving without stable association with ER (bottom row). Scale bar, 0.5 μm. c, Distribution of the ER-PO MOCs’ standard deviation (S.D.) versus their mean values, n = 137. d, Time-lapse images of a tubular ER generation event by hitchhiking on a moving PO. Scale bar, 1 μm. Gamma value, 0.7 for rDL GI-SIM images of ER, and 0.8 for rDL GI-SIM images of POs.

Source data

Extended Data Fig. 6 Visualizing the behaviors and interactions of NPM1 and RPA49 during cell mitosis via rDL LLS-SIM.

a, 3-D rendering of three-color time-lapse images of fibrillar centre (FC, labeled by Halo-RPA49) and granular component (GC, labeled by mEmerald-NPM1), and chromosome (labeled by mCherry-H2B) at different stages of mitosis in a HeLa cell, knocking-in mEmerald-NPM1 and stably expressing mCherry-H2B and Halo-RPA49. b, Time-lapse max intensity projections of NPM1 and RPA49 showing that the dissociation of RNA polymerase I subunits (e.g., RPA49) from the innermost FC preceded the disassembly of the outermost GC during prophase to prometaphase. c, Time-lapse max intensity projections of NPM1 and RPA49 showing that small nucleolar-derived foci (NDFs) of NPM1 coalesced into giant ones (indicated by arrows in the second row), while the innermost RPA49 marked FC foci of large size could be partitioned into several independent small foci during telophase (arrows in the third row). Scale bar, 6 μm (a), 3 μm (b), 2 μm (c).

Extended Data Fig. 7 Temporally and Spatially interleaved self-supervised learning of rDL denoising networks.

a, Representative LLSM images of Golgi SiT-vesicles denoised by TiS-rDL self-supervised and rDL supervised learning models. Depth color-coded maximum intensity projections (MIPs) of input noisy raw data (left), the images stack denoised by the TiS-rDL model (middle) and by the supervised learning model (right). These results illustrate the performance of TiS-rDL denoising model is as good as the rDL model trained with the supervision of high-SNR images. b, c, Representative TiS-rDL denoised images of Golgi SiT-vesicles (b) and statistical analysis in terms of PSNR under the conditions of temporally down-sampled at different factors (c), n = 50. Center line, medians; limits, 75% and 25%; whiskers, maximum and minimum. Noisy raw images and images denoised by the supervised model are shown for reference. These results indicate that the decrease of temporal sampling rate gradually affects the output fidelity of the TiS-rDL method. d, e, Schematic of SiS-rDL denoising network training (d) and inference (e). f, g, Representative outputs (f) and PSNR comparisons (g) of SiS-rDL denoising models with and without gap-amending regularization (GAR) in terms of ER and Mito in mitotic cells at different axial sampling intervals of 50, 100, 200, 300, 400, 500, and 600 nm, n = 31 for ER, 42 for Mito. Center line, medians; limits, 75% and 25%; whiskers, 95% and 5%. These results show that GAR effectively improves the performance of SiS-rDL models; The PSNR of SiS-rDL denoised images decreases by less than 4% before the axial sampling rate is lower than Nyquist criterium, that is, 300 nm for LLSM, which suggests the general applicability of the SiS-rDL method for 3D imaging. Scale bar, 2 μm (a, b), 5 μm (f), 3 μm (zoom-in regions of f).

Source data

Extended Data Fig. 8 Three-color rDL LLSM images of ER, H2B, and Golgi apparatus at different stages of mitosis.

Three-color 3-D rendering (left panel) and single slices of X-Y view (right panel) showing the remodeling process of Golgi apparatus, as well as their interactions with ER at different stages of mitosis (A-H). Scale bar, 10 μm.

Extended Data Fig. 9 3-D rendering of three-color rDL LLSM images of ER, H2B, and mitochondria (Mito) at different stages of mitosis.

For each stage of mitosis, images of volume rendering (first row), single slices of X-Y view (second row), color-coded in z axis surface rendering of Mito in X-Y view (third row) and X-Z view (forth row), respectively, are shown to visualize the segregation process of Mito and their interaction dynamics with ER during mitosis. Scale bar, 8 μm.

Extended Data Fig. 10 Sustained live imaging enabled by rationalized deep learning microscopy.

a, Flowchart of the rDL SIM denoising and reconstruction algorithm. Scale bar, 1 μm. b, Synopsis of rDL imaging modalities, including (i) rDL TIRF-SIM, rDL GI-SIM, rDL 3D-SIM, and rDL LLS-SIM that utilize the physical model of SIM to guide the network training and inference processes; (ii) rDL-TiS LLSM and rDL-SiS LLSM, which utilize the spatial/temporal continuity of acquired biological data to implement self-supervised denoising, yielding comparable results to the supervised methods. The rDL methods enable investigations into the fine spatial details, rapid kinetics and long-time dynamics of a variety of bioprocesses, showing great promise for shedding light on diverse biological phenomena.

Supplementary information

Supplementary Information

Glossary of Abbreviations, Supplementary Notes 1–3, Supplementary Figs. 1–12, Supplementary Tables 1–4, Captions for Supplementary Videos 1–17 and Supplementary References

Reporting Summary

Supplementary Video 1

Spatial and temporal resolution degradation of rDL SIM versus Hessian-SIM. SIM images reconstructed by Hessian-SIM (μ = 150, σ = 1) (left panel) and rDL SIM (middle panel) of CCPs in a SUM159 cell for 149 timepoints at 1-second intervals. The corresponding GT-SIM images (right panel) are shown for reference. The trajectory of a CCP is delineated to illustrate the temporal degradation effect by Hessian-SIM.

Supplementary Video 2

Comparison of raw SIM images denoising by deep learning algorithms with and without physical model rationalization. Top left: raw SIM images acquired at the illumination patterns of 3-phase × 3-orienation, which are the inputs of the rDL and CARE denoising DNNs. Top right: the raw SIM images denoised with CARE network without physical model rationalization. Bottom left: the raw SIM images denoised with rDL network rationalized with SIM’s physical model. Bottom right: the raw SIM images acquired at high SNR condition serving as the GT for reference

Supplementary Video 3

The coordinated remodeling dynamics of F-actin and myosin-IIA over hour-long settling process after dropping a U2OS cell onto the coverslip. rDL TIRF-SIM records the adhesion and spreading dynamics of a U2OS cell co-expressing mEmerald-Lifeact (green) and mCherry-myosin-IIA (magenta) for 749 timepoints at 5-second intervals.

Supplementary Video 4

Long-term rDL TIRF-SIM imaging and tracking of CCPs. Formation, maturation and internalization of endogenous clathrin-EGFP in a live SUM159 cell for 5,500 timepoints with a time duration of 45.8 minutes at 1-second intervals.

Supplementary Video 5

Ciliary beating dynamics by rDL GI-SIM of ultra-high spatiotemporal resolution. Ciliary beating behaviors recorded at the spatiotemporal resolution of 97 nm and 684 Hz for ~60,000 frames without notable phototoxicity and photobleaching effects in live mEPCs.

Supplementary Video 6

The transportation dynamics of the IFT trains along cilia by two-color rDL GI-SIM. The spatiotemporal coordination and remodeling of the IFT trains in live mEPCs expressing EGFP-IFT81 (red) and stained by LysoTracker (blue) as seen by rDL GI-SIM for 1,946 timepoints at 0.09-second intervals.

Supplementary Video 7

A set of examples of IFT train transportation and remodeling dynamics along the three groups of MT tracks in the axoneme of live mEPCs. Right: two-color rDL GI-SIM imaging of live mEPCs expressing EGFP-IFT81 (red) and stained by LysoTracker (blue). Left: kymography of the IFT trains indicated by the arrows in right. The IFT trains are resolved into three groups that transported along the MT tracks residing at the left (green), middle (magenta) and right (yellow) sides of the ciliary axoneme.

Supplementary Video 8

Dynamic interactions between ER and POs. Two-color rDL GI-SIM imaging of ER–PO contacts at high spatiotemporal resolution of 97 nm and 182 Hz for long duration of 45,000 timepoints in a COS-7 cell expressing mEmerald-KDEL (blue) and mCherry-SKL (orange).

Supplementary Video 9

Dynamic interactions between ER and the condensates of cGAS-DNA. Two-color rDL GI-SIM imaging of HEK293T cell stably expressing cGAS-Halo and mEmerald-KDEL for 1,000 timepoints at 1-second intervals, revealing the phase separation process of cGAS-DNA condensates growing progressively and their interaction with the surrounding ER tubules.

Supplementary Video 10

Examples of cGAS-DNA condensates stably associated with ER. Two-color rDL GI-SIM imaging at 1-second intervals in a second HEK293T cell stably expressing cGAS-Halo and mEmerald-KDEL, revealing that cGAS-DNA condensates are tightly associated with adjacent ER tubules or even corralled by ER polygons over time.

Supplementary Video 11

Dynamic interactions between ER and cGAS-DNA in another HEK293T cell. Long-time rDL GI-SIM for 2,000 timepoints at 1-second intervals in another HEK293T cell stably expressing cGAS-Halo and mEmerald-KDEL, revealing that cGAS-DNA condensates traffic over a long range and sometimes changes the direction of their movement, during which the ER keeps contacting with the moving cGAS-DNA condensates.

Supplementary Video 12

Volumetric dynamics of mitochondrial cristae by rDL 3D-SIM. Long-time rDL 3D-SIM imaging of a COS-7 cell expressing PHB2-mEmerald for 448 timepoints at 3-second intervals.

Supplementary Video 13

Disassemble and reassemble process of nucleoli over the entire mitosis as seen by rDL LLS-SIM. Three-color volumetric SR imaging of NPM1, RPA49 and chromosomes at the speed of 12 seconds per three-color volume for 748 timepoints from a HeLa cell of knocking-in mEmerald-NPM1 and stably expressing mCherry-H2B and Halo-RPA49, revealing the spatiotemporally coordinated multi-phase separation dynamics of nucleoli during mitosis.

Supplementary Video 14

Ultra-fast whole-cell imaging of the dynamic behaviors of SiT-Golgi vesicles in 3D color-coded representation as seen by rDL LLSM. Volumetric imaging at 10 Hz (1,000 z-slices per second) reveals the ultra-dynamics of SiT-Golgi vesicles across the entire intracellular space of an R1 mammalian ependymal cell expressing SiT-Halo-JF642 over 10,000 timepoints.

Supplementary Video 15

The remodeling and segregating dynamics of Golgi apparatus during mitosis. Three-color rDL LLSM imaging for 1,000 timepoints at 6-second intervals in a HeLa cell stably expressing calnexin-mEmerald (ER in gray), H2B-mCherry (chromosome in green) and SiT-Halo (Golgi in yellow).

Supplementary Video 16

Dynamic interactions between ER and mitochondria during mitosis. Three-color rDL LLSM imaging for 890 timepoints at 6-second intervals in a HeLa cell stably expressing calnexin-mEmerald (ER in gray), H2B-mCherry (chromosome in green) and Mito-Halo (mitochondria in blue).

Supplementary Video 17

Self-aggregation and deaggregation dynamics of lysosomes (Lyso) during mitosis. Three-color rDL LLSM imaging for 879 timepoints at 6-second intervals in a HeLa cell stably expressing calnexin-mEmerald (ER in gray), H2B-mCherry (chromosome in green) and Lamp1-Halo (Lyso in orange).

Source data

Source Data Fig. 1

Statistical Source Data

Source Data Fig. 2

Statistical Source Data

Source Data Fig. 3

Statistical Source Data

Source Data Fig. 5

Statistical Source Data

Source Data Fig. 6

Statistical Source Data

Source Data Extended Data Fig. 1

Statistical Source Data

Source Data Extended Data Fig. 2

Statistical Source Data

Source Data Extended Data Fig. 5

Statistical Source Data

Source Data Extended Data Fig. 7

Statistical Source Data

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiao, C., Li, D., Liu, Y. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat Biotechnol 41, 367–377 (2023). https://doi.org/10.1038/s41587-022-01471-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41587-022-01471-3

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing