Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning

Nature Methods volume 20, pages 1939–1948 (2023)Cite this article

Subjects

Abstract

Single-molecule localization microscopy (SMLM) has revolutionized biological imaging, improving the spatial resolution of traditional microscopes by an order of magnitude. However, SMLM techniques require long acquisition times, typically a few minutes, to yield a single super-resolved image, because they depend on accumulation of many localizations over thousands of recorded frames. Hence, the capability of SMLM to observe dynamics at high temporal resolution has always been limited. In this work, we present DBlink, a deep-learning-based method for super spatiotemporal resolution reconstruction from SMLM data. The input to DBlink is a recorded video of SMLM data and the output is a super spatiotemporal resolution video reconstruction. We use a convolutional neural network combined with a bidirectional long short-term memory network architecture, designed for capturing long-term dependencies between different input frames. We demonstrate DBlink performance on simulated filaments and mitochondria-like structures, on experimental SMLM data under controlled motion conditions and on live-cell dynamic SMLM. DBlink’s spatiotemporal interpolation constitutes an important advance in super-resolution imaging of dynamic processes in live cells.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: DBlink concept.
Fig. 2: Generation and analysis of simulated filament data.
Fig. 3: Static structure reconstruction during global motion.
Fig. 4: Tracking dynein motors dynamically moving on microtubules.
Fig. 5: Qualitative comparison of DBlink with other state-of-the-art methods.
Fig. 6: Reconstruction of a 12.5 min long video of mitochondria dynamics in a live cell.

Similar content being viewed by others

Data availability

The datasets analyzed during the current study are available from the corresponding author upon request. The datasets generated during this study are available online: https://doi.org/10.5281/zenodo.7023414.

Code availability

The code is available online at https://github.com/alonsaguy/DBlink.

References

  1. Hell, S. W. & Wichmann, J. Breaking the diffraction resolution limit by stimulated-emission—stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 19, 780–782 (1994).

    CAS  PubMed  Google Scholar 

  2. Gustafsson, M. G. L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc. 198, 82–87 (2000).

    CAS  PubMed  Google Scholar 

  3. Sauer, M. & Heilemann, M. Single-molecule localization microscopy in eukaryotes. Chem. Rev. 117, 7478–7509 (2017).

    CAS  PubMed  Google Scholar 

  4. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

    CAS  PubMed  Google Scholar 

  5. Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  6. Sharonov, A. & Hochstrasser, R. Wide-field subdiffraction imaging by accumulated binding of diffusing probes. Proc. Natl Acad. Sci. USA 103, 18911–18916 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Schnitzbauer, J., Strauss, M. T., Schlichthaerle, T., Schueder, F. & Jungmann, R. Super-resolution microscopy with DNA-PAINT. Nat. Protoc. 12, 1198–1228 (2017).

    CAS  PubMed  Google Scholar 

  8. Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

    CAS  PubMed  Google Scholar 

  9. Wang, Y. et al. Blind sparse inpainting reveals cytoskeletal filaments with sub-Nyquist localization. Optica 4, 1277–1284 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  10. Barentine, A. E. S. et al. An integrated platform for high-throughput nanoscopy. Nat. Biotechnol. https://doi.org/10.1038/s41587-023-01702-1 (2023).

  11. Liu, Z., Lavis, L. D. & Betzig, E. Imaging live-cell dynamics and structure at the single-molecule level. Mol. Cell 58, 644–659 (2015).

    CAS  PubMed  Google Scholar 

  12. Nehme, E., Weiss, L. E., Michaeli, T. & Schechtman, Y. Deep-STORM: super-resolution single-molecule microscopy by deep learning.Optica 5, 458–464 (2018).

    CAS  Google Scholar 

  13. Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 734–740 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  14. Speiser, A. et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Nat. Methods 18, 1082–1090 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Nixon-Abell, J. et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral ER. Science 354, aaf3928 (2016).

    PubMed  PubMed Central  Google Scholar 

  16. Priessner, M. et al. Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging. Preprint at bioRxiv https://doi.org/10.1101/2021.11.02.466664 (2021).

  17. Chen, R. et al. Single-frame deep-learning super-resolution microscopy for intracellular dynamics imaging. Nat. Commun. 14, 2854 (2023).

    CAS  PubMed  PubMed Central  Google Scholar 

  18. Dertinger, T., Colyer, R., Iyer, G., Weiss, S. & Enderlein, J. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI). Proc. Natl Acad. Sci. USA 106, 22287–22292 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  19. Gustafsson, N. et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  20. Agarwal, K. & Macháň, R. Multiple signal classification algorithm for super-resolution fluorescence microscopy. Nat. Commun. 7, 13752 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  21. Laine, R. F. et al. High-fidelity 3D live-cell nanoscopy through data-driven enhanced super-resolution radial fluctuation. Preprint at bioRxiv https://doi.org/10.1101/2022.04.07.487490 (2022).

  22. Ovesný, M., Křížek, P., Borkovec, J., Švindrych, Z. & Hagen, G. M. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).

    PubMed  PubMed Central  Google Scholar 

  23. Su, Y.-T., Lu, Y., Chen, M. & Liu, A.-A. Spatiotemporal joint mitosis detection using CNN-LSTM network in time-lapse phase contrast microscopy images. IEEE Access 5, 18033–18041 (2017).

    Google Scholar 

  24. Yu, Y., Si, X., Hu, C. & Zhang, J. A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput. 31, 1235–1270 (2019).

    PubMed  Google Scholar 

  25. Shariff, A., Murphy, R. F. & Rohde, G. K. A generative model of microtubule distributions, and indirect estimation of its parameters from fluorescence microscopy images. Cytometry A 77, 457–466 (2010).

    PubMed  PubMed Central  Google Scholar 

  26. Banterle, N., Bui, K. H., Lemke, E. A. & Beck, M. Fourier ring correlation as a resolution criterion for super-resolution microscopy. J. Struct. Biol. 183, 363–367 (2013).

    CAS  PubMed  Google Scholar 

  27. Descloux, A., Grußmayer, K. S. & Radenovic, A. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods 16, 918–924 (2019).

    CAS  PubMed  Google Scholar 

  28. Niekamp, S., Coudray, N., Zhang, N., Vale, R. D. & Bhabha, G. Coupling of ATPase activity, microtubule binding, and mechanics in the dynein motor domain. EMBO J. 38, e101414 (2019).

    PubMed  PubMed Central  Google Scholar 

  29. Kompa, J. et al. Exchangeable HaloTag ligands for super-resolution fluorescence microscopy. J. Am. Chem. Soc. 145, 3075–3083 (2023).

    CAS  PubMed  PubMed Central  Google Scholar 

  30. Lefebvre, A. E. Y. T., Ma, D., Kessenbrock, K., Lawson, D. A. & Digman, M. A. Automated segmentation and tracking of mitochondria in live-cell time-lapse images. Nat. Methods 18, 1091–1102 (2021).

    CAS  PubMed  Google Scholar 

  31. Yang, X. et al. Mitochondrial dynamics quantitatively revealed by STED nanoscopy with an enhanced squaraine variant probe. Nat. Commun. 11, 3699 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Friedman, J. R. & Nunnari, J. Mitochondrial form and function. Nature 505, 335–343 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  33. Quiles, J. M. & Gustafsson, Å. B. The role of mitochondrial fission in cardiovascular health and disease. Nat. Rev. Cardiol. 19, 723–736 (2022).

    CAS  PubMed  PubMed Central  Google Scholar 

  34. Kleele, T. et al. Distinct fission signatures predict mitochondrial degradation or biogenesis. Nature 593, 435–439 (2021).

    CAS  PubMed  Google Scholar 

  35. Tachibana, R. et al. Design of spontaneously blinking fluorophores for live-cell super-resolution imaging based on quantum-chemical calculations. Chem. Commun. 56, 13173–13176 (2020).

    CAS  Google Scholar 

  36. Möckl, L., Roy, A. R. & Moerner, W. E. Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]. Biomed. Opt. Express 11, 1633 (2020).

    PubMed  PubMed Central  Google Scholar 

  37. Matlock, A., Zhu, J. & Tian, L. Multiple-scattering simulator-trained neural network for intensity diffraction tomography. Opt. Express 31, 4094–4107 (2023).

    PubMed  Google Scholar 

  38. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    CAS  PubMed  Google Scholar 

  39. Spahn, C., Grimm, J. B., Lavis, L. D., Lampe, M. & Heilemann, M. Whole-Cell, 3D, and multicolor STED imaging with exchangeable fluorophores. Nano Lett. 19, 500–505 (2019).

    CAS  PubMed  Google Scholar 

  40. Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 5998–6008 (2017).

    Google Scholar 

  41. Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. Preprint at https://doi.org/10.48550/arXiv.2010.11929 (2020).

  42. Wensel, T. G., Potter, V. L., Moye, A., Zhang, Z. & Robichaux, M. A. Structure and dynamics of photoreceptor sensory cilia. Pflug. Arch. 473, 1517–1537 (2021).

    CAS  Google Scholar 

  43. Guggenheim, E. J. et al. Comparison of confocal and super-resolution reflectance imaging of metal oxide nanoparticles. PLoS ONE 11, e0159980 (2016).

    PubMed  PubMed Central  Google Scholar 

  44. van der Zwaag, D. et al. Super resolution imaging of nanoparticles cellular uptake and trafficking. ACS Appl. Mater. Interfaces 8, 6391–6399 (2016).

    PubMed  Google Scholar 

  45. Pujals, S., Feiner-Gracia, N., Delcanale, P., Voets, I. & Albertazzi, L. Super-resolution microscopy as a powerful tool to study complex synthetic materials. Nat. Rev. Chem. 3, 68–84 (2019).

    Google Scholar 

  46. Valli, J. et al. Seeing beyond the limit: a guide to choosing the right super-resolution microscopy technique. J. Biol. Chem. 297, 100791 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  47. Mahecic, D. et al. Mitochondrial membrane tension governs fission. Cell Rep. 35, 108947 (2021).

    CAS  PubMed  Google Scholar 

  48. Mahecic, D. et al. Event-driven acquisition for content-enriched microscopy. Nat. Methods 19, 1262–1267 (2022).

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

Live-cell data of dynamic microtubules were generously shared with us by Y. Urano and M. Kamiya (University of Tokyo). Data containing dynein moving on microtubule filaments were generously shared with us by S. Niekamp and R. Vale (University of California, San Francisco). We thank R. Henriques (Instituto Gulbenkian de Ciência, Portugal, and UCL Honorary Chair of Computational and Optical Biophysics) for his valuable input. We thank J. Kompa and K. Johnsson (MPI Medical Research, Heidelberg, and EPFL) for kindly providing the plasmid COX8A-HaloTag7. M.H. and S.J. acknowledge funding by the International Max-Planck Research School on Cellular Biophysics (IMPRS-CBP). M.H. acknowledges funding by LOEWE (FCI) and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): SFB1177; INST 161/926-1 FUGG. A.S. has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 802567-ERC-Five-Dimensional Localization Microscopy for Sub-Cellular Dynamics. Y.S. is supported by the Zuckerman Foundation.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Technion–Israel Institute of Technology, Haifa, Israel

    Alon Saguy, Onit Alalouf & Yoav Shechtman

  2. Russell Berrie Nanotechnology Institute, Technion–Israel Institute of Technology, Haifa, Israel

    Nadav Opatovski

  3. Institute of Physical and Theoretical Chemistry, Goethe-University Frankfurt, Frankfurt, Germany

    Soohyen Jang & Mike Heilemann

  4. Institute of Physical and Theoretical Chemistry, IMPRS on Cellular Biophysics, Goethe-University Frankfurt, Frankfurt, Germany

    Soohyen Jang & Mike Heilemann

Authors
  1. Alon Saguy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Onit Alalouf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nadav Opatovski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Soohyen Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mike Heilemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yoav Shechtman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.S. and Y.S. devised the main manuscript concept, developed the neural network architecture, planned the manuscript experiments and wrote the paper with the help of all contributing authors. O.A. has prepared microtubule samples for static STORM imaging. N.O. contributed to the design and execution of the static STORM experiments. S.J. and M.H. performed dynamic super-resolution imaging in living cells.

Corresponding author

Correspondence to Yoav Shechtman.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Methods thanks Jaroslaw Jacak, Doory Kim and Thomas Pengo for their contribution to the peer review of this work. Primary Handling Editor: Rita Strack, in collaboration with the Nature Methods team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–10, Tables 1–3, Discussion and Video 1–17 captions.

Reporting Summary

Supplementary Video 1

Super spatiotemporal resolution reconstruction of simulated filaments. Temporal resolution of the reconstruction is one reconstructed frame per ten simulated blinking frames.

Supplementary Video 2

DBlink reconstruction of a static STORM experiment exhibiting unwanted drift. Temporal resolution, 5 s (0.2 fps).

Supplementary Video 3

DBlink reconstruction of a static STORM experiment exhibiting global motion due to camera rotation. Temporal resolution, 0.8 s (1.25 fps).

Supplementary Video 4

Raw data of dynein motors (white) moving on static microtubules (red).

Supplementary Video 5

DBlink reconstruction based on Deep-STORM localizations of dynein motors moving on static microtubules reconstruction generated by ThunderSTORM22. Temporal resolution, 50 s (0.02 fps).

Supplementary Video 6

Deep-STORM reconstructions of live-cell microtubules when summing localizations over temporal windows of lengths 500, 100 and 20 frames, followed by DBlink reconstruction at super spatiotemporal resolution. Spatial resolution, 30 nm; temporal resolution, 15 ms (66.6 fps).

Supplementary Video 7

Reconstruction of live-cell ER. Comparison between DBlink and DECODE. DBlink spatial resolution, 30 nm; temporal resolution, 15 ms (66.6 fps).

Supplementary Video 8

Reconstruction of live-cell microtubules. Comparison between DBlink at 66.6 fps and eSRRF at 0.66 and 3.33 fps.

Supplementary Video 9

DBlink reconstruction of live-cell mitochondria over an extended experiment duration of 12.5 min at super spatiotemporal resolution. Spatial resolution, 75 nm; temporal resolution, 500 ms (2 fps).

Supplementary Video 10

Focus on two regions of interest from the live-cell mitochondria sample. Spatial resolution, 75 nm; temporal resolution, 50 ms (20 fps).

Supplementary Video 11

Comparison between mitochondria training data and DBlink reconstruction of experimental data. The reconstruction contains new structures and motions never seen in the training stage, demonstrating the generalizability of the DBlink model.

Supplementary Video 12

Confidence map of reconstructed data in simulation.

Supplementary Video 13

Confidence map of reconstructed live-cell microtubule experiment. Spatial resolution, 30 nm; temporal resolution, 15 ms (66.6 fps).

Supplementary Video 14

Confidence map of reconstructed live-cell mitochondria experiment. Spatial resolution, 75 nm; temporal resolution, 500 ms (2 fps).

Supplementary Video 15

Performance comparison between one-directional and bidirectional LSTM.

Supplementary Video 16

Simulated videos of filament data. At a certain timepoint of the experiment some filaments appear, and after some time they disappear. Left, localization maps (input to DBlink). Center, DBlink reconstruction. Right, ground truth.

Supplementary Video 17

DBlink reconstruction of simulated data based on STED imaging of mitochondrial dynamics. Left to right, input localization maps; DBlink reconstruction; ground truth data generated by STED experiment; simulated diffraction limited data. Scale bar, 2 μm.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saguy, A., Alalouf, O., Opatovski, N. et al. DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning. Nat Methods 20, 1939–1948 (2023). https://doi.org/10.1038/s41592-023-01966-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-023-01966-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing