Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Live-cell imaging powered by computation

Nature Reviews Molecular Cell Biology (2024)Cite this article

Subjects

Abstract

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Live-cell imaging and deep learning: overview, examples and limitations.
Fig. 2: Comparison of denoising methods.
Fig. 3: Deconvolution and deep learning for image restoration.

Similar content being viewed by others

References

  1. Ducret, A., Quardokus, E. M. & Brun, Y. V. MicrobeJ, a tool for high throughput bacterial cell detection and quantitative analysis. Nat. Microbiol. 1, 16077 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Levet, F. et al. SR-Tesseler: a method to segment and quantify localization-based super-resolution microscopy data. Nat. Methods 12, 1065–1071 (2015).

    Article  CAS  PubMed  Google Scholar 

  3. Rizk, A. et al. Segmentation and quantification of subcellular structures in fluorescence microscopy images using Squassh. Nat. Protoc. 9, 586–596 (2014).

    Article  CAS  PubMed  Google Scholar 

  4. Ulman, V. et al. An objective comparison of cell-tracking algorithms. Nat. Methods 14, 1141–1152 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Axelrod, D., Koppel, D. E., Schlessinger, J., Elson, E. & Webb, W. W. Mobility measurement by analysis of fluorescence photobleaching recovery kinetics. Biophys. J. 16, 1055–1069 (1976).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  6. Elson, E. L. & Magde, D. Fluorescence correlation spectroscopy. I. Conceptual basis and theory. Biopolymers 13, 1–27 (1974).

    Article  CAS  Google Scholar 

  7. Gahlmann, A. & Moerner, W. E. Exploring bacterial cell biology with single-molecule tracking and super-resolution imaging. Nat. Rev. Microbiol. 12, 9–22 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Saxton, M. J. & Jacobson, K. Single-particle tracking: applications to membrane dynamics. Annu. Rev. Biophys. Biomol. Struct. 26, 373–399 (1997).

    Article  CAS  PubMed  Google Scholar 

  9. Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 12, e1005177 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  PubMed  Google Scholar 

  12. Laissue, P. P., Alghamdi, R. A., Tomancak, P., Reynaud, E. G. & Shroff, H. Assessing phototoxicity in live fluorescence imaging. Nat. Methods 14, 657–661 (2017).

    Article  CAS  PubMed  Google Scholar 

  13. Grimm, J. B. et al. A general method to improve fluorophores for live-cell and single-molecule microscopy. Nat. Methods 12, 244–250 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Lukinavičius, G. et al. A near-infrared fluorophore for live-cell super-resolution microscopy of cellular proteins. Nat. Chem. 5, 132–139 (2013).

    Article  PubMed  Google Scholar 

  15. Shaner, N. C. et al. Improving the photostability of bright monomeric orange and red fluorescent proteins. Nat. Methods 5, 545–551 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Gustafsson, M. G. L. et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J. 94, 4957–4970 (2008).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  17. Hell, S. W. Far-field optical nanoscopy. Science 316, 1153–1158 (2007).

    Article  ADS  CAS  PubMed  Google Scholar 

  18. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

    Article  ADS  CAS  PubMed  Google Scholar 

  19. Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Denk, W., Strickler, J. H. & Webb, W. W. Two-photon laser scanning fluorescence microscopy. Science 248, 73–76 (1990).

    Article  ADS  CAS  PubMed  Google Scholar 

  21. Huisken, J., Swoger, J., Del Bene, F., Wittbrodt, J. & Stelzer, E. H. K. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305, 1007–1009 (2004).

    Article  ADS  CAS  PubMed  Google Scholar 

  22. Möckl, L., Roy, A. R., Petrov, P. N. & Moerner, W. E. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet. Proc. Natl Acad. Sci. USA 117, 60–67 (2020).

    Article  ADS  PubMed  Google Scholar 

  23. Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

    Article  CAS  PubMed  Google Scholar 

  24. Huang, T. S., Yang, G. J. & Tang, G. Y. A fast two-dimensional median filtering algorithm. IEEE Trans. Signal. Process. 27, 13–18 (1979).

    Article  Google Scholar 

  25. Strong, D. & Chan, T. Edge-preserving and scale-dependent properties of total variation regularization. Inverse Problems 19, S165–S187 (2003).

    Article  ADS  MathSciNet  Google Scholar 

  26. Buades, A., Coll, B. & Morel, J.M. A non-local algorithm for image denoising. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). 60-65 (2005).

  27. Huang, X. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451–459 (2018).

    Article  CAS  PubMed  Google Scholar 

  28. Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  ADS  CAS  PubMed  Google Scholar 

  29. Chen, J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Nat. Methods 18, 678–687 (2021).

    Article  ADS  CAS  PubMed  Google Scholar 

  30. Fang, L. et al. Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).

    Article  CAS  PubMed  Google Scholar 

  32. Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat. Biotechnol. 41, 367–377 (2023).

    Article  CAS  PubMed  Google Scholar 

  33. Gustafsson, M. G. L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc. 198, 82–87 (2000).

    Article  CAS  PubMed  Google Scholar 

  34. Chen, B. C. et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  35. York, A. G. et al. Instant super-resolution imaging in live cells and embryos via analog image processing. Nat. Methods 10, 1122–1130 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Hoebe, R. A. et al. Controlled light-exposure microscopy reduces photobleaching and phototoxicity in fluorescence live-cell imaging. Nat. Biotechnol. 25, 249–253 (2007).

    Article  CAS  PubMed  Google Scholar 

  37. Chu, K. K., Lim, D. & Mertz, J. Enhanced weak-signal sensitivity in two-photon microscopy by adaptive illumination. Opt. Lett. 32, 2846–2848 (2007).

    Article  ADS  PubMed  Google Scholar 

  38. Li, B., Wu, C., Wang, M., Charan, K. & Xu, C. An adaptive excitation source for high-speed multiphoton microscopy. Nat. Methods 17, 163–166 (2020).

    Article  CAS  PubMed  Google Scholar 

  39. Staudt, T. et al. Far-field optical nanoscopy with reduced number of state transition cycles. Opt. Express 19, 5644–5657 (2011).

    Article  ADS  PubMed  Google Scholar 

  40. Chakrova, N., Canton, A. S., Danelon, C., Stallinga, S. & Rieger, B. Adaptive illumination reduces photobleaching in structured illumination microscopy. Biomed. Opt. Express 7, 4263–4274 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  41. Göttfert, F. et al. Strong signal increase in STED fluorescence microscopy by imaging regions of subdiffraction extent. Proc. Natl Acad. Sci. USA 114, 2125–2130 (2017).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  42. Heine, J. et al. Adaptive-illumination STED nanoscopy. Proc. Natl Acad. Sci. USA 114, 9797–9802 (2017).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  43. Dreier, J. et al. Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo.Nat. Commun. 10, 556 (2019).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  44. Vinçon, B., Geisler, C. & Egner, A. Pixel hopping enables fast STED nanoscopy at low light dose. Opt. Express 28, 4516–4528 (2020).

    Article  ADS  PubMed  Google Scholar 

  45. Štefko, M., Ottino, B., Douglass, K. M. & Manley, S. Autonomous illumination control for localization microscopy. Opt. Express 26, 30882–30900 (2018).

    Article  ADS  PubMed  Google Scholar 

  46. Chiron, L. et al. CyberSco.Py an open-source software for event-based, conditional microscopy. Sci. Rep. 12, 11579 (2022).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  47. Almada, P. et al. Automating multimodal microscopy with NanoJ-Fluidics. Nat. Commun. 10, 1223 (2019).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  48. Edelstein, A. D. et al. Advanced methods of microscope control using μManager software. J. Biol. Methods 1, e10 (2014).

    Article  PubMed  Google Scholar 

  49. Pinkard, H., Stuurman, N., Corbin, K., Vale, R. & Krummel, M. F. Micro-Magellan: open-source, sample-adaptive, acquisition software for optical microscopy. Nat. Methods 13, 807–809 (2016).

    Article  CAS  PubMed  Google Scholar 

  50. Fox, Z. R. et al. Enabling reactive microscopy with MicroMator. Nat. Commun. 13, 2199 (2022).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  51. Casas Moreno, X. et al. An open-source microscopy framework for simultaneous control of image acquisition, reconstruction, and analysis. HardwareX 13, e00400 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Schermelleh, L. et al. Super-resolution microscopy demystified. Nat. Cell Biol. 21, 72–84 (2019).

    Article  CAS  PubMed  Google Scholar 

  53. Wu, Y. & Shroff, H. Multiscale fluorescence imaging of living samples. Histochem. Cell Biol. 158, 301–323 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Sarder, P. & Nehorai, A. Deconvolution methods for 3-D fluorescence microscopy images. IEEE Signal. Process. Mag. 23, 32–45 (2006).

    Article  ADS  Google Scholar 

  55. Lucy, L. B. An iterative technique for the rectification of observed distributions. Astron. J. 79, 745–754 (1974).

    Article  ADS  Google Scholar 

  56. Richardson, W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972).

    Article  ADS  Google Scholar 

  57. Sage, D. et al. DeconvolutionLab2: an open-source software for deconvolution microscopy. Methods 115, 28–41 (2017).

    Article  CAS  PubMed  Google Scholar 

  58. Zhao, W. et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat. Biotechnol. 40, 606–617 (2022).

    Article  CAS  PubMed  Google Scholar 

  59. Li, Y. et al. Incorporating the image formation process into deep learning improves network performance. Nat. Methods 19, 1427–1437 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  60. Chhetri, R. K. et al. Whole-animal functional and developmental imaging with isotropic spatial resolution. Nat. Methods 12, 1171–1178 (2015).

    Article  CAS  PubMed  Google Scholar 

  61. Wu, Y. et al. Spatially isotropic four-dimensional imaging with dual-view plane illumination microscopy. Nat. Biotechnol. 31, 1032–1038 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  62. Wu, Y. et al. Multiview confocal super-resolution microscopy. Nature 600, 279–284 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  63. Ingaramo, M. et al. Richardson-Lucy deconvolution as a general tool for combining images with complementary strengths. ChemPhysChem 15, 794–800 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Kazemipour, A. et al. Kilohertz frame-rate two-photon tomography. Nat. Methods 16, 778–786 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).

    Article  CAS  PubMed  Google Scholar 

  66. Li, X. et al. Three-dimensional structured illumination microscopy with enhanced axial resolution. Nat. Biotechnol. 41, 1307–1319 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Weigert, M., Royer, L., Jug, F. & Myers, G. Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. Preprint at https://doi.org/10.48550/arXiv.1704.01510 (2017).

  68. Hampson, K. M. et al. Adaptive optics for high-resolution imaging. Nat. Rev. Methods Prim. 1, 68 (2021).

    Article  CAS  Google Scholar 

  69. Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).

    Article  CAS  PubMed  Google Scholar 

  70. Wang, K. et al. Rapid adaptive optical recovery of optimal resolution over large volumes. Nat. Methods 11, 625–628 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  71. Liu, T. L. et al. Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms. Science 360, eaaq1392 (2018).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  72. Rodríguez, C. et al. An adaptive optics module for deep tissue multiphoton imaging in vivo. Nat. Methods 18, 1259–1264 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  73. Streich, L. et al. High-resolution structural and functional deep brain imaging using adaptive optics three-photon microscopy. Nat. Methods 18, 1253–1258 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  74. Lin, R., Kipreos, E. T., Zhu, J., Khang, C. H. & Kner, P. Subcellular three-dimensional imaging deep through multicellular thick samples by structured illumination microscopy and adaptive optics. Nat. Commun. 12, 3148 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  75. Zheng, W. et al. Adaptive optics improves multiphoton super-resolution imaging. Nat. Methods 14, 869–872 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Saha, D. et al. Practical sensorless aberration estimation for 3D microscopy with deep learning. Opt. Express 28, 29044–29053 (2020).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  77. Vinogradova, K. & Myers, E.W. Estimation of optical aberrations in 3D microscopic bioimages. 7th International Conference on Frontiers of Signal Processing, ICFSP. 97-103 (2022).

  78. Feng, B. Y. et al. NeuWS: neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media. Sci. Adv. 9, eadg4671 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  79. Balzarotti, F. et al. Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes. Science 355, 606–612 (2017).

    Article  ADS  CAS  PubMed  Google Scholar 

  80. Deguchi, T. et al. Direct observation of motor protein stepping in living cells using MINFLUX. Science 379, 1010–1015 (2023).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  81. Wolff, J. O. et al. MINFLUX dissects the unimpeded walking of kinesin-1. Science 379, 1004–1010 (2023).

    Article  ADS  CAS  Google Scholar 

  82. Royer, L. A. et al. Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms. Nat. Biotechnol. 34, 1267–1278 (2016).

    Article  CAS  PubMed  Google Scholar 

  83. Pinkard, H. et al. Learned adaptive multiphoton illumination microscopy for large-scale immune response imaging. Nat. Commun. 12, 1916 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  84. Durand, A. et al. A machine learning approach for online automated optimization of super-resolution optical microscopy. Nat. Commun. 9, 5247 (2018).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  85. Sheppard, C. J. R. Super-resolution in confocal imaging. Optik 80, 53–54 (1988).

    Google Scholar 

  86. Müller, C. B. & Enderlein, J. Image scanning microscopy. Phys. Rev. Lett. 104, 74–83 (2010).

    Article  Google Scholar 

  87. York, A. G. et al. Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy. Nat. Methods 9, 749–754 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  88. Schulz, O. et al. Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy. Proc. Natl Acad. Sci. USA 110, 21000–21005 (2013).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  89. De Luca, G. M. R. et al. Re-scan confocal microscopy: scanning twice for better resolution. Biomed. Opt. Express 4, 2644–2656 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  90. Roth, S., Sheppard, C. J. R., Wicker, K. & Heintzmann, R. Optical photon reassignment microscopy (OPRA). Opt. Nanoscopy 2, 5 (2013).

    Article  Google Scholar 

  91. Azuma, T. & Kei, T. Super-resolution spinning-disk confocal microscopy using optical photon reassignment. Opt. Express 23, 15003–15011 (2015).

    Article  ADS  CAS  PubMed  Google Scholar 

  92. Hofmann, M., Eggeling, C., Jakobs, S. & Hell, S. W. Breaking the diffraction barrier in fluorescence microscopy at low light intensities by using reversibly photoswitchable proteins. Proc. Natl Acad. Sci. USA 102, 17565–17569 (2005).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  93. Chmyrov, A. et al. Nanoscopy with more than 100,000 ‘doughnuts’. Nat. Methods 10, 737–740 (2013).

    Article  CAS  PubMed  Google Scholar 

  94. Bodén, A. et al. Volumetric live cell imaging with three-dimensional parallelized RESOLFT microscopy. Nat. Biotechnol. 39, 609–618 (2021).

    Article  PubMed  Google Scholar 

  95. Lelek, M. et al. Single-molecule localization microscopy. Nat. Rev. Methods Prim. 1, 39 (2021).

    Article  CAS  Google Scholar 

  96. Speiser, A. et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Nat. Methods 18, 1082–1090 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  97. Mahecic, D. et al. Event-driven acquisition for content-enriched microscopy. Nat. Methods 19, 1262–1267 (2022).

    Article  CAS  PubMed  Google Scholar 

  98. Alvelid, J., Damenti, M., Sgattoni, C. & Testa, I. Event-triggered STED imaging. Nat. Methods 19, 1268–1275 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  99. Tsien, R. Y. The green fluorescent protein. Annu. Rev. Biochem. 67, 509–544 (1998).

    Article  CAS  PubMed  Google Scholar 

  100. Lambert, T. J. FPbase: a community-editable fluorescent protein database. Nat. Methods 16, 277–278 (2019).

    Article  CAS  PubMed  Google Scholar 

  101. Cheng, S. et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci. Adv. 7, eabe0431 (2021).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  102. Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803.e19 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  103. McRae, T. D., Oleksyn, D., Miller, J. & Gao, Y.-R. Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning. PLoS One 14, e0225410 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  104. Seo, J. et al. PICASSO allows ultra-multiplexed fluorescence imaging of spatially overlapping proteins without reference spectra measurements. Nat. Commun. 13, 2475 (2022).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  105. Digman, M. A., Caiolfa, V. R., Zamai, M. & Gratton, E. The phasor approach to fluorescence lifetime imaging analysis. Biophys. J. 94, L14–L16 (2008).

    Article  CAS  PubMed  Google Scholar 

  106. Lanzanò, L. et al. Encoding and decoding spatio-temporal information for super-resolution microscopy. Nat. Commun. 6, 6701 (2015).

    Article  ADS  PubMed  Google Scholar 

  107. Scipioni, L., Rossetta, A., Tedeschi, G. & Gratton, E. Phasor S-FLIM: a new paradigm for fast and robust spectral fluorescence lifetime imaging. Nat. Methods 18, 542–550 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  108. Smith, J. T., Ochoa, M. & Intes, X. UNMIX-ME: spectral and lifetime fluorescence unmixing via deep learning. Biomed. Opt. Express 11, 3857–3874 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    Article  CAS  PubMed  Google Scholar 

  110. Saguy, A. et al. DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning. Nat. Methods 20, 1939–1948 (2023).

    Article  CAS  PubMed  Google Scholar 

  111. Milias-Argeitis, A. et al. In silico feedback for in vivo regulation of a gene expression circuit. Nat. Biotechnol. 29, 1114–1116 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  112. Emiliani, V., Cohen, A. E., Deisseroth, K. & Häusser, M. All-optical interrogation of neural circuits. J. Neurosci. 35, 13917–13926 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  113. Schmidt, M. & Lipson, H. Distilling free-form natural laws from experimental data. Science 324, 81–85 (2009).

    Article  ADS  CAS  PubMed  Google Scholar 

  114. Colón-Ramos, D. A., La Riviere, P., Shroff, H. & Oldenbourg, R. Transforming the development and dissemination of cutting-edge microscopy and computation. Nat. Methods 16, 667–669 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  115. Conrad, C. et al. Micropilot: automation of fluorescence microscopy-based imaging for systems biology. Nat. Methods 8, 246–249 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  116. Ouyang, W. et al. BioImage model zoo: a community-driven resource for accessible deep learning in bioimage analysis. Preprint at bioRxiv https://doi.org/10.1101/2022.06.07.495102 (2022).

  117. Katona, G. et al. Fast two-photon in vivo imaging with three-dimensional random-access scanning in large tissue volumes. Nat. Methods 9, 201–208 (2012).

    Article  CAS  PubMed  Google Scholar 

  118. Krull, A., Buchholz, T. O. & Jug, F. Noise2void – learning denoising from single noisy images. Preprint at https://doi.org/10.48550/arXiv.1811.10980 (2019).

  119. Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

    Article  CAS  PubMed  Google Scholar 

  120. Greener, J. G., Kandathil, S. M., Moffat, L. & Jones, D. T. A guide to machine learning for biologists. Nat. Rev. Mol. Cell Biol. 23, 40–55 (2022).

    Article  CAS  PubMed  Google Scholar 

  121. Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article  ADS  PubMed  Google Scholar 

  122. Wang, Z., Simoncelli, E.P. & Bovik, A.C. Multiscale structural similarity for image quality assessment. In 37th Asilomar Conference on Signals, Systems & Computers, 2003 Vol. 2, 1398–1402 (2003).

  123. Zhang, R., Isola, P., Efros, A.A., Shechtman, E. & Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 586–595 (2018).

  124. Lehtinen, J. et al. Noise2Noise: learning image restoration without clean data. In 35th International Conference on Machine Learning, ICML 2018 4620–4631 (2018).

  125. Kefer, P. et al. Performance of deep learning restoration methods for the extraction of particle dynamics in noisy microscopy image sequences. Mol. Biol. Cell 32, 903–914 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  126. Wu, Y. et al. Reflective imaging improves spatiotemporal resolution and collection efficiency in light sheet microscopy. Nat. Commun. 8, 1452 (2017).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  127. Shaevitz, J. W. & Fletcher, D. A. Enhanced three-dimensional deconvolution microscopy using a measured depth-varying point-spread function. J. Opt. Soc. Am. A 24, 2622–2627 (2007).

    Article  ADS  Google Scholar 

  128. Yanny, K., Monakhova, K., Shuai, R. W. & Waller, L. Deep learning for fast spatially varying deconvolution. Optica 9, 96–99 (2022).

    Article  ADS  Google Scholar 

  129. Guo, M. et al. Rapid image deconvolution and multiview fusion for optical microscopy. Nat. Biotechnol. 38, 1337–1346 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank Grant Kroeschell and Richard Ikegami (Shroff lab) as well as Jiji Chen (NIH Advanced Imaging and Microscopy Resource) for help with figure preparation. This work was supported by the Howard Hughes Medical Institute (HHMI) and the École Polytechnique Fédérale de Lausanne (EPFL).

Author information

Authors and Affiliations

  1. Janelia Research Campus, Howard Hughes Medical Institute (HHMI), Ashburn, VA, USA

    Hari Shroff

  2. Department of Applied Physics and Science for Life Laboratory, KTH Royal Institute of Technology, Stockholm, Sweden

    Ilaria Testa

  3. Fondazione Human Technopole (HT), Milan, Italy

    Florian Jug

  4. Institute of Physics, School of Basic Sciences, Swiss Federal Institute of Technology Lausanne (EPFL), Lausanne, Switzerland

    Suliana Manley

Authors
  1. Hari Shroff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilaria Testa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Florian Jug
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Suliana Manley
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article.

Corresponding author

Correspondence to Suliana Manley.

Ethics declarations

Competing interests

H.S. is co-inventor on US patent 9,696,534, owned by NIH and licensed to VisiTech International and Yokogawa Electric Corporation, describing multi-focal and analogue implementations of structured illumination microscopy (SIM), including the instant SIM mentioned here. H.S. has also filed invention disclosures on four-beam SIM and multi-view confocal microscopy, both of which rely on the deep learning strategies mentioned here.

Peer review

Peer review information

Nature Reviews Molecular Cell Biology thanks Ricardo Henriques, who co-reviewed with Estibaliz Gómez de Mariscal, Lothar Schermelleh and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

CSBDeep: https://csbdeep.bioimagecomputing.com

Figshare for ref. 103: https://doi.org/10.6084/m9.figshare.c.4537607

FIJI n2v: https://imagej.net/plugins/n2v

FIJI nlm: https://imagej.net/plugins/non-local-means-denoise/

GitHub 3D-RCAN: https://github.com/AiviaCommunity/3D-RCAN

GitHub autopilot: https://microscopeautopilot.github.io/

GitHub CSBDeep: https://github.com/CSBDeep/CSBDeep

GitHub DL-SR: https://github.com/qc17-THU/DL-SR

GitHub etSTED: https://github.com/jonatanalvelid/etSTED-widget-base

GitHub JiLabAO: https://github.com/JiLabUCBerkeley/JiLabAO

GitHub Knerlab: https://github.com/Knerlab

GitHub n2v: https://github.com/juglab/n2v

GitHub phasenet: https://github.com/mpicbg-csbd/phasenet

GitHub prevedel-lab/AO: https://github.com/prevedel-lab/AO.git

GitHub PSSR: https://github.com/BPHO-Salk/PSSR

GitHub pycudasirecon: https://github.com/tlambert03/pycudasirecon

GitHub rDL-SIM: https://github.com/qc17-THU/rDL-SIM

GitHub Richardson-Lucy-Net: https://github.com/MeatyPlus/Richardson-Lucy-Net

GitHub SIMreconProject: https://github.com/eexuesong/SIMreconProject

GitHub Sparse-SIM: https://github.com/WeisongZhao/Sparse-SIM

GitHub Testa Lab: https://github.com/TestaLab

GitHub UNMIX-ME: https://github.com/jasontsmith2718/UNMIX-ME

Google Code msim: http://code.google.com/p/msim/

Micro-manager EDA plugin: https://pypi.org/project/eda-plugin/

Napari n2v: https://www.piwheels.org/project/napari-n2v/

Napari nlm: https://www.napari-hub.org/plugins/napari-nlm

Skimage nlm: https://scikit-image.org/docs/stable/auto_examples/filters/plot_nonlocal_means.html

Zenodo Henry Pinkard (2020): https://doi.org/10.5281/zenodo.4314107

Glossary

Aberrations

Distortions in images generated by an optical system due to deviations in the properties of real optical components, such as lenses, mirrors and filters, as compared with theoretical models or due to refractive index variations in the sample.

Artificial neural network

(ANN). A computational model composed of interconnected nodes and layers, designed to loosely mimic the structure and function of the brain.

Binning

The denoising process of combining data from adjacent pixels in an image, which results in fewer pixels.

Bleed-through

Overlap in the emission spectra of two distinct fluorophores, leading to detection of both at the same wavelength.

Centroid

The centre position of an object, corresponding to the weighted mean of pixel intensity values.

Continuity

Property of an object consisting of containing no resolved gaps in space (spatial continuity) or time (temporal continuity).

Crosstalk

Undesired mixing of signals. For example, overlap in the excitation spectra of two distinct fluorophores, leading to excitation of both by the same wavelength of light. Or, in multi-focal microscopy, fluorescence from one excitation spot contributing to the signal of neighbouring regions.

Deconvolution

Since an image is blurred by the convolution of the fluorescently labelled object with the point spread function of the microscope, this image processing method attempts to computationally reverse this effect.

Deep learning

A class of machine learning algorithms based on artificial neural networks containing multiple data processing layers.

Depletion doughnut

A doughnut-shaped illumination light used to turn off the fluorescence in the periphery of the focal spot.

Downsampling

Reducing the sampling rate, for example, spatially or temporally.

Dwell time

The time a focused laser beam is applied to each location in the specimen being imaged.

Fluorescence lifetime

The time a fluorophore spends in the excited state before emitting a photon and returning to the ground state.

Gaussian readout noise

Noise that follows a Gaussian distribution and is independent of pixel intensity values, for example, noise generated by a camera chip when it converts charge into voltage.

Ground truth

The target of a deep learning model, for example, a label against which the predictions of a model are compared during training.

Hallucinations

Neural network outputs that look plausible but have no basis in the input data.

Lattice light-sheet

Light-sheet generated by scanning a 2D lattice of structured light known as Bessel beams.

Light-sheet microscopy

Method in which a thin slice of a specimen is illuminated perpendicular to the imaging orientation.

Linearity

The property of two quantities (for example, intensities) being linearly proportional, such that their values are related by a multiplicative constant.

Low-pass filter

An operation that passes frequencies below a cut-off value, which corresponds to retaining lower-resolution features in an image.

Mean absolute error

(MAE). The mean of the absolute value of the difference between measured and predicted values, for example, ground truth pixel intensity values and those output by a network. Used as a metric of how well a model captures the data.

Mean squared error

(MSE). The mean squared difference between measured and predicted values, for example, ground truth pixel intensity values and those output by a network. Used as a metric of how well a model captures the data.

Median filtering

An operation that replaces the intensity value in a pixel with the median value of its neighbours.

Multiphoton microscopy

Method in which multiple photons must be simultaneously absorbed by a single fluorophore to bring it into its excited state.

Mutual information

A measure of the extent to which two quantities depend on one another, related to how precisely one quantity can be predicted based on the value of the other.

Nyquist–Shannon sampling theorem

Principle that defines the maximum spacing between measurements that will be sufficient to determine a given frequency component within a signal; for example, to resolve dynamics at a timescale of T seconds, the time between images should be less than T/2.

Peak signal-to-noise ratio

(PSNR). The ratio between the squared maximum possible signal in an image and the mean squared error. This is reported in units of decibels, so the logarithm of the ratio is taken and multiplied by 10. Used as a metric of how well a model captures the data.

Photon budget

Fluorescence signal detected from an object of interest during an experiment, typically finite due to photobleaching.

Point spread function

The intensity distribution of a point-like source when imaged through a microscope.

Poisson noise

Noise (that is, shot noise) that follows a Poisson distribution, for example, arising from measuring photons because they are discrete particles.

Pyramid of frustration

Concept illustrating the tradeoffs in fluorescence microscopy, where each axis defines one measurement property such as signal-to-noise ratio or spatial or temporal resolution. The fixed photon budget implies that improving along one dimension leads to degradation along another.

Reference datasets

Data used within a field to compare the performance of algorithms, in benchmarking comparisons.

Reversible saturable optical fluorescence transitions

(RESOLFT). A super-resolution technique suitable for live-cell imaging and based on reversibly switching fluorescent probes and patterned illumination.

Sampling

Recording a signal in a discontinuous manner, at specific locations or times.

Signal-to-noise ratio

(SNR). The ratio between signal and noise that can be estimated on a per-pixel basis as the mean intensity value divided by the standard deviation of the intensity.

Single-molecule localization microscopy

(SMLM). A class of super-resolution microscopy techniques based on imaging single molecules whose signals have been isolated, then combining their sub-pixel locations to form a composite image.

Spatial frequencies

Just as a function can be decomposed into a sum of sines and cosines (compared with Fourier transform), an image can be decomposed into a sum of waves with different spatial frequencies. These represent the image at different resolution levels, with higher spatial frequencies describing finer image details and lower spatial frequencies describing coarser details.

Spatial resolution

The smallest distance at which two features can be distinguished.

Spectrum

Response of a fluorophore as a function of wavelength of light. The excitation spectrum reflects the capacity of light absorbed at different wavelengths to generate fluorescence of a particular wavelength, while the emission spectrum reflects the emission of light at different wavelengths following excitation at a particular wavelength.

Spinning disk confocal microscopy

Method in which an array of focused excitation laser beams is produced by an array of microlenses on a disk that spins to scan the specimen. Out-of-focus emission light is rejected by a confocal array of pinholes.

Statistical distance

An objective score that summarizes statistical differences between two objects, for example, between a prediction and the training set in machine learning. Possibilities include ‘total variational distance’ and ‘Kullback–Liebler divergence’.

Stimulated emission depletion

(STED) microscopy. A super-resolution technique based on patterned illumination, typically doughnut shaped, which is used to deplete the fluorescence of commonly used fluorescent probes.

Structural similarity index measure

(SSIM). A measure of how similar two images are based on distortions that humans tend to perceive: the weighted product of luminance (average brightness), contrast (standard deviation of pixel intensity values) and structure (cross-covariance).

Structured illumination microscopy

(SIM). A class of super-resolution microscopy techniques that use patterned excitation light combined with optical or digital image processing to recover information below the diffraction limit.

Super-resolution

Imaging techniques that achieve spatial resolutions surpassing the diffraction limit of light.

Synthetic data

Data generated computationally using a model. Can be combined with real data to create semi-synthetic data.

Temporal resolution

The time between consecutive images of the same part of the specimen.

Total internal reflection fluorescence

Method in which a specimen is illuminated at the coverslip-media interface by an evanescent field generated by a laser at an incident angle sufficient to cause total internal reflection.

Training data

Data used to train a machine learning algorithm to make predictions based on supervised learning. The quality and quantity of training data are key determinants of the performance of artificial neural networks.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shroff, H., Testa, I., Jug, F. et al. Live-cell imaging powered by computation. Nat Rev Mol Cell Biol (2024). https://doi.org/10.1038/s41580-024-00702-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41580-024-00702-6

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing