Article

Deep learning massively accelerates super-resolution localization microscopy

Received:
Accepted:
Published:

Abstract

The speed of super-resolution microscopy methods based on single-molecule localization, for example, PALM and STORM, is limited by the need to record many thousands of frames with a small number of observed molecules in each. Here, we present ANNA-PALM, a computational strategy that uses artificial neural networks to reconstruct super-resolution views from sparse, rapidly acquired localization images and/or widefield images. Simulations and experimental imaging of microtubules, nuclear pores, and mitochondria show that high-quality, super-resolution images can be reconstructed from up to two orders of magnitude fewer frames than usually needed, without compromising spatial resolution. Super-resolution reconstructions are even possible from widefield images alone, though adding localization data improves image quality. We demonstrate super-resolution imaging of >1,000 fields of view containing >1,000 cells in 3 h, yielding an image spanning spatial scales from 20 nm to 2 mm. The drastic reduction in acquisition time and sample irradiation afforded by ANNA-PALM enables faster and gentler high-throughput and live-cell super-resolution imaging.

  • Subscribe to Nature Biotechnology for full access:

    $250

    Subscribe

Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.

References

  1. 1.

    et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

  2. 2.

    , & Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–796 (2006).

  3. 3.

    et al. High-density three-dimensional localization microscopy across large volumes. Nat. Methods 13, 359–365 (2016).

  4. 4.

    et al. Precisely and accurately localizing single emitters in fluorescence microscopy. Nat. Methods 11, 253–266 (2014).

  5. 5.

    , , & Fast, three-dimensional super-resolution imaging of live cells. Nat. Methods 8, 499–505 (2011).

  6. 6.

    et al. Video-rate nanoscopy using sCMOS camera-specific single-molecule localization algorithms. Nat. Methods 10, 653–658 (2013).

  7. 7.

    et al. Fast live simultaneous multiwavelength four-dimensional optical microscopy. Proc. Natl. Acad. Sci. USA 107, 16016–16022 (2010).

  8. 8.

    Light-sheet fluorescence microscopy for quantitative biology. Nat. Methods 12, 23–26 (2015).

  9. 9.

    , , & Simultaneous multiple-emitter fitting for single molecule super-resolution imaging. Biomed. Opt. Express 2, 1377–1393 (2011).

  10. 10.

    , , , & Bleaching/blinking assisted localization microscopy for superresolution imaging using standard fluorescent molecules. Proc. Natl. Acad. Sci. USA 108, 21081–21086 (2011).

  11. 11.

    , & Single-molecule-based super-resolution images in the presence of multiple fluorophores. Nano Lett. 11, 5090–5096 (2011).

  12. 12.

    , , & Faster STORM using compressed sensing. Nat. Methods 9, 721–723 (2012).

  13. 13.

    et al. Bayesian localization microscopy reveals nanoscale podosome dynamics. Nat. Methods 9, 195–200 (2012).

  14. 14.

    , & Beyond Rayleigh's criterion: a resolution measure with application to single-molecule microscopy. Proc. Natl. Acad. Sci. USA 103, 4457–4462 (2006).

  15. 15.

    , & Deep learning. Nature 521, 436–444 (2015).

  16. 16.

    Sparse and Redundant Representations: from Theory to Applications in Signal and Image Processing (Springer, 2010).

  17. 17.

    & Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

  18. 18.

    Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).

  19. 19.

    et al. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods 12, 717–724 (2015).

  20. 20.

    , , & Image-to-image translation with conditional adversarial networks. Preprint at (2016).

  21. 21.

    , & in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. (eds., Navab, N., Hornegger, J., Wells, W.M. & Frangi, A.F.), 234–241 (Springer; 2015).

  22. 22.

    et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014).

  23. 23.

    , , & Mitosis detection in breast cancer histology images with deep neural networks. Med. Image Comput. Comput. Assist. Interv. 2013, 8150 (2013).

  24. 24.

    & Mean squared error: love it or leave it? A new look at Signal Fidelity Measures. IEEE Signal Process. Mag. 26, 98–117 (2009).

  25. 25.

    , , & Loss functions for image restoration with neural networks. IEEE Trans. Comput. IMAGING 3, 47–57 (2017).

  26. 26.

    et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).

  27. 27.

    , , , & Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).

  28. 28.

    , , & Inferring the physical properties of yeast chromatin through Bayesian analysis of whole nucleus simulations. Genome Biol. 18, 81 (2017).

  29. 29.

    & How does taxol stabilize microtubules? Curr. Biol. 5, 900–908 (1995).

  30. 30.

    et al. Microtubule motors regulate ISOC activation necessary to increase endothelial cell permeability. J. Biol. Chem. 282, 34801–34808 (2007).

  31. 31.

    , , & Ultrastructural immunocytochemical distribution of tubulin in cultured cells treated with microtubule inhibitors. Cell Biol. Int. Rep. 1, 177–183 (1977).

  32. 32.

    , , , & Super-resolution microscopy with DNA-PAINT. Nat. Protoc. 12, 1198–1228 (2017).

  33. 33.

    et al. Super-resolution imaging visualizes the eightfold symmetry of gp210 proteins around the nuclear pore complex and resolves the central channel with nanometer resolution. J. Cell Sci. 125, 570–575 (2012).

  34. 34.

    et al. Nuclear pore scaffold structure analyzed by super-resolution microscopy and particle averaging. Science 341, 655–658 (2013).

  35. 35.

    et al. Nuclear pore complex plasticity during developmental process as revealed by super-resolution microscopy. Sci. Rep. 7, 14732 (2017).

  36. 36.

    et al. TOM22, a core component of the mitochondria outer membrane protein translocation pore, is a mitochondrial receptor for the proapoptotic protein Bax. Cell Death Differ. 14, 785–794 (2007).

  37. 37.

    et al. Super-resolution imaging reveals distinct chromatin folding for different epigenetic states. Nature 529, 418–422 (2016).

  38. 38.

    , & Extracting microtubule networks from superresolution single-molecule localization microscopy data. Mol. Biol. Cell 28, 333–345 (2017).

  39. 39.

    et al. Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes. Nature 464, 721–727 (2010).

  40. 40.

    et al. Localization-based super-resolution imaging meets high-content screening. Nat. Methods 14, 1184–1190 (2017).

  41. 41.

    & The imaging tsunami: computational opportunities and challenges. Curr. Opin. Syst. Biol. 4, 105–113 (2017).

  42. 42.

    , & Optical imaging. Expansion microscopy. Science 347, 543–548 (2015).

  43. 43.

    et al. Iterative expansion microscopy. Nat. Methods 14, 593–599 (2017).

  44. 44.

    , & Correlated light and electron microscopy: ultrastructure lights up!. Nat. Methods 12, 503–513 (2015).

  45. 45.

    , , & Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319, 810–813 (2008).

  46. 46.

    , , & Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science 317, 1749–1753 (2007).

  47. 47.

    et al. Multiplexed 3D cellular super-resolution imaging with DNA-PAINT and Exchange-PAINT. Nat. Methods 11, 313–318 (2014).

  48. 48.

    , , & Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics. Nat. Methods 5, 417–423 (2008).

  49. 49.

    & What uncertainties do we need in Bayesian deep learning for computer vision? Adv. Neural Inf. Process. Syst. 30, 5580–5590 (2017).

  50. 50.

    , & in 2016 IEEE International Workshop on Acoustic Signal Enhancement, IWAENC 2016 (IEEE, 2016) .

  51. 51.

    , & Rectifier nonlinearities improve neural network acoustic models. Proc. 30th Int. Conf. Mach. Learn. 30, 3 (2013).

  52. 52.

    et al. Least squares generative adversarial networks. ICCV 2794–2802 (2017) .

  53. 53.

    & Adam: a method for stochastic optimization. ICLR 1–15. (2015).

  54. 54.

    & TensorFlow: biology's gateway to deep learning? Cell Syst. 2, 12–14 (2016).

  55. 55.

    & An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. 5, 1266–1271 (1996).

  56. 56.

    , & Best practices for convolutional neural networks applied to visual document analysis. Seventh Int. Conf. Doc. Anal. Recognition, 2003. Proceedings 1, 958–963 (2003).

  57. 57.

    Fast parallel algorithms for short-range molecular dynamics. J. Comput. Phys. 117, 1–19 (1995).

  58. 58.

    et al. Direct stochastic optical reconstruction microscopy with standard fluorescent probes. Nat. Protoc. 6, 991–1009 (2011).

  59. 59.

    et al. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat. Methods 7, 339–340 (2010).

  60. 60.

    et al. Superresolution imaging of HIV in infected cells with FlAsH-PALM. Proc. Natl. Acad. Sci. USA 109, 8564–8569 (2012).

  61. 61.

    , , , & Computer control of microscopes using μmanager. Curr. Protoc. Mol. Biol. 92, 14.20.1–14.20.17 (2010).

  62. 62.

    , , , & ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).

  63. 63.

    , & Globally optimal stitching of tiled 3D microscopic image acquisitions. Bioinformatics 25, 1463–1465 (2009).

  64. 64.

    , , & Dynamic multiple-target tracing to probe spatiotemporal cartography of cell membranes. Nat. Methods 5, 687–694 (2008).

Download references

Acknowledgements

We thank the following colleagues for useful discussions and suggestions and/or critical reading of the manuscript: C. Leduc, S. Etienne-Manneville, S. Lévêque-Fort, N. Bourg, A. Echard, J.-B. Masson, T. Rose, P. Hersen, F. Mueller, M. Cohen, Z. Zhang, and P. Kanchanawong. We also thank the four anonymous reviewers for their constructive criticism, which led to significant improvements of ANNA-PALM. We further thank O. Faklaris, J. Sellés and M. Penrad (Institut Jacques Monod), and F. Montel (Ecole Normale Supérieure de Lyon) for providing Xenopus nuclear pore data, J. Bai (Institut Pasteur) for TOM22 antibodies, and C. Leterrier for fixation protocols. We thank E. Rensen and C. Weber for help with experiments and suggestions, B. Lelandais for help with PALM image processing, J.-B. Arbona for polymer simulations and J. Parmar for suggestions that led to the name ANNA-PALM. We thank the IT service of Institut Pasteur, including J.-B. Denis, N. Joly, and S. Fournier, for access to the HPC cluster and relevant assistance, and T. Huynh for help with GPU computing. This work was funded by Institut Pasteur, Agence Nationale de la Recherche grant (ANR 14 CE10 0018 02), Fondation pour la Recherche Médicale (Equipe FRM, DEQ 20150331762), and the Région Ile de France (DIM Malinf). We also acknowledge Investissement d'Avenir grant ANR-16-CONV-0005 for funding a GPU farm used in this work. A.A. and X.H. are recipients of Pasteur-Roux fellowships from Institut Pasteur. W.O. is a scholar in the Pasteur–Paris University (PPU) International PhD program.

Author information

Affiliations

  1. Institut Pasteur, Unité Imagerie et Modélisation, Paris, France.

    • Wei Ouyang
    • , Andrey Aristov
    • , Mickaël Lelek
    • , Xian Hao
    •  & Christophe Zimmer
  2. UMR 3691, CNRS, Paris, France

    • Wei Ouyang
    • , Andrey Aristov
    • , Mickaël Lelek
    • , Xian Hao
    •  & Christophe Zimmer
  3. C3BI, USR 3756, IP CNRS, Paris, France.

    • Wei Ouyang
    • , Andrey Aristov
    • , Mickaël Lelek
    • , Xian Hao
    •  & Christophe Zimmer

Authors

  1. Search for Wei Ouyang in:

  2. Search for Andrey Aristov in:

  3. Search for Mickaël Lelek in:

  4. Search for Xian Hao in:

  5. Search for Christophe Zimmer in:

Contributions

W.O. conceived the method, developed ANNA-PALM software and web application, and performed experiments and analyses. A.A., M.L., and X.H. performed experiments. C.Z. conceived the method, supervised the project, and wrote the manuscript.

Competing interests

W.O. and C.Z. are listed as inventors on European patent applications EP17306022 and EP18305225.7 filed by Institut Pasteur.

Corresponding author

Correspondence to Christophe Zimmer.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–15, Supplementary Tables 1–4, Supplementary Note 1 Neural network architecture—A-net

  2. 2.

    Life Sciences Reporting Summary

Videos

  1. 1.

    ANNA-PALM reconstruction quality improves with increasing acquisition time

    This video shows an animated version of Figure 3, where the number of frames in the sparse PALM image (b) increases from k=0 to k=30,000. The quality of the corresponding ANNA-PALM reconstruction (e) increases with frame number, as shown by the merged image (h), where the ANNA-PALM reconstruction is shown in red and the dense PALM image obtained from K=30,000 frames (c) is shown in green. Panels a, c, d, f, g, i are identical to the corresponding panels in Figure 3.

  2. 2.

    High-throughput super-resolution imaging with ANNA-PALM

    This video shows a zoom-in into a mosaic image covering a 1.8 mm × 1.8 mm field, assembled from 1,089 individual fields of view. Top left: widefield mosaic image assembled from 1,089 widefield images. Top right: sparse PALM mosaic image assembled from 1,089 individual sparse PALM images, each of which was obtained from k=1,000 diffraction limited frames. Bottom left: mosaic of ANNA-PALM images reconstructed from the widefield images alone. Bottom right: mosaic ofANNA-PALM images reconstructed from the widefield images and the sparse PALM images combined.