Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Deep learning enables fast and dense single-molecule localization with high accuracy

### Subjects

A Publisher Correction to this article was published on 21 September 2021

## Abstract

Single-molecule localization microscopy (SMLM) has had remarkable success in imaging cellular structures with nanometer resolution, but standard analysis algorithms require sparse emitters, which limits imaging speed and labeling density. Here, we overcome this major limitation using deep learning. We developed DECODE (deep context dependent), a computational tool that can localize single emitters at high density in three dimensions with highest accuracy for a large range of imaging modalities and conditions. In a public software benchmark competition, it outperformed all other fitters on 12 out of 12 datasets when comparing both detection accuracy and localization error, often by a substantial margin. DECODE allowed us to acquire fast dynamic live-cell SMLM data with reduced light exposure and to image microtubules at ultra-high labeling density. Packaged for simple installation and use, DECODE will enable many laboratories to reduce imaging times and increase localization density in SMLM.

This is a preview of subscription content, access via your institution

## Relevant articles

• ### Fast DNA-PAINT imaging using a deep neural network

Nature Communications Open Access 27 August 2022

• ### Global fitting for high-accuracy multi-channel single-molecule localization

Nature Communications Open Access 06 June 2022

## Access options

\$32.00

All prices are NET prices.

## Data availability

All data can be downloaded from https://doi.org/10.25378/janelia.14674659. Raw data and bead frames are available for Figs. 2a–e and 4a–h and Extended Data Figs. 2, 3, 4a, 5 and 8. Localizations and performance metrics (for DECODE and CSpline/DeepSTORM3D when applicable) are available for Figs. 2a–e, 4a–h and 5 and Extended Data Figs. 2, 3, 4a, 7 and 8. The parametrization of the simulation for Fig. 2a–e is available and can be used to generate data. Raw data and bead frames, as well as performance metrics for Fig. 3. are publicly available at http://bigwww.epfl.ch/smlm/challenge2016/. Raw data and bead frames for Fig. 5 and Extended Data Fig. 7 are available on request from the authors of ref. 30. All other data supporting the findings of this study are available from the corresponding authors upon reasonable request. Source data are provided with this paper.

## Code availability

DECODE is available as Supplementary Software. Updated versions can be found at https://github.com/TuragaLab/DECODE.

## References

1. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

2. Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–796 (2006).

3. Van de Linde, S. et al. Direct stochastic optical reconstruction microscopy with standard fluorescent probes. Nat. Protocols 6, 991–1009 (2011).

4. Babcock, H. P. & Zhuang, X. Analyzing single molecule localization microscopy data using cubic splines. Sci. Rep. 7, 552 (2017).

5. Babcock, H., Sigal, Y. M. & Zhuang, X. A high-density 3d localization algorithm for stochastic optical reconstruction microscopy. Opt. Nanoscopy 1, 6 (2012).

6. Ovesny, M., Krizek, P., Borkovec, J., Svindrych, Z. & Hagen, G. M. Thunderstorm: a comprehensive ImageJ plug-in for palm and storm data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).

7. Sage, D. Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software. Nat. Methods 16, 387–395 (2019).

8. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

9. Ching, T. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15, 20170387 (2018).

10. Weigert, M. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090 (2018).

11. Le, T. A., Baydin, A. G., Zinkov, R., and Wood, F. Using synthetic data to train neural networks is model-based reasoning. In Proc. International Joint Conference on Neural Networks (IJCNN) 3514–3521 (IEEE, 2017).

12. Möckl, L., Roy, A. R. & Moerner, W. E. Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments. Biomed. Opt. Express 11, 1633–1661 (2020).

13. Zhang, P. et al. Analyzing complex single-molecule emission patterns with deep learning. Nat. Methods 15, 913–916 (2018).

14. Kim, T., Moon, S. & Xu, K. Information-rich localization microscopy through machine learning. Nat. Commun. 10, 996 (2019).

15. Möckl, L., Roy, A. R., Petrov, P. N. & Moerner, W. E. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network bgnet. Proc. Natl Acad. Sci. USA 117, 60–67 (2020).

16. Zelger, P. et al. Three-dimensional localization microscopy using deep learning. Opt. Express 26, 33166–33179 (2018).

17. Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 734–740 (2020).

18. Boyd, N., Jonas, E., Babcock, H. P. & Recht, B. Deeploco: fast 3D localization microscopy using neural networks. Preprint at bioRxiv https://doi.org/10.1101/267096 (2018).

19. Chen, B.-C. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).

20. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Navab, N. et al.) 234–241 (Springer, 2015); https://doi.org/10.1007/978-3-319-24574-4_28

21. Rieger, B. & Stallinga, S. The lateral and axial localization uncertainty in super-resolution light microscopy. Chem. Phys. Chem. 15, 664–670 (2014).

22. Chao, J., Ward, E. S. & Ober, R. J. Fisher information theory for parameter estimation in single molecule microscopy: tutorial. JOSA A 33, B36–B57 (2016).

23. Li, Y. et al. Real-time 3D single-molecule localization using experimental point spread functions. Nat. Methods 15, 367–369 (2018).

24. Small, A. & Stahlheber, S. Fluorophore localization algorithms for super-resolution microscopy. Nat. Methods 11, 267–279 (2014).

25. P.J. Nieuwenhuizen, R. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).

26. Diekmann, R. et al. Optimizing imaging speed and excitation intensity for single-molecule localization microscopy. Nat. Methods 17, 909–912 (2020).

27. Wäldchen, S., Lehmann, J., Klein, T., Van De Linde, S. & Sauer, M. Light-induced cell damage in live-cell super-resolution microscopy. Sci. Rep. 5, 15348 (2015).

28. Thevathasan, J. V. et al. Nuclear pores as versatile reference standards for quantitative superresolution microscopy. Nat. Methods 16, 1045–1053 (2019).

29. Dempsey, G. T., Vaughan, J. C., Chen, K. H., Bates, M. & Zhuang, X. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging. Nat. Methods 8, 1027–1036 (2011).

30. Legant, W. R. et al. High-density three-dimensional localization microscopy across large volumes. Nat. Methods 13, 359–365 (2016).

31. Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning library. In Proc. Advances in Neural Information Processing Systems (NeurIPS) Vol. 32, 8024–8035 (2019).

32. Ries, J. SMAP: a modular super-resolution microscopy analysis platform for SMLM data. Nat. Methods 17, 870–872 (2020).

33. von Chamier, L. et al. Democratising deep learning for microscopy with ZeroCostDL4Mic. Nat. Commun. 12, 2276 (2021).

34. Odena, A., Dumoulin, V. & Olah, C. Deconvolution and checkerboard artifacts. Distill https://distill.pub/2016/deconv-checkerboard/ (2016).

35. Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). Preprint at https://arxiv.org/abs/1511.07289 (2016).

36. Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

37. Weigert, M. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

38. Annibale, P., Vanni, S., Scarselli, M., Rothlisberger, U. & Radenovic, A. Quantitative photo activated localization microscopy: unraveling the effects of photoblinking. PLoS ONE 6, e22678 (2011).

39. Huang, F. Video-rate nanoscopy using scmos camera–specific single-molecule localization algorithms. Nat. Methods 10, 653–658 (2013).

40. Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. Preprint at https://arxiv.org/abs/1711.05101 (2019).

41. Banterle, N., Bui, K. H., Lemke, E. A. & Beck, M. Fourier ring correlation as a resolution criterion for super-resolution microscopy. J. Struct. Biol. 183, 363–367 (2013).

42. Perlin, K. An image synthesizer. Comput. Graph. (ACM) 19, 287–296 (1985); https://doi.org/10.1145/325165.325247

## Acknowledgements

S.C.T. is supported by the Howard Hughes Medical Institute. J.R., L.-R.M. and P.H. were supported by the European Molecular Biology Laboratory, the European Research Council (grant no. CoG-724489 to J.R.) and the National Institutes of Health Common Fund 4D Nucleome Program (grant no. U01 EB021223 to J.R.). J.H.M. and A.S. were supported by the German Research Foundation (DFG) through Germany’s Excellence Strategy (EXC-Number 2064/1, project no. 390727645) and the German Federal Ministry of Education and Research (BMBF, project no. ‘ADMIMEM’, FKZ 01IS18052). W.R.L. acknowledges support from the Searle Scholars Program, the Beckman Young Investigator Program, an National Institutes of Health New Innovator Award (no. DP2GM136653) and the Packard Fellows Program. We thank D. Sage for useful discussions, U. Boehm, D. Greenberg and P. Ramesh for comments on the manuscript, and E. Betzig and J. Lippincott Schwartz for kindly sharing data with us. We are grateful to C. Leterrier for extensive testing and useful suggestions for improving the DECODE library, and to U. Boehm for helping create DECODE tutorials. We thank D. Olbris and D. Kutra for assistance with automation of building and deploying DECODE across multiple platforms.

## Author information

Authors

### Contributions

A.S., L.-R.M., J.H.M., J.R. and S.C.T. conceived the project, analyzed the results and wrote the paper with input from all authors. A.S. and L.-R.M. wrote the software. U.M., P.H. and J.R. acquired and analyzed the biological data. A.K. provided supervision. C.J.O. and W.R.L. provided the LLS data and helped with analysis.

### Corresponding authors

Correspondence to Jakob H. Macke, Jonas Ries or Srinivas C. Turaga.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Peer review information Nature Methods thanks Alex Herbert and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Extended data

### Extended Data Fig. 1 Architecture.

The DECODE network consists of two stacked U-Nets20 with identical layouts (the three networks depicted on the left share parameters). The frame analysis module extracts informative features from three consecutive frames. These features are integrated by the temporal context module. Both U-Nets have two up- and downsampling stages and 48 filters in the first stage. Each stage consists of three fully convolutional layers with 3 × 3 filters. In each downsampling stage, the resolution is halved, and the number of filters is doubled, vice versa in each upsampling stage. Blue arrows show skip connections. Following the temporal context module three output heads with two convolutional layers each produce the output maps which have the same spatial dimensions as the input frames. The first head predicts the Bernoulli probability map p, the second head the spatial coordinates of the detected emitter Δx, Δy, Δz and its intensity N and the third head the associated uncertainties σx, σy, σz, σN. An optional fourth output head can be used for background prediction.

### Extended Data Fig. 2 Impact of grouping across grouping radius for different averaging weights.

Predictions in consecutive frames are grouped when they are closer to each other than the given grouping radius. A grouping radius of 0 nm corresponds to not performing any grouping. Predictions within a group are assigned a common set of emitter coordinates which is calculated as weighted average of their individual coordinates. We compare three different options for the weighted average: Uniform weighting (‘None’, solid lines); Weighting by the inferred number of photons for CSpline and DECODE or the inferred confidence for DeepSTORM3D (‘photons’, dotted line); Weighting by the predicted DECODE σ values, where the x,y and z values are individually weighted by $${\sigma }_{x,y,z}^{-2}$$. a, b): 3D efficiencies across grouping radii. Grouping is especially useful in the low density setting (a) where DECODE without temporal context (DECODE single) with a correctly set grouping radius can match the performance of DECODE with temporal context (DECODE multi) without grouping. This is, however, only the case when weighting by the uncertainty estimates that DECODE provides. Using grouping on top of DECODE multi offers little additional benefit. c, d): Number of groups divided by the number of localizations. Detecting all emitters and correctly grouping them would result in a ratio of 1:3 as on average each emitter is visible in three consecutive frames. See methods and Supplementary Table 1 for additional details on training and evaluation.

### Extended Data Fig. 3 Comparison of performance metrics across densities and SNRs.

DECODE outperforms DeepSTORM3D and CSpline across densities and SNRs. See methods and Supplementary Table 1 for additional details on training and evaluation.

### Extended Data Fig. 4 Comparison of localization error and CRLB for single-emitter fitting.

The r.m.s.e. achieved by DECODE and its predicted σ values closely match the single emitter CRLB in every dimension. CSpline is also able to achieve the CRLB, which has been shown for iterative MLE fitters before. In contrast the resolution that DeepSTORM3D can achieve is limited by its output representation and the size of the super-resolution voxels. a): Data simulated with high SNR (20,000 photons) and random z. r.m.s.e. and DECODE σ averaged over 10 nm bins. b): Data simulated with fixed z (0 nm) and varying SNR levels. See methods and Supplementary Table 1 for additional details on training and evaluation.

### Extended Data Fig. 5 Comparison of reconstruction quality on experimental STORM data.

Reconstructions by DECODE and the DeepSTORM3D on a subset of data shown in Fig. 4g. Histograms show within pixel distribution of localizations in x and y as well as the z coordinate in nm. DeepSTORM3D has 4 significant peaks in the subpixel distribution, corresponding to the fourfold upsampling it uses for its network output. These are visible as grid artifacts in the reconstructions. In contrast the DECODE localizations are evenly distributed and no artifacts are visible. Scale bars 0.5 μm.

### Extended Data Fig. 6 Comparison of computation times.

a) Measured as the time it takes to analyze a 64 × 64 pixel frame with varying emitter densities. Trained DECODE and DeepSTORM3D models were evaluated using a NVIDIA RTX2080Ti GPU. Computation time includes the network forward pass and postprocessing and does not include training time. CSpline was evaluated on an Intel(R)Xeon(R) CPU E5-2697 v3. b) Computation time per simulated emitter. The computation time of CSpline scales with the number emitters while the two deep learning based approaches scale with the number (and size) of the analyzed frames. GPU-based DECODE is about 20 times faster than GPU-based DeepSTORM3D and outperforms CPU-based CSpline even at low densities.

### Extended Data Fig. 7 DECODE reduces acquisition times in LLS-PAINT.

DECODE reconstruction of 35,000 frames (a) results in the same number of localizations as the Standard reconstruction of 70,000 frames (b). As DECODE detects twice as many localizations as the traditional analysis, it needs only approx. half of the frames for a high-quality reconstruction.

### Extended Data Fig. 8 Removing Pixelation artifacts.

Dim, dense out-of-focus localizations have a bias towards the pixel center (a,c). This is apparent as a non-uniform distribution of the sub-pixel positions in x and y (bottom row). This bias is not visible if every localization is rendered as a Gaussian with a standard deviation equal to the predicted uncertainty s (b,g). Filtering according to the detection probability reduces the artifact (d). Filtering according to the predicted uncertainty σ (f) or the fluorophore z-position (e) also removes the pixelation artifact. Scale bars 10 μm (a,b) and 1 μm (c-g). The overview images (a,b) are rendered with a pixel size of 10 nm, the zoom-ins (c-g) with a with a pixel size of 4 nm. The camera used to record the data has a pixel size of 117 × 127 nm.

### Extended Data Fig. 9 Performance as a function of deep network training time.

Convergence of the accuracy of DECODE for several performance metrics. Runtimes are measured on a single nVidia RTX 2080 Ti GPU. The estimated training achievable with the maximum of 12 hours possible on the free tier of Google Colab is shown in green range (assuming that a Google Colab GPU is 2 × − 4 × slower than the nVidia RTX 2080 Ti GPU). This suggests that acceptable performance is achievable using DECODE and Google Colab at minimal cost, no GPU needed. Metrics evaluated for prediction > 0.5 detection probability estimate without sigma filtering. Training data was simulated at high SNR (as described in Fig. 2c) at an average density of 1 μm−2.

### Extended Data Fig. 10 DECODE provides accurate background and signal predictions.

Shown on simulated data with inhomogeneous background of various length scales. First row: sample frames. Second row: background values simulated using Perlin noise42. Third row: background values inferred by a DECODE network that was trained on 40 × 40 pixel sized simulations with uniform background. Fourth row: Scatter plot of inferred photon counts over simulated photon counts. Scale bars are 10 μm.

## Supplementary information

### Supplementary Information

Supplementary Note, Figs. 1–6 and Table 1.

### Supplementary Video 1

Fast live-cell SMLM on the Golgi apparatus labeled with α-mannosidase II-mEos3.2 (Fig. 4d).

### Supplementary Video 2

Fast live-cell SMLM on the endoplasmic reticulum labeled with calnexin-mEos3.2 (Fig. 4e).

### Supplementary Video 3

Fast live-cell SMLM on the endoplasmic reticulum labeled with calnexin-mEos3.2 (SFig 3b).

### Supplementary Video 4

Fast live-cell SMLM on the endoplasmic reticulum labeled with calnexin-mEos3.2 (SFig 3c).

### Supplementary Software

Reviewed version of the software.

## Source data

### Source Data Fig. 2

Statistical source data Fig. 2a,c,e.

### Source Data Fig. 4

Statistical source data Fig. 4c.

## Rights and permissions

Reprints and Permissions

Speiser, A., Müller, LR., Hoess, P. et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Nat Methods 18, 1082–1090 (2021). https://doi.org/10.1038/s41592-021-01236-x

• Accepted:

• Published:

• Issue Date:

• DOI: https://doi.org/10.1038/s41592-021-01236-x

• ### Fast DNA-PAINT imaging using a deep neural network

• Kaarjel K. Narayanasamy
• Johanna V. Rahm
• Mike Heilemann

Nature Communications (2022)

• ### Global fitting for high-accuracy multi-channel single-molecule localization

• Yiming Li
• Wei Shi
• Jonas Ries

Nature Communications (2022)

• ### Avoiding a replication crisis in deep-learning-based bioimage analysis

• Romain F. Laine
• Ignacio Arganda-Carreras
• Guillaume Jacquemet

Nature Methods (2021)