Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

Real-time 3D single-molecule localization using experimental point spread functions

Nature Methods volume 15, pages 367–369 (2018)Cite this article

Subjects

Abstract

We present a real-time fitter for 3D single-molecule localization microscopy using experimental point spread functions (PSFs) that achieves minimal uncertainty in 3D on any microscope and is compatible with any PSF engineering approach. We used this method to image cellular structures and attained unprecedented image quality for astigmatic PSFs. The fitter compensates for most optical aberrations and makes accurate 3D super-resolution microscopy broadly accessible, even on standard microscopes without dedicated 3D optics.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Reconstruction of cellular structures with high 3D resolution using the real-time fitter for experimental PSFs.
Figure 2: Accurate 3D positions from a 2D data set with an unmodified PSF.

Similar content being viewed by others

References

  1. Huang, B., Wang, W., Bates, M. & Zhuang, X. Science 319, 810–813 (2008).

    Article  CAS  Google Scholar 

  2. Pavani, S.R.P. et al. Proc. Natl. Acad. Sci. USA 106, 2995–2999 (2009).

    Article  CAS  Google Scholar 

  3. Baddeley, D., Cannell, M.B. & Soeller, C. Nano Res. 4, 589–598 (2011).

    Article  CAS  Google Scholar 

  4. Shechtman, Y., Sahl, S.J., Backer, A.S. & Moerner, W.E. Phys. Rev. Lett. 113, 133902 (2014).

    Article  Google Scholar 

  5. Juette, M.F. et al. Nat. Methods 5, 527–529 (2008).

    Article  CAS  Google Scholar 

  6. Deschout, H. et al. Nat. Methods 11, 253–266 (2014).

    Article  CAS  Google Scholar 

  7. York, A.G., Ghitani, A., Vaziri, A., Davidson, M.W. & Shroff, H. Nat. Methods 8, 327–333 (2011).

    Article  CAS  Google Scholar 

  8. Quirin, S., Pavani, S.R.P. & Piestun, R. Proc. Natl. Acad. Sci. USA 109, 675–679 (2012).

    Article  CAS  Google Scholar 

  9. Liu, S., Kromann, E.B., Krueger, W.D., Bewersdorf, J. & Lidke, K.A. Opt. Express 21, 29462–29487 (2013).

    Article  Google Scholar 

  10. Tahmasbi, A., Ward, E.S. & Ober, R.J. Opt. Express 23, 7630–7652 (2015).

    Article  CAS  Google Scholar 

  11. Mlodzianoski, M.J., Juette, M.F., Beane, G.L. & Bewersdorf, J. Opt. Express 17, 8264–8277 (2009).

    Article  CAS  Google Scholar 

  12. Kirshner, H., Vonesch, C. & Unser, M. in 2013 IEEE 10th International Symposium on Biomedical Imaging 588–591 (IEEE, 2013).

  13. Babcock, H.P. & Zhuang, X. Sci. Rep. 7, 552 (2017).

    Article  Google Scholar 

  14. Deng, Y. & Shaevitz, J.W. Appl. Opt. 48, 1886–1890 (2009).

    Article  Google Scholar 

  15. Schnitzbauer, J., Strauss, M.T., Schlichthaerle, T., Schueder, F. & Jungmann, R. Nat. Protoc. 12, 1198–1228 (2017).

    Article  CAS  Google Scholar 

  16. van de Linde, S. et al. Nat. Protoc. 6, 991–1009 (2011).

    Article  CAS  Google Scholar 

  17. Ovesný, M., Krřížek, P., Borkovec, J., Švindrych, Z. & Hagen, G.M. Bioinformatics 30, 2389–2390 (2014).

    Article  Google Scholar 

  18. Franke, C., Sauer, M. & van de Linde, S. Nat. Methods 14, 41–44 (2017).

    Article  CAS  Google Scholar 

  19. Nieuwenhuizen, R.P.J. et al. Nat. Methods 10, 557–562 (2013).

    Article  CAS  Google Scholar 

  20. Huang, F. et al. Nat. Methods 10, 653–658 (2013).

    Article  CAS  Google Scholar 

  21. Huang, F. et al. Cell 166, 1028–1040 (2016).

    Article  CAS  Google Scholar 

  22. Aguet, F., Van De Ville, D. & Unser, M. Opt. Express 13, 10503–10522 (2005).

    Article  Google Scholar 

  23. von Appen, A. et al. Nature 526, 140–143 (2015).

    Article  CAS  Google Scholar 

  24. Jia, S., Vaughan, J.C. & Zhuang, X. Nat. Photonics 8, 302–306 (2014).

    Article  CAS  Google Scholar 

  25. Smith, C.S., Joseph, N., Rieger, B. & Lidke, K.A. Nat. Methods 7, 373–375 (2010).

    Article  CAS  Google Scholar 

  26. Guizar-Sicairos, M., Thurman, S.T. & Fienup, J.R. Opt. Lett. 33, 156–158 (2008).

    Article  Google Scholar 

  27. Unser, M., Aldroubi, A. & Eden, M. IEEE Trans. Signal Process. 41, 821–833 (1993).

    Article  Google Scholar 

  28. von Diezmann, A., Lee, M.Y., Lew, M.D. & Moerner, W.E. Optica 2, 985–993 (2015).

    Article  Google Scholar 

  29. Holtzer, L., Meckel, T. & Schmidt, T. Appl. Phys. Lett. 90, 53902 (2007).

    Article  Google Scholar 

  30. Press, W., Teukolsky, S., Vetterling, W. & Flannery, B. Numerical Recipes in C: The Art of Scientific Computing (Cambridge Univ. Press, Cambridge, UK, 1992).

  31. Laurence, T.A. & Chromy, B.A. Nat. Methods 7, 338–339 (2010).

    Article  CAS  Google Scholar 

  32. Przybylski, A., Thiel, B., Keller-Findeisen, J., Stock, B. & Bates, M. Sci. Rep. 7, 15722 (2017).

    Article  Google Scholar 

  33. Kirshner, H., Aguet, F., Sage, D. & Unser, M. J. Microsc. 249, 13–25 (2013).

    Article  CAS  Google Scholar 

  34. Li, Y. et al. bioRxiv Preprint at https://www.biorxiv.org/content/early/2017/08/10/172643 (2017).

  35. Doyon, J.B. et al. Nat. Cell Biol. 13, 331–337 (2011).

    Article  CAS  Google Scholar 

  36. Xu, K., Zhong, G. & Zhuang, X. Science 339, 452–456 (2013).

    Article  CAS  Google Scholar 

  37. Otsuka, S. et al. eLife 5, 1–23 (2016).

    Article  Google Scholar 

  38. Deschamps, J., Rowald, A. & Ries, J. Opt. Express 24, 28080–28090 (2016).

    Article  CAS  Google Scholar 

  39. Ries, J., Kaplan, C., Platonova, E., Eghlidi, H. & Ewers, H. Nat. Methods 9, 582–584 (2012).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank J. Mehl for helping with the sample preparation and data acquisition, S. van der Linde (University of Glasgow) for help with the TRABI analysis, E. Klotzsch (University of New South Wales) for the Zeiss Elyra bead stacks, M. Lampe (ALMF, EMBL Heidelberg) for help with the acquisition on the Leica GSD, system and D. Drubin (University of California, Berkeley, Berkeley, California, USA) for the gift of SK-MEL-2 hCLTA(EN)/DNM2(EN) cells. This work was supported by the European Research Council (ERC CoG-724489 to M.M. and J.R.), the Deutsche Forschungs Gemeinschaft (DFG RI 2380/2 to J.R. and J.D.), the EMBL Interdisciplinary Postdoc Programme (EIPOD) under Marie Curie Actions COFUND (Y.L.), the 4D Nucleome/4DN NIH Common Fund (U01 EB021223 to J.E. and J.R.), and the European Molecular Biology Laboratory (Y.L., M.M., P.H., J.D., U.M., B.N., V.J.S., J.E., and J.R.).

Author information

Authors and Affiliations

  1. Cell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany

    Yiming Li, Markus Mund, Philipp Hoess, Joran Deschamps, Ulf Matti, Bianca Nijmeijer, Vilma Jimenez Sabinina, Jan Ellenberg & Jonas Ries

  2. Molecular and Cellular Therapeutics, Royal College of Surgeons in Ireland, Dublin, Ireland

    Ingmar Schoen

Authors
  1. Yiming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markus Mund
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Philipp Hoess
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joran Deschamps
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ulf Matti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bianca Nijmeijer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Vilma Jimenez Sabinina
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jan Ellenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ingmar Schoen
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jonas Ries
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.L. and J.R. conceived the approach, developed the methods, wrote the software, and analyzed the data. M.M. imaged clathrin-coated pits. P.H. imaged nuclear pore complexes. U.M. carried out DNA-PAINT imaging of microtubules. B.N., V.J.S., and J.E. contributed the Nup107 cell line. I.S. contributed the custom-made DNA-PAINT antibodies. J.D. acquired the aberrated PSFs and helped with the data analysis. Y.L., M.M., and J.R. wrote the manuscript with input from all other authors.

Corresponding author

Correspondence to Jonas Ries.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Experimental unmodified 2D and astigmatic 3D PSFs.

(a) The averaged experimental unmodified 2D PSF used in this study. PSFs at axial positions from -800 nm to 800 nm are shown. The x-z reconstruction represents a vertical cross-section along the line depicted in the image corresponding to the axial position 0 nm. (b) Same as (a), but for the averaged experimental astigmatic 3D PSF. Scale bars 1 μm.

Supplementary Figure 2 The localization precision of the fitter achieves minimum uncertainty with the cspline interpolated experimental astigmatic PSF.

Single molecule images were simulated using an experimental astigmatic PSF model with 5000 photons/localization (panel a) or 500 photons/localization (panel b) and 10 background photons/pixel and fitted with the interpolated cubic spline model. We then evaluated the x, y and z localization precisions at different axial positions as the standard deviation of the error between fitted 3D positions and ground truth of the 10000 simulated molecules at each axial position (symbols). Additionally, we calculated the root-mean-square error between fitted positions and ground truth (dots). The localization precision as well as the localization accuracy of the fitted positions achieves the estimated Cramér–Rao lower bound (CRLB) (denoted by lines) in all 3 dimensions. Only for dimmer fluorophores and large defocus, a small fraction (<5%) of fluorophores that converged to a z position close to the starting z position (focus), result in a deteriorated average localization accuracy. The CRLB is evaluated as the diagonal element of the inverse of the Fisher information matrix with the fitted parameters and cspline interpolated experimental PSF model.

Supplementary Figure 3 Experimental localization precision compared with the Cramér–Rao lower bound (CRLB), measured by repeated imaging of a single bead.

For each z-position, spaced 50 nm apart, 50 images were acquired. The upper panel shows the fitted z-position vs. the objective z position and the lower panel compares the experimental localization precision (standard deviation of the fitted z position) with the CRLB estimated by the software. (a): The EM gain was switched off, median photons per localization were 28650. (b): The EM gain was switched off, median photons per localization were 7060. For fitting of a, b, the CCD readout noise was taken into account by fitting the data with the sCMOS noise model, using a pixel-independent noise calibrated from a stack of dark images (21.7 photoelectrons). (c): An EM gain of 200 was used, median photons per localization were 6500. The beads were fitted with a noise model that approximates the EM excess noise by Poisson noise1. To this end, we divided the photon numbers prior to fitting by the noise excess factor F2, which approaches 2 for large EM gain. This approximation seems to overestimate the noise; thus, the experimental uncertainty is slightly below the CRLB.

1. Fullerton, S., Bennett, K., Toda, E. & Takahashi, T. Optimization of precision localization microscopy using CMOS camera technology. Proc of SPIE 8228: 82280T–1 (2012).

Supplementary Figure 4 Experimental profiles and localization precisions for the data used in this paper.

The insets denote the regions where the profiles were taken. Dotted lines: Fit with two Gaussians with a distance d and a standard deviation s in nm. Scale bars 100 nm.

Supplementary Figure 5 Comparison of our fitter to previous work, using the example of immunolabeled microtubules.

(a) Front view reconstructions of immunolabeled microtubules taken from the literature2–4. The hollow structure of the microtubules can only be resolved in the interferometric 4-Pi methods. (b) Front view reconstructions of microtubules presented in this work. We are able to resolve hollow, cylinder-like structure of microtubules in both DNA-PAINT and dSTORM imaging. (c) Sketch of a microtubule labeled with antibodies to illustrate its dimensions. Scale bars 100 nm. (a, top) Adapted with permission from Ref 2, Springer Customer Service Centre GmbH: Springer Nature, Nature Photonics, copyright 2015. (a, middle and bottom) Adapted with permission from Refs 3 and 4, respectively, under creative commons license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0), cropped from original and added scale bar.

2. Verdeny-Vilanova, I. et al. 3D motion of vesicles along microtubules helps them to circumvent obstacles in cells. J. Cell Sci. 130, 1904–1916 (2017).

3. Bourg, N. et al. Direct optical nanoscopy with axially localized detection. Nat Phot. 9, 587–593 (2015).

4. Huang, F. et al. Ultra-High Resolution 3D Imaging of Whole Cells. Cell 166, 1028–1040 (2016).

Supplementary Figure 6 3D astigmatic dSTORM image of microtubules.

Microtubules in U-2 OS cells, labeled with anti-alpha tubulin primary and Alexa Fluor 647-coupled secondary antibodies, were imaged using dSTORM with a cylindrical lens. Upper panel: x-y top view. Lines denote regions for which 250 nm x-z (side view) cross-sections are shown in the lower panels. Corresponding localization precisions and profiles can be found in Supplementary Figure 4. Scale bars: 1 μm (top view), 100 nm (side views).

Supplementary Figure 7 Convergence and robustness of Levenberg–Marquardt versus the Newton iterative method.

(a), (b): Levenberg-Marquardt converges faster than Newton. We compared for the L-M and Newton method the change of the MLE cost function at each iteration. is defined as in ref. 5. The iteration stops when the relative change of decreases by less than 10−6. The number of iterations is 35±4 for the Newton method and 12±1 for the L-M method, which indicates the L-M method is more efficient than the custom Newton solver. (a) Update of during each iteration. (b) Relative change of during each iteration. Newton iterative scheme as in Smith et al6.

(c): The Levenberg-Marquardt algorithm is more robust with respect to starting parameters than the Newton method. The localization precision is plotted in dependence on the photons/localization. Both algorithms can achieve the CRLB for molecules with more than 100 photons when the centroid of the fitting window is used as a start parameter. We then purposely offset the starting parameter for x, y by 2 pixels. Whereas the final localization precision for the L-M algorithm is robust with respect to the start parameters, the Newton method often fails to converge for a wrong starting parameter estimate.

Simulation parameters: Elliptical Gaussian PSF model with a sigma 1.5 and 2.5 pixels in x and y, respectively. 5 background photons per pixel. 500 photons per localization was used for (a) and (b). Fitting parameters: ROI size 13 × 13 pixels, 1000 molecules per data point, maximum number of iterations was 100.

5. Laurence, T. A. & Chromy, B. A. Efficient maximum likelihood estimator fitting of histograms. Nat. Methods 7, 338–339 (2010).

6. Smith, C. S., Joseph, N., Rieger, B. & Lidke, K. a. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).

Supplementary Figure 8 Localization precision and accuracy for a Gaussian versus an experimental PSF model.

Single molecule images were simulated using an experimental PSF model with 5000 photons/localization and 10 background photons/pixel. For the cspline 3D and elliptical Gaussian fit, an experimental astigmatic PSF was used for the simulation. For the cspline 2D fit, an unmodified experimental PSF was used. We then evaluated both localization precision and accuracy in 3D at different axial positions from 1000 simulated molecules at each axial position. The localization precision is calculated as the standard deviation of the difference between the fitted positions and ground truth positions, and the localization accuracy is calculated as the root mean square error between the fitted positions and ground truth positions. (a) Localization accuracy and (b) localization precision in z for different models at different axial positions. For the elliptical Gaussian PSF model, two algorithms were used for z localization as described in Online Methods. (c, d) are the same as (a, b) but in lateral direction.

Supplementary Figure 9 Comparison of the experimental PSF fit with a Gaussian fit for astigmatic data on the nuclear complex.

(a) Nup107-SNAP-BG-AF647 in U-2 OS cells, imaged using dSTORM with a cylindrical lens (top view). (b) 500 nm side view cross-section of the region denoted in (a) using the experimental PSF fit. (c) Side view reconstruction of the region denoted in (a) using the Gaussian z fit6. Scale bars 1 μm.

6. Smith, C. S., Joseph, N., Rieger, B. & Lidke, K. a. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).

Supplementary Figure 10 Photon count underestimation by Gaussian fit of experimental PSFs.

Single molecule images were simulated using an experimental astigmatic PSF model with 5000 photons/localization and 10 background photons/pixel. The simulated single molecule images were fitted with a cspline interpolated PSF model (a) and an elliptical Gaussian model (b). Accuracy (root mean square error) and precision (standard deviation) of fitted photons/localization at different axial positions for (a) cspline fit and (b) elliptical Gaussian fit. Though the returned photon precision of the Gaussian fit is close to the CRLB, the returned accuracy is much larger than the precision, which indicates a strong systematic error in estimating the photons using the Gaussian PSF model. In contrast, for the cspline interpolated PSF model the accuracy, precision and CRLB of the photons/localization agree well with each other.

Supplementary Figure 11 A noisy PSF model leads to stripe artifacts.

(a) Overview image of microtubules in a U-2 OS cell. The axial positions were color coded according to the color scale bar. (b) y-z plot of the boxed region in (a) when the PSF was modeled using only a single bead without smoothing. (c) y-z plot of the boxed region in (a) when PSF was robustly modeled with many beads across different fields of view and with regularization, as described in Online Methods. (d) z-profiles of (b) and (c). The stripe artifact is avoided using the proper averaging and regularization. Scale bars: (a) 1 μm, (b), (c) 100 nm.

Supplementary Figure 12 Computational speed of different fitting routines.

(a) Comparison of the speed of the fitter presented in this work and previously published fitters. A 13×13 pixel window size was used for benchmarking. The fitting speeds of Babcock’s code and Kirshner’s code were measured using the example data provided in the software package by the authors. The fitting speed of ThunderSTORM was estimated using 1,000 frames of dSTORM data from Supplementary Figure 6. We measured the overall time difference between an analysis with a stringent detection cutoff and one with a lower threshold. We then divided the difference in number of fitted localization by the time difference to estimate the number of fits per second.

(b) Comparison of the speed of the Newton method and the L-M algorithm using an elliptical Gaussian PSF model. The effect of the readout noise variance for sCMOS camera data was also evaluated. Even though the L-M algorithm has higher complexity as it evaluates every iteration to determine the damping factor, it is still slightly faster than Newton method due to the fact that less iterations are needed. (c) Comparison of the speed of astigmatic Gauss z fit and cspline z fit. The GPU code is overall more than 100 times faster than the CPU code running on a single thread. It is interesting to notice that the spline fit is even faster than the Gauss z fit in the CPU code while it is slower than the Gauss z fit in GPU code. This is probably because the spline coefficients are accessed much more frequently in the GPU than the CPU, which slows down the performance.

Supplementary Figure 13 The sCMOS noise model avoids readout-noise-induced bias.

Single-molecule images were simulated considering pixel-dependent readout noise, as is present in sCMOS cameras. The readout noise of the overall pixels is set as a Gaussian distribution with a mean of 1.4 e− and a sigma with 0.2 e−. (a) The map of the readout noise. The readout noise of the pixel located 2 pixels to the up of the center is set as 30 e−. (b) Simulated single molecule with the readout noise as shown in (a). Each molecule was simulated as an experimental PSF, 200 photons/localization and 2 background photons/pixel. The positions of the molecules were all placed in the center (6, 6) of the window. (c) Scatter plot of the 1000 fitted positions of the molecules as shown in (b) fitted with the EMCCD noise model. (d) Scatter plot of the 1000 fitted positions of the molecules as shown in (b) fitted with the sCMOS noise model.

Supplementary Figure 14 The cspline fitter works on reference data, as well as with commercial microscopes, advanced PSF engineering approaches, and highly aberrated PSFs.

(a), (b) Fit of the 2016 SMLM Challenge (http://bigwww.epfl.ch/smlm) training data set with our fitter for experimental PSFs. a) Bi-directional fitting of the 2D training dataset (MT0.N1.LD-2D). The panels show top-view and side-view reconstructions and slices through the PSF used for fitting. (b) Fitting of the Double-Helix PSF training dataset (MT0.N1.LD-DH). Panels as for (a). Scale bars: 1 μm.

(c)-(h) Spline interpolated PSF models were derived from bead stacks and used to fit 6 individual beads from a different stack. (c) Zeiss Elyra, using a phase-ramp PSF7. (d) Leica SR GSD 3D with astigmatic PSF. (e) Double-Helix PSF from the SMLM challenge (MT0.N1.LD-DH). (f)-(h) Experimental aberrated PSFs generated with a deformable mirror on a custom microscope.

7. Baddeley, D., Cannell, M. B. & Soeller, C. Three-dimensional sub-100 nm super-resolution imaging of biological samples using a phase ramp in the objective pupil. Nano Res. 4, 589–598 (2011).

Supplementary Figure 15 The cspline fitter can improve the resolution on commercial microscopes.

Nup107-SNAP-BG-AlexaFluor647 in U-2 OS cells, imaged using dSTORM on a commercial Leica SR GSD 3D microscope. (a) overview, (b) side-view reconstruction using the cspline fitter with an experimentally derived PSF model, (c) as (b), but using the coordinates fitted with the Leica LAS X software.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–15

Life Sciences Reporting Summary

Supplementary Software 1

Source code, example code, and example data set

Supplementary Software 2

Compiled software

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Mund, M., Hoess, P. et al. Real-time 3D single-molecule localization using experimental point spread functions. Nat Methods 15, 367–369 (2018). https://doi.org/10.1038/nmeth.4661

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.4661

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics