We present a real-time fitter for 3D single-molecule localization microscopy using experimental point spread functions (PSFs) that achieves minimal uncertainty in 3D on any microscope and is compatible with any PSF engineering approach. We used this method to image cellular structures and attained unprecedented image quality for astigmatic PSFs. The fitter compensates for most optical aberrations and makes accurate 3D super-resolution microscopy broadly accessible, even on standard microscopes without dedicated 3D optics.
Subscribe to Journal
Get full journal access for 1 year
only $21.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Huang, B., Wang, W., Bates, M. & Zhuang, X. Science 319, 810–813 (2008).
Pavani, S.R.P. et al. Proc. Natl. Acad. Sci. USA 106, 2995–2999 (2009).
Baddeley, D., Cannell, M.B. & Soeller, C. Nano Res. 4, 589–598 (2011).
Shechtman, Y., Sahl, S.J., Backer, A.S. & Moerner, W.E. Phys. Rev. Lett. 113, 133902 (2014).
Juette, M.F. et al. Nat. Methods 5, 527–529 (2008).
Deschout, H. et al. Nat. Methods 11, 253–266 (2014).
York, A.G., Ghitani, A., Vaziri, A., Davidson, M.W. & Shroff, H. Nat. Methods 8, 327–333 (2011).
Quirin, S., Pavani, S.R.P. & Piestun, R. Proc. Natl. Acad. Sci. USA 109, 675–679 (2012).
Liu, S., Kromann, E.B., Krueger, W.D., Bewersdorf, J. & Lidke, K.A. Opt. Express 21, 29462–29487 (2013).
Tahmasbi, A., Ward, E.S. & Ober, R.J. Opt. Express 23, 7630–7652 (2015).
Mlodzianoski, M.J., Juette, M.F., Beane, G.L. & Bewersdorf, J. Opt. Express 17, 8264–8277 (2009).
Kirshner, H., Vonesch, C. & Unser, M. in 2013 IEEE 10th International Symposium on Biomedical Imaging 588–591 (IEEE, 2013).
Babcock, H.P. & Zhuang, X. Sci. Rep. 7, 552 (2017).
Deng, Y. & Shaevitz, J.W. Appl. Opt. 48, 1886–1890 (2009).
Schnitzbauer, J., Strauss, M.T., Schlichthaerle, T., Schueder, F. & Jungmann, R. Nat. Protoc. 12, 1198–1228 (2017).
van de Linde, S. et al. Nat. Protoc. 6, 991–1009 (2011).
Ovesný, M., Krřížek, P., Borkovec, J., Švindrych, Z. & Hagen, G.M. Bioinformatics 30, 2389–2390 (2014).
Franke, C., Sauer, M. & van de Linde, S. Nat. Methods 14, 41–44 (2017).
Nieuwenhuizen, R.P.J. et al. Nat. Methods 10, 557–562 (2013).
Huang, F. et al. Nat. Methods 10, 653–658 (2013).
Huang, F. et al. Cell 166, 1028–1040 (2016).
Aguet, F., Van De Ville, D. & Unser, M. Opt. Express 13, 10503–10522 (2005).
von Appen, A. et al. Nature 526, 140–143 (2015).
Jia, S., Vaughan, J.C. & Zhuang, X. Nat. Photonics 8, 302–306 (2014).
Smith, C.S., Joseph, N., Rieger, B. & Lidke, K.A. Nat. Methods 7, 373–375 (2010).
Guizar-Sicairos, M., Thurman, S.T. & Fienup, J.R. Opt. Lett. 33, 156–158 (2008).
Unser, M., Aldroubi, A. & Eden, M. IEEE Trans. Signal Process. 41, 821–833 (1993).
von Diezmann, A., Lee, M.Y., Lew, M.D. & Moerner, W.E. Optica 2, 985–993 (2015).
Holtzer, L., Meckel, T. & Schmidt, T. Appl. Phys. Lett. 90, 53902 (2007).
Press, W., Teukolsky, S., Vetterling, W. & Flannery, B. Numerical Recipes in C: The Art of Scientific Computing (Cambridge Univ. Press, Cambridge, UK, 1992).
Laurence, T.A. & Chromy, B.A. Nat. Methods 7, 338–339 (2010).
Przybylski, A., Thiel, B., Keller-Findeisen, J., Stock, B. & Bates, M. Sci. Rep. 7, 15722 (2017).
Kirshner, H., Aguet, F., Sage, D. & Unser, M. J. Microsc. 249, 13–25 (2013).
Li, Y. et al. bioRxiv Preprint at https://www.biorxiv.org/content/early/2017/08/10/172643 (2017).
Doyon, J.B. et al. Nat. Cell Biol. 13, 331–337 (2011).
Xu, K., Zhong, G. & Zhuang, X. Science 339, 452–456 (2013).
Otsuka, S. et al. eLife 5, 1–23 (2016).
Deschamps, J., Rowald, A. & Ries, J. Opt. Express 24, 28080–28090 (2016).
Ries, J., Kaplan, C., Platonova, E., Eghlidi, H. & Ewers, H. Nat. Methods 9, 582–584 (2012).
We thank J. Mehl for helping with the sample preparation and data acquisition, S. van der Linde (University of Glasgow) for help with the TRABI analysis, E. Klotzsch (University of New South Wales) for the Zeiss Elyra bead stacks, M. Lampe (ALMF, EMBL Heidelberg) for help with the acquisition on the Leica GSD, system and D. Drubin (University of California, Berkeley, Berkeley, California, USA) for the gift of SK-MEL-2 hCLTA(EN)/DNM2(EN) cells. This work was supported by the European Research Council (ERC CoG-724489 to M.M. and J.R.), the Deutsche Forschungs Gemeinschaft (DFG RI 2380/2 to J.R. and J.D.), the EMBL Interdisciplinary Postdoc Programme (EIPOD) under Marie Curie Actions COFUND (Y.L.), the 4D Nucleome/4DN NIH Common Fund (U01 EB021223 to J.E. and J.R.), and the European Molecular Biology Laboratory (Y.L., M.M., P.H., J.D., U.M., B.N., V.J.S., J.E., and J.R.).
The authors declare no competing financial interests.
Integrated supplementary information
(a) The averaged experimental unmodified 2D PSF used in this study. PSFs at axial positions from -800 nm to 800 nm are shown. The x-z reconstruction represents a vertical cross-section along the line depicted in the image corresponding to the axial position 0 nm. (b) Same as (a), but for the averaged experimental astigmatic 3D PSF. Scale bars 1 μm.
Supplementary Figure 2 The localization precision of the fitter achieves minimum uncertainty with the cspline interpolated experimental astigmatic PSF.
Single molecule images were simulated using an experimental astigmatic PSF model with 5000 photons/localization (panel a) or 500 photons/localization (panel b) and 10 background photons/pixel and fitted with the interpolated cubic spline model. We then evaluated the x, y and z localization precisions at different axial positions as the standard deviation of the error between fitted 3D positions and ground truth of the 10000 simulated molecules at each axial position (symbols). Additionally, we calculated the root-mean-square error between fitted positions and ground truth (dots). The localization precision as well as the localization accuracy of the fitted positions achieves the estimated Cramér–Rao lower bound (CRLB) (denoted by lines) in all 3 dimensions. Only for dimmer fluorophores and large defocus, a small fraction (<5%) of fluorophores that converged to a z position close to the starting z position (focus), result in a deteriorated average localization accuracy. The CRLB is evaluated as the diagonal element of the inverse of the Fisher information matrix with the fitted parameters and cspline interpolated experimental PSF model.
Supplementary Figure 3 Experimental localization precision compared with the Cramér–Rao lower bound (CRLB), measured by repeated imaging of a single bead.
For each z-position, spaced 50 nm apart, 50 images were acquired. The upper panel shows the fitted z-position vs. the objective z position and the lower panel compares the experimental localization precision (standard deviation of the fitted z position) with the CRLB estimated by the software. (a): The EM gain was switched off, median photons per localization were 28650. (b): The EM gain was switched off, median photons per localization were 7060. For fitting of a, b, the CCD readout noise was taken into account by fitting the data with the sCMOS noise model, using a pixel-independent noise calibrated from a stack of dark images (21.7 photoelectrons). (c): An EM gain of 200 was used, median photons per localization were 6500. The beads were fitted with a noise model that approximates the EM excess noise by Poisson noise1. To this end, we divided the photon numbers prior to fitting by the noise excess factor F2, which approaches 2 for large EM gain. This approximation seems to overestimate the noise; thus, the experimental uncertainty is slightly below the CRLB.1. Fullerton, S., Bennett, K., Toda, E. & Takahashi, T. Optimization of precision localization microscopy using CMOS camera technology. Proc of SPIE 8228: 82280T–1 (2012).
Supplementary Figure 4 Experimental profiles and localization precisions for the data used in this paper.
The insets denote the regions where the profiles were taken. Dotted lines: Fit with two Gaussians with a distance d and a standard deviation s in nm. Scale bars 100 nm.
Supplementary Figure 5 Comparison of our fitter to previous work, using the example of immunolabeled microtubules.
(a) Front view reconstructions of immunolabeled microtubules taken from the literature2–4. The hollow structure of the microtubules can only be resolved in the interferometric 4-Pi methods. (b) Front view reconstructions of microtubules presented in this work. We are able to resolve hollow, cylinder-like structure of microtubules in both DNA-PAINT and dSTORM imaging. (c) Sketch of a microtubule labeled with antibodies to illustrate its dimensions. Scale bars 100 nm. (a, top) Adapted with permission from Ref 2, Springer Customer Service Centre GmbH: Springer Nature, Nature Photonics, copyright 2015. (a, middle and bottom) Adapted with permission from Refs 3 and 4, respectively, under creative commons license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0), cropped from original and added scale bar.2. Verdeny-Vilanova, I. et al. 3D motion of vesicles along microtubules helps them to circumvent obstacles in cells. J. Cell Sci. 130, 1904–1916 (2017).3. Bourg, N. et al. Direct optical nanoscopy with axially localized detection. Nat Phot. 9, 587–593 (2015).4. Huang, F. et al. Ultra-High Resolution 3D Imaging of Whole Cells. Cell 166, 1028–1040 (2016).
Microtubules in U-2 OS cells, labeled with anti-alpha tubulin primary and Alexa Fluor 647-coupled secondary antibodies, were imaged using dSTORM with a cylindrical lens. Upper panel: x-y top view. Lines denote regions for which 250 nm x-z (side view) cross-sections are shown in the lower panels. Corresponding localization precisions and profiles can be found in Supplementary Figure 4. Scale bars: 1 μm (top view), 100 nm (side views).
Supplementary Figure 7 Convergence and robustness of Levenberg–Marquardt versus the Newton iterative method.
(a), (b): Levenberg-Marquardt converges faster than Newton. We compared for the L-M and Newton method the change of the MLE cost function at each iteration. is defined as in ref. 5. The iteration stops when the relative change of decreases by less than 10−6. The number of iterations is 35±4 for the Newton method and 12±1 for the L-M method, which indicates the L-M method is more efficient than the custom Newton solver. (a) Update of during each iteration. (b) Relative change of during each iteration. Newton iterative scheme as in Smith et al6.(c): The Levenberg-Marquardt algorithm is more robust with respect to starting parameters than the Newton method. The localization precision is plotted in dependence on the photons/localization. Both algorithms can achieve the CRLB for molecules with more than 100 photons when the centroid of the fitting window is used as a start parameter. We then purposely offset the starting parameter for x, y by 2 pixels. Whereas the final localization precision for the L-M algorithm is robust with respect to the start parameters, the Newton method often fails to converge for a wrong starting parameter estimate.Simulation parameters: Elliptical Gaussian PSF model with a sigma 1.5 and 2.5 pixels in x and y, respectively. 5 background photons per pixel. 500 photons per localization was used for (a) and (b). Fitting parameters: ROI size 13 × 13 pixels, 1000 molecules per data point, maximum number of iterations was 100.5. Laurence, T. A. & Chromy, B. A. Efficient maximum likelihood estimator fitting of histograms. Nat. Methods 7, 338–339 (2010).6. Smith, C. S., Joseph, N., Rieger, B. & Lidke, K. a. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).
Supplementary Figure 8 Localization precision and accuracy for a Gaussian versus an experimental PSF model.
Single molecule images were simulated using an experimental PSF model with 5000 photons/localization and 10 background photons/pixel. For the cspline 3D and elliptical Gaussian fit, an experimental astigmatic PSF was used for the simulation. For the cspline 2D fit, an unmodified experimental PSF was used. We then evaluated both localization precision and accuracy in 3D at different axial positions from 1000 simulated molecules at each axial position. The localization precision is calculated as the standard deviation of the difference between the fitted positions and ground truth positions, and the localization accuracy is calculated as the root mean square error between the fitted positions and ground truth positions. (a) Localization accuracy and (b) localization precision in z for different models at different axial positions. For the elliptical Gaussian PSF model, two algorithms were used for z localization as described in Online Methods. (c, d) are the same as (a, b) but in lateral direction.
Supplementary Figure 9 Comparison of the experimental PSF fit with a Gaussian fit for astigmatic data on the nuclear complex.
(a) Nup107-SNAP-BG-AF647 in U-2 OS cells, imaged using dSTORM with a cylindrical lens (top view). (b) 500 nm side view cross-section of the region denoted in (a) using the experimental PSF fit. (c) Side view reconstruction of the region denoted in (a) using the Gaussian z fit6. Scale bars 1 μm.6. Smith, C. S., Joseph, N., Rieger, B. & Lidke, K. a. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).
Single molecule images were simulated using an experimental astigmatic PSF model with 5000 photons/localization and 10 background photons/pixel. The simulated single molecule images were fitted with a cspline interpolated PSF model (a) and an elliptical Gaussian model (b). Accuracy (root mean square error) and precision (standard deviation) of fitted photons/localization at different axial positions for (a) cspline fit and (b) elliptical Gaussian fit. Though the returned photon precision of the Gaussian fit is close to the CRLB, the returned accuracy is much larger than the precision, which indicates a strong systematic error in estimating the photons using the Gaussian PSF model. In contrast, for the cspline interpolated PSF model the accuracy, precision and CRLB of the photons/localization agree well with each other.
(a) Overview image of microtubules in a U-2 OS cell. The axial positions were color coded according to the color scale bar. (b) y-z plot of the boxed region in (a) when the PSF was modeled using only a single bead without smoothing. (c) y-z plot of the boxed region in (a) when PSF was robustly modeled with many beads across different fields of view and with regularization, as described in Online Methods. (d) z-profiles of (b) and (c). The stripe artifact is avoided using the proper averaging and regularization. Scale bars: (a) 1 μm, (b), (c) 100 nm.
(a) Comparison of the speed of the fitter presented in this work and previously published fitters. A 13×13 pixel window size was used for benchmarking. The fitting speeds of Babcock’s code and Kirshner’s code were measured using the example data provided in the software package by the authors. The fitting speed of ThunderSTORM was estimated using 1,000 frames of dSTORM data from Supplementary Figure 6. We measured the overall time difference between an analysis with a stringent detection cutoff and one with a lower threshold. We then divided the difference in number of fitted localization by the time difference to estimate the number of fits per second.(b) Comparison of the speed of the Newton method and the L-M algorithm using an elliptical Gaussian PSF model. The effect of the readout noise variance for sCMOS camera data was also evaluated. Even though the L-M algorithm has higher complexity as it evaluates every iteration to determine the damping factor, it is still slightly faster than Newton method due to the fact that less iterations are needed. (c) Comparison of the speed of astigmatic Gauss z fit and cspline z fit. The GPU code is overall more than 100 times faster than the CPU code running on a single thread. It is interesting to notice that the spline fit is even faster than the Gauss z fit in the CPU code while it is slower than the Gauss z fit in GPU code. This is probably because the spline coefficients are accessed much more frequently in the GPU than the CPU, which slows down the performance.
Single-molecule images were simulated considering pixel-dependent readout noise, as is present in sCMOS cameras. The readout noise of the overall pixels is set as a Gaussian distribution with a mean of 1.4 e− and a sigma with 0.2 e−. (a) The map of the readout noise. The readout noise of the pixel located 2 pixels to the up of the center is set as 30 e−. (b) Simulated single molecule with the readout noise as shown in (a). Each molecule was simulated as an experimental PSF, 200 photons/localization and 2 background photons/pixel. The positions of the molecules were all placed in the center (6, 6) of the window. (c) Scatter plot of the 1000 fitted positions of the molecules as shown in (b) fitted with the EMCCD noise model. (d) Scatter plot of the 1000 fitted positions of the molecules as shown in (b) fitted with the sCMOS noise model.
Supplementary Figure 14 The cspline fitter works on reference data, as well as with commercial microscopes, advanced PSF engineering approaches, and highly aberrated PSFs.
(a), (b) Fit of the 2016 SMLM Challenge (http://bigwww.epfl.ch/smlm) training data set with our fitter for experimental PSFs. a) Bi-directional fitting of the 2D training dataset (MT0.N1.LD-2D). The panels show top-view and side-view reconstructions and slices through the PSF used for fitting. (b) Fitting of the Double-Helix PSF training dataset (MT0.N1.LD-DH). Panels as for (a). Scale bars: 1 μm.(c)-(h) Spline interpolated PSF models were derived from bead stacks and used to fit 6 individual beads from a different stack. (c) Zeiss Elyra, using a phase-ramp PSF7. (d) Leica SR GSD 3D with astigmatic PSF. (e) Double-Helix PSF from the SMLM challenge (MT0.N1.LD-DH). (f)-(h) Experimental aberrated PSFs generated with a deformable mirror on a custom microscope.7. Baddeley, D., Cannell, M. B. & Soeller, C. Three-dimensional sub-100 nm super-resolution imaging of biological samples using a phase ramp in the objective pupil. Nano Res. 4, 589–598 (2011).
Nup107-SNAP-BG-AlexaFluor647 in U-2 OS cells, imaged using dSTORM on a commercial Leica SR GSD 3D microscope. (a) overview, (b) side-view reconstruction using the cspline fitter with an experimentally derived PSF model, (c) as (b), but using the coordinates fitted with the Leica LAS X software.
About this article
Cite this article
Li, Y., Mund, M., Hoess, P. et al. Real-time 3D single-molecule localization using experimental point spread functions. Nat Methods 15, 367–369 (2018). https://doi.org/10.1038/nmeth.4661
Nature Methods (2020)
Nature Methods (2020)
Annual Review of Biomedical Engineering (2020)
Nature Communications (2020)
Towards mapping the 3D genome through high speed single-molecule tracking of functional transcription factors in single living cells