The quality of super-resolution images obtained by single-molecule localization microscopy (SMLM) depends largely on the software used to detect and accurately localize point sources. In this work, we focus on the computational aspects of super-resolution microscopy and present a comprehensive evaluation of localization software packages. Our philosophy is to evaluate each package as a whole, thus maintaining the integrity of the software. We prepared synthetic data that represent three-dimensional structures modeled after biological components, taking excitation parameters, noise sources, point-spread functions and pixelation into account. We then asked developers to run their software on our data; most responded favorably, allowing us to present a broad picture of the methods available. We evaluated their results using quantitative and user-interpretable criteria: detection rate, accuracy, quality of image reconstruction, resolution, software usability and computational resources. These metrics reflect the various tradeoffs of SMLM software packages and help users to choose the software that fits their needs.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
BMC Bioinformatics Open Access 05 June 2023
Nature Communications Open Access 18 May 2023
PhotoniX Open Access 05 January 2023
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).
Rust, M.J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795 (2006).
Hess, S.T., Girirajan, T.P. & Mason, M.D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 91, 4258–4272 (2006).
Balzarotti, F. & Stefani, F.D. Plasmonics meets far-field optical nanoscopy. ACS Nano 6, 4580–4584 (2012).
Walder, R., Nelson, N. & Schwartz, D.K. Super-resolution surface mapping using the trajectories of molecular probes. Nat. Commun. 2, 515 (2011).
Manley, S., Gunzenhäuser, J. & Olivier, N. A starter kit for point-localization super-resolution imaging. Curr. Opin. Chem. Biol. 15, 813–821 (2011).
Schermelleh, L., Heintzmann, R. & Leonhardt, H. A guide to super-resolution fluorescence microscopy. J. Cell Biol. 190, 165–175 (2010).
Sauer, M. Localization microscopy coming of age: from concepts to biological impact. J. Cell Sci. 126, 3505–3513 (2013).
Moerner, W.E. New directions in single-molecule imaging and analysis. Proc. Natl. Acad. Sci. 104, 12596–12602 (2007).
Small, A. & Stahlheber, S. Fluorophore localization algorithms for super-resolution microscopy. Nat. Methods 11, 267–279 (2014).
Endesfelder, U. & Heilemann, M. Art and artifacts in single-molecule localization microscopy: beyond attractive images. Nat. Methods 11, 235–238 (2014).
Wolter, S., Endesfelder, U., van de Linde, S., Heilemann, M. & Sauer, M. Measuring localization performance of super-resolution algorithms on very active samples. Opt. Express 19, 7020–7033 (2011).
Kirshner, H., Aguet, F., Sage, D. & Unser, M. 3-D PSF fitting for fluorescence microscopy: implementation and localization application. J. Microsc. 249, 13–25 (2013).
Köthe, U., Herrmannsdoerfer, F., Kats, I. & Hamprecht, F.A. SimpleSTORM: a fast, self-calibrating reconstruction algorithm for localization microscopy. Histochem. Cell Biol. 141, 613–627 (2014).
Ovesný, M., Krizek, P., Borkovec, J., Svindrych, Z. & Hagen, G.M. ThunderSTORM: a comprehensive ImageJ plugin for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).
Ovesný, M., Krízek, P., Svindrych, Z. & Hagen, G.M. High density 3D localization microscopy using sparse support recovery. Opt. Express 22, 31263–31276 (2014).
Ma, H., Kawai, H., Toda, E., Zeng, S. & Huang, Z.-L. Localization-based super-resolution microscopy with an sCMOS camera part III: camera embedded data processing significantly reduces the challenges of massive data handling. Opt. Lett. 38, 1769–1771 (2013).
Wang, Y. et al. Localization events-based sample drift correction for localization microscopy with redundant cross-correlation algorithm. Opt. Express 22, 15982–15991 (2014).
Mandula, O., Šestak, I.Š., Heintzmann, R. & Williams, C.K. Localisation microscopy with quantum dots using non-negative matrix factorisation. Opt. Express 22, 24594–24605 (2014).
Rieger, B. & Stallinga, S. The lateral and axial localization uncertainty in super-resolution light microscopy. ChemPhysChem 15, 664–670 (2014).
Nieuwenhuizen, R.P.J. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).
Mukamel, E.A., Babcock, H. & Zhuang, X. Statistical deconvolution for superresolution fluorescence microscopy. Biophys. J. 102, 2391–2400 (2012).
Abraham, A.V., Ram, S., Chao, J., Ward, E.S. & Ober, R.J. Quantitative study of single molecule location estimation techniques. Opt. Express 17, 23352–23373 (2009).
Thompson, R.E., Larson, D.R. & Webb, W.W. Precise nanometer localization analysis for individual fluorescent probes. Biophys. J. 82, 2775–2783 (2002).
Ober, R.J., Ram, S. & Ward, E.S. Localization accuracy in single-molecule microscopy. Biophys. J. 86, 1185–1200 (2004).
Holden, S.J. et al. High throughput 3D super-resolution microscopy reveals Caulobacter crescentus in vivo Z-ring organization. Proc. Natl. Acad. Sci. USA 111, 4566–4571 (2014).
Baddeley, D., Cannell, M.B. & Soeller, C. Visualization of localization microscopy data. Microsc. Microanal. 16, 64–72 (2010).
Babcock, H., Sigal, Y. & Zhuang, X. A high-density 3D localization algorithm for stochastic optical reconstruction microscopy. Opt. Nanoscopy 1, 1–10 (2012).
Li, Y., Ishitsuka, Y., Hedde, P.N. & Nienhaus, G.U. Fast and efficient molecule detection in localization-based super-resolution microscopy by parallel adaptive histogram equalization. ACS Nano 7, 5207–5214 (2013).
Zhu, L., Zhang, W., Elnatan, D. & Huang, B. Faster STORM using compressed sensing. Nat. Methods 9, 721–723 (2012).
Holden, S.J., Uphoff, S. & Kapanidis, A.N. D.A.O.S.T.O.R.M.: an algorithm for high-density super-resolution microscopy. Nat. Methods 8, 279–280 (2011).
Hoogendoorn, E. et al. in Focus on Microscopy (FOM2013) (Maastricht, the Netherlands, 2013).
Min, J. et al. FALCON: fast and unbiased reconstruction of high-density super-resolution microscopy data. Sci. Rep. 4, 4577 (2014).
Kim, K.S. et al. in Proceedings of the 10th International Conference on Sampling Theory and Applications (SAMPTA) (Bremen, Germany, 2013).
Grüll, F., Kirchgessner, M., Kaufmann, R., Hausmann, M. & Kebschull, U. in 2011 International Conference on Field Programmable Logic and Applications (FPL) 1–5 (2011).
Anthony, S.M. & Granick, S. Image analysis with rapid and accurate two-dimensional Gaussian fitting. Langmuir 25, 8152–8160 (2009).
Smith, C.S., Joseph, N., Rieger, B. & Lidke, K.A. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).
Brede, N. & Lakadamyali, M. GraspJ: an open source, real-time analysis package for super-resolution imaging. Opt. Nanoscopy 1, 11 (2012).
Babcock, H.P., Moffitt, J.R., Cao, Y. & Zhuang, X. Fast compressed sensing analysis for super-resolution imaging using L1-homotopy. Opt. Express 21, 28583–28596 (2013).
Starr, R., Stahlheber, S. & Small, A. Fast maximum likelihood algorithm for localization of fluorescent molecules. Opt. Lett. 37, 413–415 (2012).
Quan, T. et al. Ultra-fast, high-precision image analysis for localization-based super resolution microscopy. Opt. Express 18, 11867–11876 (2010).
Ma, H., Long, F., Zeng, S. & Huang, Z.-L. Fast and precise algorithm based on maximum radial symmetry for single molecule localization. Opt. Lett. 37, 2481–2483 (2012).
Niu, L. & Yu, J. Investigating intracellular dynamics of FtsZ cytoskeleton with photoactivation single-molecule tracking. Biophys. J. 95, 2009–2016 (2008).
Shtengel, G. et al. Interferometric fluorescent super-resolution microscopy resolves 3D cellular ultrastructure. Proc. Natl. Acad. Sci. USA 106, 3125–3130 (2009).
Henriques, R. et al. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat. Methods 7, 339–340 (2010).
Parthasarathy, R. Rapid, accurate particle tracking by calculation of radial symmetry centers. Nat. Methods 9, 724–726 (2012).
Boulanger, J. et al. Patch-based non-local functional for denoising fluorescence microscopy image sequences. IEEE Trans. Med. Imaging 29, 29 (2010).
Andersson, S.B. Localization of a fluorescent source without numerical fitting. Opt. Express 16, 18714–18724 (2008).
Kechkar, A., Nair, D., Heilemann, M., Choquet, D. & Sibarita, J.-B. Real-time analysis and visualization for single-molecule based super-resolution microscopy. PLoS ONE 8, e62918 (2013).
Watanabe, S., Bennett, K., Takahashi, T. & Takeshima, T. in Focus on Microscopy (FOM2013) (Maastricht, the Netherlands, 2013).
Hinterdorfer, P. & Oijen, A.V. The Handbook of Single-Molecule Biophysics (Springer, 2009).
Huang, F., Schwartz, S.L., Byars, J.M. & Lidke, K.A. Simultaneous multiple-emitter fitting for single molecule super-resolution imaging. Biomed. Opt. Express 2, 1377–1393 (2011).
Wang, Y., Quan, T., Zeng, S. & Huang, Z.-L. PALMER: a method capable of parallel localization of multiple emitters for high-density localization microscopy. Opt. Express 20, 16039–16049 (2012).
Babcock, H., Sigal, Y.M. & Zhuang, X. A high-density 3D localization algorithm for stochastic optical reconstruction microscopy. Opt. Nanoscopy 1, 6 (2012).
Cox, S. et al. Bayesian localization microscopy reveals nanoscale podosome dynamics. Nat. Methods 9, 195–200 (2012).
Janesick, J.R. Photon Transfer (SPIE Publications, 2007).
Huang, B., Wang, W., Bates, M. & Zhuang, X. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319, 810–813 (2008).
Pavani, S.R.P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. Proc. Natl. Acad. Sci. USA 106, 2995–2999 (2009).
Juette, M.F. et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples. Nat. Methods 5, 527–529 (2008).
Ram, S., Prabhat, P., Ward, E.S. & Ober, R.J. Improved single particle localization accuracy with dual objective multifocal plane microscopy. Opt. Express 17, 6881–6898 (2009).
Mortensen, K.I., Churchman, L.S., Spudich, J.A. & Flyvbjerg, H. Optimized localization analysis for single-molecule tracking and super-resolution microscopy. Nat. Methods 7, 377–381 (2010).
Bennett, K., Takahashi, T., Sage, D. & Huang, Z. in Fourth Single Molecule Localisation Microscopy Symposium (SMLMS′14) (London, UK, 2014).
Huang, F. et al. Video-rate nanoscopy using sCMOS camera-specific single-molecule localization algorithms. Nat. Methods 10, 653–658 (2013).
Deschout, H. et al. Precisely and accurately localizing single emitters in fluorescence microscopy. Nat. Methods 11, 253–266 (2014).
Banterle, N., Bui, K.H., Lemke, E.A. & Beck, M. Fourier ring correlation as a resolution criterion for super-resolution microscopy. J. Struct. Biol. 183, 363–367 (2013).
Rosenthal, P.B. & Henderson, R. Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy. J. Mol. Biol. 333, 721–745 (2003).
Carpenter, A.E., Kamentsky, L. & Eliceiri, K.W. A call for bioimaging software usability. Nat. Methods 9, 666–670 (2012).
We thank N. Olivier for providing the experimental data and R. Nieuwenhuizen for his technical assistance in running the Fourier ring correlation. We thank also P. Thévenaz for critical reading and for his assistance in writing the manuscript. We thank the participants in the ISBI 2013 localization microscopy challenge: S. Anthony, S. Andersson, T. Ashley, D. Baddeley, K. Bennett, J. Boulanger, N. Brede, L. Dai, L. Fiaschi, F. Gruell, G. Hagen, R. Henriques, A. Herbert, S. Holden, E. Hoogendoorn, B. Huang, Z.-L. Huang, A. Kechkar, K. Kim, M. Kirchgessner, U. Koethe, P. Krizek, M. Lakadamyali, Y. Li, K. Lidke, R. McGorty, L. Muresan, R. Parthasarathy, B. Rieger, H. Rouault, M. Sauer, J.-B. Sibarita, I. Smal, A. Small, S. Stahlheber, Y. Tang, Y. Wang, S. Watanabe, S. Wolter, J.C. Ye and C. Zimmer. This work was supported by the Biomedical Imaging Group, the School of Engineering at the Ecole Polytechnique Fédérale de Lausanne, the European Research Council (ERC) FUN-SP project (267439), the ERC Starting Grant PALMassembly (243016) and the Eurobioimaging Project (WP11).
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Comparative results for detection rate and accurary for the long sequence datasets.
Plot the detection rate (Jaccard) vs. the accuracy of every software and plot the efficiency line for every dataset. The crosses indicates the results of the lower-bound software (CenterOfGravity) that we have developped.
Supplementary Figure 2 Comparative results for detection rate and accurary for the high-density datasets.
Plot the detection rate (Jaccard) vs. the accuracy of every software and plot the efficiency line for every dataset.
Plots of the 2 by 2 cross-correlation of the four quantitative criteria, detection rate (JAC), accuracy (ACC), image quality assessment (SNR), and image resolution (FRC). The results of all evaluated software are plotted in a different color for every dataset, the 3 long-sequence datasets (LS) and the high-density datasets (HD). The position of the crosses indicates the average per dataset and the length of its arms is equal to the standard deviation.
Supplementary Figures 1–3 and Supplementary Notes 1–3 (PDF 1570 kb)
Visual results, comparison to ground-truth (ZIP 60834 kb)
Numerical results, values, grades, and ranking (XLSX 177 kb)
Java software to compare two sets of localization (ZIP 78 kb)
HD1 100 frames of the contest dataset HD1 (MOV 599 kb)
HD2 100 frames of the contest dataset HD2 (MOV 818 kb)
HD3 100 frames of the contest dataset HD3 (MOV 1218 kb)
LS1 100 frames of the contest dataset LS1 (MOV 744 kb)
LS2 100 frames of the contest dataset LS2 (MOV 965 kb)
LS3 100 frames of the contest dataset LS3 (MOV 1895 kb)
About this article
Cite this article
Sage, D., Kirshner, H., Pengo, T. et al. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat Methods 12, 717–724 (2015). https://doi.org/10.1038/nmeth.3442
This article is cited by
BMC Bioinformatics (2023)
Nature Photonics (2023)
Nature Communications (2023)
Micro and Nano Systems Letters (2022)