Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Quantitative evaluation of software packages for single-molecule localization microscopy

Nature Methods volume 12, pages 717–724 (2015)Cite this article

Subjects

Abstract

The quality of super-resolution images obtained by single-molecule localization microscopy (SMLM) depends largely on the software used to detect and accurately localize point sources. In this work, we focus on the computational aspects of super-resolution microscopy and present a comprehensive evaluation of localization software packages. Our philosophy is to evaluate each package as a whole, thus maintaining the integrity of the software. We prepared synthetic data that represent three-dimensional structures modeled after biological components, taking excitation parameters, noise sources, point-spread functions and pixelation into account. We then asked developers to run their software on our data; most responded favorably, allowing us to present a broad picture of the methods available. We evaluated their results using quantitative and user-interpretable criteria: detection rate, accuracy, quality of image reconstruction, resolution, software usability and computational resources. These metrics reflect the various tradeoffs of SMLM software packages and help users to choose the software that fits their needs.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Construction of the bio-inspired data.
Figure 2: Accuracy versus detection rate for each tested software.
Figure 3: Rendering of software results versus ground truth at various scales.
Figure 4: Illustration of an assessment report.

References

  1. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

    CAS  PubMed  Google Scholar 

  2. Rust, M.J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  3. Hess, S.T., Girirajan, T.P. & Mason, M.D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 91, 4258–4272 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  4. Balzarotti, F. & Stefani, F.D. Plasmonics meets far-field optical nanoscopy. ACS Nano 6, 4580–4584 (2012).

    CAS  PubMed  Google Scholar 

  5. Walder, R., Nelson, N. & Schwartz, D.K. Super-resolution surface mapping using the trajectories of molecular probes. Nat. Commun. 2, 515 (2011).

    PubMed  Google Scholar 

  6. Manley, S., Gunzenhäuser, J. & Olivier, N. A starter kit for point-localization super-resolution imaging. Curr. Opin. Chem. Biol. 15, 813–821 (2011).

    CAS  PubMed  Google Scholar 

  7. Schermelleh, L., Heintzmann, R. & Leonhardt, H. A guide to super-resolution fluorescence microscopy. J. Cell Biol. 190, 165–175 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  8. Sauer, M. Localization microscopy coming of age: from concepts to biological impact. J. Cell Sci. 126, 3505–3513 (2013).

    CAS  PubMed  Google Scholar 

  9. Moerner, W.E. New directions in single-molecule imaging and analysis. Proc. Natl. Acad. Sci. 104, 12596–12602 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  10. Small, A. & Stahlheber, S. Fluorophore localization algorithms for super-resolution microscopy. Nat. Methods 11, 267–279 (2014).

    CAS  PubMed  Google Scholar 

  11. Endesfelder, U. & Heilemann, M. Art and artifacts in single-molecule localization microscopy: beyond attractive images. Nat. Methods 11, 235–238 (2014).

    CAS  PubMed  Google Scholar 

  12. Wolter, S., Endesfelder, U., van de Linde, S., Heilemann, M. & Sauer, M. Measuring localization performance of super-resolution algorithms on very active samples. Opt. Express 19, 7020–7033 (2011).

    PubMed  Google Scholar 

  13. Kirshner, H., Aguet, F., Sage, D. & Unser, M. 3-D PSF fitting for fluorescence microscopy: implementation and localization application. J. Microsc. 249, 13–25 (2013).

    CAS  PubMed  Google Scholar 

  14. Köthe, U., Herrmannsdoerfer, F., Kats, I. & Hamprecht, F.A. SimpleSTORM: a fast, self-calibrating reconstruction algorithm for localization microscopy. Histochem. Cell Biol. 141, 613–627 (2014).

    PubMed  Google Scholar 

  15. Ovesný, M., Krizek, P., Borkovec, J., Svindrych, Z. & Hagen, G.M. ThunderSTORM: a comprehensive ImageJ plugin for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).

    PubMed  PubMed Central  Google Scholar 

  16. Ovesný, M., Krízek, P., Svindrych, Z. & Hagen, G.M. High density 3D localization microscopy using sparse support recovery. Opt. Express 22, 31263–31276 (2014).

    PubMed  Google Scholar 

  17. Ma, H., Kawai, H., Toda, E., Zeng, S. & Huang, Z.-L. Localization-based super-resolution microscopy with an sCMOS camera part III: camera embedded data processing significantly reduces the challenges of massive data handling. Opt. Lett. 38, 1769–1771 (2013).

    PubMed  Google Scholar 

  18. Wang, Y. et al. Localization events-based sample drift correction for localization microscopy with redundant cross-correlation algorithm. Opt. Express 22, 15982–15991 (2014).

    PubMed  PubMed Central  Google Scholar 

  19. Mandula, O., Šestak, I.Š., Heintzmann, R. & Williams, C.K. Localisation microscopy with quantum dots using non-negative matrix factorisation. Opt. Express 22, 24594–24605 (2014).

    PubMed  Google Scholar 

  20. Rieger, B. & Stallinga, S. The lateral and axial localization uncertainty in super-resolution light microscopy. ChemPhysChem 15, 664–670 (2014).

    CAS  PubMed  Google Scholar 

  21. Nieuwenhuizen, R.P.J. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  22. Mukamel, E.A., Babcock, H. & Zhuang, X. Statistical deconvolution for superresolution fluorescence microscopy. Biophys. J. 102, 2391–2400 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  23. Abraham, A.V., Ram, S., Chao, J., Ward, E.S. & Ober, R.J. Quantitative study of single molecule location estimation techniques. Opt. Express 17, 23352–23373 (2009).

    CAS  PubMed  Google Scholar 

  24. Thompson, R.E., Larson, D.R. & Webb, W.W. Precise nanometer localization analysis for individual fluorescent probes. Biophys. J. 82, 2775–2783 (2002).

    CAS  PubMed  PubMed Central  Google Scholar 

  25. Ober, R.J., Ram, S. & Ward, E.S. Localization accuracy in single-molecule microscopy. Biophys. J. 86, 1185–1200 (2004).

    CAS  PubMed  PubMed Central  Google Scholar 

  26. Holden, S.J. et al. High throughput 3D super-resolution microscopy reveals Caulobacter crescentus in vivo Z-ring organization. Proc. Natl. Acad. Sci. USA 111, 4566–4571 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  27. Baddeley, D., Cannell, M.B. & Soeller, C. Visualization of localization microscopy data. Microsc. Microanal. 16, 64–72 (2010).

    CAS  PubMed  Google Scholar 

  28. Babcock, H., Sigal, Y. & Zhuang, X. A high-density 3D localization algorithm for stochastic optical reconstruction microscopy. Opt. Nanoscopy 1, 1–10 (2012).

    Google Scholar 

  29. Li, Y., Ishitsuka, Y., Hedde, P.N. & Nienhaus, G.U. Fast and efficient molecule detection in localization-based super-resolution microscopy by parallel adaptive histogram equalization. ACS Nano 7, 5207–5214 (2013).

    CAS  PubMed  Google Scholar 

  30. Zhu, L., Zhang, W., Elnatan, D. & Huang, B. Faster STORM using compressed sensing. Nat. Methods 9, 721–723 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  31. Holden, S.J., Uphoff, S. & Kapanidis, A.N. D.A.O.S.T.O.R.M.: an algorithm for high-density super-resolution microscopy. Nat. Methods 8, 279–280 (2011).

    CAS  PubMed  Google Scholar 

  32. Hoogendoorn, E. et al. in Focus on Microscopy (FOM2013) (Maastricht, the Netherlands, 2013).

  33. Min, J. et al. FALCON: fast and unbiased reconstruction of high-density super-resolution microscopy data. Sci. Rep. 4, 4577 (2014).

    PubMed  PubMed Central  Google Scholar 

  34. Kim, K.S. et al. in Proceedings of the 10th International Conference on Sampling Theory and Applications (SAMPTA) (Bremen, Germany, 2013).

  35. Grüll, F., Kirchgessner, M., Kaufmann, R., Hausmann, M. & Kebschull, U. in 2011 International Conference on Field Programmable Logic and Applications (FPL) 1–5 (2011).

  36. Anthony, S.M. & Granick, S. Image analysis with rapid and accurate two-dimensional Gaussian fitting. Langmuir 25, 8152–8160 (2009).

    CAS  PubMed  Google Scholar 

  37. Smith, C.S., Joseph, N., Rieger, B. & Lidke, K.A. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods 7, 373–375 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  38. Brede, N. & Lakadamyali, M. GraspJ: an open source, real-time analysis package for super-resolution imaging. Opt. Nanoscopy 1, 11 (2012).

    Google Scholar 

  39. Babcock, H.P., Moffitt, J.R., Cao, Y. & Zhuang, X. Fast compressed sensing analysis for super-resolution imaging using L1-homotopy. Opt. Express 21, 28583–28596 (2013).

    PubMed  PubMed Central  Google Scholar 

  40. Starr, R., Stahlheber, S. & Small, A. Fast maximum likelihood algorithm for localization of fluorescent molecules. Opt. Lett. 37, 413–415 (2012).

    PubMed  Google Scholar 

  41. Quan, T. et al. Ultra-fast, high-precision image analysis for localization-based super resolution microscopy. Opt. Express 18, 11867–11876 (2010).

    PubMed  Google Scholar 

  42. Ma, H., Long, F., Zeng, S. & Huang, Z.-L. Fast and precise algorithm based on maximum radial symmetry for single molecule localization. Opt. Lett. 37, 2481–2483 (2012).

    PubMed  Google Scholar 

  43. Niu, L. & Yu, J. Investigating intracellular dynamics of FtsZ cytoskeleton with photoactivation single-molecule tracking. Biophys. J. 95, 2009–2016 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Shtengel, G. et al. Interferometric fluorescent super-resolution microscopy resolves 3D cellular ultrastructure. Proc. Natl. Acad. Sci. USA 106, 3125–3130 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Henriques, R. et al. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat. Methods 7, 339–340 (2010).

    CAS  PubMed  Google Scholar 

  46. Parthasarathy, R. Rapid, accurate particle tracking by calculation of radial symmetry centers. Nat. Methods 9, 724–726 (2012).

    CAS  PubMed  Google Scholar 

  47. Boulanger, J. et al. Patch-based non-local functional for denoising fluorescence microscopy image sequences. IEEE Trans. Med. Imaging 29, 29 (2010).

    Google Scholar 

  48. Andersson, S.B. Localization of a fluorescent source without numerical fitting. Opt. Express 16, 18714–18724 (2008).

    CAS  PubMed  Google Scholar 

  49. Kechkar, A., Nair, D., Heilemann, M., Choquet, D. & Sibarita, J.-B. Real-time analysis and visualization for single-molecule based super-resolution microscopy. PLoS ONE 8, e62918 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  50. Watanabe, S., Bennett, K., Takahashi, T. & Takeshima, T. in Focus on Microscopy (FOM2013) (Maastricht, the Netherlands, 2013).

  51. Hinterdorfer, P. & Oijen, A.V. The Handbook of Single-Molecule Biophysics (Springer, 2009).

  52. Huang, F., Schwartz, S.L., Byars, J.M. & Lidke, K.A. Simultaneous multiple-emitter fitting for single molecule super-resolution imaging. Biomed. Opt. Express 2, 1377–1393 (2011).

    PubMed  PubMed Central  Google Scholar 

  53. Wang, Y., Quan, T., Zeng, S. & Huang, Z.-L. PALMER: a method capable of parallel localization of multiple emitters for high-density localization microscopy. Opt. Express 20, 16039–16049 (2012).

    PubMed  Google Scholar 

  54. Babcock, H., Sigal, Y.M. & Zhuang, X. A high-density 3D localization algorithm for stochastic optical reconstruction microscopy. Opt. Nanoscopy 1, 6 (2012).

    Google Scholar 

  55. Cox, S. et al. Bayesian localization microscopy reveals nanoscale podosome dynamics. Nat. Methods 9, 195–200 (2012).

    CAS  Google Scholar 

  56. Janesick, J.R. Photon Transfer (SPIE Publications, 2007).

  57. Huang, B., Wang, W., Bates, M. & Zhuang, X. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319, 810–813 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  58. Pavani, S.R.P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. Proc. Natl. Acad. Sci. USA 106, 2995–2999 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  59. Juette, M.F. et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples. Nat. Methods 5, 527–529 (2008).

    CAS  PubMed  Google Scholar 

  60. Ram, S., Prabhat, P., Ward, E.S. & Ober, R.J. Improved single particle localization accuracy with dual objective multifocal plane microscopy. Opt. Express 17, 6881–6898 (2009).

    CAS  PubMed  Google Scholar 

  61. Mortensen, K.I., Churchman, L.S., Spudich, J.A. & Flyvbjerg, H. Optimized localization analysis for single-molecule tracking and super-resolution microscopy. Nat. Methods 7, 377–381 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  62. Bennett, K., Takahashi, T., Sage, D. & Huang, Z. in Fourth Single Molecule Localisation Microscopy Symposium (SMLMS′14) (London, UK, 2014).

  63. Huang, F. et al. Video-rate nanoscopy using sCMOS camera-specific single-molecule localization algorithms. Nat. Methods 10, 653–658 (2013).

    CAS  PubMed  PubMed Central  Google Scholar 

  64. Deschout, H. et al. Precisely and accurately localizing single emitters in fluorescence microscopy. Nat. Methods 11, 253–266 (2014).

    CAS  PubMed  Google Scholar 

  65. Banterle, N., Bui, K.H., Lemke, E.A. & Beck, M. Fourier ring correlation as a resolution criterion for super-resolution microscopy. J. Struct. Biol. 183, 363–367 (2013).

    CAS  PubMed  Google Scholar 

  66. Rosenthal, P.B. & Henderson, R. Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy. J. Mol. Biol. 333, 721–745 (2003).

    CAS  PubMed  Google Scholar 

  67. Carpenter, A.E., Kamentsky, L. & Eliceiri, K.W. A call for bioimaging software usability. Nat. Methods 9, 666–670 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank N. Olivier for providing the experimental data and R. Nieuwenhuizen for his technical assistance in running the Fourier ring correlation. We thank also P. Thévenaz for critical reading and for his assistance in writing the manuscript. We thank the participants in the ISBI 2013 localization microscopy challenge: S. Anthony, S. Andersson, T. Ashley, D. Baddeley, K. Bennett, J. Boulanger, N. Brede, L. Dai, L. Fiaschi, F. Gruell, G. Hagen, R. Henriques, A. Herbert, S. Holden, E. Hoogendoorn, B. Huang, Z.-L. Huang, A. Kechkar, K. Kim, M. Kirchgessner, U. Koethe, P. Krizek, M. Lakadamyali, Y. Li, K. Lidke, R. McGorty, L. Muresan, R. Parthasarathy, B. Rieger, H. Rouault, M. Sauer, J.-B. Sibarita, I. Smal, A. Small, S. Stahlheber, Y. Tang, Y. Wang, S. Watanabe, S. Wolter, J.C. Ye and C. Zimmer. This work was supported by the Biomedical Imaging Group, the School of Engineering at the Ecole Polytechnique Fédérale de Lausanne, the European Research Council (ERC) FUN-SP project (267439), the ERC Starting Grant PALMassembly (243016) and the Eurobioimaging Project (WP11).

Author information

Authors and Affiliations

  1. Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

    Daniel Sage, Hagai Kirshner & Michael Unser

  2. Center for Genomic Regulation, Barcelona, Spain

    Thomas Pengo

  3. Howard Hughes Medical Institute, University of California (UCSF), San Francisco, California, USA

    Nico Stuurman

  4. Department of Cellular and Molecular Pharmacology, UCSF, San Francisco, California, USA

    Nico Stuurman

  5. Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

    Junhong Min

  6. Laboratory of Experimental Biophysics, EPFL, Lausanne, Switzerland

    Suliana Manley

Authors
  1. Daniel Sage
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hagai Kirshner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Pengo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nico Stuurman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junhong Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Suliana Manley
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Unser
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

D.S., H.K., T.P., J.M. and N.S. conceived the project. D.S. developed the project and organized the challenge with contribution from all authors. D.S. and H.K. wrote the code for the simulated data and analyzed the results. S.M. and M.U. directed the project. D.S. and H.K. wrote the manuscript with input from all authors.

Corresponding author

Correspondence to Daniel Sage.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Comparative results for detection rate and accurary for the long sequence datasets.

Plot the detection rate (Jaccard) vs. the accuracy of every software and plot the efficiency line for every dataset. The crosses indicates the results of the lower-bound software (CenterOfGravity) that we have developped.

Supplementary Figure 2 Comparative results for detection rate and accurary for the high-density datasets.

Plot the detection rate (Jaccard) vs. the accuracy of every software and plot the efficiency line for every dataset.

Supplementary Figure 3 Correlation of the quantitative assessement criteria.

Plots of the 2 by 2 cross-correlation of the four quantitative criteria, detection rate (JAC), accuracy (ACC), image quality assessment (SNR), and image resolution (FRC). The results of all evaluated software are plotted in a different color for every dataset, the 3 long-sequence datasets (LS) and the high-density datasets (HD). The position of the crosses indicates the average per dataset and the length of its arms is equal to the standard deviation.

Supplementary information

Combined PDF

Supplementary Figures 1–3 and Supplementary Notes 1–3 (PDF 1570 kb)

Supplementary Data 1

Visual results, comparison to ground-truth (ZIP 60834 kb)

Supplementary Data 2

Numerical results, values, grades, and ranking (XLSX 177 kb)

Supplementary Software 1

Java software to compare two sets of localization (ZIP 78 kb)

Supplementary Video 1

HD1 100 frames of the contest dataset HD1 (MOV 599 kb)

Supplementary Video 2

HD2 100 frames of the contest dataset HD2 (MOV 818 kb)

Supplementary Video 3

HD3 100 frames of the contest dataset HD3 (MOV 1218 kb)

Supplementary Video 4

LS1 100 frames of the contest dataset LS1 (MOV 744 kb)

Supplementary Video 5

LS2 100 frames of the contest dataset LS2 (MOV 965 kb)

Supplementary Video 6

LS3 100 frames of the contest dataset LS3 (MOV 1895 kb)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sage, D., Kirshner, H., Pengo, T. et al. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat Methods 12, 717–724 (2015). https://doi.org/10.1038/nmeth.3442

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.3442

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing