Abstract
Methods that fuse multiple localization microscopy images of a single structure can improve signaltonoise ratio and resolution, but they generally suffer from template bias or sensitivity to registration errors. We present a templatefree particlefusion approach based on an alltoall registration that provides robustness against individual misregistrations and underlabeling. We achieved 3.3nm Fourier ring correlation (FRC) image resolution by fusing 383 DNA origami nanostructures with 80% labeling density, and 5.0nm resolution for structures with 30% labeling density.
Access options
Subscribe to Journal
Get full journal access for 1 year
$227.00
only $18.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
from$8.99
All prices are NET prices.
Data availability
Localization data are available at https://doi.org/10.4121/uuid:0d42a28ff62541a3ba7725e397685466.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
 1.
Klein, T., Proppert, S. & Sauer, M. Histochem. Cell Biol. 141, 561–575 (2014).
 2.
Nieuwenhuizen, R. P. J. et al. Nat. Methods 10, 557–562 (2013).
 3.
Löschberger, A. et al. J. Cell Sci. 125, 570–575 (2012).
 4.
Szymborska, A. et al. Science 341, 655–658 (2013).
 5.
Tang, G. et al. J. Struct. Biol. 157, 38–46 (2007).
 6.
Kudryashev, M., CastañoDíez, D. & Stahlberg, H. Comput. Struct. Biotechnol. J. 1, e201207002 (2012).
 7.
Van Engelenburg, S. B. et al. Science 343, 653–656 (2014).
 8.
Gray, R. D. M., Mercer, J. & Henriques, R. J. Vis. Exp. 2017, e55471 (2017).
 9.
Schnitzbauer, J., Strauss, M. T., Schlichthaerle, T., Schueder, F. & Jungmann, R. Nat. Protoc. 12, 1198–1228 (2017).
 10.
Salas, D. et al. Proc. Natl Acad. Sci. USA 114, 9273–9278 (2017).
 11.
Burgert, A., Letschert, S., Doose, S. & Sauer, M. Histochem. Cell Biol. 144, 123–131 (2015).
 12.
van de Linde, S., Wolter, S., Heilemann, M. & Sauer, M. J. Biotechnol. 149, 260–266 (2010).
 13.
FoxRoberts, P. et al. Nat. Commun. 8, 13558 (2017).
 14.
Broeken, J. et al. Methods Appl. Fluoresc. 3, 014003 (2015).
 15.
Henderson, R. Proc. Natl Acad. Sci. USA 110, 18037–18041 (2013).
 16.
Govindu, V. in Proc. 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition: CVPR 2004 (eds Davis, L. et al.) 684–691 (IEEE, Piscataway, NJ, 2004).
 17.
Jian, B. & Vemuri, B. C. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1633–1645 (2011).
 18.
Strauss, M. T., Schueder, F., Haas, D., Nickels, P. C. & Jungmann, R. Nat. Commun. 9, 1600 (2018).
 19.
Huang, F., Schwartz, S. L., Byars, J. M. & Lidke, K. A. Biomed. Opt. Express 2, 1377–1393 (2011).
 20.
Nieuwenhuizen, R. P. et al. PLoS One 10, e0127989 (2015).
 21.
Smith, C. S., Joseph, N., Rieger, B. & Lidke, K. A. Nat. Methods 7, 373–375 (2010).
 22.
Richardson, S. & Green, P. J. J. R. Stat. Soc. Series B Stat. Methodol. 59, 731–792 (1997).
 23.
Ye, K. & Lim, L.H. SIAM J. Appl. Algebra Geometry 1, 507–535 (2017).
 24.
Hartley, R., Aftab, K. & Trumpf, J. in Proc. IEEE Conference on Computer Vision and Pattern Recognition 2011 3041–3048 (IEEE, Piscataway, NJ, 2011).
 25.
Govindu, V. M. in Computer Vision—ACCV 2006 (eds Narayanan, P. J., Nayar, S. K. & Shum, H. Y.) 457–466 (Springer, Berlin, 2006).
Acknowledgements
We thank A. Chatterjee for providing the code for transformation averaging. This work was supported by the European Research Council (Nano@cryo, grant no. 648580 to H.H. and B.R.; MolMap, grant no. 680241 to R.J.), the eScience Center (path finder grant 027016P04 to B.v.W. and B.R.), the NIH (grants 1R21EB019589 and P50GM085273 to K.A.L. and M.F.), the New Mexico Spatiotemporal Modeling Center (K.A.L. and M.F.), the International Max Planck Research School for Molecular and Cellular Life Sciences (IMPRSLS; to M.T.S.), the Max Planck Society (R.J.), the Max Planck Foundation (R.J.), the DFG (Emmy Noether Program; DFG JU 2957/11 to R.J.), the SFB 1032 (Nanoagents for the spatiotemporal control of molecular and cellular reactions; to R.J.) and the Center for Nanoscience (CeNS; R.J.). K.A.L. and M.F. acknowledge the UNM Center for Advanced Research Computing, supported in part by the National Science Foundation, for providing highperformance computing resources.
Author information
Affiliations
Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands
 Hamidreza Heydarian
 , Sjoerd Stallinga
 & Bernd Rieger
Department of Physics and Center for Nanoscience, Ludwig Maximilian University, Munich, Germany
 Florian Schueder
 , Maximilian T. Strauss
 & Ralf Jungmann
Max Planck Institute of Biochemistry, Martinsried, Germany
 Florian Schueder
 , Maximilian T. Strauss
 & Ralf Jungmann
Netherlands eScience Center, Amsterdam, the Netherlands
 Ben van Werkhoven
Department of Physics and Astronomy, University of New Mexico, Albuquerque, NM, USA
 Mohamadreza Fazel
 & Keith A. Lidke
Authors
Search for Hamidreza Heydarian in:
Search for Florian Schueder in:
Search for Maximilian T. Strauss in:
Search for Ben van Werkhoven in:
Search for Mohamadreza Fazel in:
Search for Keith A. Lidke in:
Search for Ralf Jungmann in:
Search for Sjoerd Stallinga in:
Search for Bernd Rieger in:
Contributions
S.S. and B.R. conceived the project. H.H. performed simulations. B.v.W. wrote GPU code. M.F. and K.A.L. provided multiemitter fitting data. M.T.S., F.S. and R.J. designed DNA origami and acquired images. H.H., S.S. and B.R. wrote the paper, and all authors commented on the paper.
Competing interests
The authors declare no competing interests.
Corresponding author
Correspondence to Bernd Rieger.
Integrated supplementary information
Supplementary Figure 1 Comparison of simulated 50% DOL samples with different average localization uncertainties with a sample from the 50% DOL experimental dataset.
(ac) Simulated samples with mean localization uncertainties of 1.2 nm, 1.8 nm and 2.3 nm respectively. (d) A sample from the experimental dataset with an average localization uncertainty of 0.8 nm. The comparison reveals that the sample from experimental dataset (d) is more similar to figure (c) in visual appearance with larger localization uncertainty rather than figure (a). This difference in the computed uncertainty for the experimental data is likely due to a residual drift after drift correction of about 12 nm.
Supplementary Figure 2 Particlefusion performance for 256 simulated TUD logos with different labeling densities.
The output of the algorithm at different steps of the particle fusion for (ac) Alltoall registration outputs. (df) Outlier removal. (gi) Bootstrapping registrations. The final reconstructions in (gi) are the results of fusing 256 logos with around 470,000, 230,000 and 145,000 localizations respectively. In all the reconstructions in each column, the number of localizations is the same. A comparison of the images (ai) and Fig. 2 indicates a good match between simulation and experiment. The final results in (h) and (i), however, have better contrast than those from experiment. This is attributed to the effects of residual drift and of false positive localizations, which are not taken into account in the simulations.
Supplementary Figure 3 Particlefusion performance for 256 simulated PAINT TUD logos with different photon counts and 100% DOL.
(ac) Groundtruth reconstructions for photon counts of 1000, 500 and 200 or equivalently average localization uncertainties of 5, 7 and 11 nm, respectively. (df) Our reconstructions (gi) Reconstructions using EMAN.2 with 115, 202 and 147 included particles and the minimum of three classes for the class averaging. While the quality of reconstructions is close to the groundtruth both for our method and EMAN.2, the docking sites are hardly resolved as the mean localization uncertainties for all datasets is larger or equal than the minimum binding site distance (5 nm for our datasets).
Supplementary Figure 4 Comparison of the fusion of experimental 80% DOL particles with localizations based on singleemitter and multiemitter fitting and with and without filtering out localizations that are too far away from any other localization.
(a) Reconstruction of single emitter fitted particles without filtering (1,017,559 localizations). (b) Reconstruction of filtered (with neighborhood parameter r = 0.015 pixels) single emitter fitted particles (788,875 localizations). (c) Reconstruction of multiemitter fitted particles without filtering (2,591,464 localizations). (d) Reconstruction of filtered (with neighborhood parameter r = 0.015 pixels) multiemitter fitted particles with filtering (548,091 localizations). Comparing figures (b) and (d) suggests that the combination of multiemitter fitting and filtering can yield an improved reconstruction with better resolvability of the binding sites, probably because of a reduction of false positive localizations.
Supplementary Figure 5 The effect of bleaching rate on particlefusion performance for 65% DOL on simulated STORMtype datasets with a photon count of 5,000 and 1,000 recorded frames.
(ac) Groundtruth fusion of 256 particles with bleaching rates corresponding to an average number of localizations per biding site equal to ~33, ~13 and ~7. (df) Our reconstructions. (gi) Reconstructions using EMAN.2 with 95, 51 and 53 included particles (minimum of three classes for the class averaging). A higher bleaching rate results in a lower number of localizations per sites and a less uniform distribution, which degrades the particle fusion performance. Our method outperforms EMAN.2 at higher bleaching rates.
Supplementary Figure 6 Comparison of particlefusion performance by our method and EMAN.2 on simulated STORM datasets as a function of DOL (bleaching rate corresponding to an average of ~33 localizations per binding site and 5,000 photons per localization event).
(ac) Groundtruth fusion of 256 particles (df) Our reconstructions. (gi) Reconstructions using EMAN.2 with 71, 90 and 128 included particles (minimum of three classes for the class averaging). With the chosen bleaching rate, the average number of localizations per particle for the three datasets is similar to the corresponding simulated PAINT data in Supplementary Fig. 2. While for 100% DOL the reconstruction of STORM data is as good as for PAINT and close to the groundtruth, successful reconstructions require a DOL of at least ~50%. Our method outperforms EMAN.2 for all degrees of underlabeling.
Supplementary Figure 7 The effect of the filtering setting on localizations that are too far away from any other localization on the final reconstruction for the experimental 80% DOL dataset.
(a) The fusion of 383 TUD logos filtered with the parameter r=3×.0075 pixel size resulted in 939,707 localizations. (b) The fusion with filter parameter r=.0075 pixel size resulted in 151,729 localizations. The figures show that decreasing the neighborhood parameter (r) will identify more localizations as false positives. While in figure (a) only 10% of the localizations are discarded, more than 75% of all localizations are identified as false positives in (b), which appears too much to be considered as a correct assessment of false positives.
Supplementary Figure 8 Histogram of the number of localizations per particle in 80%, 50% and 30% DOL datasets with fitted normal distributions.
The mean and variance of the normal distributions are as follows: 80% DOL: mean 2.66 × 10^{3}, variance 1.94 × 10^{5}; 50% DOL: mean 9.85 × 10^{2}, variance 3.24 × 10^{4}; 30% DOL: mean 4.53 × 10^{2}, variance 7.74 × 10^{3}. The variance exceeds the mean as the overall distribution is a convolution of the distribution of the number of localizations per binding site (which can be assumed to be Poissonian) and the distribution of active binding sites given the average DOL.
Supplementary Figure 9 Histograms of the distribution of the inconsistency between the estimated relative angles after Liealgebraic averaging and the initial relative angles from the alltoall registration for the experimental datasets.
The distribution typically is a mix of three contributions: a peak with width of a few to 20 deg around the correct angle (~0 deg), a peak around 180 deg, due to the closetosymmetric shape of the `TUD' logo, and a uniform background distribution. The fraction of erroneous pair registrations (the second and third component) increases with decreasing DOL, and is a significant fraction of the total number of pair registrations. This is the reason why the redundancy of the alltoall registration is a necessary ingredient of the particle fusion process.
Supplementary Figure 10 Effect of repeated bootstrapping registration step on 50% DOL simulated data.
(a) The bootstrapping output at the first iteration. (b) The output after the second iteration. Iterating the bootstrapping step can further improve the result until convergence. For 50% DOL, the registration already converges after the second iteration.
Supplementary Figure 11 Histogram of the overall registration error (rotation angle) for the simulated 100% DOL dataset, which included 256 TUD logos.
A normal distribution with a standard deviation of 0.9 degree (red curve) is shown for comparison. Since the length of the bounding box of the logo is 70 nm, this amount of error will maximally result in a displacement of ±0.55 nm at the edges of the logo. This is smaller than the distance of the binding sites (5 nm) and very close to the mean localization uncertainty. Consequently, the impact of the overall outlier removal is minimal on the final reconstruction as depicted in Supplementary Fig. 2g.
Supplementary Figure 12 The effect of averaging over a subset of pair registrations on the final reconstruction for the 80% DOL experimental data.
(a) The result of averaging over all 73153 elements of the alltoall registration matrix. (b) The result of averaging over 1200 (less than 2%) elements of the alltoall registration matrix. Both reconstructions have 788875 localizations as a result of fusing 383 TUD logos with 80% DOL. Below 2%, the reconstruction is still possible for this dataset, however, the final logo then becomes blurry especially at the edges where the effect of the registration error is severe (see Supplementary Video 9).
Supplementary Information
Supplementary Text and Figures
Supplementary Figures 1–12 and the Supplementary Note
Reporting Summary
Supplementary Video 1
Data fusion of 80% DOL filtered TUD logos. The onebyone addition of all 383 aligned 80% DOL filtered particles demonstrates how SNR is increased by fusion of more particles.
Supplementary Video 2
Samples from 80% DOL filtered TUD DNA origami nanostructures with the output of each step of the particlefusion pipeline. The video shows the contrast between the low SNR of initial raw particles and the final superparticle as the result of particle fusion.
Supplementary Video 3
Data fusion of 50% DOL filtered TUD logos. The onebyone addition of all 442 aligned 50% DOL filtered particles demonstrates how SNR is increased by fusion of more particles. Compare with Supplementary Video 1 for the effect of the different labeling density.
Supplementary Video 4
Samples from 50% DOL filtered TUD DNA origami nanostructures with the output of each step of the particlefusion pipeline. The video shows the contrast between the low SNR of initial raw particles and the final superparticle as the result of particle fusion.
Supplementary Video 5
Data fusion of 30% DOL filtered TUD logos. The onebyone addition of all 549 aligned 30% DOL filtered particles demonstrates how SNR can be increased by fusion of more particles. Compare with Supplementary Videos 1 and 3 for the effect of the different labeling density.
Supplementary Video 6
Samples from 30% DOL TUD DNA origami nanostructures with the output of each step of the particlefusion pipeline. The video shows the contrast between the low SNR of initial raw particles and the final superparticle as the result of particle fusion. Note that the characters of the TUD logo are totally unrecognizable in the initial particles.
Supplementary Video 7
Data fusion of 80% DOL unfiltered TUD logos. The video highlights the effect of leaving out filtering on false positive localizations and subsequently the final reconstruction; compare to Supplementary Video 1.
Supplementary Video 8
Samples from 80% DOL unfiltered TUD DNA origami nanostructures with the output of each step of the particlefusion pipeline. Compared with Supplementary Video 2, this video demonstrates the effect of leaving out filtering on false positive localizations on initial raw particles and the final reconstruction.
Supplementary Video 9
The effect of averaging a small subset of the elements of the alltoall registration matrix on the particlefusion output for 80% DOL dataset. For highly labeled datasets, averaging a small subset of registrations (video shows the range from 0.05% to 100%) can yield highquality reconstructions at low computational time. All reconstructions include all 383 TUD logos.
Supplementary Software
MATLAB scripts for templatefree 2D particle fusion in localization microscopy.
Rights and permissions
To obtain permission to reuse content from this article visit RightsLink.
About this article
Further reading

1.
Singleparticle analysis for fluorescence nanoscopy
Nature Methods (2018)