sCMOS imagers are currently utilized (replacing EMCCD imagers) to increase the acquisition speed in super resolution localization microscopy. Single-photon avalanche diode (SPAD) imagers feature frame rates per bit depth comparable to or higher than sCMOS imagers, while generating microsecond 1-bit-frames without readout noise, thus paving the way to in-depth time-resolved image analysis. High timing resolution can also be exploited to explore fluorescent dye blinking and other photophysical properties, which can be used for dye optimization. We present the methodology for the blinking analysis of fluorescent dyes on experimental data. Furthermore, the recent use of microlenses has enabled a substantial increase of SPAD imager overall sensitivity (12-fold in our case), reaching satisfactory values for sensitivity-critical applications. This has allowed us to record the first super resolution localization microscopy results obtained with a SPAD imager, with a localization uncertainty of 20 nm and a resolution of 80 nm.
Abbe and Rayleigh defined and formulated the diffraction-limited optical resolution1. Several techniques were developed to overcome Abbe’s limit2, such as ground-state depletion and single-molecule return (GSDIM3, also known as dSTORM4), a single molecule super resolution localization microscopy (SMLM) technique based on the on and off states of fluorescence molecules. Fluorophores bonded to biological structures are embedded in buffers, inducing oxidation and reduction and leading to stochastic blinking. The centers of the point spread functions (PSFs) of single fluorophores are determined in one imaging frame with a sparse number of fluorophores that are emitting photons. By subsequently imaging these emissions, a super resolved pointillistic image is formed3. Similar techniques relying on a sparse number of emitting fluorophores per frame, such as photo-activated localization microscopy (PALM, using laser controlled activation of emitting fluorophores) or stochastic optical reconstruction microscopy (STORM), can also be used.
Highly sensitive EMCCD (electron multiplying charge coupled device) imagers were initially used in SMLM instrumentation. Their high sensitivity enhanced the uniformity of the PSFs, increasing the accuracy of PSF centers. Because of limited speed and the presence of excess noise in EMCCDs, a number of researchers started to use faster sCMOS (scientific complementary metal oxide semiconductor) imagers, eventually enabling video rate localization nanoscopy5. In this case, faster fluorophores were required5,6, and faster instrumentation became critical for a dye’s photophysical characterization and optimization7,8,9. To date, the maximal reported frame rate available to SMLM is still limited to 3 000 fps6. Single-photon avalanche diode (SPAD) imagers provide even faster frame rates at zero readout noise and could represent an appealing solution for widefield analysis of blinking dyes.
SPAD imagers are arrays of SPADs10 that are known for high timing resolution and are thus suitable for time resolved applications such as fluorescence lifetime11. These imagers can also be used to count individual photons, so as to form gray scale images at high frame rates and with high photon response uniformity12,13. When compared to charge accumulating imagers like EMCCD or sCMOS, one of the main advantages of SPAD imagers is an early digitalization of photon counts within the pixel13, at the cost of a somewhat lower fill factor – unless recovered by means of microlenses, for instance – and a lower photon detection probability (PDP). A theoretical analysis comparing the three imager types for SMLM applications, including simulations, was presented by Krishnaswami et al.14.
This article reports on the use of microlens-enabled photon counting SPAD imagers in SMLM. We firstly describe an implementation of a photon counting SPAD imager, called SwissSPAD12,15 and the optical setup used for the experiments. We then analyze the optimal frame time in SMLM and present the methodology to analyze fluorophore blinking at an unprecedented time resolution of 6.4 μs. Finally, we show the first SMLM images recorded with a SPAD imager.
A SPAD is a pn junction reverse-biased above the breakdown voltage VBD by a voltage known as excess bias. Such a device is said to operate in Geiger mode. The sensitivity of a SPAD, quantified by PDP, and its noise, quantified by dark count rate (DCR), depend on the actual excess bias, albeit in different ways. Since photoresponse in a SPAD is influenced by VBD, a uniform breakdown voltage across the die ensures low photoresponse nonuniformity (PRNU), and thus high PDP uniformity. Usually PRNU is very low in SPAD arrays due to low variability of VBD in modern CMOS processes13. Because of the early digitalization, the SPAD noise sources are limited to DCR, afterpulsing and crosstalk. SPAD imagers can have very fast frame rates and programmable bit depth resolution without compromising their performance.
SwissSPAD is an array of 512 × 128 SPAD pixels; Fig. 1a shows a micrograph of the sensor and a detail of the pixels. The pixel comprises a 1-bit memory and a mechanism to quench and recharge the SPAD, thereby achieving global gating over the sensor with deep-subnanosecond accuracy and sub-5 ns width. The 1-bit memory is used to hold the photon hit information, which is subsequently read out using a row select transistor.
Readout is performed using a rolling shutter system with a 6.4 μs frame period. 1-bit frames are sent to a data acquisition board including a field-programmable gate array (FPGA), where multiple frames can be accumulated to form gray level images of programmable bit depth. For an 8-bit image with 255 grayscale values, 255 1-bit frames are needed, leading to a frame rate of 613 fps. For 1 000 fps, 156 1-bit frames are added. When ignoring the nonlinear effect of the imager response13, it implies a maximal count of 156 photons per pixel within 1 ms. The 1-bit frame period also determines an upper limit to the effective dead time of the SPAD, since recharge of the 1-bit memory is performed only at the beginning of each frame. If more than one photon is detected by the SPAD within a frame time, it will be interpreted as a single count, introducing a nonlinear response at higher photon rates. To a certain degree, this effect can be corrected for13.
In SwissSPAD, the fill factor, i.e. the ratio between the photosensitive area and the overall area of the pixel, is limited to 5%. It can, however, be increased by building a microlens array on the imager to concentrate light on the photosensitive area16. By using microlenses, we achieved a concentration factor of 12 for partially collimated light, leading to an effective fill factor of 60% and a peak photon detection efficiency (PDE) of 20% at 450 nm13 for an with a f/# of 16. The median DCR is about 200 cps at room temperature with 2% of noisy pixels in the array12,13.
GSDIM experimental setup
We used a dual port Leica SR GSD super resolution microscope (Leica Microsystems, Wetzlar, Germany), and initially placed an Andor iXon3 897 BV EMCCD (Andor Technology, Belfast, UK) with a pixel pitch of 16 μm on one port and SwissSPAD with a pixel pitch of 24 μm on the other. We later used a pco.edge 4.2 sCMOS imager (PCO, Kelheim, Germany) in combination with SwissSPAD. The emission light can be directed to either the EMCCD/sCMOS or SwissSPAD. The objective and the tube lens (HCX PL APO 160 × /1.43 Oil CORR GSD) magnifications are 100 × and 1.6 × , respectively. When installing sCMOS, we changed to a 160 × objective with a 1 × tubelens. Yielding a total magnification of 160, the effective EMCCD and SwissSPAD pixel sizes are therefore 100 nm and 150 nm, respectively. When using sCMOS, a demagnificator was used to yield 100 nm pixel size.
For SMLM measurements, we used human bone osteosarcoma epithelial cells (U2OS) stained for microtubuli with Alexa 647, in Vectashield (Vector Laboratories, Burlingame, USA) embedding resin or MEA buffer17, as well as fibroblast cells stained for actin with Alexa 647 together with an OxEA buffer18, and GATTAquant nanorulers19,20 (GATTAquant, Braunschweig, Germany).
We first recorded fluorescence intensity images to compare the imager sensitivity. The ratio between the collected numbers of photons - NSwissSPAD/NEMCCD - was found to be 12%13. This ratio is lower than the ratio between the peak sensitivities of the two imagers because the emission spectra of dyes are more appropriate for EMCCDs. Figure 1b shows a SwissSPAD multicolor fluorescence image of a Convallaria rhizome (Lily-of-the-Valley) specimen stained with Safranin and Fast Green, having peak excitation wavelengths of 530 nm and 620 nm, respectively. SwissSPAD has an optimum wavelength range from 400 to 600 nm, with a peak at 450 nm, and considerable sensitivity until 750 nm12. The EMCCD has its peak between 500 and 700 nm, with a range from 350 to 900 nm. Two consecutive images were taken using different filters and combined to form a single multicolor fluorescence image. It should be noted that Fig. 1b and a 320 × 24021 array represent the state-of-the-art in SPAD image quality when compared with images taken with other recent SPAD imagers featuring lower resolution, uniformity and/or fill factor22,23,24. The 320 × 240 SPAD imager features comparable resolution and 26.8% native fill factor, but also asymmetrical pixels and a maximum 1-bit frame rate of kiloframes per second21.
Optimal frame duration
When increasing the speed of the fluorophore blinking to achieve fast SMLM acquisitions, it is not a priori clear what frame duration should be used.
Figure 2 shows a fluorophore photon count rate Irate in the presence of a background Brate, corresponding to the background accumulated by all pixels, which are contributing to Irate. Note that more than one pixel contributes to the PSF. The blinking time is assumed to be TON and is here assumed to be entirely embedded in a frame. In the general case, the SNR as a function of Tframe is:
For a constant TON value, the maximum SNR is reached for Tframe = TON. This changes when TON is an exponential random variable with a decay constant τON, which is the case for the emission time of the fluorophores. First, the estimated E(SNR) with the above assumption for the probability density function of TON, , is calculated as:
An example of SNR as a function of Tframe when TON is an exponential random variable with a typical decay constant τON = 10 ms is shown in Fig. 3.
The optimal Tframe is found solving:
The derivation does not yield an explicit Tframe. The optimal Tframe is a function of Irate and Brate and can be seen in Fig. 3c. If Brate = 0, Tframe should be chosen so that Tframe > max(TON), however, since TON is a random variable, the optimal Tframe should be infinite. On the other hand, for Brate > Irate, Tframe should be chosen so as to minimize the noise effect, thus Tframe = τON.
SwissSPAD achieves a short frame time without negative effects on the imager performance, such as clock induced charge noise and readout noise. The fast frame time can be used to analyze the optimal Tframe. To the best of our knowledge, this analysis has not been carried out before, also due to the fact that the performance of charge accumulating imagers changes as a function of readout speed. Figure 4a and b show ThunderSTORM25 super resolution results of simulated data with τON = 10 ms. The background per pixel is constantly increasing when increasing Tframe. The fluorophore intensity is also increasing but starts saturating after Tframe > τON. The number of localizations decreases rapidly until Tframe = τON (Fig. 4a), while the uncertainty curve reaches its minimum at Tframe > τON (Fig. 4b). Figure 4b is in good agreement with Fig. 3b. Figure 4c and d show experimental SwissSPAD data. The experimental curves resemble the simulated curves, although there is an additional effect of rejecting dim emissions (and partial merging of localizations) causing an artificial uncertainty decrease. Merging close localizations in consecutive frames had insignificant influence on the results presented in Fig. 4, since ThunderSTORM averages the positions, imaged sizes, and backgrounds, and sums the intensity. Adding photons from multiple frames and then calculating the position and uncertainty could however yield a decrease in uncertainty26, but it requires intense processing, currently not implemented in super resolution software.
Figure 5 shows simulated SMLM results on emissions in a circle using three different Tframe, where the theoretical optimum is between 18 and 19 ms. Short frame times result in larger number of localizations (also faulty ones) with high localization uncertainty, while long frame times result in localization rejection and worse localization uncertainty.
When using higher power intensities to reach fast emission times, one of the most important procedures is to characterize the dye’s blinking to set the optimal operating point. An example of blinking is shown in Fig. 6a. On states that appear continuous could additionally exhibit fast blinking (Fig. 6b) not clearly observable with 1–10 ms resolution time frame. This fast blinking can however be detected when using Poisson statistics. To estimate the on and off averages as well as the corresponding emission and background bands (see Fig. 6a), we first used the whole signal sample length to estimate A, where:
and thus λ1, i.e. the average value of the background noise as random variable with Poisson distribution, where:
Note that the whole signal includes the background and the emission signal, and that min(signal) and max(signal) are easily found. The emission sample length, i.e. the total length of the signal above max(background), is then used to find B, where:
and thus λ2, i.e. the average value of the emission signal and background noise as random variable with Poisson distribution.
In the presence of additional fast blinking (due to a triplet state or additional dark states), the photon response will indeed not follow Poisson statistics with a constant emission rate, as it is clear from Fig. 6b. The emission changes in time fall clearly outside the estimated Poisson band.
To analyze such a fast blinking phenomena, shorter accumulation times must be used. However, due to the decrease of accumulation time, the SNR is reduced as well, leading to overlapping of the signal and background bands. We therefore used two methods for molecule blinking analysis: a thresholding method, which is suitable for signals with two distinguishable signal intensities (molecule on and molecule off), where the bands are well separated, and an autocorrelation method, which can be used for continuous signals, in cases where the SNR is not high enough to distinguish clearly between the on and off states27.
After the ThunderSTORM25 analysis, we extracted the single molecule photon response for the U2OS sample with MEA buffer. We used 5 × 5 pixels to cover the whole molecule emission PSF. Examples of measured blinking with different accumulation times, with estimated emission and background bands, are shown in Fig. 6c–f.
We expect the emissions to have two blinking components: slow and fast. The number of emitting fluorophores per frame is sparse due to the slow component. The fast component is not an application necessity, and is induced by the triplet state or other dark states. τOFF is defined as the time between two emissions of the same molecule. We recorded images with three different laser intensities and firstly used the thresholding method to extract τON and τOFF of the slow blinking component. Fitted monoexponential distributions for the on time have τON,thresh = 6.1 ms, τON,thresh = 3.1 ms and τON,thresh = 2.0 ms for laser powers of 5.7 kW/cm2, 8.5 kW/cm2 and 11.3 kW/cm2 respectively. τON,thresh decreases with increase of laser intensities. Although off times over a wide range of laser intensities should be analyzed as a sum of three exponential distributions6, we fitted off data monoexponentially as an indication of changes over different laser intensities. The distributions for the off time have τOFF,thresh = 13.4 ms, τOFF,thresh = 5.7 ms and τOFF,thresh = 3.3 ms for laser powers of 5.7 kW/cm2, 8.5 kW/cm2 and 11.3 kW/cm2 respectively. The threshold was set at the upper noise bound because of possible multiple emissions with different photon intensities and additional fast blinking (see Fig. 6). Single outliers over the upper noise bound were rejected. A fast sCMOS camera can extract τON,thresh and τOFF,thresh in the range of 1–10 ms, but cannot investigate the additional fast blinking in the μs range, whereas a SPAD imager such as SwissSPAD can.
The additional fast blinking can lower the overall molecule emission. When compared to our previous work28 where we used the thresholding to estimate the beginning and end of course emission, here we used the information of the beginning and end of the molecule emission from 8-bit video ThunderSTORM analysis. The 8-bit data was formed by accumulating the original 1-bit data in the PC. We than returned to the 1-bit data with 6.4 μs time resolution and used autocorrelation to explore if additional fast blinking is present. Note that no information was lost in the process. Since we expect the blinking to be exponentially distributed, the autocorrelation curve is fitted to extract a decay constant of the exponential autocorrelation curve τauto. The drawback of using 1-bit data is that the emission and noise bands (as shown in Fig. 6) are not easily distinguishable and τauto will be a combination of the fast on and off lifetimes (if present), i.e.:
In our U2OS sample with microtubuli stained with Alexa 647 and with MEA buffer, we measured 68%, 74% and 70% emissions with 0 < τauto < 1 ms. The rest of the blinks feature a τauto = −∞, indicating no blinking (autocorrelation is flat), and a minor number of outlying τauto because of the low number of emission photons. The averages τauto of this fast blinking were 58 μs, 48 μs and 54 μs for laser intensities of 5.7 kW/cm2, 8.5 kW/cm2 and 11.3 kW/cm2 respectively (Fig. 7a,b and c). The U2OS sample stained with Alexa 647, but with a Vectashield embedding resin, had 46%/53 μs (Fig. 7d), 55%/58 μs and 41%/57 μs for laser intensities of 5.7 kW/cm2, 8.5 kW/cm2and 11.3 kW/cm2 respectively. A Fibroblast sample stained with Alexa 647 in an OxEA buffer18 had an average τauto of 28 μs, where 20% emissions had 0 < τauto < 1 ms (Fig. 7e). When using Atto 647 in an OxEA buffer, we detected 27% emissions with 0 < τauto < 1 ms, with an average of 49 μs (Fig. 8f).
Data shows that laser power has a small effect on the fast blinking. On the contrary, buffers change blinking parameters significantly, indicating that the fast blinking is caused by the triplet state. The two different dyes also have a different blinking behavior. The extracted τauto (Fig. 7) - with an unprecedented time resolution range down to 6.4 μs - can be used for the photophysical analysis of a dye and its optimization. Timing parameters could also be used to estimate pH or concentration values7,29.
Super resolution images
We compared SwissSPAD, sCMOS and EMCCD super resolution measurements. When comparing SwissSPAD and EMCCD/sCMOS we alternately recorded frames with the two imagers. To compare SwissSPAD and sCMOS, we also used GATTAquant PAINT 80 R nanorulers, where three emitters in each nanoruler are separated by 80 nm20. The SwissSPAD has a PDE of around 9% at the emission wavelength of Alexa Fluor 647 and Atto 655.
The SwissSPAD video was first pre-processed in MATLAB to correct for the sensor’s nonlinear photon response and DCR, on a pixel-by-pixel basis, employing the following correction scheme13:
where C represents the corrected count rate, CM the measured count rate, and Treadout = 6.4 μs the dead time of the pixel.
The recorded 1-bit, 6.4 μs frames were then binned to form 10 ms frames, background was subtracted30 and images were analyzed with ThunderSTORM25. Figure 8a shows a super resolved image of actin in a fibroblast sample, stained with Alexa Fluor 647, used with an OxEA buffer. Figure 8b shows the corresponding widefield image. Figure 8c–f show GATTAquant PAINT 80 R nanoruler data of SwissSPAD and sCMOS, respectively. Images where reconstructed using 5 000 10 ms frames. When zoomed, one can note the resolvability of 80 nm distances between single emitters, where sCMOS has a 2–3 times finer resolution. Gyongy et al. showed the benefits of aggregating short frames with a single GATTAquant nanoruler imaged with a SPAD imager26,31, but without super resolution images.
Figure 8g–i show microtubuli from a fibroblast sample, which was stained with Alexa Fluor 647, and prepared in OxEA buffer. The sCMOS and SwissSPAD images contain 40 000 localizations. Although the effect is minimized by preprocessing (see Equation 8), some of the noisy pixels distort localizations. Highly noisy pixels, known as screamers, constitute 2% of the overall pixel population13; interpolation is used to minimize localization distortions. The interpolation of hot pixels caused loss of data and a localization uncertainty increase. This effect was verified by the software, resulting in a bias of the localization position (away from the true position).
Figure 8j–l show a SwissSPAD super resolution image of microtubuli in an U2OS cell stained with Alexa Fluor 647, in Vectashield. It was obtained during a 70 second exposure and contains 90 000 localizations.
Thanks to the implementation of microlenses that boosted the SwissSPAD PDE, we obtained the first SMLM images recorded with a SPAD imager. The sCMOS typically collected 800 photons with a localization uncertainty of 10 nm (for Fig. 8g), while SwissSPAD had 100 photons collected with 20 nm uncertainty (for Fig. 8h). For the measurements with EMCCD, the typical estimated number of collected photons was 1800, and the typical estimated localization uncertainty about 15 nm (Fig. 8j), while SwissSPAD had 200 photons collected and 30 nm uncertainty (Fig. 8k). The emphasis should be placed on the comparison between the SPAD and one of the two other imagers. A direct comparison between the sCMOS and EMCCD images and uncertainty results is not completely fair since the sCMOS imaged samples with OxEA, and the EMCDD imaged samples with Vectashield. The localization uncertainty is estimated using ThunderSTORM, employing the Thompson et al. formula32 for sCMOS and SPAD, and Quan et al. formula33 for EMCCD (as to include the multiplication noise). It is worth mentioning that, although the EMCCD collected 10 times more photons, the localization uncertainty is only about two times better. Ten times more photons should yield √10 = 3.16 better localization uncertainty, but the excess noise lowers this by a factor of √26, resulting in √5 = 2.23 better localization uncertainty. CMOS SPAD structures reaching a PDP of 40% between 440 and 620 nm have been published34, but not yet implemented as SPAD imagers. A theoretical analysis did actually show that SPAD imagers with the same sensitivity as EMCCD and sCMOS imagers will feature superior localization accuracy because of absence of excess and readout noise14, assuming that the dark noise uniformity is similar to conventional CMOS imagers.
SPAD imagers have by some been regarded as unsuitable for applications where sensitivity is critical. This belief was driven by the lack of high fill factor sensors. With the introduction of SPAD imagers with improved fill factor, we could demonstrate that super resolution localization microscopy can exploit the high timing resolution provided by this type of imagers. We showed the first super resolution localization microscopy results obtained with a SPAD imager, with an estimated localization uncertainty of 20 nm and resolution better than 80 nm. We investigated the optimal frame time and concluded that it is longer than the average blinking time, and dependent on the emission intensity and background intensity ratio.
We also presented the instrumentation and methodology for a systematic widefield blinking analysis. We believe that the characterization of very fast blinking is critical for future developments in super resolution, with the goal of high acquisition speeds for the best possible localization uncertainty. Finally, we presented data where the MEA buffer allowed for 70% of emission with additional fast blinking in the μs range, whereas OxEA allowed for 20%. To the best of our knowledge, this is the first comprehensive widefield analysis of blinking with microsecond timing resolution and the first performed on a SPAD imager.
State-of-the-art SPAD imagers have a competitive advantage when combining spatial and temporal resolution, but still don’t reach the same spatial resolution as EMCCD and sCMOS due to the lower PDE. Further developments of the SPAD imagers will yield a higher PDE and a smaller pixel pitch, which is expected to further increase the noise uniformity and lower the number of hot pixels.
How to cite this article: Antolovic, I. M. et al. SPAD imagers for super resolution localization microscopy enable analysis of fast fluorophore blinking. Sci. Rep. 7, 44108; doi: 10.1038/srep44108 (2017).
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We are grateful to Daisy Picavet and Emilie Desclos for the sample preparation and microscope handling. We thank the Dutch Technology Foundation STW and Leica Microsystems for funding this research. I.M.A. is supported by STW project 12153. Work at EPFL was supported, in part, by the Swiss National Science Foundation and by the SwissEx project. The microlenses on the SwissSPAD chip were fabricated by CSEM Muttenz, Switzerland. The authors thank Xilinx Inc. for the generous donation of Virtex™ 4 FPGAs used in this work.
This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/