To increase the temporal resolution and maximal imaging time of super-resolution (SR) microscopy, we have developed a deconvolution algorithm for structured illumination microscopy based on Hessian matrixes (Hessian-SIM). It uses the continuity of biological structures in multiple dimensions as a priori knowledge to guide image reconstruction and attains artifact-minimized SR images with less than 10% of the photon dose used by conventional SIM while substantially outperforming current algorithms at low signal intensities. Hessian-SIM enables rapid imaging of moving vesicles or loops in the endoplasmic reticulum without motion artifacts and with a spatiotemporal resolution of 88 nm and 188 Hz. Its high sensitivity allows the use of sub-millisecond excitation pulses followed by dark recovery times to reduce photobleaching of fluorescent proteins, enabling hour-long time-lapse SR imaging of actin filaments in live cells. Finally, we observed the structural dynamics of mitochondrial cristae and structures that, to our knowledge, have not been observed previously, such as enlarged fusion pores during vesicle exocytosis.
Recent years have witnessed an exponential growth in the use of super-resolution (SR) fluorescence microscopy, providing new mechanistic insights into many biological processes1,2,3,4. However, because the light dose required for SR imaging is usually markedly higher than that used in conventional microscopy, it has long been recognized that live-cell SR microscopy techniques are severely limited by photobleaching and phototoxicity5. Therefore, for cell biology studies, ultrafast SR microscopy techniques and long-term live-cell SR microscopy strategies remain to be developed.
The development of brighter and more photostable dyes is often regarded as a crucial step toward long-term SR imaging6,7. However, the fluorescent tagging of cellular proteins at a high density might not always be possible, and the high illumination intensity used for photostable dyes might interact with endogenous chromophores to generate photodamage5. The fluorescence emission of fluorescent tags also saturates at high excitation powers, for example, at ∼1.5 kW/cm2 for EGFP8. Simply increasing the illumination power over a threshold induces nonlinear photobleaching of EGFP and many other fluorophores and reduces imaging duration9.
Live-cell SR imaging at millisecond (ms) temporal resolution is even more challenging due to the amount of emission photons that need to be collected during such short exposures. The temporal resolution is ultimately bound by the maximal emission photon flux rates from fluorophores excited under saturated illumination conditions. Despite the temporal resolution of 32–125 Hz reported for SR microscopy based on the on-and-off kinetics of single molecules10,11, these methods suffer from substantial phototoxicity and photodamage due to the use of an excitation laser power in the kW–MW/cm2 range12. The fidelity of entire structures from a small field of view (FOV) reconstructed from a ms exposure time is also questionable due to the intrinsically stochastic nature of SR image formation.
Structured illumination microscopy (SIM) is more photon efficient than other SR methods in increasing the spatial resolution13,14,15, and can operate at temporal resolutions up to 100 Hz13,16,17,18,19,20. However, because SIM reconstruction is essentially an ill-posed inverse problem that is prone to reconstruction artifacts21, the fidelity and the quantitative nature of conventional Wiener reconstruction have been challenged22. A number of iterative deconvolution algorithms, including total variance (TV)23 and Richardson–Lucy (RL)24,25, have been proposed to reduce reconstruction artifacts. However, their performance in the reconstruction of raw images with different signal-to-noise ratios (SNRs) remains unclear. Increasing the excitation laser power to obtain raw images with a sufficiently high SNR helps reduce such artifacts; however, this leads to phototoxicity and photobleaching, which limits the maximum number of usable consecutive frames that can be obtained from a time-lapse imaging experiment. To date, ultrafast SIM microscopy has been used to record a maximum of 200∼300 consecutive time points (Supplementary Table 1)13,16,17,18,19, and as a result, SIM methods are not commonly used by cell biologists for long-term time-lapse SR imaging under physiological conditions.
To address this problem, here we have developed an ultrasensitive Hessian-SIM microscopy method that is compatible with sub-ms exposures as it increases the fidelity of image reconstruction from low SNR images (Supplementary Note 1). Hessian-SIM has a spatial resolution of 88 nm at a maximal frame rate of 188 Hz. The high sensitivity allowed us to record two orders of magnitude more consecutive time points than previously reported, enabling the imaging of fast processes with high temporal resolution or the recording of long SR time-lapse experiments.
To obtain the optimal trade-off between resolution and photon budget, we have built a high numerical aperture (NA, 1.7), total internal reflection fluorescence structured illumination microscope (TIRF-SIM)13,15, which used a sCMOS camera with 82% peak quantum efficiency (ORCA-Flash4.0 V2, Hamamatsu) to detect the emission fluorescence. To generate and switch between excitation patterns created by various diffraction-limited gratings, we used a ferroelectric liquid crystal spatial light modulator (SLM) with a high frame rate, one polarization rotator (PR) assembled from four half-wave plates (HWPs) and two BK7 glasses to eliminate the switching time required to change the illumination polarization (Fig. 1a,b and Supplementary Fig. 1)18. At the emission path, we designed a synchronization paradigm that efficiently coordinates the pattern generation of the SLM and the camera readout interval (Supplementary Fig. 2).
Artifacts during the reconstruction of SR-SIM images
Artifacts due to imperfections in the optical system are nicely summarized in a recent paper26 and will not be discussed here. Using high-contrast raw images of stationary fluorescent beads captured by a well-aligned SIM microscope, conventional algorithms are able to accurately determine the reconstruction parameters and generate SR images with minimal artifacts through Wiener filtering (resolution of 88 ± 0.6 nm, n = 20, Supplementary Fig. 3). Upon imaging of mobile objects, because nine images are required for the reconstruction of one SR image, motion artifacts could manifest if the distance that the object moved between two consecutive exposures is larger than the resolution of SIM. Simply increasing the sampling speed helps to reduce motion artifacts (Supplementary Fig. 4), and we have given the upper speed limits for moving structures to be reconstructed without motion artifacts at different sampling rates in Supplementary Note 2.
However, increasing the sampling speed leads to shorter exposures of raw images. Under sub-ms exposures, fluorescent images of biological samples are often corrupted with substantial noise, which leads to reconstruction artifacts that are difficult to be disentangled from real features. By superimposing synthesized Gaussian noise onto high-contrast images of fluorescent beads, we systematically analyzed the impact of noise on the reconstruction of SR images (Supplementary Note 2). When the noise in the raw images exhibited a s.d. with a similar amplitude to that of the average signal, the wave vectors estimated using conventional algorithms began to show 2% deviation from the real value (Supplementary Fig. 5). Because 0.2–0.5% errors in the wave vectors resulted in the appearance of point-like emitters as rings at the edge of the FOV (Supplementary Fig. 6 and Supplementary Table 2), we hypothesized that SR images reconstructed from raw images with a poor signal-to-noise contrast would be corrupted with fixed-pattern artifacts due to inaccurately determined reconstruction parameters (Supplementary Note 2).
In addition, even though accurate parameters were used, artifacts were still observed in SR images reconstructed from raw images with increased noise using the Wiener algorithm (Supplementary Fig. 5a,c). In real experiments, various types of noise from the image-detection device, including readout noise, dark current and shot noise, are always superposed on the real fluorescent signals. Deconvolution amplifies white noise during the reconstruction process, which causes random, non-continuous artifacts in the spatial and temporal domains (Supplementary Fig. 5c). With raw images with a high signal-to-noise contrast, these artifacts are efficiently suppressed by the Wiener deconvolution (Fig. 2e and Supplementary Fig. 3). However, for biological samples captured under sub-ms exposures or that have been severely photobleached, these types of noise will dominate, resulting in the generation of intrinsic artifacts that contribute significantly to the artifacts observed during real experiments21,22.
Accurate determination of pattern wave vectors
To precisely estimate the reconstruction parameters from raw images with poor signal-to-noise contrast, we developed an amplitude-normalized cross-correlation method27 combined with multiple-frame averaging of low-contrast raw images to estimate wave vectors (Supplementary Note 1). In short, we normalized the amplitude and only used the phase information to perform the cross-correlation. For synthetic images containing Gaussian noise with a s.d. close to the average signal of the fluorescence beads, the conventional cross-correlation method failed, whereas our normalized cross-correlation yielded the accurate phase vector (Supplementary Fig. 5b). Compared with the method of filtering the low-frequency region to reduce the contribution from low-frequency noise in calculating the cross-relation23, our method yielded comparable results without arbitrarily setting the threshold. However, the estimated wave vector deviated significantly from the correct value when the s.d. of the added noise was 1.6-fold higher than the magnitude of the average signal (Supplementary Fig. 5b). Because the pattern wave vectors, starting phase, and modulation depth remained unchanged for a long period of time, we classified several continuous time-lapse raw images (9 × N frames) into nine sets according to their patterns and phases and calculated an average image from each set. For images with mean signal intensities that were less than 1/3 of the s.d. of the noise, this method surpassed previous algorithms and achieved a 99.993% precision in determining the pattern wave vector (Fig. 1c, Supplementary Note 1, and Supplementary Fig. 5b).
Hessian deconvolution algorithm
In contrast to linear Wiener reconstruction that uses no a priori knowledge, we proposed using the continuity of biological structures in the xyt axes to constrain the iterative deconvolution process. In short, because the full-width half maximum (FWHM) of the point spread function (PSF) of the SIM is ∼2.9 pixels (∼90 nm), sparse pointillist structures should be continuous within that region along the x and y axes. Moreover, if the distance that the biological sample moved between two consecutive time points was smaller than the lateral resolution of SIM, structures were considered to be continuous along the t axis. Artifacts due to random Gaussian noise, however, are mostly non-continuous along the xyt axes. Therefore, we iteratively adjusted the value of one pixel according to its neighboring pixels located one pixel away in the xyt dimensions during the reconstruction. To avoid oversharpening the boundaries between different regions with the TV penalty23, we adopted a Hessian penalty for the fluorescence variations along the x, y, and t directions, which enabled smoother transitions between different regions in the final reconstruction (Fig. 1c)28. The detailed equations are shown in Supplementary Note 1, where μ is the regularization parameter reflecting the relative weight of the fidelity term to the Hessian penalty. We also designed the variable σ to impose an additional constraint on the t axis, which could be turned off in cases of resolving rapidly moving objects to avoid temporary blurring. Right choice of μ and σ is important for balancing between effective denoising and reduced spatial resolution (Supplementary Note 3 and Supplementary Fig. 7), which help remove excessive random, non-continuous artifacts while preserving the shapes when tested with fluorescent beads (Supplementary Fig. 5c).
Hessian-SIM outperforms other deconvolution algorithms
Next, we have benchmarked the performance of Hessian-SIM with those of five existing algorithms, our Wiener implementation, the Wiener deconvolution in fairSIM25, the RL deconvolution in fairSIM25, TV reconstruction23, and the running average of three consecutive images from our Wiener reconstruction (RA Wiener) on synthetic images that contained structures of various shapes and spatial frequency contents and corrupted with excessive noise (Supplementary Fig. 8). We calculated the peak signal-to-noise ratio (PSNR) of reconstructed SR images obtained from raw images corrupted with noise according to the ground-truth SR image (reconstructed from noiseless raw images). Among all algorithms tested, Hessian-SIM achieved the highest PSNRs in all conditions (Supplementary Table 3), which supports its superior performance over other algorithms in obtaining high-fidelity SR images from raw images of low SNR.
Genetically encoded fluorescent probes in live cells emit markedly fewer photons than fluorescent beads, and are more prone to artifacts during SIM reconstruction21,22. During 7-ms exposures, densely packed actin filaments in human umbilical vein endothelial cells (HUVECs) labeled with Lifeact-EGFP were resolved without artifacts simply by Wiener deconvolution13, and Hessian deconvolution did not further improve the image quality (Fig. 2). In contrast, SR images reconstructed from raw images obtained from 0.5-ms exposures exhibited spurious small structures outside of filaments and larger fluctuations in fluorescence along actin filaments (Fig. 2a–d, experimental conditions given in Supplementary Table 4), similar to the synthetic data (Supplementary Fig. 8b); these artifacts were effectively removed by Hessian deconvolution. Because there is no ground truth in real experimental data, we used the variances of the fluorescence orthogonal to (V⊥) and along (V∥) actin filaments to estimate the magnitude of the artifacts. In constructing raw images captured during 7-ms exposures to 0.5-ms exposures, SR actin filaments reconstructed by the Wiener deconvolution suffered from about sevenfold and twofold increases in the V⊥ and V∥ values, respectively (Fig. 2e). In contrast, Hessian deconvolution reduced the V⊥ and V∥ values in SR images captured during 0.5-ms exposures to 1/11 (V⊥) and 1/2 (V∥) of those obtained with the Wiener deconvolution, at the expense of a marginal reduction in the spatial resolution (Wiener: 86 ± 1 nm, Hessian: 88 ± 1 nm, n = 8, Fig. 2c). Therefore, Hessian-SIM-generated SR actin filaments were similar to those obtained with the conventional Wiener-SIM at a photon dose only 1/12 of the latter. The Hessian algorithm also outperformed other algorithms by a large margin: the V⊥ in the SR images reconstructed with the Hessian algorithm from raw images obtained during 0.5-ms exposures was 32% of those obtained using the TV approach, which was identified as the second-best algorithm in terms of reducing artifacts23; in contrast, the RA Wiener algorithm generated SR images with an average V⊥ value that was approximately sixfold higher than that obtained with the Hessian algorithm (Fig. 2e), which shows that the Hessian algorithm reduces artifacts more efficiently than temporally smoothing the data.
We subsequently captured 61,200 consecutive raw images of actin during 0.5-ms exposures in live HUVECs, which corresponded to 6,800 SR images (∼1 min duration, Supplementary Video 1). As the average actin fluorescence progressively decreased to ∼55% of its initial value during the recording, artifacts in SR images reconstructed with the Wiener algorithm also significantly increased (V⊥, 176%; V∥, 139%). This increase in artifacts was largely suppressed by Hessian deconvolution, yielding continuous filament structures at the end of the experiment that were significantly better than those obtained with other reconstruction algorithms (Fig. 2f,g).
Next, we simultaneously imaged Lifeact-EGFP and SiR-tubulin29 in live HUVECs (Fig. 3a and Supplementary Video 2) and EB3-EGFP and SiR-tubulin in living insulin-secreting INS-1 cells (Fig. 3b and Supplementary Video 3) at a temporal resolution of ∼49 Hz. The spatial resolution of SiR-tubulin is approximately 113–115 nm in various cells (exact numbers in the legend to Fig. 3), providing a performance benchmark for the consistent resolution of the same structure in different cells. Despite a low illumination intensity (∼8 W/cm2 for the 647-nm laser, Supplementary Table 4), all of the densely packed structures that were imaged (actin filaments, microtubule meshworks, and puncta of tubulin-capping protein) were better resolved with minimal artifacts by Hessian deconvolution compared with other methods (Fig. 3c). Thus, compared with other algorithms, Hessian-SIM reconstructs actin filaments with high fidelity using a low photon budget, which substantially extends the ability to obtain usable SR images from time-lapse experiments.
Photobleaching determined by exposure times and mean powers
In live cells, the recording of 6,800 consecutive SR images of densely packed actin filaments is an order of magnitude longer than that of previous reports, which may involve mechanisms other than simple reduction of the total photon dose. To explore the underlying mechanisms, we compared the photobleaching kinetics during pulsed exposures in the sub-ms range (short exposures) with those obtained during exposure in the ms range (normal exposures). During conditions of a 7-ms exposure followed by a 4-ms recovery time per frame (10 Hz SIM imaging under 107 W/cm2 illumination), ∼50% of the fluorescence of Lifeact-EGFP was bleached rapidly after imaging for 160 time points, which translated into the reconstruction of severely aberrated SR images using the Wiener algorithm (Fig. 4a and Supplementary Videos 4 and 5). The addition of an interval of 900 ms between two consecutive time points (1 Hz, Supplementary Fig. 9a) slowed down the photobleaching, resulting in a twofold increase in the residue fluorescence at the 500th time point compared with that obtained with 10-Hz imaging (Fig. 4b,c). Thus, the mean illumination power (illumination intensity multiplied by the fraction of on-illumination per duty cycle), instead of the total light dose received, determines the speed of photobleaching under the pulsed illumination mode. Notably, the mean illumination power in 188-Hz Hessian-SIM is similar to that obtained in 10-Hz, 7-ms exposure SIM (63.4 W/cm2 versus 66.15 W/cm2), even though the residue fluorescence intensities of the former were substantially higher than those of the latter at all examined time points (Fig. 4b,c). The addition of variable recovery times between two time points of excitation pulses during 0.2-ms exposures reduced the mean illumination power in a dose-dependent fashion. Because EGFPs usually have typical lifetimes in the triplet state of ∼25 μs8, a sub-ms dark recovery time following the pulsed excitation might efficiently depopulate EGFPs to the non-excited basal state and extend the lifetime of EGFP before bleaching. The imaging of fewer EGFP molecules within the FOV further reduced the bleaching process because the fluorescence intensity of sparsely labeled EB3-EGFP puncta remained stable at 50% and 70% of the initial value after imaging for more than 3,000 time points under 188 Hz and 1 Hz, respectively (Supplementary Fig. 9b,c and Supplementary Videos 6 and 7). Because EB3-EGFP puncta moved slowly at a speed of 0.18 ∼ 0.23 μm/s (Supplementary Fig. 10), 1-Hz frame rate is sufficient for the visualization of their dynamics. Because many cellular structures move in the cytosol with similar speeds, Hessian-SIM with sub-ms exposures may enable time-lapse SR imaging of these genetically labeled structures with minimal perturbations to the internal cellular status for unlimited periods of time, which cannot be achieved with any existing SR methods. As pulsed excitation has also been associated with reduced phototoxicity in imaging experiments using fluorophores such as mCherry9 and fluo-3 (ref. 30), we suspect that the prolonged SR imaging with Hessian-SIM under sub-ms exposures may be extended to other fluorophores in general.
Visualizing millisecond dynamics and enlarged fusion pores
Rapid dynamic processes require ultrafast SR imaging. For example, in INS-1 cells labeled with VAMP2-mNeonGreen (Supplementary Fig. 11), vesicles moving at speeds of 7 μm/s and 4 μm/s could be resolved as spots in any frame, and their sizes approximated the PSF of Hessian-SIM with frame rates of 97 and 188 Hz. Flat sheets of peripheral endoplasmic reticulum (ER), which was recently found to be composed of small matrices, rapidly move over time19. Consistent with this finding, we identified small loops within ER sheets labeled by KDEL-EGFP in HEK293 cells (Supplementary Fig. 12 and Supplementary Video 8). Sometimes these loops could reach a speed as high as ∼7 μm/s, which required a frame rate higher than 78 Hz to avoid reconstruction artifacts (Supplementary Fig. 12c).
The release of neurotransmitters and hormones from secretory vesicles requires fusion of the vesicular membrane with the plasma membrane31,32, which involves the distortion and controlled disruption of the lipid bilayer via formation of a fusion pore. These intermediate structures, which occur at a nanometer scale and have lifetimes of milliseconds33,34, to our knowledge, have never been visualized in vivo and remain a great challenge in live-cell SR microscopy. By labeling exocytotic vesicles with VAMP2-pHluorin in INS-1 cells35, we were able to continuously record stimulation-triggered exocytosis for more than 10 min at a frame rate of 97 Hz (∼540,000 consecutive raw images, Supplementary Video 9), and revealed a biphasic glucose-stimulated insulin-secretion process (Supplementary Fig. 13a). To further increase the temporal resolving power, we used overlapping frames to create a 'rolling' SIM reconstruction (Fig. 1c), which did provide additional structural and dynamic information, as explained in detail in Supplementary Note 1 and improved the frame rate by threefold (Fig. 5f, Supplementary Fig. 14, and Supplementary Table 5).
At the improved frame rate, we were able to dissect single-vesicle fusion events labeled with VAMP2-pHluorin into four kinetically distinct steps: an initial rapid increase in the fluorescence intensity (fast rise, duration t1), a stage of slower increase (slow rise, duration t2), a stage in which the fluorescence remained elevated (plateau phase, duration t3), and the return of fluorescence to the baseline value (decay phase, time constant τ) (Fig. 5a,b). In contrast, the fusion of single vesicles labeled with NPY-pHluorin exhibited only the fast-rise and decay phases; the slow-rise and plateau phases were absent (Fig. 5b). The average t1 values for the VAMP2-pHluorin- and NPY-pHluorin-labeled fusion events were not different (24 ± 0.8 ms, n = 179 versus 20 ± 2 ms, n = 36, 16 pixels in diameter; see also Fig. 5b,d). Given a minimal size of ∼3.5 nm for three dimensions of the NPY-pHluorin structure36, the fast-rise phase appeared to represent the early expansion of a small fusion pore when vesicular H+ efflux occurs in the absence of NPY diffusion. Its duration is consistent with the results of previous experiments, in which small neurotransmitters, such as 5-HT and ATP, were released from insulin granules with a half-life of 5–8 ms by electrophysiological recordings33,34. The slow rise phase is likely due to the movement of fusing vesicle toward the plasma membrane and further expansion of the pore, which introduces an increased amount of the vesicular membrane into the evanescent field and full discharge of luminal NPY-pHluorin. Constriction of the pore must occur in the plateau phase to prevent the diffusion of vesicular proteins, and this constriction is relieved by final dilation of the pore, an effect that is followed by full exchange between the vesicular membrane and the plasma membrane. Fusion intermediates other than those presented at the first stage involve a fusion pore that is larger than 3.5 nm, and pores of this size are difficult to probe using indirect electrophysiological methods.
Notably, 21% of the fusion events (38 of 183) exhibited ring structures (∼192 ± 10 nm in diameter, “fusion with ring,” FR, Fig. 5c and Supplementary Video 10), resembling the ∼150-nm fusion pores of dense-core vesicles observed with rapid-freezing electron microscopy three decades ago37. These events were distinct from the conventional brightening and dispersion of diffraction-limited VAMP2-pHluorin puncta (79%, FWHM ∼103 ± 2 nm, n = 145, “fusion with no ring”, FNR, Fig. 5a). Based on two lines of evidence, we conclude that larger vesicles are involved in the FR type of exocytosis, whereas smaller vesicles are committed to FNR events. First, the peak fluorescence intensities of all fusion events could be described by two Gaussian functions, and the peak fluorescence intensities of the FR and FNR events were each fitted by one of the Gaussian peaks (Supplementary Fig. 13b). The mean peak intensity of FR events was 1.7-fold higher than that of FNR events, suggesting that FR events involve vesicles with larger membrane surface (Fig. 5d). Second, the average t1 and τ values for FR events were 1.6- to 2.4-fold longer than those found for FNR events. Because the rate of change in fluorescence was the same in the two populations (Supplementary Table 6), more time is needed for the dequenching or diffusion of VAMP2 in vesicles undergoing FR events. On average, ring structures appeared at the transition between t2 and t3 intermediates (61 ± 12 ms after the initial pore opening) in FR events and remained during the entire pore constriction stage (418 ± 88 ms) before diffusing away from the fusion site. More FR events than FNR events displayed pore constriction intermediates, and FR events remained in the constriction intermediate phase for longer periods of time (Fig. 5e and Supplementary Table 6). Therefore, these ring structures represent enlarged fusion pores that slow down the pore dilation of large vesicles, and do not restrict the release of vesicular contents, thus are different from nm-size fusion pores inferred previously.
Imaging mitochondrial cristae
Mitochondrial cristae are complicated and dynamic structures that are challenging for SR microscopy38. Using Hessian 2D-SIM, we clearly resolved mitochondrial cristae structures in live COS-7 cells and monitored their dynamics for more than 800 frames, which led to appreciation of dynamic changes in cristae structure during mitochondrial fission and fusion (Fig. 6a–c and Supplementary Videos 11, 12, 13). Moreover, we could identify the inter-cristae mergence in a single non-fusion mitochondrion labeled with both a chemical dye (MitoTracker Green) and a protein marker (PHB2-mScarlet) (Fig. 6d and Supplementary Video 14). Despite the intensive application of other SR microscopy techniques in imaging mitochondria38,39, the visualization of such mitochondrial inner membrane dynamics has remained difficult, highlighting the importance of illumination of minimal photon dose in combination with live-cell SR imaging.
In addition to the high-fidelity reconstruction of fluorescent structures exhibiting a low SNR, the Hessian algorithm can also be applied to structures that emit a high photon flux but have a low internal contrast. In live or fixed liver sinusoidal endothelial cells (LSECs) labeled with DiI, hollow-shaped artifacts smaller than the resolution of the microscope in Wiener-reconstructed images were eliminated by the Hessian algorithm (Supplementary Fig. 15). A concern of iterative SIM reconstruction is that the deconvolved structures depend on their surroundings, which might distort the signals in a nonlinear manner14. However, we have measured the peak fluorescence intensities during vesicle exocytosis through Hessian-SIM and found that these are tightly correlated with those measured with TIRF microscopy (Supplementary Fig. 16), indicating that Hessian-SIM is quantitatively accurate. Our Hessian algorithm can also be used to process images generated with other types of SIM instruments, such as to remove artifacts in Wiener-reconstructed SR images under a two-dimensional SIM (Supplementary Fig. 17 and Supplementary Video 15), or a commercial Nikon three-dimensional SIM15 (Supplementary Fig. 18). To facilitate these types of application, we have assembled a flowchart to summarize the optimal procedure, which incorporates SIMcheck software40 and our own experiences (Supplementary Note 3).
Hessian deconvolution can also be combined with 3D-SIM or lattice light sheet SIM41 to reduce photon budgets for volumetric SR imaging in live samples. However, as the extension of resolution in 3D requires an increase in photons in a cubic manner, brighter and more stable fluorescence probes are still needed for long-term SR imaging. To further increase the spatial resolution and to image thick samples, stimulated emission depletion may be a good tool as it has already been used to image scattering tissues such as living mouse brain42. Hessian deconvolution could also be applied to nonlinear SIM13,43 to further increase spatial resolution.
The visualization of the enlarged fusion pores and related fusion intermediates points toward another direction of application. The development of SLMs with faster transition times and the use of brighter fluorescent probes might even confer sub-ms temporal resolvability to Hessian-SIM in the future, which may enable even faster cellular dynamics, such as the spreading of Ca2+ nanodomains originating from single Ca2+ channel44 and the propagation of membrane potentials along axons to be captured in high definition.
In summary, in comparison to current SR microscopy techniques, Hessian-SIM excels in extending the spatiotemporal resolution and the imaging duration with a notably reduced illumination intensity. Under an illumination intensity one order of magnitude lower than that required in conventional point-scanning confocal microscopy (∼kW/cm2)8, Hessian-SIM enables sub-90 nm SR imaging with an ms frame rate in live cells labeled with commonly used fluorophores and allows consecutive SR imaging for thousands of time points. The microscope can be used for the imaging of different cellular organelles, and the algorithm can be widely adapted to different SIM modalities.
Cell maintenance and preparation.
HUVECs were isolated and cultured in M199 medium (Thermo Fisher Scientific, 31100035) supplemented with fibroblast growth factor, heparin, and 20% FBS (FBS) or in Endothelial Cell Medium (ECM) (ScienCell, 1001) containing endothelial cell growth supplement (ECGS) and 10% FBS. The cells were infected with a retrovirus system to express Lifeact-EGFP. The transfected cells were cultured for 24 h, detached using trypsin-EDTA, seeded onto poly-L-lysine-coated coverslips (H-LAF10L glass, reflection index: 1.788, thickness: 0.15 mm, customized), and cultured in an incubator at 37 °C with 5% CO2 for an additional 20–28 h before the experiments. To label tubulin in HUVECs, we followed a previously described protocol29, in which the cells were incubated with SiR-tubulin (Cytoskeleton, Inc. CY-SC006) at a concentration of 1 μM in growth medium at 37 °C for 2 h and imaged without washing.
INS-1 cells were cultured as described previously33 and transfected with VAMP2-pHluorin/EB3-EGFP/NPY-pHluorin/VAMP2-mNeonGreen using the Lipofectamine 2000 reagent (Thermo Fisher Scientific, 11668019) according to the manufacturer's instructions. After transfection, the cells were detached using trypsin-EDTA, seeded onto poly-L-lysine-coated coverslips, and cultured in an incubator at 37 °C with 5% CO2 for an additional 20–28 h before the experiments. We followed the above-described protocol to label tubulin in INS-1 cells with SiR-tubulin. For experiments conducted in INS-1 cells, we stimulated the cells with a solution containing 70 mM KCl and 20 mM glucose33 and observed the cytoskeletal dynamics and vesicle fusion events under the TIRF-SIM microscope.
HEK293 cells were cultured in high-glucose Dulbecco's Modified Eagle's Medium (DMEM) (HyClone, SH30022.01) supplemented with 10% FBS, 50 U/ml penicillin, and 50 μg/ml streptomycin (Thermo Fisher Scientific, 15140122) and were transfected with KDEL-EGFP/STIM1-mKO using Lipofectamine 2000.
LSECs were isolated and plated onto 100 μg/ml collagen-coated coverslips and cultured in high-glucose DMEM supplemented with 10% FBS, 1% L-glutamine, 50 U/ml penicillin, and 50 μg/ml streptomycin in an incubator at 37 °C with 5% CO2 for 6 h before imaging. Live cells were incubated with DiI (100 μg/ml, Biotium, 60010) for 15 min at 37 °C, whereas fixed cells were fixed with 4% formaldehyde at room temperature for 15 min before labeling with DiI.
COS-7 cells were cultured in high-glucose DMEM supplemented with 10% FBS and 1% 100 mM sodium pyruvate solution (Sigma-Aldrich, S8636) in an incubator at 37 °C with 5% CO2, detached using trypsin-EDTA, seeded onto poly-L-lysine-coated coverslips, and cultured for an additional 12–18 h before the experiments. To label mitochondria, COS-7 cells were incubated with 500 nM MitoTracker Green FM (Thermo Fisher Scientific, M7514) in high-glucose DMEM at 37 °C for 30 min before being washed and imaged in HBSS solution containing Ca2+, Mg2+ but no phenol red (Thermo Fisher Scientific, 14025076). All the above-described cells were tested for mycoplasma contamination before use.
To clean the coverslips (H-LAF10L glass, reflection index: 1.788, diameter: 26 mm, thickness: 0.15 mm, customized) for the live cell imaging, we immersed the coverslips in 10% Alconox (Sigma-Aldrich) and sonicated for 20 min. After we rinsed them in deionized water, the coverslips were sonicated in acetone for 15 min and then sonicated again in 1 M NaOH or KOH for 20 min. Finally, we rinsed the coverslips with deionized water, followed by sonication three times for at least 5 min each time. The washed coverslips were stored in 95–100% ethanol at 4 °C.
TIRF-SIM and 2D-SIM.
Our system is schematically illustrated in Figure 1a. It is based on a commercially available inverted fluorescence microscope (IX83, Olympus) equipped with a TIRF objective (Apo N 100X/1.7 HI Oil, Olympus) and a multiband dichroic mirror (DM, ZT405/488/561/640-phase R; Chroma). We used three lasers (Sapphire 488LP-200, Coherent; Sapphire 561LP-200, Coherent; and MRL-III -640-150, IL photonics) as the light sources, and an acoustic optical tunable filter (AOTF, AA Opto-Electronic, France) to combine, switch, and adjust the illumination power of the lasers. A collimating lens (focal length: 10 mm, Lightpath) was used to couple the lasers to a polarization maintaining single mode fiber (QPMJ-3AF3S, Oz Optics). The output lasers were then collimated by an objective lens (L1, CFI Plan Apochromat Lambda 2× N.A. 0.10, Nikon) and diffracted by pure phase grating consisting of a polarizing beam splitter (PBS), an HWP and the SLM. The diffraction beams were then focused by another achromatic lens (L2, AC508-250, Thorlabs) onto the intermediate pupil plane, where a carefully designed stop mask was placed to block the zero-order beam and other stray light and to permit the passage of ±1 order beam pairs only. We constructed a polarization rotator (PR) and placed it after the stop mask to maximize the modulation contrast of the illumination pattern while eliminating the switching time between different excitation polarizations (Fig. 1b). The light then passed through another lens (L3, AC254-125, Thorlabs) and a tube lens (L4, ITL200, Thorlabs) to focus on the back focal plane of the objective lens, and the light interfered at the sample plane after passing the objective lens. The emitted fluorescence collected by the same objective was passed through a dichroic mirror (DM), an emission filter, and another tube lens (L5) in the microscope body, split by an image splitter (W-VIEW GEMINI, Hamamatsu, Japan) and captured by a sCMOS camera (Flash 4.0 V2, Hamamatsu, Japan).
Nine raw frames illuminated with a periodic pattern of parallel lines and shifted through three phases for each of three orientation angles were used to reconstruct one SR image. We used ferroelectric liquid crystal on silicon SLM (SXGA-3DM or QXGA-3DM, Fourth Dimension Display) to generate the illumination patterns (Supplementary Fig. 1), which were formed by the interference of the ±1 order of the diffraction beam from the SLM. The radius of the annulus on the back focal plane of the objective lens was 5.2 mm and 3.1 mm for TIRF-SIM and 2D-SIM, corresponding to excitation NAs of 1.45 and 0.9, respectively. To maximize the modulation contrast of the signal, it was necessary to change the polarization of the two input beams to S-polarization. This polarization is often adjusted with a liquid crystal retarder13, which requires a ms time scale for switching between the two polarizations, thereby reducing the imaging speed. Similar to a previous report18, we designed a PR to eliminate the time for the switching between different polarizations. The PR was a segmented HWP consisting of six parts: two BK7 glasses marked with red orientation passes, and four HWPs with the fast axis marked with blue and green orientation passes, as shown in the upper panel of Figure 1b. After passing the homemade PR, the linearly polarized light beams illuminating different positions on the plate adopted a polarization tangential to the radius of the plate (Fig. 1b, lower panel, Supplementary Fig. 1), which eliminated the switching time between different illumination polarizations.
For a region of 256 × 128 pixels on the sCMOS camera, the time sequence for the synchronization of the system enabled a minimal exposure time and a transition time of ∼0.5 ms and ∼0.65 ms, respectively (Supplementary Fig. 2), corresponding to a raw data acquisition rate of 873 Hz, which translated to a frame rate of 97 Hz in SIM imaging. By reducing the imaging region to 256 × 72 pixels on the sCMOS and changing the SLM from SXGA-3DM to QXGA-3DM, we can further increase the raw data acquisition rate to 1,692 Hz (with exposure and transition times of ∼0.2 ms and ∼0.39 ms, respectively), corresponding to a frame rate of 188 Hz in SIM imaging.
Image processing and statistical analysis.
The image processing was primarily performed using ImageJ software (1.51s). Four plugins of ImageJ were used: fairSIM was used to reconstruct the images by Wiener deconvolution in fairSIM and RL deconvolution; SIMcheck was used to examine the quality of the raw images obtained using SIM microscopes; PSF Generator was used to generate simulated PSF; MTrackJ was used to analyze the speed of single ER loop and EB3. A bleaching correction method based on the histogram matching built in ImageJ was used to process all data and videos except those shown in Figures 2a,b, and 5 and Supplementary Videos 6, 7, 9, and 10. The reconstruction of the images acquired by TIRF-SIM or 2D-SIM were processed using customized MATLAB software (2012a). All data were plotted and final images were prepared using Igor Pro software (6.11, WaveMetrics, Lake Oswego, OR). Quantitative data are presented as box-and-whisker plots (center line, average; limits, 75% and 25%; whiskers, maximum and minimum) or mean ± s.e.m. in graphs and tables. The differences between groups were compared using either the two-side Student's t-test, or the Mann–Whitney rank-sum test when the data did not pass the normality test. The asterisks *, ** and *** denote statistical significance with P < 0.05, 0.01, and 0.001, respectively.
Life Sciences Reporting Summary.
Further information on experimental design is available in the Life Sciences Reporting Summary.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We thank B.-C. Chen and C. Shan for commenting on the optics and biological experiments and B.-C. Chen, T. Maritzen, and P. Cheng for reading the manuscript and providing suggestions. The work was supported by grants from the National Science and Technology Major Project Program (2016YFA0500400), National Natural Science Foundation of China (31327901, 31521062, 31570839, 31428004, 61375018, 61672253 and 91750203), the Major State Basic Research Program of China (2013CB531200), and Beijing Natural Science Foundation (L172003).
The HUVEC labeled with Lifeact-EGFP was imaged for 6,800 consecutive time points at 97 Hz. Scale bar: 2 μm.
The HUVEC labeled with SiR-tubulin (magenta) and LifeactEGFP (green) was imaged continuously at 49 Hz. Scale bar: 2 μm.
The INS-1 cell labeled with SiR-tubulin (magenta) and EB3-EGFP (green) was imaged continuously at 49 Hz. Scale bar: 2 μm.
Conventional SIM and Hessian-SIM compared at 10 Hz with 7ms and 0.2ms exposure time for 3,000 consecutive time points. Scale bar: 2 μm.
Conventional SIM and Hessian-SIM compared at 1 Hz with 7ms and 0.2ms exposure time for 3,000 consecutive time points. Scale bar: 2 μm.
INS-1 cell labeled with EB3-EGFP was imaged at 188 Hz with for 3,000 consecutive time points. Scale bar: 2 μm.
The INS-1 cell labeled with EB3-EGFP was imaged at 1 Hz for 3,000 consecutive time points. Scale bar: 2 μm.
The HEK293 cell was labeled with KDEL-EGFP, and imaged at 188 Hz for 6,800 consecutive time points. Scale bars: the first one is 200 nm and the later one is 2 μm.
The INS-1 cell was labeled with VAMP2-pHluorin, and stimulated with glucose and KCl. Fusion events were detected for more than 10 min at 97 Hz frame rate (~540,000 consecutive raw images). Videos of three different durations (2-12, 280-290, and 519-529 s) were shown on the top, middle, and bottom, respectively. Scale bar: 2 μm.
The INS-1 cell was labeled with VAMP2-pHluorin. Scale bar: 200 nm.
The mitochondria in the COS-7 cell was labeled with MitoTracker Green, and imaged under 0.5-ms exposure with an initial illumination intensity of ~18 W/cm2 light intensity (which increased by 0.05% during each SIM image). Scale bar: 2 μm.
The COS-7 cell was labeled with MitoTracker Green. The boxed region in the left is magnified and shown on the right. Scale bars: 1 μm (left) and 0.2 μm (right).
The COS-7 cell was labeled with MitoTracker Green. The boxed region in the left is magnified and shown on the right. Scale bars: 1 μm (left) and 0.5 μm (right).
The COS-7 cell was labeled with PHB2-mScarlet and MitoTracker Green. Boxed regions in the left are magnified and shown on the right. Scale bars: 1 μm (left), 0.5 μm (center) and 0.2 μm (right).
97 Hz 2D-Hessian-SIM imaging of cytosolic actin dynamics in a HUVEC for 2,000 consecutive time points.
The HUVEC labeled with Lifeact-EGFP was imaged for 2,000 consecutive time points at 97 Hz. Scale bar: 2 μm.
About this article
Nature Methods (2018)