Structured illumination microscopy relies on reconstruction algorithms to yield super-resolution images. Artifacts can arise in the reconstruction and affect the image quality. Current reconstruction methods involve a parametrized apodization function and a Wiener filter. Empirically tuning the parameters in these functions can minimize artifacts, but such an approach is subjective and produces volatile results. We present a robust and objective method that yields optimal results by two straightforward filtering steps with Richardson-Lucy-based deconvolutions. We provide a resource to identify artifacts in 2D-SIM images by analyzing two main reasons for artifacts, out-of-focus background and a fluctuating reconstruction spectrum. We show how the filtering steps improve images of test specimens, microtubules, yeast and mammalian cells.
Since the introduction of structured illumination microscopy (SIM) as a super-resolution imaging technique a reconstruction algorithm has been required1,20. The tasks of the reconstruction algorithm are threefold. First, the extraction of the domains that contain the high resolution information. Second, the extraction of the reconstruction parameters, namely the spatial frequency and the phase of the illumination pattern. Finally, the correct placement of the domains in Fourier space to assemble the spectrum of the super-resolution image . The first task is carried out by solving a linear system of equations. The reconstruction parameters are usually estimated, either through cross-correlations or through a linear regression1,2, using the information redundancy between the high frequency domains and the diffraction limited frequencies . To achieve a correct assembly of , an accurate estimate of and are important as they determine the solutions of and their correct placement respectively. Consequently, most publications introducing reconstruction methods are set in a context where the values of and are optimized in order to avoid artifacts due to a wrong estimate2,21,22,23,24.
We analyze the out-of-focus background and fluctuations in the image spectrum as important reasons for artifacts. The out-of-focus background generates periodic patterns in the image background3,4, while ripples in the image spectrum spuriously highlights or attenuates certain frequencies that translate into side lobe artifacts that diminish the quality of the super-resolved image . Our analysis leads to a reconstruction approach where we implement a pre- and a post-reconstruction filtering steps to minimize the artifacts. The initial filtering is performed on the raw images. It prevents the artifacts originating from the out-of-focus signal and facilitates the recovery of , even when it lies beyond the cut-off frequency of the detection objective. The second filtering is performed on the reconstruction and compensates the uneven weighting of , yielding a smoother spectrum that improves the high frequency details in the reconstruction.
In the most commonly applied SIM reconstruction procedure proposed by Gustaffson et al.1, the domains are combined through a Wiener filter while an apodization function is used to remove the ringing artifacts produced by this filter. The apodization function has evolved into a parametrized function whose shape is empirically tuned to enhance the reconstruction and eliminate artifacts. In this sense, the reconstruction has become reliant on the appointed apodization function and is still a heavily debated issue2,3,4,5,6,7,8,9. In our method, we perform two filtering steps with the Richardson-Lucy (RL) deconvolution. The advantage of this deconvolution algorithm over the Wiener filter is that it intrinsically accounts for the Poisson noise characteristic of the photon counting process in cameras.
In our reconstruction approach we completely prescind from any parameter tuning and only rely on the unbiased filtering steps to reduce artifacts and produce user-independent 2D-SIM reconstructions. We demonstrate the positive effects with images of diverse biological samples acquired with our set-up and an external data set.
Two-dimensional SIM super-resolution reconstruction rationale
In 2D-SIM a sample is illuminated with a sinusoidal pattern along different orientations. The acquisition process uses three different orientations and at least three phases for each orientation, i.e. a minimal set of nine images per plane. If the period of the pattern is To, its spatial frequency is given by . The pattern orientation is determined by the unit vector (j = 1, 2, 3). The three pattern phases for each j orientation are denoted as . Although these phases can be arbitrary, uniform steps are preferred to achieve uniform illumination of the whole sample. Thus the phases are equally distributed within the [0, 2π] interval, ; (m = 0, 1, 2), with representing the initial phase of the pattern in the j orientation. The intensity profile of a pattern at a given point in space is:
with a constant a proportional to the illumination intensity.
The images in the 9-tuple recorded in each plane are represented by and Fourier transforms by . The linear system in equation (2) relates with domains of the object/sample spectrum that are not accessible in conventional wide-field microscopy. These domains contribute new spatial frequencies, which in turn increase the resolution by extending the effective support of the microscope’s optical transfer function (OTF) (see Fig. 1). Solving equation (2) provides the extended domains as well as the central domain that contains the diffraction limited information20.
The set of extended domains is available due to the intensity modulation provided by the illumination pattern. As explicitly stated in their argument they are shifted into the support of the OTF by the vectors . Therefore, when a super-resolution image is assembled, the extended domains have to be shifted to their correct location in Fourier space by the vectors . The super-resolution image is then given by equation (3), where the shifting properties of the Fourier transform have been employed to shift the extended domains through the exponential factors.
will yield an image with a major resolution gain in the j direction. To gain resolution along the other orientations, the extended domains along each j orientation have to be calculated from equation (2) and then assembled using the following equation:
A typical footprint of an image spectrum is shown in Fig. 1c. The approximated perimeter of the extended domains is outlined in white and the central domain in black. The cut-off frequency of the OTF determines the radius of all domains. The resolution gain in is determined by the content of spatial frequencies in beyond the cut-off frequency.
The spectrum seen in Fig. 1c corresponds to a simulated 2D-SIM image acquisition of 1000 fluorescent beads 115 nm in diameter. The illumination pattern was simulated with equation (1) using a period of To~260 nm, a = 1000 and patterns oriented along 0°, 60° and 120°. All orientations have the same initial phase . The pixel size is 57.6 nm, the emission wavelength is 515 nm and the detection system is a 63x/NA 1.0 water immersion objective lens. The pixel size, emission wavelength and detection parameters were chosen to resemble the experimental conditions in our coherent structured illumination light sheet-based fluorescence microscope (csiLSFM), which creates an illumination pattern by interfering two light sheets (Supplementary Material Fig. S1)14.
Correlation as a tool for reconstruction optimization
Good estimates of the pattern spatial frequency and the pattern’s initial phase are essential before reconstructing an image with equation (4). The spatial frequency accomplishes the correct translation of the extended domains. The initial phase defines the coefficients of the linear system in equation (2) and determines the correct solutions for the extended domains.
There is usually a certain extent of overlap between the central and the different extended domains. It is this overlap of information that can be used to estimate the pattern spatial frequency . The cross-correlation of the central domain with an extended domain,
leads to a peak where the information overlap is maximal. The peak position provides a good estimate of . Figure 1d illustrates this situation by correlating some of the domains in Fig. 1c. The cross-correlation in equation (5) has not only been used to estimate but also to optimize the values of the phases 2. We extract the initial phases by carrying out a normalized cross-correlation between the reconstruction and the wide-field image. This approach evaluates the similarity of the features contained in both images.
Pattern spatial frequency estimation
Before carrying out the correlation between and , one must obtain these domains by solving equation (2). Their solutions for , (m = 0, 1, 2) are:
Equations (6 and 7) offer information for the reconstruction: First, is not a function of the initial phase ϕj, and second, the solution of incorporates ϕj in an exponential factor. Therefore, ϕj only acts as a scaling factor to the norm of the correlation . The position of the peak that determines is independent of the chosen initial phase in equation (2). Consequently, we arbitrarily pick a value of the initial phase ϕj and then solve equation (2) to obtain and .
The spatial frequency is calculated as:
The pixel-size is given in units of length, image length is the number of pixels along one axis of a square grid, (kx,ky) is the peak coordinate and is the grid center coordinate.
Pattern phase estimation through reconstruction evaluation
So far, we established that the value of the initial phase is not important to recover the spatial frequency from equation (5), yet its correct value is essential for a good super-resolution reconstruction. The reason is that the exponential factor in equation (7) determines the phase of the image through its imaginary argument, it is well known that the image phase is key in the perception and preservation of image features in the real space10. In Fig. 2a,b the sample introduced in Fig. 1a is reconstructed for the 60° orientation using the correct initial phase and an incorrect phase . The degradation of the reconstruction using the wrong initial phase (Fig. 2b) is obvious since features that are not original appear in the image.
To recover the correct value of the initial phase , we perform a normalized cross-correlation between the reconstruction and the wide-field Io (equation (9)). This correlation is only evaluated at the origin (xo, yo), i.e. at offset zero, and it is carried out for each of the values in a given discrete interval . Hence, a reconstruction is performed for each of the values in , by considering them as possible initial phase values to solve equation (2). Then each reconstruction follows equation (9) and the cross-correlation value is allocated in Rj, which becomes a function of . The correct initial phase is then given by the argument that produces the maximum value of Rj.
The cross-correlation compares with the diffraction limited image Io, which is calculated using . The rationale of using equation (9) to estimate the initial phase is that it also works as an assessment step for the reconstruction, since, even though has a higher resolution, it should still share similar features with the ground truth established by Io. Therefore, a good reconstruction is maximally correlated with Io. On the other hand, using a wrong estimate of leads to a distorted reconstruction reducing its similarity with Io, which in turn causes a drop in the value of the correlation.
A curve R2(Φ) is displayed in Fig. 2a corresponding to the reconstruction of our sample in Fig. 1a. In this case, Φ is a regular interval consisting of 200 points. A red dot marks the position of the maximum in the curve, which, as expected, is located at . The minimum value occurs at associated to the “worst” reconstruction (see Fig. 2c). Magnifications of the area enclosed in the white square (Fig. 2b) are shown for values of , 0 and (Fig. 2d–f). Anisotropy of the beads is expected, since we consider just one orientation for the reconstruction. Please notice, how the reconstruction using the correct phase (Fig. 1d) becomes degraded when wrong initial phases are used (Fig. 1e,f). This fact is also reflected in the R2(Φ) curve with the values of the correlation dropping around . This approach to estimate was tested on both sparse and dense samples giving good results as demonstrated with the successful reconstructions of the α-tubulin and mitochondria samples (Fig. 3b,d third and fourth columns). Each was estimated independently by its corresponding curve (see Supplementary Figure S6).
Initial deconvolution of O j,m reduces artifacts and enhances detection of reconstruction parameters
A real wide-field image always contains noise and out-of-focus fluorescence. Therefore, the cross-correlations in equations (5 and 9) are affected, which in turn makes the retrieval of the pattern spatial frequency and its initial phase difficult.
This issue has been addressed in ref. 2 where weighted cross-correlations have been used to diminish the effects of noise. The applied weighting functions, which rely on signal-to-noise ratios, were obtained either empirically or experimentally. On the other hand, weighting functions do not address the out-of-focus fluorescence issue directly, which is important in thick samples, and if not removed becomes a major source of artifacts. We experimentally observed the artifacts as quasi-periodic patterns (Fig. 3/second column) and also verified this effect with simulations (Fig. 4b). As Fig. 4 shows, these patterns appear in the regions where strong background fluorescence is present. The blurry regions in the wide-field image (Fig. 4a,d) directly relate to the occurrence of artifacts in the same regions of the reconstruction (Fig. 4b,e). Similar artifacts have been previously reported and associated to raw images (Oj,m) with low signal-to-noise ratio and strong out-of-focus signal3,4,12.
We propose a straightforward approach to remove artifacts in the reconstruction related to the out-of-focus fluorescence. We implement the Richardson-Lucy algorithm to deconvolve the set of Oj,m images with the PSF of the imaging system prior to solving equation (2). Additionally, this initial deconvolution diminishes the weighting effect of the OTF and enhances the detection of the peak resulting from the cross-correlation in equation (5). This enhancement is important especially when the frequency of the structured illumination pattern is beyond the diffraction limit since the overlap between and becomes small.
Initial deconvolution of O j,m reduces out-of-focus signal related artifacts in
The reduction of artifacts due to the initial deconvolution is illustrated in Fig. 4. Out-of-focus fluorescence has been simulated in our synthetic sample shown in Fig. 1a by adding a layer of 200 randomly distributed out-of-focus beads 115 nm in diameter. These beads are blurred by a PSF four times larger than the PSF of the emission wavelength (515 nm) of the in-focus beads. Structured illumination was simulated with the illumination and detection conditions summarized above. A set of Oj,m images is obtained and used to reconstruct . The wide-field and the images are shown in Fig. 4a,b, respectively. A second reconstruction was carried out but deconvolving all Oj,m images with 10 iterations of the Richardson-Lucy algorithm and the PSF of the detection system. Comparing the reconstructions in Fig. 4b,c proves that the initial deconvolution step suppresses the artifacts due to the out-of-focus fluorescence and provides a cleaner background. The effects on with and without the initial deconvolution are not just noticeable in simulations but also in experiments as compared in the second and third columns of Fig. 3. Non-initially deconvolved raw images produce reconstructions with seemingly periodic artifacts imprinted in the regions with background fluorescence, while applying the initial deconvolution step produces reconstructions with no artifacts.
When processing 3D stacks reconstructed in a plane-by-plane manner a 2D deconvolution is still preferred over a 3D deconvolution, since the latter appears to result in deconvolution related artifacts due to the non-applicability of spatial shift invariance (Supplementary Figure S8).
Initial deconvolution enhances the detection of
The initial deconvolution not only reduces periodic artifacts but it also facilitates the detection of the peak that determines . Especially for those data sets that have been illuminated under patterns with periods beyond the diffraction limit of the detection objective. This situation is illustrated in Fig. 5 where the norm of the cross-correlation of and is displayed without (Fig. 5a) and with initial deconvolution (Fig. 5b). Both images were normalized to have a maximum value of one for comparison. The results shown in Fig. 5 were calculated from the dataset of a human umbilical vein endothelial cell (HUVEC) stained with α-tubulin (Fig. 3b and Supplementary Material Fig. S6). The dataset was recorded with our csiLSFM set-up, with an illumination pattern of 215 nm in period, meaning that the norm of its spatial frequency was 1.2 times larger than the cut-off frequency of our detection lens.
The white dashed insets in Fig. 5 indicate the region where the peak is expected but is not detectable in Fig. 5a. The situation is different for the initially deconvolved data since a local maximum is clear in Fig. 5b. An automated detection of the peak is implemented by defining a region of interest (ROI), which makes the peak the global maximum in the cross-correlation of and . This ROI is a ring-shaped mask of e.g. 0.9ρ internal radius and 1.1ρ external radius, in which the mask has a value of 1 and 0 everywhere else. ρ denotes an expected estimate of the illumination pattern spatial frequency . Figure 5c shows the mask applied to the image of Fig. 5b. ρ is estimated from a calibration curve, e.g. in our set-up the spatial frequency is a function of the mirror angular displacement α (Supplementary Material Fig. S2). The calibration curve needs to be determined only once, upon system alignment and by using small fluorescent beads as test specimens.
The main mechanism, by which initial deconvolution enhances the peak occurrence is that deconvolution of the raw images increases their spectrum extent, which in turn increases the overlap of the and domains. A more fundamental explanation is that the deconvolution increases the effective modulation of the raw images by enhancing the differences between the raw images. Such enhancement facilitates the extraction of the spatial frequencies that form the extended domains (Eq. 7, Supplementary Figure S11). This becomes a feature when is larger than the cut-off frequency of the detection objective, since the overlap area of the domains decreases as increases.
Continued reconstruction deconvolution further enhances image quality
Although we increased the image resolution by adding the extended domains to the central domain in equation (4), this operation might not yield optimal results as it induces different frequency weights in different regions of the spectrum due to the arithmetic addition. For example, the overlap of and is large with smaller resolution gains, so the assembled spectrum has a major weighting in the overlap region. The low frequency content of the reconstruction is then emphasized. On the other hand, a high resolution gain implies less overlap of and . Such reduced overlap generates a fluctuating profile with valleys where the information drops due to the lower weighting at the edges of the OTF (Fig. 6i and Supplementary Material Fig. S4). This information drop in the spectrum translates into side lobes in the real space image (Fig. 6e), an effect which is especially obvious in high resolution gains (>2)5,13,17. Therefore, further improvement of the SIM reconstruction can be achieved by adjusting the weight of certain frequencies. This step is usually performed by multiplying the spectrum by an apodization function to tune the final appearance of the reconstruction3,5,6 (Supplementary Material Figures S3 and S5). In our reconstruction approach we simply deconvolve with an effective point spread function (PSFSIM), to equalize the weights of the high frequency and low frequency contents in the final image. The PSFSIM is formed by applying equation 4 to the PSF of the microscope and the obtained set of :
The deconvolution uses at most five iterations the Richardson-Lucy algorithm. How the reconstruction is improved by the continued deconvolution is demonstrated in Fig. 6 with 40 nm fluorescent beads and an illumination pattern with a period of 183 nm (wide-field Fig. 6a). The spectrum of a reconstruction, without undergoing initial or continued deconvolution, presents seven prominent spots (Fig. 6i). The prominence of such regions results in side lobes around the bead in the reconstruction (Fig. 6e). The unequal weighting of such a spectrum is slightly equalized after applying the continued deconvolution (Fig. 6j) and results in the reduction of the side lobes (Fig. 6f). Better results are achieved when the reconstruction is carried out using initially deconvolved raw images (Fig. 6g). The spectrum of this reconstruction (Fig. 6k) has a smoother profile than the ones in Fig. 6i,j. Best reconstruction results are obtained when using both the initial and continued deconvolution (Fig. 6h), there the side lobes are practically removed as validated by the intensity profile plot (Fig. 6d magenta line). The spectrum also has a more balanced profile without any sudden increases or decreases of values (Fig. 6l).
The continued deconvolution maximizes the resolution gain by equalizing both the low and the high spatial frequency content (Supplementary Figure S10). Its application to the reconstructions of biological sample is shown in the fourth column of Fig. 3. The contrast of the images is highly improved allowing a better identification of features in the sample, in comparison to the ones without the continued deconvolution (third column in Fig. 3). For 3D stacks reconstructed plane-by-plane a 3D continued deconvolution can be applied to achieve a resolution gain along the axial direction (Supplementary Figure S9). The PSFSIM for such 3D deconvolution is calculated with equation 10 using the 3D PSF of the detection objective.
Algorithm implementation on csiLSFM images
We summarize our reconstruction method in the flow chart in Fig. 7. MATLAB code based on it has been scripted. The code has been used successfully in 2D samples and 3D samples reconstructed plane-by-plane (Fig. 3). For each plane an input of 9 raw images is required, reconstructing a plane of 256x256 pixels takes about 10 seconds when using an interval Φ of 200 points, in a CPU equipped with a 3.1 GHz processor and 16 GB RAM. The reconstruction time is reduced to three seconds if only 30 points, which suffice for a good reconstruction, are used.
Several examples of applying our methodology in SIM reconstruction are shown in Fig. 3. Images were taken with our csiLSFM set-up where the diffraction limit of the detection objective is approximately 260 nm. Independent of the nature of the sample, one can track the quasi-periodic artifacts due to the out-of-focus blur visible in the wide-field images (first column) in the reconstructions (second column). These artifacts do not arise when the initial deconvolution is applied, as demonstrated by the reconstructions in the third column. Finally, the reconstructions in the third column are further improved in contrast and resolution by applying the continued deconvolution. The improvement of the image due to the continued deconvolution step is particularly noticeable in Fig. 3d (red box). Many structures are now easily identifiable. Reconstructions of samples in Fig. 3b,c are noteworthy as the illumination patterns had periods of 215 nm and 250 nm respectively, i.e. frequencies 1.2 and 1.1 times larger than the cut-off frequency of the objective lens. These images confirm the application of our reconstruction methodology using pattern frequencies beyond the diffraction limit.
We introduce a robust and user-independent pipeline for reconstructing 2D-SIM images, i.e. it does not require the empirical tuning of any parametrized functions to achieve optimal reconstructions. Instead, we rely on a deconvolution applied to the raw images prior to the reconstruction process and on a further deconvolution performed on the reconstructed image. Both deconvolutions use the Richardson-Lucy algorithm, which is an automated iterative method that only requires an approximated PSF and is included in many software packages. Furthermore, the algorithm is very proficient in deconvolving SIM data sets as shown in15,16.
These two steps remove the commonly seen periodic and side lobe artifacts, which arise even when using a correct estimate of the reconstruction parameters. The first deconvolution also facilitates the recovery of illumination patterns with spatial frequencies beyond the diffraction limit. In our case, we were able to recover spatial frequencies of illumination patterns with 183 nm periods using an objective of 260 nm resolution (Fig. 6). In that case the resolution is improved by a factor of 2.4, which is more than the two-fold improvement available in conventional SIM, and consistent with TIRF-SIM microscopes where such fine patterns are also used5,17,18.
The main advantage of our method is that it directly leads to an optimal image by eliminating the artifacts with the deconvolution steps. As a result, our method is objective and reproducible. In contrast, a parameter tuning method does not remove such artifacts effectively and can lead to degraded images, depending on the set of chosen parameters (Supplementary Figure S5). Thus, requiring intense tuning to achieve a proper reconstruction. As an example of this situation, we show in Supplementary Figure S7 an image reconstructed with our method and three reconstructions produced by an ImageJ/Fiji plug-in based on the tuning of a Wiener filter and an apodization function19. After several attempts to optimize the image we were not able to achieve a reconstruction of the quality provided by our method. Additionally our method was tested in an external TIRF-SIM data set to demonstrate its compatibility with images produced by other SIM systems (Supplementary Figure S12).
We also applied the method to 3D-SIM data sets, leading to the same conclusions, i.e. objective reconstructions with reduced artifacts (Supplementary Figure S8). In summary, the proposed method provides an objective procedure that simplifies the image reconstruction task for researchers using SIM, especially the newly interested and non-experienced.
How to cite this article: Perez, V. et al. Optimal 2D-SIM reconstruction by two filtering steps with Richardson-Lucy deconvolution. Sci. Rep. 6, 37149; doi: 10.1038/srep37149 (2016).
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We thank Prof. Dr. Robert Ernst and Stephanie Ballweg (BMLS, Goethe Universität Frankfurt am Main) for the yeast cell line used in this article and valuable comments, Sven Plath for substantial assistance and the Goethe Universität’s workshop for building some of the hardware. We also thank Eric Snapp for kindly providing the plasmid pRS415-ER-sfGFP-HDEL used in the yeast cell and Dr. Jürgen Neumann (Biozentrum der Ludwig-Maximilians Universität München) for providing the data set acquired with the OMX system used in the Supplementary Material. The research was funded by the Cluster of Excellence for Macromolecular Complexes (CEF-MC, speaker Volker Dötsch) granted to the Goethe Universität Frankfurt am Main by the Deutsche Forschungsgemeinschaft (DFG, EXC-115).
About this article
Nature Biotechnology (2018)