Abstract
Fourier ring correlation (FRC) has recently gained popularity among fluorescence microscopists as a straightforward and objective method to measure the effective image resolution. While the knowledge of the numeric resolution value is helpful in e.g., interpreting imaging results, much more practical use can be made of FRC analysis—in this article we propose blind image restoration methods enabled by it. We apply FRC to perform image denoising by frequency domain filtering. We propose novel blind linear and nonlinear image deconvolution methods that use FRC to estimate the effective pointspreadfunction, directly from the images. We show how FRC can be used as a powerful metric to observe the progress of iterative deconvolution. We also address two important limitations in FRC that may be of more general interest: how to make FRC work with single images (within certain practical limits) and with threedimensional images with highly anisotropic resolution.
Introduction
Reliable and realistic estimation of spatial resolution in fluorescence microscopy images has for decades been a subject of passionate scientific debate. With the development of fluorescence nanoscopy techniques^{1} this debate has resurfaced with newfound fervor, e.g., see refs. ^{2,3}.
Typically resolution is estimated by measuring either the minimum resolvable distance between two adjacent structures in an image—as per the classical Rayleigh/Abbe/Sparrow resolution definitions—or, alternatively it can be estimated from the intensity profile analysis, such as simple fullwidthhalfmaximum (FWHM) or more complex fitting^{4} of subresolved sized structures. In order to perform either one of the two measurements, suitable structures need to be subjectively identified and manually measured; ideally, the measurements should be repeated at several positions to gain some statistical basis for the estimate. This task is both tedious and errorprone.
Fourier ring correlation (FRC)^{5,6} and Fourier shell correlation (FSC)^{7}—essentially, FRC generalized to 3D—have for decades been used to estimate image resolution in electron cryomicroscopy. Recently FRC was adapted for optical nanoscopy, by us^{8} and others^{9,10}, to address the issues with the traditional resolution assessment methods. It is based on a normalized crosscorrelation histogram measure calculated in the frequency domain between two images of the same regionofinterest, with independent noise realizations. For FRC/FSC calculation, the spatial frequency spectra of the two images are divided into bins, which produces a series of concentric rings/shells in the polarform frequency domain images (hence the names). The FRC/FSC histogram is formed by calculating a correlation value for each bin according to
where F_{1} and F_{2} are the Fourier transforms of the two images and r_{i} the ith frequency bin. The image resolution in FRC/FSC is defined from the histogram, as a cut off frequency at which the crosscorrelation value drops below a preset threshold value. Advantages of the FRC/FSC are that it is fully automatic, quantitative, and depends both on sample and microscope characteristics. It is also less prone to subjective bias and measurement errors, although the choice of the appropriate resolution threshold criterion still requires some input from the researcher, as no single solution seems to be correct in all applications^{9,11}.
While the resolution estimation certainly is an interesting application in itself, in our view, the true potential of FRC/FSC is in much more practical tasks. In this paper we show several examples of advanced image restoration methods that leverage FRC/FSC measures. We apply FRC to perform image denoising by frequency domain filtering. We propose novel blind linear and iterative image deconvolution methods that use FRC/FSC measurements to estimate the effective pointspreadfunction (PSF) of the microscope, directly from the images, with no need for prior knowledge of the instrument characteristics. The deconvolution is shown to work exquisitely with both two and threedimensional (2D/3D) images. We also show how FRC can be used as a powerful metric to observe the progress of iterative deconvolution tasks.
Results
The role of FRC in image restoration
In image restoration tasks, in one way or another, one tries to improve the quality of an image, by increasing the contrast between signal and noise, i.e., the signaltonoise ratio (SNR). The FRC measure defines a cut off frequency in the frequency domain, beyond which (at higher frequencies than the cutoff), there are no details with sufficient SNR to be discernible. The knowledge of the cut off frequency alone (i.e., resolution in the spatial domain), can already help in several image restoration and analysis tasks, such as frequency domain filtering, autofocusing and image quality assessment. On the other hand, it is a common practice to interrelate the spatial resolution with the FWHM of the microscope’s PSF. Thus, once one knows the resolution (FRC), it is possible to form an estimate of the PSF. Naturally, an assumption needs to be made regarding the shape of the PSF—here, we approximate the PSF as a simple Gaussian function, which should be reasonable in most cases^{12}. This makes it possible to implement blind image deconvolution (deblurring) algorithms that leverage FRC to obtain an estimate of the PSF. In iterative deconvolution tasks it may also be of interest to update the PSF after each iteration, or to evaluate the quality of the deconvolution results. The principle of using FRC in image restoration and tasks is illustrated in Fig. 1a. The list of tasks mentioned in the Fig. 1a is by no means intended to be exhaustive, but it just simply reflects subjects that we touch upon in this paper.
In Fig. 1b,c an example is shown about the use of FRC in Fourier domain lowpass filtering with a noisy confocal image of HeLa cell tubulin cytoskeleton (pixel size 51 nm). The result shown in Fig. 1c was obtained with an ideal lowpass filter, which simply removes all the frequencies that are higher than then FRC cutoff (183 nm^{−1}). The ideal filter works excellently: it is able to practically remove all the highfrequency noise, with a faint lowfrequency noise pattern visible in the background (from scanning, laser fluctuation etc.). The use of an ideal filter is not typically recommended, as introducing such sharp edges into the frequency domain may produce artefacts (ringing effects) in the filtering results. No such effects can be seen here, supposedly because signal power is very low at frequencies higher than the FRC threshold. In Supplementary Fig. 1 results obtained with three different Fourier domain filters ideal, Butterworth and Gaussian are compared. In addition a result for the ideal filter with a theoretical cut off value is shown. All the filters were able to significantly reduce the noise, with no apparent effect on the fine image details, but the ideal filter with FRC cutoff is clearly the most effective.
In Fig. 1d an example is shown, of 2D blind Wiener deconvolution, with FRCbased PSF. The PSF is based on a simple Gaussian model, in which the FRC resolution is used as FWHM value (\({\mathrm{FWHM}} = 2\sqrt {2{\mathrm{ln}}2} \sigma\)). The Wiener deconvolution result shows dramatic enhancement of contrast and apparent resolution, but Wiener filtering does not really have any effect on the background level. It is possible to further tune the performance of a Wiener filter with the regularization parameter value. In Fig. 1d rather gentle regularization (SNR = 0.1) was used to produce the crisp looking result; the value was chosen subjectively.
FRC enabled blind iterative deconvolution
For traditional FRC analysis two images of the very same regionofinterest and with independent noise realizations are needed. There are various ways to obtain these two images, as discussed, e.g., in reference^{8}. In iterative image deconvolution, however, especially if one wants to explore the feedback connection illustrated in Fig. 1a—for progress estimation or iterative PSF updates—the twoimage requirement is not optimal, as it would require simultaneously, in a synchronous manner, running deconvolution on two images. This translates into complex software implementation, a significant computational overhead, and possible hardware resource issues, especially when dealing with large images. And of course, one does not always have two identical images in the first place. For this reason, we wanted to figure out, whether it would be possible to find a method to calculate FRC from a single image (oneimage FRC), with reasonable compromises, in terms of frequency bandwidth and accuracy.
In order to perform FRC analysis on a single image, one needs to find a way to form statistically independent image subsets—that share the same details, but different noise realizations—by some form of subsampling. As described in Fig. 2a, we propose to do this by dividing a single image into four subsets, i.e., twoimage pairs. The first pair is formed by taking every pixel with (even, even) row/column indexes to form one subimage and (odd, odd) indexes to the other. The second image pair is formed from pixels at (even, odd) and (odd, even) indexes. The dimensions of the four subimages are identical, exactly half of the size of the original image. FRC can be calculated from either one of the image pairs alone, but we noticed that averaging two measurements helps to deal with special spectral domain symmetries (Supplementary Fig. 2) that arise when details in an image are oriented predominantly in one direction. With 3D images (FSC) the same splitting method is used, except that in the axial direction (z), layers are summed pairwise to maintain image proportions; we get back to the 3D measurements later.
Because subsampling inevitably leads to loss of information, one might be keen to think that such a method is only feasible on significantly oversampled data, i.e., with sampling density much higher than the Nyquist limit (\(d_{{\mathrm{pixel}}} \le d_{{\mathrm{min}}}/2\sqrt 2\); d_{pixel} for pixel size; d_{min} for expected image resolution); we use the \(\sqrt 2\) factor in our Nyquist limit definition to ensure sufficient sampling in all directions, assuming square pixels on a rectangular sampling grid^{13}. However, as illustrated by the diagonal lines in Fig. 1a, the proposed subsampling pattern introduces a shift between the two subimages in each of the twoimage pairs, which as described in detail in Supplementary Note 1, demonstrates itself as a rather interesting exponential modulation in the frequency domain. This modulation has the property of compressing the FRC curve; by compression one means that details are shifted to a lower frequency than where they actually should be. In order to make any use of this effect, however, we needed to figure out how the compression works, i.e., where are the frequencies shifted. We achieved this by calibrating the oneimage FRC, against the “gold standard” twoimage FRC, at the cut off frequency defined by the 1/7 resolution threshold^{8,9}; the same threshold was used in both one and twoimage FRC. The principle of the calibration is explained in detail in Supplementary Note 2, and the results are shown in Supplementary Fig. 3. After the calibration, the one and twoimage FRC work in a very similar manner, as shown in Fig. 2b, up to the Nyquist limit. It should be noted that the calibration is only valid for the stated 1/7 threshold; we do not attempt to correct the entire FRC curves. For a different threshold, a similar process should be repeated. This however does not stand in the way of using oneimage FRC in the image restoration applications that we envision for it here.
Having established how to calculate FRC on single images, we then put the method to work in iterative Richardson–Lucy (RL) devonvolution. In our RL implementation, the FRC is calculated after each iteration in order to assess the effective resolution of the deconvolution results, similar to what was done in^{14}. We implemented the RL algorithm in two ways: in regular RL the same FRCbased PSF is used throughout the iteration process, whereas in the Adjustive RL the PSF is updated after each iteration based on FRC. The latter case was inspired by classical iterative blind deconvolution algorithms^{15,16}, in which one starts with some sort of a rough PSF estimate that is updated at every iteration. Such a scheme may be beneficial if the initial guess does not fit the data very well. When using FRC for the PSF estimation, the first guess is always based on the actual data, but for example when the original data is very noisy (low SNR), FRC may initially underestimate the resolution. Updating the PSF throughout the iteration process may however lead to overestimation of the PSF, and thus using some sort of a constraint may be beneficial. To that end, in Adjustive RLTV we added total variation (TV) regularization (λ_{TV} = 5 × 10^{−4})^{17} to the Adjustive RL algorithm.
The blind RL deconvolution results with the three methods are shown in Fig. 3a with a microtubulesstained HeLa cell confocal image (pixel size 56 nm). The regular RL deconvolution, with fixed PSF produces the sharpest looking results. The Adjustive RL and Adjustive RLTV both improve the quality of the original image, but at least in this instance, there seems to be no benefit from updating the PSF during iteration. In Fig. 3b the FRC resolution is plotted as a function of iteration count, for each of the three algorithms. The measures confirm the subjective observations: the regular RL produces the highest resolution results. It is also interesting to observe that the TV regularization helps the Adjustive RL algorithm to converge much more quickly, but the improvement stops at a similar level with the nonregularized Adjustive RL.
From the FRC measures overall, it is evident that the effective resolution in the deconvolution results tends to converge to a nearly fixed value after a number of iterations. This insight motivated us to look into another longstanding problem with RL and other iterative deconvolution algorithms: that one does not really know when the algorithm should be stopped^{17,18,19}. Quite commonly this is decided by trial and error, by running the algorithm with different iteration counts, and picking the best looking result. Several parameters have been proposed as well to observe the progress of deconvolution, but sadly, they are not very reliable or commonly used. Therefore we wanted to see, if we could use FRC for that, and found out that the first derivative (∇Resolution) of the curve shown in Fig. 3c is very well suited for that—it quantifies the rate of change of the FRC resolution as a function of deconvolution iterations. The most obvious iteration to stop the deconvolution would be at the maximum resolution, i.e., when the derivative reaches zero. In Fig. 3 Adjustive RL converges after 27 iterations, Adjustive RLTV after 17, and RL after 92 iterations. However, as is evident from the RL deconvolution, reaching the maximum resolution may take a long time, and on the other hand, very little improvement is made (based on the FRC measures) as the deconvolution approaches convergence. Thus, the derivative zero may be an overly stringent condition. Therefore we highlight two alternative thresholds in Fig. 3c (∇ Resolution ≥ −1 nm it^{−1}, ∇ Resolution ≥ −0.2 nm it^{−1}) that one may want to use instead. The first threshold stops the iteration, when most of the resolution gain has been made, whereas at the second threshold, the deconvolution approaches nearly complete convergence. The specified resolution thresholds assume a nanometer length scale, which is meaningful in optical microscopy. One of course needs to adjust for appropriate physical units, when working with images with significantly higher or lower resolution. The 1 nm it^{−1} threshold corresponds to roughly 0.5% and 0.02 nm it^{−1} to 1 permille of the resolution. In Supplementary Fig. 4 the deconvolution results with the RL algorithm at the three thresholds are compared (∇ Resolution ≥ −1 nm it^{−1}, −0.2 nm it^{−1}, and 0 nm it^{−1}). Already at the lower −1 nm it^{−1} threshold the results are very good. In Supplementary Fig. 5 similar results were obtained with a vimentin stained cell image (confocal, pixel size 29 nm) and in Supplementary Fig. 6 with a widefield image of a vimentin stained cell. In all of the three examples (Fig. 3d, e; Supplementary Fig. 5e, f; Supplementary Fig. 6e, f) two alternative deconvolution progress parameters τ_{1} and η_{k} are plotted as well: both indicate some sort of convergence, but no other qualitative information can be derived from them. The η_{k} measure in Fig. 3d works exactly the same with all three algorithms; τ_{1} reacts to the regularization, but cannot tell a difference between the RL and Adjustive RL algorithms.
Having established that the FRCbased deconvolution progress observation works rather well, we then wanted to test its robustness to strong background noise, which is known to affect the performance of RL (and other iterative deconvolution algorithms)^{20}; the algorithm(s) cannot make a difference between signal and background noise, and thus, as the deconvolution progresses, it eventually starts to fit the noise in the background as well as the signal^{21,22}. In order to assess how this effect becomes visible in the FRC measures, we devised a worst case simulation of sorts with an image containing mainly background noise (Supplementary Fig. 7). After the very first iteration a separate second peak appears in the FRC curves that grows in intensity as the RL algorithm tries to fit the noise into a sparse solution, which results into the dotty pattern that is shown after 30 iterations. In this case, as shown in Supplementary Fig. 7b,c, the FRC does not get caught into the second peak, but manages to measure the actual resolution, and the deconvolution stops after one iteration. It is however possible that a situation would arise, in which the noise peak is not as clearly separated from the signal, and thus we wanted to devise some sort of an indicator for abnormal behavior of the FRC measures during deconvolution. For this purpose in Supplementary Fig. 7d, we propose the derivative of the ∇ Resolution curve ∇^{2} Resolution. The reasoning behind this is that as the deconvolution converges, the slope of the ∇ Resolution should approach zero; i.e., it should not significantly increase, after it has decreased previously. The ∇^{2} Resolution crosses zero every time ∇ Resolution changes direction, thus indicating abnormal convergence behavior—it works as a failsafe of sorts.
In Supplementary Fig. 8 an RL deconvolution example of a mictotubulinlabeled HeLa cell with strong noise background is shown. It is possible to improve the performance of the RL algorithm with images with strong background, as discussed in ref. ^{20}, by adding an estimate of the background to the algorithm as a prior information. To that end in Supplementary Fig. 8, a (blind) method for estimating the background is introduced, and it is shown to greatly enhance the quality of the deconvolution result. As shown in Supplementary Fig. 8b the strong background demonstrates itself as a shoulder in the FRC curves, which appears after seven iterations. Unlike in the simulation, the background does not form a separate peak, but rather gets mixed with the signal spectrum. The FRC curves with the background estimation do not show a similar effect, and the RL deconvolution also seems to reach a much higher resolution (degree of convergence). This is confirmed by FRC measurements, shown in Supplementary Fig. 8d. The appearance of the shoulder in the FRC curves shows up as an edge in the ∇ Resolution curve (Supplementary Fig. 8d), and as a zero crossing ∇^{2} Resolution (Supplementary Fig. 8e), which is in agreement with the simulations. The η_{k} (Supplementary Fig. 8f) and τ_{1} (Supplementary Fig. 8g) do not react to the background fitting in any way.
Working with 3D images
In fluorescence microscopy the image resolution is highly anisotropic: due to limited numerical aperture of the single objective lens microscope systems, the axial resolution (direction of the optical axis) is typically at least factor of three inferior to the lateral resolution. For this reason FSC, the simple 3D expansion of FRC, is of only limited value in fluorescence microscopy applications. In order to address this issue, we developed a FSCbased method in which each Fourier Shell is divided into wedges. A single Sectioned FSC (SFSC) indexing shape consists of two such wedges that are each others mirror images (Supplementary Fig. 9b)—this is to take advantage of the symmetries in Fourier space. In order to calculate resolution values for the whole sphere, the dualwedge structure is rotated in increments of α (Supplementary Fig. 9a) around an axis located on the XY plane—for each orientation a separate crosscorrelation histogram and resolution value are calculated; the α also matches the angular size of the wedge. At α = 2π the SFSC simplifies into normal FSC. As a concept the SFSC is very similar to the recently proposed Conical FSC measure^{23}. They essentially differ in the way that the Fourier sphere is indexed to produce the directional resolution measures. The SFSC is especially tuned to observe variation of resolution, when rotating around a single axis, which makes it rather fast to calculate (few sections/volume) as well as robust (large number of voxels on every section).
We compared our new SFSC measure with FRC on a stimulated emission depletion (STED) microscopy^{24} image stack of a microtubules stained HeLa cell. The FRC measures were made by identifying apparently infocus planes in the 3D images and then calculating separate resolution values for the XY and XZ orientations. We also compared our SFSC measure against Fourier Plane Correlation (FPC) that was proposed for 3D image analysis in^{9}. In order to facilitate the manual FRC measurement and to limit the computational load in SFSC and FPC the resolution measurements were done on a cropped 300 × 300 × 30 pixel central section of the STED image (about 1/3 of the image size, 300 × 300 × 300 pixels after resampling and zero padding). As shown in Supplementary Fig. 10, the FRC and SFSC measurements are in rather good agreement. SFSC measurement at SNR_{e} = 0.5 threshold (onebit^{11}), corresponds to FRC measurement at 1/7 threshold; the onebit threshold was used in all SFSC measurements. The FPC works rather well as well, but appears to suffer from the interpolation, scanning and other artefacts in the axial direction, which as shown in Supplementary Fig. 10a, c causes it to somewhat underestimate the resolution in the axial direction.
In Fig. 4, blind Wiener filtering results with two 3D images are shown. The new SFSC measure is leveraged to produce the resolution estimates, necessary for generating the PSFs. In Fig. 4a, results obtained with a STED microscope superresolution image are shown. The images were acquired with relatively low STED depletion intensity to ensure good contrast, and to reduce photobleaching. As is typical to a STED microscope image, the resolution anisotropy between lateral (XY) and axial (Z) directions is considerable, as shown in the SFSC resolution plot in Fig. 4a. The polar plot illustrates the resolution point in micrometers, and it was calculated by rotating the SFSC section at 15° increments around the yaxis. Because STED is a bandwidth unlimited technique, the power spectrum in STED images often contains very high frequencies that unfortunately, are often hidden by noise. However, the simple blind Wiener filtering, as shown in Fig. 4a, is able to recover a surprisingly large amount of fine details. The axial haze is clearly reduced, and the effective resolution is drastically improved with previously blurred filaments, clearly visible in the results.
In Fig. 4b, the same blind Wiener filtering approach was applied to a much larger (deep) image of Pollen recorded with a confocal microscope. Only single image was available for analysis, so the diagonal splitting was used in the SFSC calculations. The deconvolution results show dramatic improvement of contrast and details—and axial haze is effectively reduced, as indicated by the depth coloring. Even crispier details can be obtained by decreasing the Wiener regularization; SNR = 0.005 was used in Fig. 4b to produce a smooth result.
Discussion
In this paper several blind image restoration methods were introduced that leverage FRC resolution measurements in different ways. In frequency domain denoising methods FRC was used to find a cut off frequency point for lowpass filtering. In deconvolution tasks, both linear (Wiener) and iterative (RL), FRC measurements were used to estimate the effective PSF, directly from the image data. There are several clear benefits of estimating the PSF with FRC. Firstly, no prior knowledge—even theoretical—is needed of the microscope or the sample. Secondly, the PSF generated via FRC is always tailored to every given image. Thirdly, the PSF estimation with FRC is a singlestep process, although it can also be updated iteratively if necessary. This advantage specifically made it possible to perform linear blind Wiener filtering in a very straightforward way, both in 2D and 3D. With larger images it might be of interest to divide the images into several smaller blocks^{25}, to adapt the PSF in the blind deconvolution to local changes of resolution (Supplementary Fig. 11).
In iterative deconvolution (RL), FRC measures were also leveraged to observe the progress and quality of the deconvolution. No other such metric to our knowledge exists in the literature. The τ_{1} and η_{k}, as well as many other measures that can be found in the literature, mainly quantify the convergence of the deconvolution algorithm, but cannot really quantitatively analyze the quality of the deconvolution results, in absence of a ground truth image. It was shown that FRC can be used to identify the deconvolution iteration at which the effective resolution is maximal—or near to it, if using the proposed thresholds. It was also shown that FRC measures can provide additional qualitative information of the deconvolution progress, e.g., related to the effect of the image background to the deconvolution quality.
In addition to deconvolution and image denoising that was the focus of this paper, there are several other image processing/analysis tasks that FRC/SFSC could be applied to. In Supplementary Fig. 12, we entertain the idea of combining FRC with other image quality assessment parameters, to produce quantitative measures of image quality for e.g., highcontent screening applications; a similar method was recently proposed for assessing the quality of localization superresolution microscopy image reconstructions^{26}. FRC is also very sensitive to the defocus: as shown in Supplementary Fig. 13, it actually behaves rather similarly to several autofocus metrics, which might be an interesting future application—and of course, a good way to identify an infocus plane at postprocessing stage. Our opensource MIPLIB software library that was used to perform all the demonstrated image analysis and processing tasks, may help in such future applications.
Methods
One and twoimage resolution measurements with 2D images
One and twoimage FRC measurements were both done according to
where F_{1} and F_{2} are the Fourier transforms of the two images and r_{i} the ith frequency bin. Prior to the FRC calculations, in twoimage case, the two images were registered by a phasecorrelation based method^{27}; in singleimage case the splitting was performed to produce the four subimages. A Hamming window was applied to each image to suppress edge effects and other spurious correlations. With both one and twoimage FRC the 1/7 resolution threshold^{8,9} was used to determine the numerical resolution value.
Resolution measurements with 3D images
Measurements on 3D images were done with our SFSC measure
where F_{1}(r, α, ϕ) and F_{2}(r, α, ϕ) denote the voxels in two Fourier transformed images that are located (I) at a given distance r_{i} from the origin and (II) within an orientation sector, defined by α and ϕ (Supplementary Fig. 9a). We compared the SFSC measures against FPC^{9} as well as FRC measures.
The SFSC/FPC/FRC analyses with the 3D images were performed with the same logic as the FRC measures on the 2D images. Before data analysis each 3D image was resampled to isotropic spacing using linear interpolation. Single image splitting was achieved similarly to the 2D case; only one diagonal was used to limit the computational effort in SFSC/FPC. In the axial direction two consecutive (z) layers were simply added together, to maintain the image proportions, and to not introduce additional offsets. Threshold curve based on SNR_{e} = 0.5 (onebit) was used with SFSC, whereas with both FPC and FRC, 1/7 threshold produced more reasonable numerical values. The SNR_{e} thresholds, proposed in^{11} can be calculated from
where T(r_{i}) is the threshold value, N(r_{i}) the number of pixels/voxels at ith Fourier ring/shell, and SNR_{e} the expected SNR value at the cut off point.
Because of the anisotropic sampling that is typical to 3D fluorescence microscopy images, the frequency axes in SFSC and FRC curves need to be corrected to compensate for it. In this paper this was achieved by multiplying the image pixel/voxel size by factor k(θ):
where z is the sampling anisotropy factor (e.g., two for an image sampled with half the sampling rate in depth (Z) with respect to the lateral (XY) direction) and θ denotes the rotation angle with respect to the XY plane, which in case of SFSC is a multiple of α. If no such correction is made, all the numerical resolution values calculated with FPC/FRC/SFSC at orientations θ ≠ 0 + nπ will be unrealistically high.
Frequency domain lowpass filtering
The frequency domain filtering was performed by first estimating the effective image resolution with FRC and then using it as a cut off frequency for a lowpass Fourier domain filter. Three different types of Fourier space filters were used in this work: (I) an ideal lowpass filter (ILPF), (II) a Butterworth lowpass filter, and (III) a Gaussian lowpass filter.
An ILPF can be defined as:
where r_{i} is a polar distance from the center of the frequency domain filter (zero frequency) and r_{th} is the distance at the cut off frequency, obtained with FRC. Frequencies after the cutoff are simply clipped to zero.
A Butterworth lowpass filter (BLPF) on the other hand is defined as:
where n denotes the degree of the filter, which controls how sharply the transition from passband (allowed frequencies) to stopband (filtered frequencies) is made. When compared to ILPF, BLPF has nearly equally flat (unity) response in the passband, but transitions to stopband more smoothly, thus avoiding a strong discontinuity at the cutoff; at the r_{th}, H(r_{i}) = 0.5.
A Gaussian lowpass filter (GLPF) is defined as:
Similarly to BLPF, the GLPF has a smooth transition from passband to stopband. The Gaussian function however, is not flat in the passband and transitions more slowly from passband to stopband, which means that it may in some cases blur the filtered images and perform less than optimally, when filtering out noise. At the r_{th}, H(r_{i}) = 0.607.
Image deconvolution
Image formation in a microscope can be described as a convolution of every object sample point with the PSF:
where i(x, y, z), h(x, y, z), and o(x, y, z) are the measured image, the PSF and the original sample object, respectively. By image deconvolution^{28} one attempts to revert the blurring effect of the microscope, and thus increase image contrast and effective resolution, by using the PSF as a prior information. Deconvolution can be performed in a single step, e.g., by Wiener or Tikhonov filtering^{29}—or then iteratively, e.g., by RL^{30,31,32}. In this paper we use the Wiener and RL algorithms.
The Wiener deconvolution algorithm is based on inverse filtering, and takes advantage of the fact that a convolution operation in the spatial domain, becomes a multiplication in the frequency domain—and thus a relatively simple singlestep deconvolution can be realized as follows:
where O(u, v, w), I(u, v, w), and H(u, v, w) are the Fourier space representations of the estimate for the original object, the observed image and the PSF. H(u, v, w)^{2} is the power spectrum of the PSF. The regularization term SNR^{−1} can also be written as N(u, v, w)^{2}/O(u, v, w)^{2}, where N(u, v, w)^{2} is the power spectrum of the noise and O(u, v, w)^{2} is the power spectrum of the original object; neither of the two terms are known, which means that usually the value is decided on casebycase basis, based on the subjective quality of the deconvolution results. The regularization factor is weighted by the power spectrum of the PSF. It will have a stronger effect at high frequencies, as the power spectrum approaches zero value.
The iterative RL algorithm can be described by
where o_{k} and o_{k+1} are the current and next object estimates, i is the original image, h is the PSF, h^{*} its mirrored version (complex conjugate) and b a background term. In the equation, the pixel indexes have been omitted to allow a simple presentation of the algorithm. b is typically set to zero, but a nonzero value can be used to correct for a strong background signal. In our example (Supplementary Fig. 8) the background was estimated by first dividing an image into two segments (signal and background) with a simple intensity based spatial mask^{33}, after which the mean intensity value of the background segment was calculated and then used as b in deconvolution.
With both Wiener filtering and RL, PSF estimate was generated on the basis of an FRC measurement on the original image data: the FRC resolution value was simply used as an FWHM value for a Gaussian PSF. With 3D images, separate values were used for lateral and axial directions. In the Adjustive RL algorithm the PSF is updated during RL iteration, by calculating the FRC resolution for each intermediate deconvolution estimate.
In addition to updating the PSF estimates (when so desired) the FRC measures of the intermediate estimates were used to observe the progress of the RL deconvolution. The deconvolution can be considered fully converged, when effective resolution reached its maximum value. However, as reaching the full convergence may take a large amount of iterations, we proposed two more practical thresholds based on the rate of change of the effective resolution (−1 nm it^{−1} and −0.2 nm it^{−1}). We compared the FRC metric against two previously published ones τ_{1} and η_{k}^{17,18,19,34}.
With τ_{1} the relative difference between two subsequent deconvolution estimates is measured:
where o_{k} is the current estimate and o_{k−1} the previous one. η_{k} is a measure of convergence
where N is the total number of pixels, N(u) is the number of pixels that are not currently converging and ε is the convergence epsilon.
TV regularization^{17} with fixed λ_{TV} = 5 × 10^{−4} was used as a smoothness constraint in our Adjustive RLTV algorithm.
Test images
The test images consist of various types of confocal as well as STED microscope images that were acquired with a variety of commercial and custombuilt microscopes. None of the samples was specifically prepared for this paper, but a short description for each is given in Supplementary Note 3.
Data availability
The image data that supports the findings of this study are available in the figshare repository, https://doi.org/10.6084/m9.figshare.c.4511663.
Code availability
The FRC/FSC measurement functions as well as all the image processing, analysis and data visualization tools used in this paper are available as opensource Python software library, called Microscope Image Processing Library (MIPLIB) at https://github.com/sakoho81/miplib. Please see Supplementary Note 4 for additional information about the software.
References
 1.
Hell, S. W. et al. The 2015 superresolution microscopy roadmap. J. Phys. D Appl. Phys. 48, 443001 (2015).
 2.
Sahl, S. J. et al. Comment on “extendedresolution structured illumination imaging of endocytic and cytoskeletal dynamics”. Science 352, 527 (2016).
 3.
Li, D. & Betzig, E. Response to comment on “extendedresolution structured illumination imaging of endocytic and cytoskeletal dynamics”. Science 352, 527 (2016).
 4.
Barentine, A. E. S., Schroeder, L. K., Graff, M., Baddeley, D. & Bewersdorf, J. Simultaneously measuring image features and resolution in livecell STED images. Biophys. J. 115, 951–956 (2018).
 5.
Van Heel, M., Keegstra, W., Schutter, W. & Van Bruggen, E. J. F. Arthropod hemocyanin structures studied by image analysis. Life Chem. Rep. Suppl. 1, 69–73 (1982).
 6.
Saxton, W. O. & Baumeister, W. The correlation averaging of a regularly arranged bacterial cell envelope protein. J. Microsc. 127, 127–138 (1982).
 7.
Harauz, G. & van Heel, M. Exact filters for general geometry three dimensional reconstruction. Optik 73, 146–156 (1986).
 8.
Tortarolo, G., Castello, M., Diaspro, A., Koho, S. & Vicidomini, G. Evaluating image resolution in stimulated emission depletion microscopy. Optica 5, 32–35 (2018).
 9.
Nieuwenhuizen, R. P. J. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).
 10.
Banterle, N., Bui, K. H., Lemke, E. A. & Beck, M. Fourier ring correlation as a resolution criterion for superresolution microscopy. J. Struct. Biol. 183, 363–367 (2013).
 11.
van Heel, M. & Schatz, M. Fourier shell correlation threshold criteria. J. Struct. Biol. 151, 250–262 (2005).
 12.
Small, A. & Stahlheber, S. Fluorophore localization algorithms for superresolution microscopy. Nat. Methods 11, 267–279 (2014).
 13.
Pawley, J. B. Point, pixels, and gray levels: digitizing image data In: J. B, Pawley (Ed.) Handbook Of Biological Confocal Microscopy. 59–79 Springer US: Boston, MA, 2006.
 14.
Arigovindan, M. et al. Highresolution restoration of 3D structures from widefield images with extreme low signaltonoiseratio. Proc. Natl Acad. Sci. USA 110, 17344–17349 (2013).
 15.
Holmes, T. J. Blind deconvolution of quantumlimited incoherent imagery: maximumlikelihood approach. J. Opt. Soc. Am. A 9, 1052–1061 (1992).
 16.
Fish, D. A., Brinicombe, A. M., Pike, E. R. & Walker, J. G. Blind deconvolution by means of the Richardson–Lucy algorithm. J. Opt. Soc. Am. A 12, 58–65 (1995).
 17.
Dey, N. et al. Richardson–Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution. Microsc. Res. Tech. 69, 260–266 (2006).
 18.
Cannell, M. B., McMorland, A. & Soeller, C . Image enhancement by deconvolution In: J. B, Pawley (Ed.) Handbook of Biological Confocal Microscopy. 488–500 Springer US: Boston, MA, 2006.
 19.
Laasmaa, M., Vendelin, M. & Peterson, P. Application of regularized Richardson–Lucy algorithm for deconvolution of confocal microscopy images. J. Microsc. 243, 124–140 (2011).
 20.
van Kempen, G. M. P. & van Vliet, L. J. Influence of background estimation on the superresolution properties of nonlinear image restoration algorithms. In ThreeDimensional and Multidimensional Microscopy: Image Acquisition and Processing VI, Vol. 3605, 179–189 (International Society for Optics and Photonics, San Jose, CA, USA, 1999).
 21.
Prasad, S. Statisticalinformationbased performance criteria for Richardson–Lucy image deblurring. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 19, 1286–1296 (2002).
 22.
White, R. L. Image restoration using the damped Richardson–Lucy method. In Instrumentation in Astronomy VIII, Vol. 2198, 1342–1348 (International Society for Optics and Photonics, Kailua, Kona, HI, USA, 1994).
 23.
Diebolder, C. A., Faas, F. G. A., Koster, A. J. & Koning, R. I. Conical fourier shell correlation applied to electron tomograms. J. Struct. Biol. 190, 215–223 (2015).
 24.
Vicidomini, G., Bianchini, P. & Diaspro, A. STED superresolved microscopy. Nat. Methods 15, 173–182 (2018).
 25.
TemerinacOtt, M. et al. Multiview deblurring for 3D images from lightsheetbased fluorescence microscopy. IEEE Trans. Image Process. 21, 1863–1873 (2012).
 26.
Culley, S. et al. Quantitative mapping and minimization of superresolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).
 27.
GuizarSicairos, M., Thurman, S. T. & Fienup, J. R. Efficient subpixel image registration algorithms. Opt. Lett. 33, 156–158 (2008).
 28.
Bertero, M., Boccacci, P., Desiderà, G. & Vicidomini, G. Image deblurring with poisson data: from cells to galaxies. Inverse Probl. 25, 123006 (2009).
 29.
Tikhonov, A. N. & Arsenin, V. I. Solutions of IllPosed Problems, Vol. 330 (Winston and Sons, Washinton DC, USA, 1977).
 30.
Richardson, W. H. Bayesianbased iterative method of image restoration. J. Opt. Soc. Am. A 62, 55–59 (1972).
 31.
Lucy, L. B. An iterative technique for the rectification of observed distributions. Astron. J. 79, 745–754 (1974).
 32.
Holmes, T. Maximumlikelihood image restoration adapted for noncoherent optical imaging. J. Opt. Soc. Am. A 5, 666–673 (1988).
 33.
Koho, S., Fazeli, E., Eriksson, J. E. & Hänninen, P. E. Image quality ranking method for microscopy. Sci. Rep. 6, 28962 (2016).
 34.
Koho, S., Deguchi, T. & Hänninen, P. E. A software tool for tomographic axial superresolution in STED microscopy. J. Microsc. 260, 208–218 (2015).
Acknowledgements
The authors thank Dr. Paolo Bianchini and Prof. Colin J. R. Sheppard (Istituto Italiano di Tecnologia) for useful discussion, and Elena Tcarenkova (University of Turku) for help with obtaining the Abberior STED images.
Author information
Affiliations
Contributions
S.K., G.T., M.C., and G.V. conceived the idea. S.K. and G.V. planned the studies and the experiments. S.K. wrote the software and performed the majority of the experiments. T.D. performed the threedimensional confocal experiments. S.K. and G.V. analyzed the data. S.K., G.T., M.C., T.D., A.D. and G.V. participated in extensive discussions during the course of the research. S.K. and G.V. wrote the manuscript. G.T., M.C., T.D., and A.D. participated in revising the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer Review Information: Nature Communications thanks Niccolo Banterle, Ryan McGorty and other anonymous reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Koho, S., Tortarolo, G., Castello, M. et al. Fourier ring correlation simplifies image restoration in fluorescence microscopy. Nat Commun 10, 3103 (2019). https://doi.org/10.1038/s4146701911024z
Received:
Accepted:
Published:
Further reading

Genetically encoded sensors enable micro and nanoscopic decoding of transmission in healthy and diseased brains
Molecular Psychiatry (2021)

Distributed Optimization for Nonrigid NanoTomography
IEEE Transactions on Computational Imaging (2021)

A bidirectional scanning method for scanning transmission Xray microscopy
Journal of Synchrotron Radiation (2021)

FRCQE: a robust and comparable 3D microscopy image quality metric for cleared organoids
Bioinformatics (2021)

Improving axial resolution in Structured Illumination Microscopy using deep learning
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences (2021)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.