Introduction

In optical imaging, the resolution is ideally limited by the light diffraction characteristics, limiting the maximal resolution to the wavelength scale. However, in many cases, imperfections of the optical elements or the medium inhomogeneity introduce phase deformations that deteriorate the image resolution from its diffraction-limited value. For instance, in biological microscopic imaging, the setup is usually optimized for a homogeneous watery medium, while this is rarely the case when imaging a real sample1.

One of the ways to improve the compromised resolution is by deconvolution. In the linear imaging model, there is a convolution relationship between the optical setup impulse response (point spread function, PSF) and the object light reflectance function. By evaluating the PSF, the image resolution can be improved by solving the inverse problem2 either directly or through an iterative process of improving the PSF from its initial conjecture3, 4, 5, 6. As even in the iterative cases, the PSF initial value strongly influences the final result, it is generally crucial to obtain a good estimate of the PSF. The second approach toward improving resolution is by measurement and correction of the imaging wavefront error (adaptive optics), a method that provides very good results for astronomical and retinal images7, 8, 9. However, when the reference beam also passes through phase-deforming media or in the cases of rapidly changing or significant phase errors, the resolution does not reach the diffraction-limited value. In these cases, other methods are usually combined for further improvement10, 11, 12, and PSF estimation approaches could be advantageous. Taken together, distortion-resilient methods for PSF estimation could provide a powerful addition to the imaging toolbox.

Here we present a new approach termed PEPSI for estimating the transverse PSF when imaging through phase-deforming media, which is based on measuring the deformation of a speckle pattern illuminating a fluorescent object. The motivation for using a speckle pattern to measure the deformations arises from their random-phase distribution: these random phases yield a pattern whose statistics are not affected by optical aberrations13 (see Supplementary Section 1 for a detailed explanation and demonstration of this aberration invariance). Therefore, by illuminating the object with the speckle pattern, an objective measure of the phase errors of the imaging path can be obtained, irrespective of the illumination path’s phase aberrations. As the speckle pattern is uniformly distributed throughout the field of view, PEPSI estimates the average PSF of the entire field of view from a single-pattern projection, and is thus suitable for dealing with dynamic-phase aberrations (not requiring prior calibrations or acquisition of multiple images14, 15). Moreover, in cases where the aberrations are non-isoplanatic, the local PSF for selected areas in the field of view can be obtained by the same analysis on those areas. To obtain a practical PSF estimation technique, we derive our solution using Wiener-type minimization, leading to a Wiener-type deconvolution algorithm for the PSF estimation (requiring only a change in the input functions) that we validate using both simulations and experiments. Finally, we show that PEPSI can be used to improve the resolution of the (phase-deformed) images using a common iterative maximum likelihood (ML)-based image reconstruction algorithm16, 17.

Materials and methods

The image of an object illuminated by a speckle pattern can be expressed using the isoplanatic imaging equation

where i is the recorded intensity at the image plane, p is the PSF, o is the object's fluorescent density spatial distribution, s is the illumination speckle pattern at the object plane and n is the noise. To simplify the following derivation, we assume that the optical setup is non-magnifying.

Having a random-phase distribution, the speckle pattern projected onto the object does not suffer from phase deformations13 (see Supplementary Section 1 for a detailed explanation and demonstration). However, the phase of the emitted wavefront is no longer random—it is made of fluorescent emission of light from a set of spots on the object plane, which are a result of constructive interference. Therefore, the incoherent emitted light is affected by the imaging-phase deformations.

To analyze the statistical properties of each single-speckle illumination, we first divide the image into M sub-frame images, each containing a single speckle (see Supplementary Section 2, for more details on the frame division process). The l-th image can be written as

Here and below, we assume that the imaging aberrations are not strong enough to cause an overlap of adjacent speckles. This assumption makes the convolution values in Equation (2) equal to zero at the edges of each l-th image. Thus, we can assume that each one-speckle image is independent of the others, and discretization of the large image into M one-speckle images is possible.

Averaging this set of one-speckle images yields

where symbolizes the discrete averaging of the function fl over all M images. As we have assumed that there is no overlap between two images of adjacent speckles, we can change the order of the integration (by the convolution operator) and averaging in Equation (3). In addition, as ol characterizes the object’s local fluorescent properties and sl characterizes the illumination, their covariance is negligible and an averaged image can be expressed as

The terms in this equation are as follows:

is the average over M different object locations of the local fluorescent densities. As the positions of the projected speckles onto the object plane are random and M >> 1, this term will be averaged out into a constant factor (denoted next by k).

is the average speckle intensity. This term is determined solely by the illumination and not by the object.

is the average noise function, obtained by averaging over M different one-speckle images.

By substituting the object’s average fluorescent density constant (k) and dividing Equation (4) by a normalization factor (c) that constrains the total intensity of to be equal to one, we get

This equation describes the averaged degraded speckle image, made of a large number of one-speckle images; it cannot be solved analytically, because it contains three unknown variables (the PSF, the averaged and factorized projected speckle, and the averaged noise). To overcome this hurdle, we replace with a computer simulation of the averaged projected speckle (denoted by sc) generated using prior knowledge on the setup’s illumination exit pupil size and magnification (see Supplementary Section 2 for implementation details). By also constraining the total intensity of sc to be equal to one, we must introduce a new function (denoted by z) that balances the average noise term and keeps the equality valid:

The equation is valid as long as and shows that the quality of the PSF estimation will be inversely proportional to the size of z: when the normalized noise term is negligible with respect to the convolution term, z is also negligible and the PSF estimation precision will improve. Indeed, the equation noise term is mostly much smaller than the convolution term because it is an average over a large number of one-speckle images (M»1) with no spatial correlation in the position of noise and because it is divided by the total intensity of the signal (c). However, at the very peripheral parts of the PSF where , the contribution of z cannot be neglected, and we expect a reduction in the precision of the PSF estimation in those areas.

To estimate the PSF, we solve an error minimization problem. Owing to the similarity of our problem to Wiener deconvolution, we adopt a similar solution and define the following least mean square error function (see Supplementary Section 3 for more details on the similarity and derivation):

where P is the Fourier transform of the PSF, G is a Fourier filter for estimating the PSF, I is the Fourier transform of the normalized image of the average speckle (that is, the left term in Equation (6)), and E denotes expectation. By minimizing the error function, we obtain the following expression for G:

where Sc is the Fourier transform of sc and N' is the Fourier transform of . This solution for the minimization problem is similar to a Wiener filter2, except that here the prior knowledge is regarding the object and not the PSF. To estimate the PSF, we multiply G by I (the Fourier transform of ), and then inverse-transform the result.

Equation (8) generally requires knowledge regarding the spectral content of n′ and of the PSF, but not having access to these values, we substitute their ratio (|N'|2/c2|P|2) by a small constant value. Substituting the explicit form of N′ into Equation (8) highlights the inaccuracy that z yields at the peripheral parts of the estimated PSF, whereas in the substantial parts the division by |c|2 in the denominator of Equation (8) yields a PSF estimation that is quite robust to high noise levels (see the next section). Consequently, we used throughout this study (simulations and experiments) a single constant ratio (0.05) found by requiring that the total intensity of the estimated PSF will be as close to unity as possible (0.95; that is, no absorption of light in the imaging part). Substituting smaller values for this ratio will yield artifacts, similar to the case of inverse-filtering process in the presence of noise.

Owing to the similarity of PEPSI estimation to Wiener deconvolution, the implementation of this method is quite simple. By having a degraded image of the speckles-illuminated object and a simulation of a clear speckle pattern, the PSF can be estimated by Wiener deconvolution. This is done by substituting the averaged degraded speckle from the image and the averaged clear speckle from the simulation into Wiener deconvolution algorithm. Note that the averaged clear speckle is substituted in the Wiener algorithm instead of the initial guess of the PSF.

Results and discussion

Validation by simulation

To test the new method performance and characteristics, we began by studying simulations of optically blurred speckle images. In each simulation, a random speckle pattern (Figure 1a) was projected onto an object (Figure 1b), the result was convolved with a blurring PSF (Figure 1c), and the final image (Figure 1d) was also corrupted by Poisson noise.

Figure 1
figure 1

Validating the method by simulation. A speckle pattern (a) is multiplied by an object fluorescent density pattern (b, scale bar=32 μm) convolved with the blurring PSF (c) and corrupted by Poisson noise (noise to signal ratio=0.014), forming the degraded image of a speckle pattern-illuminated object (d). By finding the average speckle of (d) and of a speckle pattern (similar to a), we estimate the PSF (e, scale bar=1.3 μm), which is qualitatively very similar to the main lobe of the blurring PSF (f). (g) The RMSE of the PSF estimates versus the main lobe of the blurring PSF for different noise levels in d. The insets show the estimated PSFs for the corresponding noise levels. In this analysis, the blurring PSF values were normalized to the [0 1] range (right inset, same as f), and all estimated PSFs were normalized correspondingly, and share the same color map. The noise to signal ratio was calculated by dividing the standard deviation of noise by the average signal. RMSE, root mean square error.

Full size image

To estimate the PSF, we used a simulated speckle pattern to find sc (different from the pattern in Figure 1a, to avoid getting a trivial solution). The PSF estimate (Figure 1e) is obtained by substituting the Fourier transform of sc into Equation (8) and multiplying the resultant filter frequency response (G) by the Fourier transform of , obtained from the degraded image (Figure 1d). For easy comparison, Figure 1f shows the main lobe of Figure 1c with the same color map as Figure 1e. Note that in the very corners of the estimated PSF, there are negative values—these values were expected from Equation (6) owing to the low values of sc in the periphery.

We repeated this simulation for different Poisson noise levels corrupting the image of Figure 1d. The results (Figure 1g) show the relatively strong robustness of the estimation procedure for noise to signal ratios of up to 0.25.

Experimental validation

An optical apparatus (Figure 2a) was built in order to experimentally validate the method and to demonstrate resolution improvement (next sub-section). To project a speckle pattern onto the object, a laser-illuminated diffuser (λ=532 nm) was imaged on the object plane. Both speckle projection and imaging were then performed through a phase-deforming plate. In order to yield the computer-generated clear speckle pattern, sc, knowledge of the magnification power of the setup as well as the size of the illumination exit pupil was needed13, but no prior calibration was required (see Supplementary Section 2 for additional details on the optical setup).

Figure 2
figure 2

Experimental validation of PEPSI. (a) Experimental setup used for speckle pattern projection and imaging. The phase deformation was introduced by a phase-deforming plate, and the speckle pattern was formed by a diffuser and projected onto a thin film of fluorescent dye. (b–d) The degraded speckle pattern, a computer-generated speckle pattern (based on the size of the illumination exit pupil and optical setup's magnification) and the estimated PSF (scale bar=3.6 μm), respectively. (e) By convolving the 'clear' image of a 4 μm diameter fluorescent bead (diffuser and phase plate removed) with the estimated PSF, a degraded image is obtained (f) that can be compared with its optically acquired degraded image (g): the RMSE between the degraded bead images is only 0.039, showing their good agreement. To avoid discrepancy due to noise, we applied an adaptive noise removal filter to the bead's clear (e, left) and optically acquired degraded (g) images, which mostly removed noise in the bead's surroundings without significantly changing it. (h) Three more cases validating the estimated PSF's accuracy: the optically acquired degraded images of 6 μm beads (central column) closely match images obtained by convolving their clear bead images (left column) by the estimated PSF (right column, also lists the images' RMSE). Noise was reduced by averaging 50 frames (here without a noise removal filter). RMSE, root mean square error.

Full size image

PEPSI’s accuracy was tested by an indirect two-step method, because the numerical aperture of the setup was too small for collecting enough light to directly measure the PSF using a sub-diffraction-limited object. In the first step, we estimated the PSF by projecting speckle on a thin film of a fluorescent dye. Figure 2b–2d shows the degraded image of the speckles, the corresponding computer-generated ‘clear’ speckle pattern and the estimated PSF, respectively. In the second step, we took two images of a fluorescent bead, one with the same aberrations as in the PSF estimation process and one clear image of the bead (without aberrations). Both images were taken by removing the diffuser from the illumination part of the optical setup, while for the clear image of the bead, the phase-deforming plate was also removed. Finally, by convolving the clear image of the bead with the estimated PSF (Figure 2e), we obtained a degraded image (Figure 2f) that can be compared with the optically acquired degraded image acquired in the second step (Figure 2g). A potentially significant drawback in this comparison arises from the noise that is an additive term in the optically acquired degraded bead (Equation (1)), while in the PEPSI-based image, it is convolved with the PSF. To reduce the effect of this discrepancy, we reduced the noise level of the optically acquired bead images by applying an adaptive noise removal filter (Figure 2e, left, and Figure 2g), which mostly reduces noise in the bead surroundings, while hardly changing its shape or intensity18 (see also Supplementary Section 4 for filter details).

Comparison between the degraded bead image formed by convolution with the estimated PSF (Figure 2f) and the optically acquired bead (Figure 2g) shows a qualitative resemblance, and the root mean square error between the two is 0.039. Figure 2h shows three more examples of validating PEPSI, each for different phase deformations. In these examples, the noise was decreased by averaging a large number of frames and not by a de-noising filter. In all of these examples, the optically degraded bead (central column) is qualitatively similar to the bead-convolved degraded image (right column) in shape, orientation and intensity levels. To demonstrate the level of phase deformation, the bead clear images are also shown (left column, note the different color bars).

Application: resolution enhancement of images

As an application for PEPSI, we applied the estimated PSF toward resolution enhancement of degraded images using both simulated and experimental data. The image reconstruction was based on an iterative ML algorithm19 that uses PEPSI-estimated PSF as an initial input. By using this familiar algorithm that constrains the final PSF to be positive, we avoid any possible degradation in the reconstruction process owing to the expected negative values in the estimated periphery of the PSF.

Figure 3a shows a degraded image of the object used for the simulations in Figure 1, convolved with the blurring PSF from Figure 1c and corrupted by additional noise. The reconstructed image is shown in Figure 3b, showing a major improvement of the resolution when using speckles PSF estimation as initial input for the ML algorithm. This is further demonstrated by examining the averaged radial power spectrum of the two images showing power enhancement at high resolvable frequencies in the reconstructed image (Figure 3c; for example, at 0.28 cycles per μm frequency the reconstructed image has five times higher power).

Figure 3
figure 3

Resolution improvement of simulated data by deconvolution based on PEPSI’s PSF. (a) Degraded image of the object in Figure 1. The image is obtained by convolving the PSF (Figure 1c) with the object (Figure 1b) and adding Poisson noise (noise to signal ratio=0.014). Scale bar=32 μm. (b) Image reconstruction based on the speckles PSF estimation (Figure 1e) as initial input for iterative ML algorithm. (c) Averaged radial power spectrum of the reconstructed image (red) versus the degraded one (cyan), showing a resolution increase of the reconstructed image by having higher power at high resolvable frequencies.

Full size image

Moreover, in Supplementary Section 5, we demonstrate that there is only a small improvement in the ML-based image reconstruction versus a non-constrained algorithm (Wiener deconvolution), which is due to minor peripheral changes of the ML final PSF estimate from the high-quality initial estimate provided by PEPSI.

To experimentally demonstrate resolution enhancement, we used the same optical setup from 'Experimental validation' in Results and Discussion section (Figure 2a), and a silicon wafer with fluorescently stained rectangles of horizontal and vertical protruding bars, and polygonal domains formed outside of the bars areas when the dye dried. We observed a significant improvement in the image resolution when using the PEPSI-based image deconvolution (Figure 4c) in comparison with the degraded image (Figure 4b), and even in comparison with the unaberrated object image (Figure 4a, hereafter referred to as ‘the clear image’).

Figure 4
figure 4

Experimental demonstration of resolution enhancement. (a) The object without addition of optical aberrations (scale bar=56 μm). (b) A degraded image of the bars. (c) The reconstructed image. (d) The PSF that was used for image reconstruction (scale bar=5.4 μm). (e) Comparison of the intensity profile, marked by dashed line in a–c. Only the PEPSI-based deconvolution resolves the first bar—the first local minimum, circled. (f) The PEPSI-based ML image reconstruction, obtained by PSF estimation of the intrinsic aberrations of the optical setup. The initial image for the reconstruction was a and speckles were measured without the phase plate. All images share the same gray scale. (g) The PSF used for image reconstruction in f. Scale bar=3.6 μm.

Full size image

Figure 4d shows the final PSF that was used for the image reconstruction, having only small difference from the PEPSI-based PSF (comparison can be found in Supplementary Section 6). The resolution improvement is also evident when examining the intensity profile of a set of five bars (Figure 4e, profile marked by dashed line in Figure 4a–4c). Although only four bars (local minima) are identified in the non-speckles-based images, in the reconstructed image all five bars are seen. Moreover, the width of the first minimum dip in the speckles deconvolution graph corresponds well to the width of the bars, as measured by a microscope (7 μm). All graphs were normalized to share the same minimum intensity, which was set to zero, and they are presented as fraction of the overall maximal intensity value. The standard deviation of the speckle-based intensity profile is 1.33 times larger than in the degraded image profile.

In comparing the intensity profile of the speckle-based deconvolution with the profile of the clear image (Figure 4e), one can see that the former gives better results than the latter. We believe that this is due to inherent aberrations in our apparatus, even before the addition of the phase-deforming plate, which were also identified and corrected by the PEPSI method. This assertion is further backed by the following reasons: (a) In the computer simulation test (Figure 3a and 3b), we started with an image with no aberrations and degraded it with a broad PSF, but the reconstruction results did not provide a better image than the initial image (Figure 1b). (b) When using the method on images that were taken with our optical setup, but without additional phase deformation (no phase plate), we also observed an improvement (Figure 4f compared with Figure 4a; the PSF resulting from ML is shown in Figure 4g).

Next, we applied the PEPSI-based deconvolution to a more complex object: a chicken breast slice stained by the fluorophore Rhodamine B. Figure 5a–5c presents the clear (without phase plate), degraded and the speckle-based reconstructed images. The resolution improvement is also evident when examining the power spectrum of the degraded and reconstructed images (Figure 5d).

Figure 5
figure 5

Demonstration of resolution improvement in a fluorophore-stained chicken breast tissue. (a) Object without addition of aberrations (scale bar=50 μm). (b) Degraded image. (c) Reconstructed image using PEPSI. (d) Corresponding power spectra of the degraded (top) and reconstructed (bottom) images, showing increase in the resolution (higher spatial frequencies, see dashed lines with corresponding minimal distances).

Full size image

Finally, we also validated PEPSI-based resolution improvement in an inverted microscope (Nikon TE2000-U, Nikon Corporation, Japan) where optical aberrations were deliberately introduced in front of the sample. In this experiment, the sample consisting of a US Air Force resolution target stained by a fluorescent dye and imaged through a 10 × objective (Figure 6a) was severely aberrated by a drop of agar on the target (Figure 6b). The speckle pattern was projected through one of the microscope’s camera ports, and the averaged projected ‘clear’ speckle (Sc) was experimentally pre-calibrated rather than calculated (an additional optional way for obtaining Sc in cases where the optical aberrations originate at the sample and with the advantage of not requiring any prior knowledge of the optical setup parameters; see Supplementary Section 2 for comparison between the two estimated PSFs). The results of the PEPSI-based image reconstruction clearly demonstrate a major resolution and contrast improvement both visually in the image itself (Figure 6c and 6d) and quantitatively at the high frequencies of the averaged radial power spectrum (Figure 6e), and also in the target’s vertical intensity profile (upper right panel of Figure 6e, profile marked by dashed line in Figure 6d).

Figure 6
figure 6

Applying PEPSI in a microscope. (a) Clear image of a fluorescently dyed section of a US Air Force resolution target, obtained by a 10 × objective. Scale bar=40 μm (b) Degraded image of the target following introduction of an aberrating agar drop. (c) PEPSI-based reconstructed image. (d) Enlarged regions of the deformed and reconstructed images. (e) Averaged radial power spectrum of the reconstructed target in c (red) versus the corresponding degraded one (cyan) showing resolution improvement by higher power at high resolvable frequencies. The inset shows the corresponding vertical intensity profile of target’s element number one (horizontal bars, see dashed lines in d) for both images.

Full size image

Conclusions

In this study, we demonstrated a calibration-free method for PSF estimation of an imaging setup that suffers from unknown-phase deformations. When projecting a speckle pattern onto the object plane, we obtain a pattern that is only affected by the phase distortions of the imaging path while being unaffected by illumination path distortions. Therefore, by a single projection of the speckle pattern, an estimate of field of view-averaged PSF is obtained. Moreover, since the method is locally true for areas where the PSF is space invariant (Equation (1)), the same analysis can, in principle, be performed for local estimation on sections of the field of view, thereby making it suitable in cases where the PSF varies across the field of view1, 9, 10, 20, 21, 22, 23, 24, 25, 26. Consequently, the lack of prior calibration as well as the ability to estimate the PSF at a single projection of speckle pattern, both for wide and local fields of view, enable the use of this approach in cases of dynamic, random and non-isoplanatic phase aberrations.

In addition to validating the PEPSI method by simulations and experiments, we also demonstrated the feasibility of the PSF estimation for resolution enhancement via deconvolution. The reconstruction is based on a numerical solution of an inverse problem, which does not necessarily have a unique solution, so we did not use the reconstructed image to analyze the precision of our PSF estimation, but merely show it as an application for the method. The study was limited to reconstruction by a common ML algorithm16, 19. We believe that by using PEPSI with other deconvolution algorithms, an even better reconstruction can be achieved.

Some of the more promising applications for this method could come from its adaptation to wide-field microscopic imaging through phase-deforming media1, 22, 23, 24 (for example Figure 6), such as in fluorescent retinal micro-imaging26, 27. The optical simplicity of this method, which only requires an additional diffuser in the microscope illumination path, enables easy integration into essentially any fluorescence microscope. The use of a PEPSI can also solve the dual-pass problem that sometimes exists in wavefront analysis of imaging setups11, 23, 28, that is, to eliminate the effect of phase deformation of the incoming wavefront. The method could also be used as an initial adjustment in adaptive optics setups, for a faster response time when a large field of view is required1, 22, 23, 24, 29, 30. Moreover, the reduction of aberrations with this approach can also be obtained whenever speckle patterns are being projected for other aspects of imaging such as super-resolution19, 31, 32, 33 or optical depth sectioning34.

The current study was limited to the cases of imaging fluorescent objects in order to avoid coherent coupling between the illumination and emitted light. However, PEPSI can be used for imaging non-fluorescent objects as long as the illumination light coherence is significantly reduced when emitted from the object; for example, it can potentially be used for imaging objects consisting of sub-wavelength scale particles.