Abstract
For several centuries, farfield optical microscopy has remained a key instrument in many scientific disciplines, including physical, chemical, and biomedical research. Nonetheless, farfield imaging has many limitations: the spatial resolution is controlled by the diffraction of light, and the imaging speed follows the Nyquist–Shannon sampling theorem. The recent development of superresolution techniques has pushed the limits of spatial resolution. However, these methods typically require complicated setups and long acquisition times and are still not applicable to deeptissue bioimaging. Here, we report imaging through an ultrathin fibre probe with a spatial resolution beyond the Abbe limit and a temporal resolution beyond the Nyquist limit simultaneously in a simple and compact setup. We use the random nature of mode coupling in a multimode fibre, the sparsity constraint and compressive sensing reconstruction. The new approach of superresolution endomicroscopy does not use any specific properties of the fluorescent label, such as depletion or stochastic activation of the molecular fluorescent state, and therefore can be used for labelfree imaging. We demonstrate a spatial resolution more than 2 times better than the diffraction limit and an imaging speed 20 times faster than the Nyquist limit. The proposed approach can significantly expand the realm of the application of nanoscopy for bioimaging.
Introduction
Optical techniques have long been recognized as indispensable tools for bioimaging. Modern microscopy demonstrates a drive towards miniaturization caused by the need to access deep tissues in vivo^{1}. Miniaturized endomicroscopy provides large depth penetration and is not limited by the interior of a hollow organ or cavity of the body. The recent emergence of spatial wavefront shaping^{2,3} has allowed a conventional stepindex multimode (MM) fibre to be utilized as an ultrathin aberrationfree imaging probe^{4,5,6,7}. The numerical aperture (NA) of MM fibres approaches 0.9^{8,9}, paving the way towards highresolution but still diffractionlimited imaging. Currently, minimally invasive endoscopes are widely used in neuroscience for in vivo deep brain imaging^{10,11,12}, as well as in clinical studies to assist in detecting cancers, to prescribe the right drugs and to monitor treatment response^{13,14}. The ongoing challenge towards highquality minimally invasive deeptissue imaging calls for a new solution that combines enhancement in the spatial and temporal resolutions with a footprint reduction.
Recent years have witnessed the development of superresolution farfield fluorescence microscopy that allows the diffractionlimited resolution to be surpassed, unveiling processes at the nanoscale level^{15}. Structured illumination microscopy utilizes spatial modulation of the fluorescence emission with patterned illumination^{16}. However, it yields a resolution improved by only a factor of two. Stochastic optical reconstruction microscopy and photoactivation localization microscopy (PALM) are based on the stochastic switching on of individual molecules at different times^{17,18}. Stimulated emission depletion (STED) microscopy increases the resolution through shrinking the pointspread function (PSF) by depleting the fluorescence emission in the periphery of the diffractionlimited spot^{19}. These techniques have specific requirements for fluorescent labels.
Parallel efforts aim to improve the imaging speed^{20}. An imaging approach with a millisecond time resolution is required to study fast processes, such as action potentials and communication in neuronal networks. Not surprisingly, advanced methods of diffractionunlimited resolution are timeconsuming, and the tradeoff between the spatial and the temporal resolution affects all superresolution techniques. Most approaches require pointbypoint scanning, and consequently, the acquisition speed is limited by the Nyquist–Shannon sampling theorem^{21}.
Achieving superresolution in vivo deep in living tissue is extremely challenging due to the limited optical access, optical aberrations, and low imaging speed. The maximum demonstrated imaging depth for superresolution is still only 120 μm below the tissue surface^{22}. Methods to enhance the resolution of multimode fibrebased imaging are rapidly being developed^{23,24}. However, it is still a great challenge to incorporate these cuttingedge superresolution technologies into a fibre format.
Here, we propose and experimentally demonstrate fluorescence imaging through a thin MM fibre probe with a spatial resolution beyond the Abbe limit and a temporal resolution beyond the Nyquist limit, meaning that superresolution and superspeed are achieved at the same time. The proposed approach does not put any specific requirements on the fluorescent labels except the brightness and can be used with any standard dye at a reasonable concentration. Moreover, overcoming the Nyquist limit paves the way towards new designs of flexible superresolution imaging probes. We exploit the random nature of mode coupling in an MM fibre, the sparsity constraint inherent to any natural image and compressive sensing integrated into a compact fibre probe. The new approach potentially allows minimally invasive deeptissue imaging with a spatial resolution more than 2 times better than the diffraction limit and an imaging speed 20 times faster than is required by the Nyquist–Shannon limit, opening a new avenue for in vivo ultrafast nanoscale imaging.
Results
Superresolution image reconstruction via compressive sensing
The Abbe diffraction limit^{25} is often considered the theoretical limit of the spatial resolution achievable by a conventional optical system. Nevertheless, it was shown that it is possible to computationally resolve infinitesimally small details^{26}. Unfortunately, the existing methods for bandwidth extrapolation^{27,28,29} are known to be extremely sensitive to noise. As a result, the diffraction limit remains a practical resolution boundary of a simple imaging system^{26,30}.
Recently, compressive sensing—a novel imaging paradigm robust to noise in the measured data—has emerged^{31,32}. Compressive sensing enables signal acquisition with a large reduction in sampling for signals that have a sparse representation, vastly reducing the number of measurements that need to be done^{33,34,35,36}. It exploits the fact that many natural signals are sparse on some basis. Compressive sensing can be used to significantly reduce the number of measurements in endomicroscopy applications^{37,38}. Remarkably, compressive sensing serves as an effective method for rejecting certain common types of noise, which typically do not have sparse transforms. As a result, superresolution imaging by using the sparsity constraint is theoretically feasible^{39,40}. The idea of superresolution imaging via compressive sensing was originally demonstrated by using artificial lowpass filtering to measure data^{41,42}. Recently, the possibility of reconstructing images in the case where the illumination transverse size is larger than the feature size of the sample was shown, though with a resolution of only tens of micrometres^{43,44}.
The spatial resolution of the compressive sensing imaging approach is fundamentally limited by sparsity. In an ideal noisefree oversampling scenario, the resolution improvement can be up to a factor of 1/(2β), where β is the sparsity parameter (the relative fraction of nonzero basis functions occupied by the original image). The sparsity parameter can be expressed as β = S/N, where S is the number of nonzero values and N is the total number of basis functions^{41,42}. However, it is important to note that the information does not have to be sparse in real space: any mathematical basis that is sufficiently uncorrelated with the sampling basis will work. The majority of natural objects are sparse on some basis. In the following sections, we also consider the practical issues limiting the spatial resolution of the proposed superresolution endomicroscopy, such as the noise, stability and orthogonality of the sensing basis, and the number of measurements.
Principles of the optical setup
The idea of MMfibrebased ultrafast superresolution endomicroscopy is sketched in Fig. 1a. The system consists of three main components: a continuouswave (cw) laser source with a scanning system, an MM fibre probe, and a singlepoint detector. Pump light is scanned across the fibre input facet, creating different illumination patterns on the fibre output. The total fluorescent response from the sample is collected by the same fibre probe, propagated back and measured by the singlepoint detector. The “Materials and methods” section and Supplementary Fig. S1 present a complete description of the experimental geometry.
The key component of the proposed approach is a stepindex MM fibre. We used two different fibres to demonstrate the independence of the proposed approach to the probe geometry. The first fibre has a silica core of 50 µm in diameter and an NA = 0.22. The Abbe diffraction limit is λ/(2NA) = 1.21 μm. The second fibre has a core of 105 µm in diameter, an NA = 0.1, and an Abbe diffraction limit of 2.66 μm. Each MM fibre supports approximately 2000 modes^{45} at a wavelength of 532 nm, leading to a Nyquist limit of approximately 4000 measurements for diffractionlimited sampling.
Sensing basis provided by mode scrambling in the multimode fibre
One of the fundamental premises underlying compressive imaging is incoherence. Incoherence means that the sampling method should ensure the lowest correlation between any element of the sensing basis and any element of the representation basis. Consequently, to have freedom of choice of the representation basis, we need a universal sensing basis. Random matrices are known to be largely uncorrelated with any fixed basis, which makes them a very good tool for compressive sensing^{31}. We create a universal basis for compressive endomicroscopy by using the random nature of mode coupling in an MM fibre^{46}. The light field in an MM fibre (stepindex circular waveguide) can be expressed as a finite sum of the socalled HE, EH, TE, and TM modes propagating at their own speeds. Different focal positions on the input fibre facet result in different specklelike mode interferences on the distal facet. An example is presented in Fig. 1a, where three locations of the input beam correspond to three different speckle patterns on the output. A total of 256 patterns are created by scanning a focused input beam over 256 points organized in a square lattice (16 × 16) on the input fibre facet. It is known that the full transmission matrix of a fibre can be measured using a similar scanning procedure^{47}, meaning that this experimental geometry potentially addresses all fibre modes. Alternatively, speckle patterns can be generated by any orthogonal bases. A Hadamard basis or a hexagonal scan lattice can be used to achieve a maximum available number of basis vectors.
First, we characterized the orthogonality of the experimentally generated basis by calculating the correlation coefficient between any two measured speckle patterns (see “Materials and methods”). The results for the first fibre are presented in Fig. 1b. The correlation between two random patterns generated in the MM fibre is close to zero, confirming the high level of orthogonality of our speckle basis. However, focal positions on the fibre facet located very close to each other lead to nonnegligible correlations between the corresponding speckle patterns. As a result, the experimentally created basis is not ideal. We want to emphasize that in all our simulations, we use these experimentally measured nonideal specklebased PSFs and demonstrate that the basis is sufficient for superresolution endomicroscopy.
Second, we analyzed the spatial frequency content of the speckle patterns produced by the MM fibre with a cutoff frequency ν_{cutoff} = 2NA/λ. The spatial power spectra were calculated by a 2D fast Fourier transform (FFT). The average onedimensional spectrum (k_{x} component) is presented in Fig. 1c by the black line. The experimental results are consistent with the theoretical predictions: the measured data at frequencies beyond ν_{cutoff} are close to zero and effectively contain only noise. A zoomedin camera image of the speckle pattern generated by the first MM fibre is presented in Supplementary Fig. S2a. Red arrows indicate the presence of dead pixels on the camera sensor that lead to highfrequency noise. To ensure that we do not introduce any highfrequency components in the simulations of our imaging system, we cut the Fourier spectrum of each measured speckle pattern at ν_{cutoff} (the red line in Fig. 1c). Lowpass filtered speckle patterns were recovered by the inverse 2D FFT. A zoomedin example of the speckle pattern after lowpass filtering is presented in Supplementary Fig. S2b. No more highfrequency components are preserved in the speckle PSF.
Finally, we measured the reproducibility and stability of the speckle patterns while the focused laser beam was switched between different locations on the fibre input. We scanned the focused beam over 100 different locations and recorded 100 different speckle patterns every 3 min. Then, we characterized the reproducibility of the speckle patterns as the crosscorrelation coefficient between the pattern measured at time equal to zero and the pattern measured after the time delay and switching between different focal positions. We found that over 45 min, the average crosscorrelation coefficient was greater than 0.95, confirming a high level of reproducibility. The results are presented in Supplementary Fig. S4.
Simulation with lowpassfiltered speckle patterns: beyond the Abbe limit
As we now demonstrate, exploiting the disorder in MM fibres in combination with the sparsity constraint allows the creation of a compact ultrafast superresolution endomicroscope. As a sample, we use a structure consisting of parallel stripes in the horizontal and vertical directions separated by 390 nm (Fig. 2a) and by 480 nm (Fig. 2b). The feature sizes are far below the diffraction limit of the probe we used (λ/2NA = 1.21 μm). The field of view (FOV) is 6.8 × 6.8 μm^{2}.
First, we simulated conventional point scan imaging using the Gaussian PSF presented in Fig. 2h. The full width at half maximum (FWHM) of the Gaussian PSF corresponds to the diffraction limit of the MM fibre. The spatial power spectrum is represented in Fig. 1c by the filled yellow area. We calculated the overlap integral between the Gaussian PSF and the sample for different positions of the pump beam. A total of 4900 values were calculated for each sample. The resulting images are presented in Fig. 2b, e for 390 and 480 nm feature sizes, respectively. Conventional microscopy does not resolve any subdiffraction details.
Second, we simulated superresolution compressive imaging through an MM fibre. As a sensing basis, we used the experimentally measured and lowpassfiltered speckle patterns. An example is presented in Fig. 2i. We calculated the response for each speckle pattern of our sensing basis. A total of only 256 values were calculated for each sample. The collected data were scaled in such a way as to represent the number of “detected” photons. We added shot noise by generating random numbers from the Poisson distribution specified by the number of photons. First, we analyzed the average number of photons, i.e., 10^{10}. To retrieve a 4900 pixel image from only 256 measurements, we used the reconstruction procedure described in the “Materials and methods” section. The acquired images are presented in Fig. 2c, f for 390 and 480 nm feature sizes, respectively. In contrast to conventional imaging, all subdiffraction details are resolved even if the distance between them is more than 3 times smaller than the diffraction limit. The spatial power spectra of the acquired images resemble those of the original sample and spread way beyond the cutoff frequency of our optical system (ν_{cutoff}); see Supplementary Fig. S3.
Robustness to noise
We analyzed the quality of superresolution compressive endomicroscopy for different noise levels. We used three samples with subdiffraction features separated by 390 nm (see Fig. 2a), 480 nm (see Fig. 2d), and 580 nm. We simulated the superresolution compressive endomicroscopy by using different average numbers of “detected” photons per measurement, ranging from 10^{2} to 10^{10} (leading to a different signaltonoise ratio according to the Poisson statistics). Supplementary Movie 1 demonstrates how the acquired image improves with an increase in the number of detected photons for the 480 nm feature size. For hundreds of photons, the image contains only noise; however, at approximately 10^{5} photons, all four stripes start to be resolved. Finally, from approximately 10^{8} photons, we obtain a good image presented in Fig. 2f.
For each acquired image, we retrieved the reconstruction fidelity by calculating the crosscorrelation coefficient, k_{s}, between the recorded image and the original sample (see “Materials and methods”). The fidelities as a function of the number of photons for the 390, 480, and 580 nm features are presented in Fig. 2g in black, red, and green, respectively. The dashed line in Fig. 2g represents the level at which we can observe subdiffraction details. The graph shows a similar behaviour for different samples, with the main distinction being in the converging quality: the 580 nm features can be imaged with high fidelity, whereas the 390 nm features can barely be resolved.
Simulation with lowpassfiltered speckle patterns: beyond the Nyquist limit
To analyze the superspeed property of our superresolution endomicroscopy, we address the following questions: how many measurements are actually required to resolve the subdiffraction features of our sample and how are the imaging speed and the resolution related to each other? In our simulations, we kept the general structure of interest the same (see Fig. 2d) and increased the FOV and the sparsity by adding zero pixels. The total length of each dimension of the FOV varied from 6.4 to 10.6 μm, and the total number of pixels N_{p} varied from 4356 to 12,100. For each sample size, we simulated the superresolution approach to endomicroscopy for total numbers of experimentally measured speckle patterns, N_{m}, from 50 to 256. We performed simulations for an average number of “detected” photons of 10^{10}. It is important to mention that we always kept the number of measurements far below the Nyquist limit, with a minimum of 0.4% to a maximum of 6% of the information, and the sample contained features 2.5 times smaller than the diffraction limit of the fibre.
For each image, we characterized the reconstruction fidelity by the crosscorrelation coefficient, k_{s}, between the acquired image and the original sample. Figure 3a shows the image quality as a function of the total number of pixels N_{p} and the total number of measurements N_{m}. Low values of the crosscorrelation coefficient (blue) represent images with a diffractionlimited resolution. An example is presented in Fig. 3b (N_{m} = 100 and N_{p} = 6400). A crosscorrelation coefficient of more than approximately 0.7 (light blue) corresponds to an image that still resolves the subdiffraction features of our object. An example is presented in Fig. 3c (N_{m} = 160 and N_{p} = 6400). Yellow (k_{s} close to 1) represents a very good reconstruction: an example for N_{m} = 250 and N_{p} = 6400 is presented in Fig. 3d. The total number of measurements N_{m} that is needed for subdiffraction imaging is linearly proportional to the total number of pixels N_{p} (white dashed line), while N_{m} ≪ N_{p}. The crosssections over two stripes in the images in Fig. 3b–d are presented in Fig. 3e by the blue, red, and green lines, respectively. The crosssection of the image for standard diffractionlimited microscopy is presented in Fig. 3e by the filled yellow area. The recorded images of the sample with a 480 nm feature size as a function of the number of measurements for N_{p} = 8100 are presented in Supplementary Movie 2. We can see the improvement in the image quality as the number of measurements increased.
Experimental demonstration of endomicroscopy beyond the Abbe and Nyquist limits
In the first set of measurements, we demonstrated superresolution and superspeed compressive endomicroscopy through the first MM fibre with NA = 0.22. The details of the imaging procedure are given in the “Materials and methods” section. Similar to our simulations, we performed only 256 measurements and reconstructed the full FOV, consisting of more than 4000 pixels. The result is presented in Fig. 4a, and a zoomedin area of interest is presented in Fig. 4b. We compared the results of the proposed approach with those of the stateoftheart method of MM fibre imaging based on complex wavefront shaping^{6}, where the image is retrieved after pointbypoint scanning of a focused laser beam along the MM fibre output facet. For wavefront shapingbased endomicroscopy, we used the same setup and the same fibre probe to retain the same numerical aperture and the resolution of the system. We did not use the same sample because the small spheres demonstrated pronounced photobleaching of the fluorescent signal that may influence the performance of the second measurement. Instead, we used a similar sample consisting of spheres with exactly the same diameter of 450 nm (standard deviation was 9 nm). The result (an enlarged view of the area of interest) is presented in Fig. 4c.
A crosssection of a fluorescent sphere imaged with the conventional laser scanning endomicroscopy approach is represented in Fig. 4d by blue circles. It follows a Gaussian shape (blue solid line) with an FWHM = 1.34 μm, which is slightly above the diffraction limit due to inaccuracies in the wavefront shaping and the distance between the fibre facet and the sample. The crosssection of a fluorescent sphere imaged with compressive superresolution endomicroscopy is represented in Fig. 4d by open red circles. In contrast to the stateoftheart method, it has very sharp edges and does not follow the conventional Gaussian shape. The FWHM of 0.99 μm was estimated using interpolation. The measured sphere diameter was 20% below the diffraction limit of our optical system. Simultaneously, we used only approximately 5% of the information, meaning the imaging speed was 20 times faster than would be needed for scanningbased imaging with any other technique.
In the second set of measurements, we experimentally analyzed the intrinsic tradeoff between the spatial resolution and the imaging speed. We repeated the compressive endomicroscopy experiment for a different number of measurements varying from 110 to 256. The results are presented in Fig. 4e, where the experimentally measured resolution of the proposed approach for two fluorescent spheres is plotted as a function of the number of measurements. The diffraction limit is presented by the solid green line, and the resolution achieved under the same setup by the stateoftheart endomicroscopy method is presented by the solid blue line. We see that by reducing the number of measurements and keeping the same sample, we reduce the resolution, as was predicted by our simulations.
We note that the number of pixels in the images where the results of the reported and conventional methods are compared (Fig. 4b, c) are not identical. Differences in the pixel number are a direct consequence of differences in the imaging techniques. For each method, the pixel size was chosen as a compromise between the imaging (or reconstruction) speed and the resolution. Here and later, we ensured that the pixel sizes were always several times smaller than the best achievable spatial resolution.
Equivalent measurements were taken for an MM fibre with a core of 105 µm and NA = 0.1. This fibre has a diffraction limit of 2.66 µm and supports approximately the same total number of modes as the first one. Consequently, despite the 2fold larger FOV, we expect a similar resolution improvement according to our simulations. To demonstrate this, we compare the performance of the proposed approach (Fig. 5a) with that of the stateoftheart endomicroscopy (Fig. 5b). We used the same setup and the same sample for both measurements.
Crosssectional plots along the dashed lines in Fig. 5a, b are represented in Fig. 5c by blue triangles and red circles, respectively. The FWHM for the proposed endomicroscopy approach is 1.93 μm, which is approximately 25% below the diffraction limit of our optical system. Simultaneously, we performed only 225 measurements, meaning the imaging speed was 20 times faster than would be needed for scanningbased imaging. The tradeoff between the spatial resolution and the imaging speed is presented in Fig. 5d, where the experimentally measured resolution averaged over four fluorescent spheres is plotted as a function of the total number of measurements. The error bars represent the standard deviation. The diffraction limit is represented by the solid green line.
Improving the resolution by reducing the FOV and the noise level
Our theoretical analysis shows that the resolution depends on the total number of measurements and can be significantly improved by reducing the total FOV for the same number of speckle patterns forming the basis. In the final set of measurements, we experimentally demonstrated this effect by restricting the image reconstruction to a smaller area of the fibre. We used a fibre with NA = 0.1 (diffraction limit of 2.66 μm) and a core diameter of 105 μm, but we reconstructed the image in only a 30 × 30 micron area on the fibre facet, while we made sure that no fluorescent spheres were present outside the reconstructed area. In this manner, we artificially mimicked a fibre with a small FOV. We used larger fluorescent spheres (approximately 1.5 μm in diameter) to reduce the noise level.
The brightfield image of the sample is presented in Fig. 6a. The separation between the left and middle spheres was approximately 1 μm, and the separation between the middle and right spheres was less than 3 μm. We repeated the experimental procedures described in the previous section. The image of the sample obtained with the stateoftheart endomicroscopy based on wavefront shaping is presented in Fig. 6b (zoomedin area of interest). We see that the left and middle spheres are completely unresolvable and that the middle and right spheres are hardly resolved, as expected. We used the same fibre probe and the same sample for the superresolution endomicroscopy experiments. The result (the enlarged area of interest) is presented in Fig. 6c. All three spheres were fully resolved.
Discussion
We have demonstrated that the compressive sensing approach in combination with an MM fibre probe can provide imaging beyond the Abbe and Nyquist limits simultaneously in an endoscopic configuration. Our simulations show that the spatial resolution can be improved by more than 3 times below the diffraction limit and simultaneously that the speed can be made more than 20 times faster than that dictated by the Nyquist limit. Supplementary Movie 3 presents experimentally acquired images as a function of the total number of measurements for superresolution compressive endomicroscopy (left) and for the stateoftheart point scan endomicroscopy based on wavefront shaping (right). The experiments were performed under the same setup, same signal level and same fibre probe. This movie clearly demonstrates the advantages of superresolution compressive endomicroscopy: significantly fewer measurements are needed to obtain a full FOV image with a resolution below the diffraction limit. In the case of the conventional approach, the same number of measurements allows visualization of only a small fraction of the FOV with a diffractionlimited resolution at best.
The best possible spatial resolution of the compressive sensing approach is fundamentally limited by the theoretical resolution enhancement factor, 1/(2β), as was shown by Gazit et al.^{41}. In our simulations and experiments, β was estimated to be 0.06 or smaller, meaning that the theoretical limit of the resolution enhancement factor was 8 or more. However, this limit was derived for an ideal noisefree oversampling scenario. We incorporated practical limitations of the resolution improvements, such as the noise, stability, and reproducibility of the basis vectors, orthogonality of the sampling basis, and imaging speed (number of measurements). We concluded (see Figs. 3, 4e, and 5d) that the imaging speed and the resolution are closely related: the better the resolution we want to obtain, the larger the number of measurements we need to perform. We carefully analyzed the effect of the shot noise (see Fig. 2g and Supplementary Movie 1). The estimation of the fundamental spatial resolution limit as a function of all these parameters is a subject of further investigation. It is important to note that we demonstrated that the spatial resolution of the proposed compressive endomicroscopy is not limited to a 2fold improvement.
We highlight that overcoming the Nyquist limit not only helps to improve the imaging speed but also opens up new avenues for the development of flexible ultrathin superresolution imaging probes. By overcoming the Nyquist limit, we significantly decrease the necessary number of input modes that provide superresolution imaging through an MM fibre. Our results show that only approximately 250 speckle patterns are needed to image a 50 μm diameter area with a resolution below the diffraction limit. This number is more than an order of magnitude smaller than the number of complex wavefronts that are needed for the stateoftheart diffractionlimited endomicroscopybased holographic control of light propagation^{12}. To beat the deleterious effects of MM fibre bending, we need to organize a stable delivery of only 250 different single modes (and not complex wavefronts) to the entry facet of the MM fibre. This process can be carried out in many ways. As an example, a short rigid MM fibre and a long bundle of singlemode fibres that are immune to bending can be fused together, where the excitation light is scanned over the entry facet of the fibre bundle. This will generate stable speckle patterns at the exit facet of the rigid MM fibre. For a fibre probe of 50 μm in diameter and 250 input modes, a fibre bundle with a 3 μm distance between cores provides a stable light delivery suitable for superresolution endomicroscopy. These fibre bundles are commercially available. Implementing a fibre bundle instead of pointbypoint scanning through a microscope objective conceptually does not change the approach.
Remarkably, the new superresolution endomicroscopy approach does not use any specific properties of the fluorescent label typical for many superresolution techniques such as depletion (STED) or stochastic activation of the molecular fluorescent (PALM). The main limiting factor related to the sample is the number of registered photons: the sample should be bright enough to provide more than 10^{4} multiplied by N_{m} photons. The method in its present form is not suitable for singlemolecule imaging because a conventional dye molecule usually emits only 10^{6} photons before being bleached. Superresolution endomicroscopy is ready to be implemented in a variety of fluorescent and labelfree modalities of microscopy.
Materials and methods
Experimental setup
All experiments were performed using standard stepindex MM fibres [Thorlabs, FG050UGA and FG105LVA] with a silica core diameter of 50 µm, NA = 0.22, and of 105 µm, NA = 0.1. The fibres were not straight, and a slight curve was introduced to improve crossmode coupling; the length of each fibre was approximately 20 cm. A detailed sketch of the experimental setup is presented in Supplementary Fig. S1. As a pump, we used the continuouswave linearly polarized second harmonic output of a Nd:YAG laser [Cobolt Samba] with a wavelength of 532 nm and power below 1 mW. As a laser scanning element for endomicroscopy, we used a spatial light modulator: 1920 × 1200 Vialux V4100 digital micromirror device (DMD). This choice stems from the ability of the DMD to perform phase control, which is required for the comparison of our new approach with the stateoftheart wavefrontshapingbased method of highresolution endomicroscopy. The beam was expanded by the telescope to match the area of the DMD. Two lenses were placed in a 4f configuration to image the DMD on the back focal plane of a 20× (NA = 0.4) or 10× (NA = 0.25) objective that coupled the light into the MM fibre. A pinhole in the Fourier plane blocked all but the 1st diffraction order, encoding the desired phase or amplitude distribution. The DMD was used to control the spatial phase (for the stateoftheart endomicroscopy) or the spatial amplitude (for the superresolution compressive endomicroscopy) of a beam on the input facet of the fibre. A 50× (NA = 0.75) or 40× (NA = 0.6) objective was used to collect light from the fibre output (transmission side). The camera imaged the output facet of the MM and was used for precalibration. The same MM fibre was used to collect the total fluorescent response and propagate it in the backward direction to the detection system. We used a dichroic mirror to separate the fluorescent signal and an additional Notch filter to remove the pump light. For detection, a standard avalanche photodiode [Thorlabs, APD410A2] was used in combination with a lownoise preamplifier [Stanford Research].
As a sample, we used fluorescent microparticles (PSFluoRedFi144–2) with a diameter of 450 ± 9 nm and fluorescent microparticles (Nile red) with a diameter of 1.1–1.5 μm. Microparticles were randomly distributed over a glass substrate.
Stateoftheart wavefrontshapingbased endomicroscopy
During the precalibration step, the complex wavefront shaping algorithm^{7} was used to create tightly focused spots on the fibre output. The phase masks corresponding to focal spots at different positions on the output fibre facet were calculated and stored. After the wavefront shaping procedure, the sample was placed as close to the fibre facet as possible without touching. The sample image was acquired in the endoscopy configuration by sequentially applying the recorded phase masks to the DMD and detecting the total fluorescent signal. As a result, the image of the sample was reconstructed pixelbypixel, and approximately 4000 measurements were required to provide an image with diffractionlimited resolution over the full FOV, according to the Nyquist–Shannon sampling theorem.
Compressive endomicroscopy
The precalibration procedure consisted of recording a set of 256 speckle patterns on the output facet of the MM fibre for different positions of the focal spot on the fibre input facet, organized in a 16 × 16 square lattice. The background signal that corresponded to each speckle pattern was also recorded. After the precalibration, the sample was placed as close to the output fibre facet as possible without touching. Superresolution and superspeed compressive endomicroscopy was performed by scanning the focal spot at the input fibre facet in the same way as during the precalibration procedure. The total fluorescent response for each speckle pattern collected by the MM fibre probe was measured by a singlepixel detector. A total of 60–256 measurements were taken in different experiments. To retrieve an image, we used the reconstruction procedure described in the following section.
Image reconstruction
The fundamental problem is to recover a vector \(x \in {\Bbb R}^n\) from data y: y = Ax + z, where A is a known m × n “sensing matrix”, which is used in the measurements, z is an unknown error term, n is the number of pixels, and m is the number of “measurements”. We are interested in the case where m ≪ n, and the distance between two neighbouring pixels Δn is smaller than the Abbe diffraction limit λ/2NA, where λ is the wavelength and NA is the numerical aperture of our optical system.
We used an established procedure as the solution to the l_{1} minimization problem^{48}. Among all objects consistent with the data, we chose the one whose coefficient sequence had a minimal l_{1} norm:
This is a standard linear programming problem that can be solved in polynomial time using one of the many existing software packages. We used the open software algorithm “l_{1} magic” from Stanford.edu^{49}.
Crosscorrelation coefficient
We calculated the correlation coefficient r between any two measured speckle patterns by using the following equation:
where a and b are images of speckles on the output facet of an MM fibre and \(\bar a\) and \(\bar b\) are their means. A total of 256 speckle patterns were created by scanning a focused input beam over 256 points organized in a square lattice (16 × 16) on the input fibre facet. Correlation coefficients were calculated for all pairs of speckle patterns, and the results are presented in Fig. 1b.
We also used the crosscorrelation coefficient to characterize the fidelity of the acquired image. The reconstruction fidelity was calculated as the crosscorrelation coefficient between the recorded image and the original sample. In that case, a and b were the recorded image and the original sample, respectively, and \(\bar a\) and \(\bar b\) were their means.
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Flusberg, B. A. et al. Fiberoptic fluorescence imaging. Nat. Methods 2, 941–950 (2005).
Vellekoop, I. M. & Mosk, A. P. Focusing coherent light through opaque strongly scattering media. Opt. Lett. 32, 2309–2311 (2007).
Vellekoop, I. M., Lagendijk, A. & Mosk, A. P. Exploiting disorder for perfect focusing. Nat. Photonics 4, 320–322 (2010).
Čižmár, T. & Dholakia, K. Shaping the light transmission through a multimode optical fibre: complex transformation analysis and applications in biophotonics. Opt. Express 19, 18871–18884 (2011).
Di Leonardo, R. & Bianchi, S. Hologram transmission through multimode optical fibers. Opt. Express 19, 247–254 (2011).
Čižmár, T. & Dholakia, K. Exploiting multimode waveguides for pure fibrebased imaging. Nat. Commun. 3, 1027 (2012).
Descloux, A., Amitonova, L. V. & Pinkse, P. W. H. Aberrations of the point spread function of a multimode fiber due to partial mode excitation. Opt. Express 24, 18501–18512 (2016).
Amitonova, L. V. et al. Highresolution wavefront shaping with a photonic crystal fiber for multimode fiber imaging. Opt. Lett. 41, 497–500 (2016).
Leite, I. T. et al. Threedimensional holographic optical manipulation through a highnumericalaperture softglass multimode fibre. Nat. Photonics 12, 33–39 (2018).
Ohayon, S. et al. Minimally invasive multimode optical fiber microendoscope for deep brain fluorescence imaging. Biomed. Opt. Express 9, 1492–1509 (2018).
DoroninaAmitonova, L. V. et al. Fiberoptic probes for in vivo depthresolved neuronactivity mapping. J. Biophotonics 3, 660–669 (2010).
Turtaev, S. et al. Highfidelity multimode fibrebased endoscopy for deep brain in vivo imaging. Light: Sci. Appl. 7, 92 (2018).
Sharma, P. et al. Realtime increased detection of neoplastic tissue in Barrett’s esophagus with probebased confocal laser endomicroscopy: final results of an international multicenter, prospective, randomized, controlled trial. Gastrointest. Endoscopy 74, 465–472 (2011).
Kiesslich, R. et al. Local barrier dysfunction identified by confocal laser endomicroscopy predicts relapse in inflammatory bowel disease. Gut 61, 1146–1153 (2012).
Maglione, M. & Sigrist, S. J. Seeing the forest tree by tree: superresolution light microscopy meets the neurosciences. Nat. Neurosci. 16, 790–797 (2013).
Heintzmann, R. & Huser, T. Superresolution structured illumination microscopy. Chem. Rev. 117, 13890–13908 (2017).
Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).
Rust, M. J., Bates, M. & Zhuang, X. W. Subdiffractionlimit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–796 (2006).
Hell, S. W. Farfield optical nanoscopy. Science 316, 1153–1158 (2007).
Dong, S. Y. et al. Resolution doubling with a reduced number of image acquisitions. Biomed. Opt. Express 6, 2946–2952 (2015).
Shannon, C. E. Communication in the presence of noise. Proc. IRE 37, 10–21 (1949).
Urban, N. T. et al. STED nanoscopy of actin dynamics in synapses deep inside living brain slices. Biophys. J. 101, 1277–1284 (2011).
Mahalati, R. N., Gu, R. Y. & Kahn, J. M. Resolution limits for imaging through multimode fiber. Opt. Express 21, 1656–1668 (2013).
Laporte, G. P. J. et al. Enhanced resolution in a multimode fiber imaging system. Opt. Express 23, 27484–27493 (2015).
Abbe, E. Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung. Arch. Mikroskop. Anat. 9, 413–468 (1873).
Goodman, J. W. Introduction to Fourier Optics. 3rd edn. (Roberts and Company Publishers: Greenwoood Village, 2005).
Harris, J. L. Diffraction and resolving power. J. Opt. Soc. Am. 54, 931–936 (1964).
Barnes, C. W. Object restoration in a diffractionlimited imaging system. J. Opt. Soc. Am. 56, 575–578 (1966).
Gerchberg, R. W. Superresolution through error energy reduction. Opt. Acta: Int. J. Opt. 21, 709–720 (1974).
Sementilli, P. J., Hunt, B. R. & Nadar, M. S. Analysis of the limit to superresolution in incoherent imaging. J. Opt. Soc. Am. A 10, 2265–2276 (1993).
Candes, E. J. & Wakin, M. B. An Introduction to compressive sampling. IEEE Signal Process. Mag. 25, 21–30 (2008).
Davenport, M. A. et al. Introduction to compressed sensing. Compressed Sensing: Theory and Applications. (Cambridge University Press, Cambridge, 2012).
Candès, E. J., Romberg, J. K. & Tao, T. Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59, 1207–1223 (2006).
Donoho, D. L. Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006).
Duarte, M. F. et al. Singlepixel imaging via compressive sampling. IEEE Signal Process. Mag. 25, 83–91 (2008).
Katz, O., Bromberg, Y. & Silberberg, Y. Compressive ghost imaging. Appl. Phys. Lett. 95, 131110 (2009).
Amitonova, L. V. & De Boer, J. F. Compressive imaging through a multimode fiber. Opt. Lett. 43, 5427–5430 (2018).
Shin, J., Bosworth, B. T. & Foster, M. A. Compressive fluorescence imaging using a multicore fiber and spatially dependent scattering. Opt. Lett. 42, 109–112 (2017).
Donoho, D. L. Superresolution via sparsity constraints. SIAM J. Math. Anal. 23, 1309–1331 (1992).
Yang, J. C. et al. Image superresolution via sparse representation. IEEE Trans. Image Process. 19, 2861–2873 (2010).
Gazit, S. et al. Superresolution and reconstruction of sparse subwavelength images. Opt. Express 17, 23920–23946 (2009).
Shechtman, Y. et al. Superresolution and reconstruction of sparse images carried by incoherent light. Opt. Lett. 35, 1148–1150 (2010).
Gong, W. L. & Han, S. S. Highresolution farfield ghost imaging via sparsity constraint. Sci. Rep. 5, 9280 (2015).
Gong, W. L. Highresolution pseudoinverse ghost imaging. Photonics Res. 3, 234–237 (2015).
Snyder, A. W. & Love, J. D. Optical Waveguide Theory. (Chapman and Hall, London, 1983).
Xiong, W. et al. Principal modes in multimode fibers: exploring the crossover from weak to strong mode coupling. Opt. Express 25, 2709–2724 (2017).
Plöschner, M., Tyc, T. & Čižmár, T. Seeing through chaos in multimode fibres. Nat. Photonics 9, 529–535 (2015).
Candes, E. J., Romberg, J. & Tao, T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489–509 (2006).
l1Magic. https://statweb.stanford.edu/~candes/software/l1magic/ (2005).
Acknowledgements
The authors acknowledge the financial support of the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) (VENI 15872).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare the following competing interests. A patent application, i.e., No. 2021837 (Imaging through multimode fibers), based on these results has been filed by VU University, with L.V.A. and J.F.d.B. as inventors.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Amitonova, L.V., de Boer, J.F. Endomicroscopy beyond the Abbe and Nyquist limits. Light Sci Appl 9, 81 (2020). https://doi.org/10.1038/s413770200308x
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1038/s413770200308x
This article is cited by

Sweptsource multimode fiber imaging
Scientific Reports (2023)

Farfield superresolution ghost imaging with a deep neural network constraint
Light: Science & Applications (2022)

Phonon imaging in 3D with a fibre probe
Light: Science & Applications (2021)