Main

One of the ultimate dreams of endoscopists would be real-time high-resolution endoscopy that combines in vivo imaging and therapeutic intervention, resulting in an endoscopic diagnosis that matches an in vitro pathological diagnosis1,2. Considerable progress has recently been made in endoscopy. Endocytoscopy, confocal laser endomicroscopy and other techniques have been developed to enable in vivo cellular imaging3,4,5,6,7,8. Meanwhile, super-resolution microscopy enables spatial resolution at much smaller scales than the subcellular level, leading to breakthroughs in the fields of biology and life sciences9,10,11,12,13. However, super-resolution microscopy often requires cumbersome optics and is thus challenging to implement in a narrow channel. In fact, one critical missing ability is to conduct robust in vivo nano-endoscopy, which is yet to be satisfactorily established. One promising strategy is to employ thin multimode fibres (MMFs) as minimally invasive probes using wavefront shaping. The mode density of MMFs is 2–3 orders of magnitude higher than those of traditional endoscopes of the same diameter14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30, but this technology has two critical limitations: the operational inflexibility due to the strong transmission dependence on the fibre’s configuration20,21,22 and the diffraction-limited resolution due to limited numerical aperture (NA)15. The dynamic change and interference between different deformation states of the fibre is a major obstacle to rapid disorder measurement, tracking and image correction. Many past studies have attempted to compensate for the effect of fibre’s dynamic movement with strategies that mainly fall into two categories (Supplementary Table 1): tracking based on slow spatial area detection23,24,31,32,33 or no tracking20,21,25,28. It is challenging to conduct imaging under bending and various operating conditions. On the other hand, complex engineering of the fibre’s distal facet is often required23,24,31,32,33. With these accumulating challenges, techniques have not yet demonstrated sufficiently high-resolution imaging under practical operational conditions due to the lack of fast tracking and super-resolution methods.

Here we propose an MMF-based spatial-frequency tracking adaptive beacon light-field encoded (STABLE) endoscope imaging (Fig. 1). The system can be used in ultrathin and ultralong channels and improves the resolution to a subdiffraction-limited value of 250 nm (λ/3NA) with high stability and robustness against the movement and deformation of the probe. STABLE endoscopy and self-designed white-light endoscopy (WLE) are integrated to achieve centimetre-to-nanometre cross-scale imaging of various samples inside narrow channels. The WLE can provide a 120° wide field-of-view (FOV) to locate areas of interests. STABLE endoscopy can provide subcellular resolution for precise analysis. In our experiments we apply the STABLE endoscopy to a number of clinical samples, including test targets, pathological samples, nanomaterials and living mice. Our study illustrates the potential to greatly expand the flexibility of STABLE endoscopy, making a substantial leap towards applications in the life sciences, biology, industrial inspection, as well as clinical diagnoses.

Fig. 1: Principle of the STABLE system.
figure 1

a, STABLE can detect and track disorders in the MMFs introduced by practical movements and manufacturing defects, thus retaining high-SNR imaging. The input light is composed of all spatial-frequency modes in designed polarization states, which are spread over the Poincaré sphere. S1 to S3 represent the three polarization components. The spatial-frequency beacon is located at the Fourier plane of the fibre proximal end. kx and ky are positions in Fourier space. b, Diagram of spatial-frequency beacon (left) and conventional spatial speckle (right) monitoring. In spatial-frequency beacon monitoring, the modulated wavefront with designed polarization is incident into the MMF, reflected by the distal facet of the fibre, and then returned to the proximal facet again, focusing on a single pixel in the spatial-frequency domain as spatial-frequency beacon. In conventional monitoring, the modulated wavefront with the same polarization is incident to the MMF, reflected by distal facet engineered, and then returned to the proximal facet again, imaging on array detection. c, Dimension reduction retrieval concept map. STABLE is a highly compressive method that takes advantage of the fact that the MMF is a radial cylindrically symmetric waveguide. This compresses the problem from a high-order problem to a much lower-order problem. d, Endoscope prototype. The black dotted line shows the probe. WLE, white-light endoscopy; LS, light source of WLE; MMF, multimode fibre.

Full size image

Principle of STABLE

Perturbations and coupling between spatial or polarization modes—or both—are inevitable and non-negligible, and are thus obstacles in the way of using MMF imaging for in vivo applications34,35,36. They have limited the MMF imaging applications under practical movement for approximately decades. We introduce the STABLE concept to eliminate the influence of unpredictable couplings and fluctuations (Fig. 1a). Taking endoscopy applications as an example, eliminating the influence of the dynamic changes requires searching the transmission matrix (TM) based on the light reflection of the closed space immediately during endoscope movement. Tracking was previously based on speckle measurement of the whole image with an engineered distal facet23,24,31,32,33, and several minutes were required to find the correct TM after the change24 (Supplementary Table 1).

In STABLE, we designed a full-vector modulation (FVM) incident wavefront that focuses the distal-end Fresnel reflection light on a single-pixel detector in the spatial-frequency domain (that is, the Fourier plane of the proximal facet of the fibre), forming the spatial-frequency beacon (Fig. 1b, left). The beacon transfers from spatial speckle tracking to single-pixel spatial-frequency beacon tracking. Its strength reflects the correlation between the current bending states and the pre-calibrated TMs. By analysing the simulation of light propagation in the fibre, and the relationship between the reflection light and spatial-frequency beacon, we find that the spatial-frequency beacon intensity detected by the single-pixel detector is directly related to the chaotic phase, polarization and amplitude, all of which are relative changes that accompanies fibre deformation (Supplementary Fig. 3). The correct fibre state can be found only when the intensity value is at its maximum. It is therefore possible to search for the correct TM under deformation by tracking the beacon.

STABLE is a highly dimensionality reduction single-pixel tracking method that takes advantage of the fact that the MMF is a radial cylindrically symmetric waveguide. Through simulations and experiments we found that the spatial-frequency beacon intensity maintains a high intensity (that is, more than half of the initial value, Supplementary Figs. 4–7 and Supplementary Note 1) under rotation, limited translation and small twist of the fibres (Fig. 2a). Measuring and updating the TM can be skipped in these scenarios (Supplementary Video 3). This compresses the problem to a much lower order (Fig. 1c). We can therefore just pre-measure the TM under different bending curvature and build a relative complete database for calibration. Furthermore, the database searching process based on single-pixel beacon tracking could eliminate the iteration of TM, which is always the bottleneck in speeding the compensation. The time taken to find the correct TM from a database (about 20–100 states) is in the order of 10−3 s. The search speed is enhanced by a factor of 104 compared with the spatial domain tracking (Fig. 2b,c, Supplementary Table 1, Supplementary Video 4 and Supplementary Fig. 10).

Fig. 2: Deformation disorder tracking and compensation in MMF imaging.
figure 2

a, Types of fibre deformations. Orange circle, bending deformation; blue circle, other three deformations (translation of a fixed curvature fibre, rotation around a central axis and twisting deformation). b, Spatial-frequency beacon intensity variation with bending at different locations in the spatial-frequency domain. The angular coordinate is the spatial-frequency, and v is the maximum spatial-frequency of the MMF. The radial coordinate represents curvature from 0 m−1 to 5 m−1. The normalized intensity represents the ratio of spatial-frequency beacon intensity in straight state to that in different curvature. c, The feedback of different probes under bending. The location of the spatial-frequency beacon is indicated by the red dashed line in b. d, Retrieval of the bending state based on spatial-frequency beacon tracking. e, The state change and imaging recovery. The first row represents different fibre states, whereas the second and third rows represent the tracking and recovery processes of MMF imaging of atherosclerotic and lung pathology slides under the corresponding states, respectively. Scale bars, 25 μm.

Full size image

STABLE process

STABLE can be divided into database creation and imaging recovery processes (Supplementary Fig. 8). The experimental system is shown in Supplementary Fig. 17.

In the creation of the STABLE database, customized for each endoscopist, we pre-measured and created a highly effective database containing a set of TMs and reflection matrices (RMs) under different bending states (Supplementary Fig. 8). The spatial-frequency beacon is generated using RMs. It describes the linear input–output response of an MMF at the same port. The RM can be obtained using the Onsager reciprocity relationship and end-facet Fresnel reflection37, as shown in equation (1):

$${\mathrm{RM}}={T}^{T} F T$$
(1)

Here, TT represents the transpose of TM and F is the end-facet Fresnel reflection matrix.

Dual feedback loops are introduced into the STABLE imaging process. The spatial-frequency beacon provides fast encoded feedback in the inner loop to track the deformation state (loop 1 in Supplementary Fig. 8b). The imaging sharpness acts as encoded feedback in the outer loop to track the focus plane (loop 2 in Supplementary Fig. 8b). Their cooperation enables highly stable imaging. The system includes an optical module, a probe and an electrical module (Fig. 1d). The transfer functions of loops 1 and 2 are as follows:

$${T}^{t+1}={T}^{t}+(1-G({T}^{t})){T}^{t}\,{\rm{for}}\,{\rm{loop}}1$$
(2)
$${H}^{t+1}={H}^{t}+(1-O({H}^{t})) {H}^{t}\,{\rm{for}}\,{\rm{loop}}2$$
(3)

Here, G and O represent the normalized intensity of the spatial-frequency beacon and image sharpness, respectively; \(H=\exp (-id\sqrt{{k}_{0}^{2}+{l}^{2}+{k}^{2}})\) is the propagation factor, which is a quadratic phase resulting in imaging plane axial movement; k and l are positions in Fourier space; d is propagation distance; k0 = 2π/λ is the wavenumber in free space; and t is the index of retrieval.

Under the framework of STABLE, the key to high-quality volumetric imaging lies in the efficient generation of beacons and the formation of high-signal-to-noise ratio (SNR) scanning points in three-dimensional space. Volumetric imaging is difficult with the conventional MMF imaging method due to the extreme complexity in alleviating mode and polarization dispersion during the generation of the TM stack at different axial distances (Supplementary Fig. 20). In this study, the mode dispersion, polarization-mode dispersion and mode-dependent losses occurring along the fibre length are resolved via FVM (equation (4)):

$${\psi }_{\mathrm{in}}^{u,v}={T}^{\mathrm{Tik}}H{\psi }_{\mathrm{out}}^{k,l},$$
(4)

where TTik is the Tikhonov regularization of T; \({\psi }_{\mathrm{in}}^{u,v}\) is the mode distribution of the input field; \({\psi }_{\mathrm{out}}^{k,l}\) is the distribution of the output field; and μ, ν and k, l represent the two-dimensional spatial-frequency factors at the two ends of the fibre.

Simulations and experiments indicate that FVM can convert the MMF function to a highly efficient programmable full-vector holographic device (Supplementary Fig. 20g–j and Supplementary Note 4). A high-contrast full-vector TM stack can therefore be obtained with a single measurement. In practical applications, the combination of TM within the acquired stack with image sharpness O in loop 2 can fix the preferable imaging planes within the depth range for stable volumetric imaging.

Furthermore, super-resolution imaging can be obtained by using FVM to perform a spatial-frequency shift super-resolution method (here, differential fluorescence microscopy) on the sample. Conventional differential fluorescence microscopy loses both resolution and the contrast in MMF. We combine the STABLE and FVM to overcome the mode dispersion and dynamic bending to have high-fidelity donut pattern. Finally, super-resolution imaging under practical movement is achieved.

In the following sections we show the versatility of STABLE in MMF through several demos. In each demonstration, STABLE combines different technical variants to highlight selected imaging effects.

Result

What makes STABLE-based endoscopy compelling is its ability to provide stable imaging under bending, folding and even twisting shapes, without the need to detect or rebuild the distal facet. This makes the diagnosis more robust and accurate for in vivo imaging inside narrow lumen or in the tissues of solid organs, as desired in clinical applications. We have the MMF with a diverse range of shapes to mimic delicate and difficult-to-reach organs, such as bronchioles and coronary. For each fibre 20–100 deformation states were measured, and we show three deformation states to demonstrate the performance. Detailed information on atherosclerotic and lung cell is preserved well under various shapes (scanning frame rate = 2.5 fps, 96 × 96-pixel frame). As shown in Fig. 2d,e and Supplementary Video 1, images can be recovered within about 1 s under operational movements, which includes tracking time, scanning time and synchronization time (as shown in Supplementary Fig. 10). Precalculating and pre-uploading were adopted to eliminate the calculating and upload time. It should be noticed that in previous studies, the bottleneck for image recovery under dynamic deformation is the slow tracking time (to the order of serval minutes). That makes it impossible to recover the image under reasonable movement; complex image recovery has therefore not yet been realized experimentally. The spatial-frequency beacon tracking speeds the tracking time by four orders of magnitude, from several minutes to submilliseconds, offering the ability to track dynamic deformation; however, there still exists other challenges in fast imaging. Synchronization is currently the major limit to recovery, and it can be improved by field programmable gate array in the future studies. STABLE is also applicable to large bending curvature (Supplementary Video 2). To explore the limit performance of STABLE, we measured the tolerance range of varying the curvature radius and twisting angle (Supplementary Fig. 35); the maximum bending curvature and twisting angle are 80 m−1 and 130°, respectively.

To further verify the stability of the proposed scheme over a long time frame, we used STABLE to simultaneously calibrate the optical path drift (Supplementary Note 1) and compensate for disorder. Over a time span of five days, we captured the different areas of the small intestine of a sheep and combined the images captured at different timestamps (Fig. 3b). In a wide FOV image of up to 1 mm (Fig. 3b–d), the goblet cells, absorptive cells and central lacteal were clearly visible. These images are consistent with the conventional wide-field micrographs (Supplementary Fig. 13). For the same focal point, stable focusing was achieved over one week (Fig. 3e), indicating that the system can work stably and consistently in the order of weeks. This aspect is crucial for long-term in situ observations in the life sciences. Moreover, local temperature changes and liquid environments inevitably affect the refractive index of the fibre and cause spherical aberration, which often brings in information distortion. By using STABLE, robust results can be obtained in such complex environments (Fig. 3a). Broad bean anther, onion pollen, basswood segregation and lily pollen were placed underwater to capture clear fluorescent images using a 50 m MMF (Fig. 3j–m). The corresponding wide-field microscope images are shown in Fig. 3f–i. The growth of ice crystals in a capillary was also recorded via MMF imaging (Supplementary Fig. 15). Without the STABLE process, the focal plane shifts rapidly owing to the temperature change at the front of the fibre. This temperature change was well compensated by our STABLE system. The front view of the ice growth inside a 500 μm channel is clearly shown. The observation of ice growth in capillary may give valuable inspiration in the investigation of plant antifreeze and microchemical reaction.

Fig. 3: Complex environments tracking and compensation in MMF imaging.
figure 3

a, Concept diagram of endoscopy in complex environment (long-term, liquid, long cavity, temperature variations). b, Image of a sheep’s small intestine captured via MMF endoscopy. The types of cells are annotated on the image. The images are captured at different times over 5 days and can be stitched to a large FOV image without distortion, indicating the long-term stability of the system. This image corresponds to the dashed box in Supplementary Fig. 13a. Scale bar, 150 μm. c, Resolution of each image before stitching. The horizontal axis indicates the resolution of the images, whereas the vertical axis represents the percentage of the total number of images. d, The SNR of each image before stitching. The vertical axis indicates the percentage of the total number of images, whereas the horizontal axis represents the SNR of the images. e, Focus intensity variation during a long period of time. f–i, Wide-field microscopic images of broad bean anther (f), onion pollen (g), basswood segregation (h) and lily pollen samples (i). Scale bar, 40 μm. j–m, The corresponding 50 m MMF images of the samples in f–i placed underwater. Scale bar, 40 μm. n, High-contrast projection through 200 m MMF. o, Imaging of fluorescent lens paper with 200 m MMF. Scale bar, 15 μm. p, Imaging of CdSe nanowire with 200 m MMF. Scale bar, 15 μm.

Source data

Full size image

To validate the stability of the method in long cavity imaging, we used a 200-m-long commercially available MMF to obtain high-contrast projection on an optical table (as shown in Fig. 3n). Using the same optical fibre, we performed fluorescence imaging on a fluorescent lens paper and a CdSe nanowire (Fig. 3o,p). This result showed that STABLE has potential applications that extend from biology to materials science. It should be noted that small fluctuations in the environments will seriously disturb the TM as the fibre length increases. We can have stable imaging under reasonable movement for a fibre length of less than 50 m. When the fibre length is longer than 50 m, STABLE can compensate for the disturbance caused by the environmental fluctuation, but imaging under practical movement has not yet been achieved.

We next performed volumetric imaging of a sample with 5-µm-diameter fluorescent spheres distributed randomly in a glue matrix. A single MMF with a core diameter of 105 μm and a 0.22 NA was used to excite the fluorescent spheres and collect the fluorescence (Fig. 4g). The fluorescent images showed that the spheres could be clearly resolved at imaging depths reaching up to about 200 µm from the fibre facet (details are given in Supplementary Fig. 20f). Moreover, the structure of the compound eye can be clearly distinguished in the volumetric fluorescence images (Fig. 4h). The proposed method can therefore enable volumetric imaging without mechanical motion.

Fig. 4: Three-dimensional super-resolution imaging and cross-scale imaging characteristics of STABLE endoscopy.
figure 4

a, Schematic of STABLE endoscopy for ultrastable slim endoluminal imaging. b, The probe of integrated endoscope. c, Pig oesophagus imaging through WLE. Scale bar, 1.8 cm. The white dashed box is the observation area for MMF imaging. d, Pig oesophagus fluorescence imaging by a wide-field microscope. Scale bar, 25 μm. e, Pig oesophagus fluorescence imaging with MMFs. Scale bar, 25 μm. f, Wide FOV navigation in the bronchial model. g, Fluorescence imaging for a fluorescent microsphere at different focal planes in the glue matrix. Z represents the distance from the end face of the fibre. h, Butterfly compound eye was captured at different focal planes. Scale bars, 28 μm. i, The deformation state change and super-resolution imaging recovery. The illustrations in the insets represent different fibre states. The left and right star resolution target images represent the super-resolution imaging without and with STABLE fluorescence emission differences, respectively. Scale bar, 8 μm. j–m, Resolution analysis of nano-endoscope. j, Comparisons of the MMF images of a USAF 1951 test target with conventional and STABLE techniques. The resolution of STABLE is 1.5-times improved. Scale bar, 3 μm. k, Comparison of the MMF images of fluorescent microsphere with conventional and STABLE techniques. The resolution of STABLE is 1.5-times improved. Scale bar, 0.5 μm. The miniature picture is a local magnified image of the dotted box in the big picture. l, Normalized intensity profiles along the white dashed lines in j. The solid black curve represents conventional imaging, whereas the solid red curve represents STABLE imaging. m, Normalized intensity profiles along lines indicated by the white arrows in k.

Full size image

Based on disorder fast tracking and FVM, spatial-frequency control can then be further used to break the diffraction limit to about λ/3NA (Supplementary Note 5). Super-resolution imaging in fibre practical motion is also verified (Fig. 4i). Two samples are used to verify the resolution of STABLE endoscopy (Fig. 4j–m). The intensity profiles along the line indicated by white arrows clearly show 780 nm linewidth in USAF 1951 test target (with NA = 0.22 MMF) and 250 nm in fluorescent microspheres sample (with effective NA = 0.65 MMF). This implies that the resolution of the STABLE endoscopy is approximately λ/3NA, which is an improvement of 1.5 times beyond its diffraction limit. Furthermore, axial subdiffraction-limited imaging can be obtained by STABLE (as shown in Supplementary Fig. 27).

Finally, for clinical analysis, localizing a target area in a complex scene is necessary. To bridge the gap between macroscopic and microscopic morphology, wide-FOV and high-resolution 3D imaging capabilities must be combined (Fig. 4a). We integrated the MMF with a self-developed WLE by placing the MMF in the biopsy channel of WLE. An integrated endoscope was used to observe a sample in the bronchus model and pig oesophagus (Supplementary Video 5 and Fig. 4b–f). Fluorescent markers in the outer bottom bronchial tubes were observed synchronously using the MMF and WLE. The WLE offered a wide FOV of approximately 120°, enabling global observation of the oesophagus and bronchus (Supplementary Fig. 16). It also functioned as a guiding tool for the MMF probe toward the diagnosis area. The MMF could thus provide submicrometre 3D imaging of the diagnosis area. The cell is clearly observed in the MMF imaging.

With STABLE, the in vivo high-resolution imaging of the gastrointestinal tracts of mice was first demonstrated for the first time (Fig. 5a). Under the guidance of WLE, MMF (NA = 0.22) enables imaging of living cells following spraying topical fluorescent stains inside the tongue, oesophagus, colon, stomach and small intestine (as shown in Fig. 5 and Supplementary Fig. 14). Based on the STABLE, subdiffraction-limited imaging showing different aspects of the small intestinal mucosa is analysed at a subcellular level (Fig. 5d,e). In a healthy colon, epithelial nuclei surround regular, round crypt openings in transepithelial sections (Fig. 5h). Crucially, with subsurface imaging, the spatial resolution degrades due to scattering and aberrations in biological tissues (Supplementary Fig. 29). This impact can be eliminated using MMF as a needle puncture to provide large-depth sampling of the submucosal (Supplementary Fig. 14f and Supplementary Video 6). Compared with those in healthy colonic tissues, the cells are specifically enriched in colon cancer tissue and the crypt structure disappeared (Fig. 5j). They are consistent with histology (Supplementary Fig. 14j,k).

Fig. 5: STABLE endoscopy for in vivo gastrointestinal imaging.
figure 5

STABLE can provide resolution similar to conventional wide-field microscopy; however, the latter cannot be used in the lumen due to its large size and rigid operation. a, Schematic of in vivo imaging. b,c, Stitching FOV fluorescence imaging through STABLE endoscopy (b) and wide-field microscopy (c) at the tongue. Scale bar, 30 μm. d, Subdiffraction-limited fluorescence image of the small intestinal mucosa. Scale bar, 5 μm. e, Normalized intensity profiles along lines indicated by white arrow in d. f, WLE images showing regions of healthy colon tissue. Scale bar, 4 mm. g, Histological sections stained with haematoxylin–eosin from healthy colonic tissue is shown. Scale bar, 300 μm. h, Stitching fluorescence image of the healthy colon (regions 1 and 2 in f) by STABLE endoscopy. Scale bar, 55 μm. i, Colon cancer imaging by STABLE endoscopy. j, Fluorescence image of the colon cancer tissue acquired using STABLE endoscopy. Scale bar, 20 μm.

Full size image

Discussion

Herein we demonstrate a straightforward practical solution for high-resolution micro-endoscopy called STABLE endoscopy, which, based on spatial-frequency-domain coding tracking, can provide scalable bending resistance for most traditional MMF endoscopes without further modification to the fibre. Fundamentally, it solves the general challenge of bending cross-talk in MMF imaging caused by combination of multiple optical physical constraints with simple coding tracking. At present, we can achieve super-resolution imaging under practical movement with a length of less than 50 m. For fibres longer than this, high-resolution imaging was obtained under environmental fluctuations with a fixed position.

As with any imaging system, STABLE imaging involves balancing trade-offs, although it sits on a more optimized trade-off line with a higher performance baseline (Supplementary Note 6). The architecture with the best trade-off should be determined according to specific experimental scenarios. Non-periodic modulation such as super-pixel-based spatial modulation38 can also be applied with STABLE (Supplementary Fig. 22).

STABLE may also be used in a nonlinear propagation regime for high peak power capable of generating coupling distortions between modes; however, certain problems must be addressed: (1) linear TM cannot describe the relationship between input and output light of MMF in a nonlinear process; and (2) new unstable problems may be caused by nonlinear effects. The STABLE process needs to be modified according to the new approach, which requires further investigation in the future.

Finally, higher super-resolution techniques are worth implementing such as stimulated emission depletion in future systems to further improve image resolution in combination with STABLE. We anticipate that STABLE endoscopy will provide an exciting novel modality for endoscopy widely used in basic and applied research in clinical medicine, biology and industry.

Ethics declarations

All in vivo experiments were approved by the local ethical review committee at the Ethics Committee of the Second Affiliated Hospital of Zhejiang University School of Medicine.