Deep optical imaging within complex scattering media

Abstract

Optical imaging has had a central role in elucidating the underlying biological and physiological mechanisms in living specimens owing to its high spatial resolution, molecular specificity and minimal invasiveness. However, its working depth for in vivo imaging is extremely shallow, and thus reactions occurring deep inside living specimens remain out of reach. This problem originates primarily from multiple light scattering caused by the inhomogeneity of tissue obscuring the desired image information. Adaptive optical microscopy, which minimizes the effect of sample-induced aberrations, has to date been the most effective approach to addressing this problem, but its performance has plateaued because it can suppress only lower-order perturbations. To achieve an imaging depth beyond this conventional limit, there is increasing interest in exploiting the physics governing multiple light scattering. New approaches have emerged based on the deterministic measurement and/or control of multiple-scattered waves, rather than their stochastic and statistical treatment. In this Review, we provide an overview of recent developments in this area, with a focus on approaches that achieve a microscopic spatial resolution while remaining useful for in vivo imaging, and discuss their present limitations and future prospects.

Key points

  • Optical microscopy is an indispensable tool in biology and medicine owing to its high spatial resolution, molecular specificity and minimal invasiveness, but it is limited to the interrogation of superficial layers for in vivo imaging.

  • The intensity of single-scattered waves used in conventional imaging decreases exponentially with depth; thus, the imaging depth limits are set by the detector dynamic range and the efficiency of the gating operations.

  • Approaches that make deterministic use of the abundant multiple-scattered (MS) waves have been proposed to enable deep optical imaging while maintaining the microscopic spatial resolving power.

  • Recording and controlling the wavefront of MS waves enables a complex scattering layer to be converted into a focusing lens, leading to the development of an ultrathin endoscope.

  • Acousto-optic interactions and wavefront sensing and/or control are integrated to exploit the large penetration depth of ultrasound and high spatial resolution of optical imaging.

  • Reflection-matrix approaches that record and process all MS waves enable the correction of sample-induced aberrations, exploitation of multiple scattering signals and suppression of multiple scattering noise, allowing for imaging at depths greater than those accessible with conventional confocal and adaptive optics microscopy.

Introduction

Optical microscopy is widely used to investigate the mechanisms underlying various physical and biological phenomena and to enable minimally invasive disease diagnosis and treatment. Unlike other imaging modalities that rely on high-energy sources, such as X-rays and electron beams, this approach does little harm to samples of interest, while producing valuable molecular information at a subcellular spatial resolution. Optical microscopy is thus used in various biological and medical applications, including the investigation of protein interactions and signalling pathways1, monitoring of the dynamics of neural circuits at the cellular level and at timescales ranging from milliseconds to months2, and the non-invasive imaging of cancer for intraoperative surgical guidance and screening3. However, a disadvantage of optical imaging is that it is extremely near-sighted because the light waves are deflected multiple times when propagating through complex scattering media. Owing to this multiple light scattering, the spatial resolving power of optical imaging degrades significantly even at depths of just a few hundred micrometres when used in bioimaging (Fig. 1a). Although there have been advances in the past two decades in improving the resolving power beyond the diffraction limit, the difficulties associated with overcoming the higher-order perturbations caused by multiple light scattering have slowed progress in increasing imaging depth4,5,6,7. Thus, advanced applications of optical imaging in the fields of life science and medicine have yet to be realized.

Fig. 1: Effects of multiple light scattering on imaging depth.
figure1

a | Imaging depth of various imaging modalities: stimulated emission depletion (STED), photoactivated localization microscopy (PALM), super-resolution optical fluctuation imaging (SOFI), optical coherence tomography (OCT), optical coherence microscopy (OCM), multiphoton microscopy (MPM), ultrasound-modulated optical tomography (UOT) and photoacoustic tomography (PAT). b | An input wave (with a signal \({{\mathcal{E}}}_{{\rm{i}}}\) and a wave vector ki) incident on a scattering medium with a target object at depth z0 undergoes scattering events, producing various types of scattered wave with the same output wave vector ko for the given ki. \({{\mathcal{E}}}_{{\rm{S}}{\rm{S}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) denotes the signal of a single-scattered (SS) wave (where τ0 is the flight time). \({{\mathcal{E}}}_{{\rm{T}}{\rm{M}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) and \({{\mathcal{E}}}_{{\rm{B}}{\rm{M}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) are multiple-scattered (MS) waves that do and do not interact with the target object, respectively, while having the same flight time as the SS wave. By contrast, \({{\mathcal{E}}}_{{\rm{D}}{\rm{C}}}({{\bf{k}}}_{{\rm{o}}}\,;\tau \ne {\tau }_{0})\) are MS waves that have different flight times to that of the SS wave. c | The relative intensities as a function of z0 for an SS wave (\({{\mathcal{E}}}_{{\rm{S}}{\rm{S}}}\)), time-gated MS waves (\({{\mathcal{E}}}_{{\rm{T}}{\rm{M}}}\) and \({{\mathcal{E}}}_{{\rm{B}}{\rm{M}}}\)) and other MS waves in typical cases in which the transport mean free path, lt, is ~10 times larger than the scattering mean free path of the medium, ls. The decay lengths for the curves were derived from ref.10. Panel a adapted from ref.149, Springer Nature Limited. Panel b adapted from ref.10, Springer Nature Limited.

To understand the challenges associated with microscopic imaging within complex scattering media, it is important to understand the way that light waves propagate in reflection-mode imaging, which is the relevant modality for in vivo imaging. Consider a target object located at a depth z0 with a plane wave with transverse wave vector ki incident on the medium (Fig. 1b). The propagation direction of the internal wave changes owing to the inhomogeneity of the medium, leading to the wave being scattered through various pathways. The signal of most interest is the single-scattered (SS) wave, often called ballistic light, denoted as \({{\mathcal{E}}}_{{\rm{SS}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) with the output transverse wave vector ko and the flight time τ0 = 2z0/c′, where c′ is the average speed of light in the medium. The SS wave experiences a single scattering event by the target object but no scattering events in the intervening medium. An ideal diffraction-limited object image can be reconstructed with the SS wave because its momentum change, ko − ki, is directly related to the structure of the object. Notably, the intensity of the SS wave is attenuated by a factor of exp(−2z/ls), where ls is the scattering mean free path of the medium (see section I in the Supplementary Information). Owing to this exponential attenuation, the SS wave is obscured by multiple-scattered (MS) waves even at a depth of just a few ls.

Except for the SS wave, the incident wave is fully converted into MS waves. There are several types of MS wave that return from the scattering medium with the same ko as that of the SS wave and are therefore indistinguishable from the SS wave based on their output propagation direction. Of these, two MS waves \({{\mathcal{E}}}_{{\rm{TM}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) and \({{\mathcal{E}}}_{{\rm{BM}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) have the same flight time as the SS wave. Here, \({{\mathcal{E}}}_{{\rm{TM}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) describes an MS wave that interacts with the target object at least once. Its momentum change is not identical to the object’s spatial frequency spectrum owing to scattering events in the complex media but is loosely related to the object’s structure because the deflection angles are 1 rad. This MS wave consists of ‘snake photons’ and is responsible for the finite size of the isoplanatic patch in adaptive optics (AO)8. \({{\mathcal{E}}}_{{\rm{BM}}}({{\bf{k}}}_{{\rm{o}}}\,;{\tau }_{0})\) indicates an MS wave that travels at depths shallower than that of the target object and thus has no interaction with the object. \({{\mathcal{E}}}_{{\rm{TM}}}\) and \({{\mathcal{E}}}_{{\rm{BM}}}\) are stronger than SS waves because their intensity is attenuated exponentially as exp(−2z/l′), where the decay length l′ is longer than ls (refs9,10). It is difficult to estimate l′ because it depends on various factors, such as the specifications of the imaging optics and the properties of the scattering medium. Nevertheless, measurements of l′ (ref.10) and the analytical estimation of the ratio between the SS wave and MS waves11 have provided insight into the interaction between these factors. The remaining MS waves are described by \({{\mathcal{E}}}_{{\rm{DC}}}({{\bf{k}}}_{{\rm{o}}}\,;\tau \ne {\tau }_{0})\) and travel along longer or shorter path lengths than that of the SS wave and thus have different flight times. The relative intensities of the SS wave and various MS waves (Fig. 1c) demonstrate the difficulty in achieving high-resolution deep optical imaging. For example, at a target depth of z0 = 10ls, which corresponds to ~1 mm in biological tissue, the intensity of the SS wave is attenuated by a factor of e–20 with respect to \({{\mathcal{E}}}_{{\rm{DC}}}\).

In the past two decades, various deep optical imaging approaches have been developed to overcome the detrimental effects of complex media on optical imaging. Early attempts at deep optical imaging have been previously reviewed12. The most straightforward approach has been to filter out the MS waves using gating operations. For example, in confocal detection13, the SS waves are selectively collected along the propagation direction, although a considerable portion of the MS waves (\({{\mathcal{E}}}_{{\rm{TM}}}\), \({{\mathcal{E}}}_{{\rm{BM}}}\) and \({{\mathcal{E}}}_{{\rm{DC}}}\)) bypass confocal gating. Temporal gating based on low-coherence interferometry removes \({{\mathcal{E}}}_{{\rm{DC}}}\), but \({{\mathcal{E}}}_{{\rm{TM}}}\) and \({{\mathcal{E}}}_{{\rm{BM}}}\) still contribute to the noise14,15. In fluorescence imaging, \({{\mathcal{E}}}_{{\rm{BM}}}\) and \({{\mathcal{E}}}_{{\rm{DC}}}\) can be removed using spectral filters because their wavelengths differ from those of the fluorescence emissions16. Alternatively, ultrasound waves are focused on the target object to modulate the light waves that interact with it17,18. Selective detection of the modulated wave can eliminate a large fraction of \({{\mathcal{E}}}_{{\rm{DC}}}\). In many cases, multiple gating operations are combined to maximally reject unwanted multiple scattering noise19,20,21. In addition to gating approaches, multiphoton imaging based on nonlinear excitation22,23,24,25,26,27,28 has proved to be effective in increasing the imaging depth (Fig. 1a). Longer-wavelength excitations reduce multiple light scattering, and the nonlinear response of the emission is favourable for excitation localization. However, multiphoton excitation is hampered by the attenuation of the excitation-beam intensity by scattering and aberrations, which is currently the main reason for imaging depth limitations.

Despite these previous efforts, the imaging depth of high-resolution optical microscopy, which has a spatial resolving power close to the ideal diffraction limit, has largely been limited to the superficial layer of biological tissue (<1 mm). This limitation is primarily due to the reliance on the readily obscured SS waves. The detector dynamic range and SS gating efficiency are two major factors in determining the achievable imaging depth in the case of optical coherence imaging10, and the sample-induced aberrations pose an additional limit (Box 1).

In addition to the noise created by the MS waves, the SS wave is prone to perturbation by the scattering medium. The SS wave experiences angle-dependent phase retardations, referred to as sample-induced aberrations, due to the inhomogeneity of the medium. The aberrations cause the focal spot to blur and degrade the peak intensity of the point spread function (PSF; see section II of the Supplementary Information). Sample-induced aberrations are unavoidable in deep optical imaging, and it is difficult to reach the gating limit because these aberrations attenuate the intensity of the SS wave, \({| {{\mathcal{E}}}_{{\rm{SS}}}| }^{2}\), in the image-formation step. Their effect is characterized by the Strehl ratio, which is defined as the ratio of the peak intensity of the PSF in an aberrated system to that of the ideal diffraction limit (the dashed blue line in the figure in Box 1 indicates the depth limit with sample-induced aberrations). AO has been adopted to remove sample-induced aberrations (see section II of the Supplementary Information)8,29 and raise the effective imaging depth limit (Box 1 figure). However, most AO microscopy methods are either prone to strong multiple scattering noise or are unable to correct high-order aberrations.

A direct approach to increasing imaging depth is to devise new types of gating mechanism, such as acoustic gate spacing30, that more effectively filter out MS waves. Another approach is to exploit the \({{\mathcal{E}}}_{{\rm{TM}}}\) waves to form an image, which raises the signal-to-noise ratio of the image, suggesting that the maximum achievable imaging depth for a given dynamic range of a detector can be improved (Box 1). Diffuse optical tomography and photoacoustic imaging are good examples of this approach, because their imaging depth is much greater than that of high-resolution optical microscopy owing to the use of MS waves31,32,33. However, their spatial resolution is worse than the diffraction limit because the stochastic treatment of MS signals results in the loss of spatial resolution. In overcoming these limitations, the main question is whether \({{\mathcal{E}}}_{{\rm{TM}}}\) waves that interact with the target object can be deterministically used to increase the imaging depth while maintaining the ideal diffraction-limited spatial resolution.

About a decade ago, it was experimentally demonstrated that MS waves can be controlled deterministically using wavefront shaping to form a sharp focus and to construct an object image even through a highly scattering medium34,35. Although this approach is not yet directly applicable to the optical imaging of targets embedded within a scattering medium, it has led to the development of various deep-optical-imaging approaches that perhaps would not otherwise have been conceivable. For example, concepts related to mesoscopic scattering theory36,37,38,39, such as short- and long-range correlations and open eigenchannels, have been developed for optical focusing and imaging40. The scattering matrix formalism (see section III of the Supplementary Information) has also been revisited as a more complete description of the light–medium interaction, opening new avenues for the manipulation of MS waves for optical focusing and imaging in scattering media41. Although these developments have not yet enabled the ultimate goal of the full deterministic use of MS waves for in vivo deep microscopic optical imaging, advances are being made towards the better use of MS waves and increasing the practicality of the proposed approaches. In this Review, we describe the principles of several key experimental methodologies that both attain microscopic spatial resolution and can potentially be applied to in vivo imaging. Because this research area is still in its developmental stage, we conclude by discussing present challenges and future prospects.

Imaging through a scattering medium

In this section, we discuss experimental approaches that make deterministic use of MS waves transmitted through a scattering layer with finite thickness. In these approaches, it is assumed that a detector can be placed on the plane behind the scattering layer for measuring the transmitted MS waves. We first discuss optical wavefront shaping techniques for focusing transmitted MS waves, along with their applications for imaging through scattering media. Subsequently, we discuss generalized approaches based on the measurement of the transmission matrix (TM) of a scattering layer and their use for endoscopic reflectance and fluorescence imaging.

Focusing multiple-scattered waves

In AO microscopy, wavefront-shaping has been used to control weakly scattered waves only, owing to the practical constraints in bioimaging, such as the deficiency of ideal guide stars and the requirement for a short image acquisition time. Control of MS waves induced by a large number of scattering events has been considered possible in principle and previously studied in acoustics42,43,44 and microwaves45,46. However, this control was realized only comparatively recently in optics. Two noteworthy examples are the demonstration of optical focusing behind a highly scattering layer34 by iterative feedback control of the incident wavefront, a process often referred to as point optimization, and the demonstration of phase conjugation through thick biological tissues35.

In the input–output relationship of a linear system, vt = Tuin (where T is the transmission matrix of the system; see section III of the Supplementary Information), the output wave vt can be controlled by adjusting the incident wave uin in the point-optimization method. A random speckle pattern is generated when a strongly scattering medium is illuminated with a plane wave (Fig. 2a). The point-optimization method adjusts the phases of the incident wavefront by using a wavefront-shaping device such that the light waves arriving at a predefined point on the opposite side of the medium constructively interfere (Fig. 2b). Consequently, the intensity at the target point can be substantially increased relative to the surrounding region. Although optimization is conducted for a single point, the overall transmission can be increased47 (Fig. 2ce), because the point-optimization process increases the contribution of eigenchannels with high transmittance48.

Fig. 2: Point optimization by wavefront shaping and its imaging applications.
figure2

a,b | Speckled transmission of a plane wave through a strongly scattering medium (panel a). A focus can be generated by shaping the wavefront of the incident wave (panel b). c,d | Speckled transmission image of a plane wave through a scattering medium (panel c) and the corresponding transmission image through a scattering medium after point optimization (panel d). e | Intensity-summed profiles in the vertical direction of panels c and d. The grey dashed line is the transmission of the plane wave in panel c, and the blue solid line is the transmission of the point-optimized wavefront in panel d. f | Instantaneous image delivery through a scattering medium. R is the distance between the object and the scattering medium, and f is the focal length of the lens. The thin scattering medium in conjunction with the spatial light modulator (SLM) serves as a ‘scattering lens’ with focal length R. The SLM shapes the wavefront of the scattered light from the object such that the image of the object is refocused on a camera. The band-pass (BP) filter rejects uncorrected wavelengths. g,h | Images of a point produced with a flat SLM (panel g) and after optimization of the SLM pattern (panel h). The inset shows the phase pattern written on the SLM for optimization. i,j | Transmitted images of an object (the letter A, shown in the inset) before (panel i) and after (panel j) point optimization. Panels a and b adapted with permission from ref.34, OSA Publishing. Panels ce adapted with permission from ref.47, APS. Panels fj adapted from ref.49, Springer Nature Limited.

One application based on point optimization is instantaneous image delivery though a scattering layer49. Point optimization through a spatial light modulator (SLM; Fig. 2f) enables the recovery of the image not only at the optimized point but also at adjacent points owing to the optical memory effect37,38,50. In one example, point optimization was used to refocus scattered light from a point source49 (Fig. 2g,h). With the optimized SLM correction, an object image that was invisible before the point optimization (Fig. 2i) could be clearly resolved (Fig. 2j). The point-optimization method has also been used to generate a focal spot sharper than the diffraction-limited size determined by the numerical aperture of a focusing lens51. Placing a scattering layer between the focusing lens and a detector increases the deflection angles, which enables the generation of a focal spot that is smaller than the diffraction limit set by a conventional lens. Indeed, using a scattering layer with a high refractive index, a sub-100-nm focal spot was generated52. Light has also been focused at a subwavelength scale by controlling the near field generated in a metamaterial53, and the control of surface plasmon polaritons has been demonstrated for a disordered array of nanoholes in a metallic film54,55,56.

In addition to iterative feedback optimization, other point-optimization algorithms have been proposed based on TM recording and optical phase conjugation57. Although liquid-crystal SLMs have been widely used (Fig. 2f), they are not fast enough to focus light in dynamic scattering media. Over the past decade, notable progress has been made in increasing the speed of focus generation and image acquisition using digital micromirror devices (DMDs)58,59, high-speed micro-electro-mechanical systems60 and ferroelectric SLMs61, with the operating time reduced to 10 μs using an acousto-optic modulator62.

Point optimization demonstrates the full control of MS waves in the sense that a diffraction-limited spot is formed, but it cannot be directly applied to deep-tissue imaging because the detector needs to be placed in the object plane. To find a wavefront that forms a focus within the scattering medium, the detector has to be embedded within the subject of interest, which is not compatible with most in vivo optical bioimaging. However, the concepts underlying point optimization have been adopted to increase the imaging depth in two-photon microscopy. In general AO microscopy, several low-order Zernike modes are controlled to increase image sharpness. In a different approach, the segments of the incident wavefront can be controlled to optimize the two-photon fluorescence from a target beacon63, similar to the point-optimization method. Conjugate AO fluorescence microscopy was also demonstrated in the in vivo imaging of neurons through an intact mouse skull over an extended corrected field of view, achieved by placing a wavefront-shaping device in the plane conjugate to the dominant aberration-inducing layer (skull), rather than in the pupil plane of the microscope64. However, these approaches work only for relatively weak scattering or a thin aberrating layer because the target beacon has to be visible to initiate the optimization process. Various approaches have been explored to overcome this limitation and increase the viability of point optimization for use in imaging within more strongly scattering media65,66,67,68.

Transmission-matrix approach to image delivery through scattering media

The TM approach is a generalization of point optimization because point optimization is equivalent to recording one column of a TM (see section III of the Supplementary Information). In the relationship vt = Tuin, the operator T is measured by sending an incident wave to each basis vector in uin. With knowledge of T, the initial object field uin at the input plane can be recovered from the distorted field vt at the output plane using the relationship uin = T–1vt. In 2010, wide-field image transfer through a scattering medium using TM measurements was first demonstrated69,70. Subsequently, the integration of a fast steering mirror into digital holographic microscopy greatly improved the resolution and sensitivity of image transfer though a scattering medium with unprecedented TM information71,72. In addition, the counterintuitive use of multiple scattering to enhance the resolution and extend the view field has been demonstrated in wide-field imaging71. These studies confirmed that the TM approach can transform the scattering medium into a high-quality optical element that can surpass conventional optics in terms of its functionality73.

For the same reason as point optimization, the TM approach cannot be directly applied to deep optical imaging. However, it is used in ultrathin endoscopes in combination with multimode optical fibres. A single multimode optical fibre can, in principle, transmit images through numerous propagating modes but scrambles the original image information owing to the dispersion and mixing of modes. Recording the TM of the fibre resolves this problem because it converts the fibre into imaging optics. Because the mode density of a multimode fibre is 1–2 orders of magnitude greater than that of the core density of bundled fibres, the diameter of an endoscope can be greatly reduced. We briefly discuss the demonstration of endoscopic reflectance and fluorescence imaging based on the TM approach.

Reflectance endoscopic imaging through a single multimode fibre was first realized in 2012 (ref.74). In endoscopic imaging, the image is distorted by the fibre twice — once on the way in and once on the way out. In this implementation, the distortion on the way in is removed using the speckle imaging technique and that on the way out is recovered by applying an inversion of the TM. A light wave was illuminated through the input plane of a fibre proximal to a camera (Fig. 3a). After being transmitted, the light was reflected by the sample at the object plane and recaptured by the same fibre. To use the fibre as an image guide, its transmission properties were calibrated using its TM measured from the object plane to the image plane (Fig. 3b). Owing to image distortions, the observed images exhibited complex speckle patterns (Fig. 3c). By applying the inverse of the measured TM, the distortion on the way out was removed. After reconstruction, however, the images still retained speckle patterns (Fig. 3d) because the distortion on the way in had not yet been resolved. As this distortion arose prior to the light–object interaction, a clean image was obtained using the speckle-imaging method, which involves the averaging of images taken from various illuminations (Fig. 3e). To demonstrate the ex vivo imaging of tissues, a rat intestinal tissue was imaged with bright-field microscopy (Fig. 3f) and with the fibre endoscope (Fig. 3g,h). Although changes in the conformation of the fibre cause a loss of TM reconstruction, an object image was recovered when the tip of the fibre facing the object was scanned over an extended field of view (Fig. 3g,h). The TM approach has also been applied to endoscopic reflectance imaging using a bundled fibre, wherein each fibre core acts as an image pixel. The main benefit of the TM approach is the removal of image pixellation caused by the gap between neighbouring fibres in a bundle75,76.

Fig. 3: TM approach to image transfer and its application in endomicroscopy.
figure3

a | Schematic showing the set-up for wide-field endoscopic imaging using a multimode optical fibre (where OP is the object plane and IP is the input plane). b | Transmission-matrix (TM) elements measured along the direction from the OP to the IP shown in panel a. (θξ,θη)T is the illumination angle of the beam impinging on the fibre at the OP. c | Reflection images of an object taken at the IP using the set-up shown in panel a for various illumination angles (θx,θy)S. d | Object images reconstructed by multiplying the images in panel b by the inverse of the TM. e | Object image recovered after averaging the reconstructed images in panel d. f | Transmission image of rat villi taken using a conventional microscope. g,h | Image of the villi shown in panel f taken using endoscopic imaging through a multimode fibre (panel g) and an image obtained by numerical refocusing of the image in panel g by 40 μm towards the fibre end (panel h). Arrows A and B indicate the villus in focus in panels h and g, respectively. Owing to the weak absorption by the villi, the reflectance endoscopic images in panels g and h provide stronger contrast than that of the transmission image in panel f. i | Schematic showing the set-up for scanning fluorescence endoscopy through a multimode optical fibre. In step 1, the TM of the fibre is measured between the input from the digital micromirror device (DMD) and the output at the camera. In step 2, images of an object (in this case, a mouse) are acquired in vivo using the calibrated fibre. j | Images of neuronal somata and neuronal processes (indicated by the white arrows) taken through direct insertion of the fibre probe 2–3 mm into a mouse brain. Time-lapse images of blood cells haemorrhaging from a burst blood vessel (bottom) acquired using the method shown in panel i. BS, beam splitter; DM, dichroic mirror; Fil, filter; GM, two-axis galvanometer scanning mirror; HW, half wave plate; ISO, isolator; L, lens; M, mirror; OL, objective lens; PBS, polarizing beam splitter; PMT, photomultiplier tube; QW, quarter-wave plate. Panels ah adapted with permission from ref.74, APS. Panels i and j adapted from ref.84, CC-BY-4.0.

Compared with reflectance endoscopic imaging, fluorescence endoscopic imaging using a multimode optical fibre is relatively straightforward because only the distortion on the way in needs to be corrected. From the measured TM of the fibre, an incident wave uin is identified that generates a focus at the distal end, where the sample is located. By shaping the incident wavefront as the identified uin, a focus is physically generated at the desired position. For point-scanning imaging, several incident wavefronts are sequentially shaped by the SLM to scan a focus at the object plane, and fluorescence emissions at each focus are collected by the same fibre77,78,79,80. A representative configuration of a focus-scanning endoscope using a multimode optical fibre is shown in Fig. 3i. After TM measurement from the DMD to the object plane located at the distal end, the appropriate phase modulations are applied to the DMD to scan the focus. DMDs have been widely used for high-speed image acquisition81,82,83, although liquid-crystal SLMs were used in earlier set-ups77 (sometimes in conjunction with an acousto-optic deflector)79. By taking advantage of the ultrathin multimode fibre probe, minimally invasive in vivo fluorescence imaging within a mouse brain was demonstrated82,84 (Fig. 3j). Confocal imaging85 and two-photon fluorescence imaging86 have also been realized using multimode fibres for deep-tissue imaging. Intensity-only TMs have been explored to eliminate the reference arm required for complex-field detection87,88, and the fluorescence imaging of a planar object was demonstrated using intensity-based speckle correlation through a fibre bundle89.

Despite recent technological advances, the bending and twisting of the fibre, which leads to deterioration of the TM, still limit the widespread application of fibre-based endoscopy. Because pre-calibration is readily undermined by small deformations or twisting, it is crucial to maintain the fibre geometry during the imaging process74,79. To dynamically compensate for fibre bending while focusing light through a multimode fibre, a virtual point source was generated at the tip of the fibre, and the speckled wavefront generated through the fibre by this virtual point source was measured at the camera plane and phase-conjugated90. Moreover, under certain bending conditions, the fibre output characteristics can be conserved and the shaped focus remains intact91. In addition, the transmission-mode properties of deformed multimode fibres have been numerically predicted92, and image reconstruction has been demonstrated under the bending deformation of a graded-index fibre93. These efforts to resolve the problems associated with bending and deformation will enable the application of ultrathin multimode fibre endoscopy in biological and biomedical fields.

In situ imaging using memory effects

The MS waves generated by a thin scattering layer have a unique correlation characteristic: the speckle patterns rotate in angle with the rotation of the illumination wave. This angular memory effect has been reported in the form of short-range angular correlations in the mesoscopic physics of wave propagation in disordered media37,38,94,95. A tilted plane wave with an angle Δθ (Fig. 4a) can be regarded as a superposition of point sources with their relative phases increasing linearly along the lateral plane. When the scattering layer is thin enough, the transmitted waves from these individual point sources maintain the same phase difference induced by the tilt. Therefore, the overall transmitted speckle pattern is also tilted by the same Δθ. After propagating over a distance s, the transmitted speckle pattern projected on a screen is shifted by Δr ≈ sΔθ. The effective range of the memory effect depends on the thickness L of the scattering layer and the wavenumber k of the incident light. This range can be determined using the theoretical correlation function:

$$C(| \Delta \theta | ,L)={\left(\frac{k| \Delta \theta | L}{\sinh (k| \Delta \theta | L)}\right)}^{2}$$
(1)
Fig. 4: Non-invasive imaging using speckle correlations and memory effects.
figure4

a | Angular memory effect for a strongly scattering layer of thickness L. Here, s is the distance between the scattering layer and the screen, Δθ is the tilt of the angle of the incident plane wave, and Δr is the shift of the speckle pattern at the screen due to the tilt. b | Translational memory effect for an anisotropic scattering medium. When the incident wave is laterally shifted, the transmitted speckle pattern is shifted by the same amount. cf | Exploitation of multiple-scattered waves for non-invasive imaging using the angular memory effect. Panel c shows the integrated fluorescence intensity I(θ) measured as a function of the incident angle θ = (θx,θy) for a fluorescent target located at the screen shown in panel a. Panel d shows the autocorrelation I\(\,\star \) I averaged over nine scans with different starting incident angles, θ0 = (θx,0,θy,0). Panel e shows a fluorescence microscopy image of an object taken without the scattering layer. Panel f shows the image of the object retrieved from the autocorrelation in panel d using a Gerchberg–Saxton-type algorithm. g,h | Scattering compensation by focus scanning holographic aberration probing (F-SHARP). Panel g shows the experimental set-up for F-SHARP. The half-wave plate (HW) and the first polarizing beam splitter split the excitation laser beam into a strong beam and a weak beam. A strong scanning field Escan(r′ − r) is generated at the focus plane of the objective lens by the beam passing through a phase stepper and is scanned by a piezo scanner across a coordinate r. A stationary field Estat (r′) is generated by the weak beam illuminating the sample at a location r′ via the galvo scanning arm (where GM is a two-axis galvanometer scanning mirror). In the experimental demonstration, the strong and weak beams were switched. For two-photon imaging, the strong beam is sent through the spatial light modulator (SLM) for aberration correction and raster scanned at the sample by the GM. Panel h shows the theoretical description of F-SHARP for measuring the scattered electric-field point spread function (EPSF). The cubic term of Escan, a δ-like function, is convolved with Estat during the two-photon fluorescence recording by the combined excitation of Estat and Escan. Both Estat(r′) and Escan(r′) are scaled by EPSF(r′), and thus Estat(r′) Escan(r′) EPSF(r′). Thus, the reconstructed electric field Erecon(r) in this recording gives EPSF(r). ϕPSF(r) is the phase of EPSF(r). il | Panel i shows the set-up for in vivo imaging of a mouse brain using F-SHARP. Panels j and k show the two-photon imaging of an interneuron that expresses green fluorescent protein 480 µm below the dura mater before (panel j) and after (panel k) the application of F-SHARP. Panel l shows the intensity plot along the dashed lines in panels j and k. Panels a and b adapted with permission from ref.96, OSA Publishing. Panels cf adapted from ref.97, Springer Nature Limited. Panels gl adapted from ref.105, Springer Nature Limited.

Similar to the angular memory effect, a translational memory effect50,96 has been reported in scattering media, with a transmitted speckle pattern maintaining its correlation along the transverse direction (Fig. 4b).

An approach that uses the angular memory effect has been reported for the non-invasive imaging of fluorescent objects hidden behind a scattering layer89,97,98,99,100,101,102. A plane wave incident at an angle θ = (θx,θy) was used to illuminate a fluorescent object placed at a distance s behind an opaque scattering layer, and the fluorescence emissions from the object were recorded on the illuminated side. The recorded signal is described by the convolution between the intensity PSF, IPSF(r), and the object function, O(r), where r = (x,y) is the position vector in object coordinates. Because IPSF(r) is similar to the δ function in conventional microscopy, the measured intensity directly resembles O(r). In the presence of aberrations and multiple scattering, IPSF(r) is highly scrambled into a speckle pattern, Isp(r), making it difficult to extract O(r). Nevertheless, the total backscattered fluorescence intensity as a function of θ (Fig. 4c) can be written in convolution form within the finite translational invariant range of Isp(r), that is, the memory range:

$$I( {\mathbf{\uptheta}})={\int }_{-\infty }^{\infty }O({\bf{r}}){I}_{{\rm{sp}}}({\bf{r}}- {\mathbf{\uptheta}}s){{\rm{d}}}^{2}{\bf{r}}=[O\ast {I}_{{\rm{sp}}}]( {\mathbf{\uptheta}})$$
(2)

O(r) was retrieved from the autocorrelation of I(θ) (Fig. 4d), which is expressed as

$$\langle I\star I\rangle (\Delta {\mathbf{\uptheta}})=[O\star O]\ast {\langle I}_{{\rm{sp}}}\star {I}_{{\rm{sp}}}\rangle $$
(3)

where \(\star \) denotes the cross-correlation product, and the angle brackets denote the average over different I(θ) taken at different starting incident angles. Because Isp is a speckle pattern, its autocorrelation is approximately a δ function. Therefore, \(\langle I\star I\rangle \) can be written as \(\langle I\star I\rangle (\Delta {\mathbf{\uptheta}})\cong [O\star O].\) O(r) can then be obtained from the autocorrelation using a Gerchberg–Saxton-type algorithm103,104. The object image retrieved from the averaged autocorrelation resembles the fluorescence image taken without a scattering layer (Fig. 4e,f).

This method is an important step towards deep optical imaging as it makes full deterministic use of MS waves and is applicable to strongly scattering media in which SS waves are almost invisible. Furthermore, it can support non-invasive imaging under epi-detection geometry without guide stars. However, this method needs a finite distance between the scattering layer and the object to secure a sufficient imaging area, and the view field is reduced as the thickness of the scattering layer is increased and, thus, the memory-effect range decreased.

The range of the translational memory effect is related to the size of the isoplanatic patch in AO microscopy. If high-resolution wavefront correction is conducted for numerous scattered modes, a tight focus inside an anisotropic scattering medium is achieved and maintained over a small but finite translational memory-effect range50. An approach termed focus scanning holographic aberration probing (F-SHARP) achieves rapid, high-resolution wavefront correction of both strong aberrations and higher-order scattered modes in living tissue105. With this approach, it was possible to measure the phase and amplitude of the scattered electric-field PSF (EPSF) and generate a tightly focused excitation beam by transmitting the optical phase conjugation of EPSF.

To measure EPSF , a nonlinear interaction between two excitation beams can be exploited (Fig. 4g). In the theoretical description of this approach, a weak stationary beam first illuminates the sample at a location r′ to form the stationary field Estat(r′). A strong beam is then scanned across coordinate r using a piezo scanner to form the scanning field Escan(r′–r). The two-photon fluorescence intensity generated by the superposition of the two beams can be written as

$$I({\bf{r}})\propto {\int | {E}_{{\rm{scan}}}({{\bf{r}}}^{{\prime} }-{\bf{r}})+{E}_{{\rm{stat}}}({{\bf{r}}}^{{\prime} })| }^{4}{\rm{d}}{{\bf{r}}}^{{\prime} }$$
(4)

When Estat is much weaker than Escan, Estat terms with a power ≥2 can be ignored and the cubic term of Escan(r′) can be considered a highly peaked δ function (Fig. 4h). The captured fluorescence intensity can then be approximated as

$$I({\bf{r}})\propto {I}_{{\rm{background}}}+{E}_{{\rm{PSF}}}({\bf{r}})+{E}_{{\rm{PSF}}}^{\ast }({\bf{r}})$$
(5)

where Estat(r) has been replaced by EPSF(r), which is extracted from I(r) using phase-shifting interferometry with a phase stepper (Fig. 4g). In the experimental demonstration of this approach, the strong beam is set to be stationary and the weak beam is scanned by a piezo scanner. The strong beam is then corrected by applying the complex conjugate of the Fourier transform of the measured EPSF to the SLM. After a finite number of consecutive correction steps, the strong beam can be transformed quickly into a sharp focus. After all the correction steps, the weak beam is blocked and the strong beam is raster scanned by a galvanometer scanning mirror to obtain two-photon images.

The effectiveness of F-SHARP microscopy was demonstrated for in vivo imaging of a mouse brain. In an anaesthetized GAD67 mouse, interneurons labelled with green fluorescent protein were imaged 480 µm below the brain surface following a craniotomy (Fig. 4i). Compared with that of conventional two-photon microscopy (Fig. 4j), the image quality was greatly enhanced after correction (Fig. 4k). The F-SHARP images exhibited a fivefold increase in the signal from the corrected region and a higher resolution (Fig. 4l). In many ways, F-SHARP resembles AO microscopy that directly detects the wavefront of fluorescence from nonlinear guide stars106. However, the interferometric detection of EPSF by introducing the nonlinear interaction between two beams can greatly increase the sensitivity of wavefront sensing. Unlike existing iterative wavefront-shaping AO methods, F-SHARP determines the wavefront correction by raster scanning EPSF using fast scanning mirrors. This approach decouples the wavefront measurement speed from the limited speed of the wavefront shapers, which is crucial for correcting a large number of scattered modes (>1,000). Further increases in imaging depth will depend on relieving the requirement for a spatially sharp initial EPSF for the method to converge.

Ultrasound-assisted optical imaging

Unlike light waves, which are prone to multiple light scattering in biological tissue, ultrasound can propagate through tissue without notable distortion up to depths of 1–100 mm (ref.107). The greater penetration depth can be translated into optical imaging through ultrasound–light interactions. Two major interaction mechanisms have been actively employed to date: the frequency shift of optical waves using ultrasound and the generation of ultrasonic waves using optical waves. In bioimaging, the former mechanism has led to the development of ultrasound-modulated optical tomography (UOT), in which an ultrasound-induced refractive index grating and movement of optical scatterers modulate the frequency of optical waves propagating through an ultrasonic focus18,108,109. The latter mechanism was used to develop photoacoustic tomography (PAT), in which absorbed optical energy produces heat and subsequently generates ultrasound from the thermal expansion of the medium18,110,111. The trade-off for the increase in penetration depth is that the resolving power of these ultrasound-mediated imaging methods is far lower than the optical diffraction limit. The resolving power is dictated by the ultrasonic diffraction limit, which is 20–1,000 μm for the commonly used ultrasound frequency range of 1–50 MHz. Higher frequency ultrasound has been considered to improve the resolution, but this approach is not practical because of the weaker ultrasound-mediated signal and the severe attenuation of the high-frequency ultrasound in biological tissue112.

Advances in the field of wavefront shaping have improved the spatial resolution of UOT-based and PAT-based approaches. Before discussing these approaches, it is important to understand how the wavefront-shaping technique34,40,113 is used to enhance the sensitivity of ultrasound–light interactions. When combined with UOT-based or PAT-based approaches, an optimal wavefront can be obtained to focus light on the ultrasonic guide star through approaches based on iterative optimization34,57, optical phase conjugation114 or a TM70. Owing to the reciprocity of light scattering, the optimal wavefront cancels the effect of light scattering in a ‘time-reversed’ manner and is focused back onto the ultrasound focus. For imaging purposes, the focused spot can be scanned through the object to yield fluorescence or optical absorption-contrast images.

Using UOT-assisted wavefront shaping, time-reversed ultrasonically encoded (TRUE) optical focusing based on optical phase conjugation was demonstrated, in which the wavefront of a frequency-shifted optical wave was directly measured and phase conjugated115. This approach (Fig. 5a) reverses the propagation of all frequency-modulated optical waves, including both SS and MS waves. Therefore, the focusing resolution is dictated by the ultrasonic diffraction limit of a few tens to hundreds of micrometres115,116,117 (Fig. 5b). To overcome the resolution limit of TRUE, two approaches have been proposed to date based on the non-uniform pressure profile of focused ultrasound waves: the iterative conjugation of frequency-shifted light and the time reversal of variance-encoded light. In the iterative approach, the measurement and conjugation process for frequency-modulated light is repeated to find an open eigenchannel that maximizes the light transmitted through the ultrasound focus118,119. Owing to the Gaussian-like shape of the ultrasound focus, the maximally transmissive channel is confined to an area close to the peak of the ultrasound focus, where the modulation is strongest. In the variance-encoding method, several input–output matrices are measured at multiple positions adjacent to ultrasound foci for thousands of different input modes120. The optimal wavefront is then calculated to maximize the variance ratio between the sum and difference fields using singular-value analysis. Typical frequency-modulated optical fields measured at different ultrasound positions are shown in Fig. 5c, along with their sum and difference. Because a single speckle grain located at the centre of an ultrasound focus statistically has the largest variance, this approach in principle realizes speckle-scale optical focusing (Fig. 5c). To date, the resolution based on the non-uniformity of an ultrasound focus is 5–10 μm, which is a threefold to fivefold increase over the ultrasonic diffraction limit118,119,120. A generalized framework has been presented for iterative and variance-encoding methods in the context of the TM121. For these approaches to operate as high-resolution microscopy techniques, however, they need to be applicable to target objects that are fully embedded within scattering media, in which the speckle grain size approaches half the optical wavelength. The technical challenge ahead is the large number of measurements required to guarantee statistical robustness; this number grows cubically with the number of optical speckle grains within the ultrasound focus121.

Fig. 5: Ultrasound-mediated optical wavefront shaping and imaging.
figure5

a | Experimental set-up of the time-reversed ultrasonically encoded (TRUE) optical focusing method. The frequency of the optical wave (f0) is shifted by the ultrasound frequency (fUS) when it travels through an ultrasound focus. The digital-phase-conjugation system selectively measures and conjugates the wavefront of the frequency-shifted optical wave. Owing to the reciprocity of light scattering, the conjugated wavefront is focused back to the position of the ultrasound. b | Images showing the typical frequency-shifted speckle field b′ (top part; the colour represents the phase and the luminance indicates the normalized amplitude) and ensemble-averaged amplitude profile (bottom part) at the plane of the ultrasound focus. c | Frequency-shifted speckle fields b1, b2, b3 and b4 were recorded at four different acoustic foci. The optimal wavefront that maximizes the variance ratio between the sum field (b1+2+3+4) and the difference fields (b2–3 and b1–4) is phase conjugated to focus the ultrasound-modulated wave to a single speckle grain located inside the ultrasound focus (bottom right part). d | Experimental set-up of a wavefront-shaping method, known as photoacoustically guided wavefront shaping (PAWS), that iteratively optimizes the nonlinear photoacoustic signal excited by dual laser pulses. e | Images showing the optimized nonlinear signal (far right), obtained using the method shown in panel d, in which light is focused on a single speckle grain inside the ultrasound detection area. sCMOS, scientific complementary metal-oxide semiconductor; SLM, spatial light modulator. Panel a adapted from ref.116, Springer Nature Limited. Panels b and c adapted from ref.120, Springer Nature Limited. Panels d and e adapted from ref.124, Springer Nature Limited.

As an alternative to adopting wavefront-shaping techniques, which place an emphasis on controlling MS optical waves, ultrasonic light modulation in coherent confocal microscopy can be employed to reject MS waves travelling outside the ultrasound focus30. In other words, acousto-optic modulation is used as a form of space gating that is similar to the temporal and confocal gating used in optical coherence microscopy122. The space-gating mediated by acousto-optic modulation acts inside the scattering medium in which the target object is located, unlike temporal gating and confocal gating, which are applied to the optical waves that return to the free space. Therefore, space gating can reject MS waves that the other gating operations cannot; as a result, the combined use of these available gating operations maximizes the visibility of SS optical waves. Because this method relies on the detection of SS waves, the resolution is guaranteed to reach the optical diffraction limit, regardless of the speckle grain size. However, owing to the exponential extinction of SS waves, the penetration depth of the gating approach will be set by the detector dynamic range limit (Box 1), whereas the approaches based on the wavefront shaping of modulated MS waves can overcome this limit if the aforementioned challenges are addressed.

Unlike the UOT-based approach, the PAT-based wavefront-shaping approach cannot directly measure the optimal wavefront. Instead, the acoustic wave generated by optical absorption is iteratively optimized or the photoacoustic TM measured by injecting light into different free-space modes at the input plane of the scattering medium. By measuring the photoacoustic TM, selective focusing on absorbing targets was demonstrated together with the scanning of an optical focus based on the optical memory effect123. However, similar to TRUE, the resolution of this approach is intrinsically limited to the ultrasound diffraction limit owing to the linearity of photoacoustic signals. To overcome this resolution limit, nonlinear photoacoustic signals have been introduced with a dual-pulse excitation based on the Grueneisen relaxation effect124. A genetic algorithm was then used to identify the optimal wavefront that maximizes the nonlinear signal. This method is referred to as photoacoustically guided wavefront shaping (PAWS; Fig. 5d). Unlike the linear photoacoustic approach, the optimization of the nonlinear signal converges to the wavefront solution confined within a single speckle grain inside the ultrasonic detection area (Fig. 5e). The Gaussian-shape sensitivity profile of the detection area can also be used without inducing a nonlinear signal125, although the enhancement is quite limited. Similar to iterative TRUE or the variance-encoding method, the improved resolution in the PAT-based approaches can reach ~5 μm (refs124,125). For further improvement, the low measurement sensitivity when the size of the speckle grains approaches half the optical wavelength needs to be addressed. Furthermore, in imaging applications, photoacoustic approaches need to be developed to enable arbitrary control of the position of the speckle-scale focus113.

Ultrasound–light interactions are uniquely valuable in biophotonics because both UOT and PAT provide direct access to the optical flux inside a scattering medium. The incorporation of wavefront-shaping techniques has added the superior resolving power of optics to ultrasound-mediated approaches, thus overcoming the conventional ultrasonic diffraction limit. Although there are practical challenges, advances in sensor and SLM technology may soon realize the full potential of ultrasound-mediated wavefront shaping and imaging in microscopic applications.

Deep imaging using a reflection matrix

Epi-detection geometry is essential for most in vivo applications, which demand the recording of a reflection matrix. Unlike conventional confocal microscopy, in which only the signal filtered by the confocal pinhole is collected for image reconstruction, reflection-matrix-based approaches make deterministic use of all of the backscattered signals arriving at non-confocal positions as well as those at confocal positions (see section III of the Supplementary Information). In this section, we first introduce methods that simultaneously suppress multiple scattering and sample-induced aberrations without using guide stars and then describe approaches based on the coherent superposition of MS waves.

Solving the inverse scattering problem using a reflection matrix

Diffraction-limited imaging requires the momentum change ko − ki induced by the target object to be measured for all possible angles supported by the objective lens. However, MS waves and sample-induced aberrations prevent the accurate tracking of ki and ko. The first experimental system for measuring a wide-field time-gated reflection matrix, R(τ0), was developed in 2015 (ref.10) (Fig. 6a). Using R(τ0), a method referred to as collective accumulation of single scattering (CASS) microscopy was proposed to selectively identify SS waves that preserve the momentum change induced by the target object. Two unique properties of SS waves are used in this process: the depth-dependent flight time and momentum conservation. Using CASS microscopy, an imaging depth of 11.5ls was demonstrated with a near-diffraction-limited spatial resolution of 1.5 μm (ref.10).

Fig. 6: Scattering and aberration suppression using a time-gated reflection-matrix approach.
figure6

a | Experimental set-up of the collective accumulation of single scattering (CASS) microscope (the closed-loop accumulation of single scattering (CLASS) method adopts a similar set-up). Low-coherence interferometry is used for time-gated detection, and off-axis digital holographic microscopy is integrated for the complex field recording. b,c | Time-gated reflection matrices obtained in position space (E(ro,ri;τ0); panel b) and wavevector k space (\({\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\); panel c). ri and ro are the input (illumination) and output (detection) position vectors in the object coordinates, respectively; ki and ko are the corresponding input and output transverse wavevectors, respectively, and τ0 is the flight time. d | The nth correction of an input pupil aberration. The phase correction \({\theta }_{{\rm{i}}}^{(n)}({{\bf{k}}}_{{\rm{i}}})\) is applied to each column of \({\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\) in panel c. e | The nth correction of an output pupil aberration. The phase correction \({\theta }_{{\rm{o}}}^{(n)}({{\bf{k}}}_{{\rm{o}}})\) is applied to each row of \({\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\). The iteration is repeated until the total intensity of the retrieved CLASS image increases sufficiently to reach a given criterion. f,g | Reconstructed images from the diagonal elements of E(ro,ri;τ0) before aberration correction (panel f) and the corresponding aberration-corrected CLASS image after five rounds of iterations (panel g). h,i | The accumulated phase-correction maps for the sample-induced aberrations in the illumination path (panel h) and detection path (panel i) after five rounds of iterations. j–l | The time-gated reflection matrix was obtained by adaptive optical synchronous angular scanning microscopy (AO-SASM) at a depth of 160 μm in the hindbrain of a zebrafish. Images showing neural fibres reconstructed before (panel j) and after (panel k) aberration correction. The white dotted boxes in panel j show segmented areas where the CLASS algorithm was individually applied. The local aberration maps in the pupil plane (panel l) correspond to the segments indicated in panel k. m | Imaging configuration (top) and bright-field image (bottom) for a 21-days-post-fertilization (dpf) living larval zebrafish. The yellow dotted box indicates the imaging area. n | In vivo volumetric imaging of the deep hindbrain of a 10-dpf larval zebrafish. For 3D rendering, aberration-corrected slice images were retrieved from the consecutive time-gated reflection matrices along axial intervals at the length of the coherence gating, then numerical propagation was applied with sufficient sampling along the depth. o,p | Dorsal-view maximum-intensity-projection images from confocal microscopy (panel o) and AO-SASM (panel p). BS, beam splitter; CCD, charge-coupled device; DG, diffraction grating; OL, objective lens; PZT, piezoelectric transducer; SLD, superluminescent diode; SM, scanning mirror. Panel a adapted from ref.10, Springer Nature Limited. Panels bi adapted from ref.126, CC-BY-4.0. Panels jp adapted from ref.127, CC-BY-4.0.

Subsequently, a method termed closed-loop accumulation of single scattering (CLASS)126 was developed that separately identifies the aberrations in SS waves incident on and reflected from an object, even in the presence of multiple light scattering. Using an experimental set-up similar to that used in CASS microscopy, complex-field images of backscattered waves, E(ro,ki;τ0), were measured at a conjugate image plane for a set of wavevectors {ki} that cover all of the orthogonal input free modes, where ro is the position vector in image coordinates. By taking the inverse Fourier transform with respect to ki, the matrix E(ro,ri;τ0) can be obtained in position space (Fig. 6b). Here, E(ro,ri;τ0) corresponds to the matrix element of R(τ0) (see section III of the Supplementary Information). The appearance of strong off-diagonal elements is due to multiple scattering and aberrations. By taking the Fourier transform of the original images, E(ro,ri;τ0) with respect to ro, the time-gated reflection matrix \({\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\) in k space is obtained (Fig. 6c). The k basis is a good basis because the phase-retarded SS wave can be decoupled from the MS waves. That is, the matrix element can be decomposed by

$${\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}};{\tau }_{0})={{\mathscr{P}}}_{{\rm{o}}}^{{\rm{a}}}({{\bf{k}}}_{{\rm{i}}}+\Delta {\bf{k}}){\mathscr{O}}(\Delta {\bf{k}}){{\mathscr{P}}}_{{\rm{i}}}^{{\rm{a}}}({{\bf{k}}}_{{\rm{i}}})+{{\mathcal{E}}}_{{\rm{M}}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}};{\tau }_{0})$$
(6)

Here, the first term on the right-hand side describes an SS wave scattered once by the target object. Therefore, its amplitude is proportional to the amplitude of the reflectance of the object for the spatial frequency Δk = ko − ki, but its phase is retarded by complex pupil functions for aberrations in the illumination and imaging paths, \({{\mathscr{P}}}_{{\rm{i}}}^{{\rm{a}}}({{\bf{k}}}_{{\rm{i}}})=\exp [-i{\phi }_{{\rm{i}}}({{\bf{k}}}_{{\rm{i}}})]\) and \({{\mathscr{P}}}_{{\rm{o}}}^{{\rm{a}}}({{\bf{k}}}_{{\rm{o}}})=\exp [-i{\phi }_{{\rm{o}}}({{\bf{k}}}_{{\rm{o}}})]\), respectively. Here, ϕi(ki) and ϕo(ko) are the sample-induced phase retardation functions in the illumination and imaging paths, respectively. The second term on the right-hand side is the contribution of all time-gated MS waves with the same wavevector as that of the SS wave: \({{\mathcal{E}}}_{{\rm{M}}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})={{\mathcal{E}}}_{{\rm{TM}}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})+{{\mathcal{E}}}_{{\rm{BM}}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\).

The main concept underlying the CLASS algorithm (Fig. 6d,e) is iterative processing of the reflection matrix to determine the phase corrections θi(ki) and θo(ko) that compensate for ϕi(ki) and ϕo(ko), respectively. In every nth iteration, the input and output correction processes occur in sequence. In the input correction process (Fig. 6d), \({\theta }_{{\rm{i}}}^{(n)}({{\bf{k}}}_{{\rm{i}}})\) is optimized to maximize the total intensity of the reconstructed target image. In the same way, \({\theta }_{{\rm{o}}}^{(n)}({{\bf{k}}}_{{\rm{o}}})\) is identified in the output correction process (Fig. 6e). The correction process mainly involves the SS waves, with the MS waves having little role because they are taken at different angles of illumination and are uncorrelated with respect to one another. As the iteration continues, the contribution of the correlated SS waves to the phase correction increases, while that of the uncorrelated MS waves remains the same. Therefore, the maximization of total intensity is almost exclusively due to the aberration correction of the SS waves. After applying the accumulated phase corrections, an aberration-corrected target image is obtained by summing the angular spectra for all of the illuminations, that is, \({{\mathcal{E}}}_{{\rm{CLASS}}}(\Delta {\bf{k}})={\sum }_{{{\bf{k}}}_{{\rm{i}}}}{\mathcal{E}}({{\bf{k}}}_{{\rm{o}}},{{\bf{k}}}_{{\rm{i}}}\,;{\tau }_{0})\). In short, the multi-angular illuminations used in the k-space reflection matrix offer a way to eliminate the influence of MS waves. In comparison with an image taken before aberration correction (Fig. 6f), a signal in an aberration-corrected CLASS image (Fig. 6g) was increased more than 20-fold126. The finest structures in the resolution target are also clearly visible, confirming the recovery of near-diffraction-limited spatial resolution. The input and output phase-correction maps identified independently by the CLASS algorithm show high-order aberration modes that ordinary AO microscopy cannot (Fig. 6h,i).

In initial studies, a drawback of CLASS microscopy was the slow acquisition rate for recording the time-gated reflection matrix owing to the use of a SLM for multi-angular illuminations. To resolve this issue, adaptive optical synchronous angular scanning microscopy (AO-SASM)127 was proposed, in which a pair of scanning mirrors is used to increase the scanning speed up to 10,000 modes per second. The high-speed recording of R(τ0) enabled aberration-free imaging of a living larval zebrafish over the entire volume of the hindbrain. Coherence imaging of neural fibres before aberration correction at a depth of 160 μm, corresponding to conventional en-face optical coherence tomography (OCT) imaging, produces a blurred image (Fig. 6j) in which the fine neural fibre structures are not well resolved because the zebrafish has many organs that are optically heterogeneous. Indeed, the sample-induced aberrations are so pronounced that they vary from one position to another. To address these position-dependent aberrations, the CLASS algorithm was individually applied to each segmented area (Fig. 6j). The combined aberration-corrected image and the local aberration maps in the pupil plane (Fig. 6k,l, respectively) show the fine neural fibres and spatial complexity of the aberrations. A clear 3D image of the whole neural network throughout the hindbrain of a 10-days-post-fertilization zebrafish was obtained by AO-SASM (Fig. 6m,n), even though the dark epidermal layer, complex skin architectures and optically heterogeneous internal organs disturb the light propagation in the imaging configuration. Only dense fascicles are visible in the confocal microscopy image in the dorsal view of the hindbrain (Fig. 6o), whereas anatomical details, including fine neuronal fibres and other subcellular architectures, are visible in the corresponding AO-SASM image (Fig. 6p).

Deep optical imaging based on the reflection matrix has an advantage over conventional AO microscopy in that it can suppress sample-induced aberrations without the need for guide stars and under the influence of strong MS noise. These features enable deep optical imaging to reach the ideal SS gate limit (Box 1 figure). These advantages are made possible because the reflection matrix deterministically records both the aberrated SS waves and MS waves arriving at non-confocal positions. Compared with computational AO approaches that rely on full-field imaging, the reflection-matrix approach offers more robust image reconstruction with respect to pupil aberrations and multiple light scattering. Conventional computational AO approaches often use a single 2D complex-field image for a specific depth128 or a 3D tomogram for a volume129,130 to correct the aberrations, which is equivalent to the acquisition of the diagonal elements only of the position-space reflection matrix. Owing to this limited acquisition, the optimization problem becomes underdetermined or ill posed, especially for reflection imaging in epi-detection geometry, for which both the input and output wavefront aberrations need to be identified. Indeed, the complete recording of the input–output response through the reflection matrix has provided the opportunity to resolve difficulties that cannot be handled using conventional imaging. However, there is still room to explore the use of the reflection matrix to solve the inverse problem associated with \({| {{\mathcal{E}}}_{{\rm{TM}}}| }^{2}\) MS waves that interact with the target object. Extending the reflection-matrix approach to solve shift-variant aberrations64,131,132 could be one option for the complete deterministic use of multiple scattering. The identification of position-dependent local aberrations (Fig. 6l) is a good starting point as they contain a small fraction of \({| {{\mathcal{E}}}_{{\rm{TM}}}| }^{2}\).

Deep imaging based on reflection-matrix eigenchannels

Identifying eigenchannels in the reflection matrix provides a unique opportunity to control the interference of MS waves. Mathematically, eigenchannels are retrieved through the singular value decomposition of the reflection matrix into R = UΣV, where Σ is a rectangular diagonal matrix with real non-negative singular values, and V and U are unitary matrices with columns that are the eigenchannels at the input and output planes, respectively. V denotes the conjugate transpose of V. Coupling light to eigenchannels with large singular values leads to constructive interference between the scattered waves133, thus increasing the reflectance in proportion to the square of the singular values. Indeed, the eigenchannels with large singular values guide incident waves to relatively high reflective target objects embedded within a scattering medium. This characteristic of eigenchannels has previously been employed in acoustics to detect reflecting objects embedded within a scattering medium134,135 in a process known as the decomposition of the time-reversal operator (abbreviated to DORT in French).

In 2011, the first optical analogue of the DORT method was reported for selectively focusing light on gold nanoparticles inside disordered media136. Owing to multiple light scattering, the particles were invisible in the reflection image (Fig. 7a). The reflection matrix R was measured using a continuous-wave laser and its singular-value distribution obtained (Fig. 7b). Four large singular values associated with the bright gold nanoparticles were identified and their output eigenchannels (v1v4) derived from the columns in V, enabling the individual nanoparticles to be clearly resolved (Fig. 7c). The images recorded on the transmission side confirmed that the waves generated through eigenchannel coupling were strongly focused on the target particles. However, one technical limitation of this early method based on a steady-state reflection matrix is its susceptibility to MS noise. The envelope of small singular values shown in Fig. 7b is due to background noise, such as stray reflections and laser fluctuations. The eigenchannels of these small singular values could be distinguished from those of target particles in a relatively weak scattering medium. In general, for a highly scattering medium, MS waves that have no interaction with the nanoparticles dominate those that do, which may leave the targets invisible, even with the use of the DORT method.

Fig. 7: Eigenchannels of a reflection matrix and their use in imaging.
figure7

ac | Selective focusing on gold nanoparticles through an aberrating layer using the eigenchannels of the steady-state reflection matrix. The gold nanoparticles are not visible in the reflection image obtained with plane-wave illumination (panel a). In the singular-value distribution of the measured reflection matrix (panel b), there are a few high singular values. The reflection images obtained using the output eigenchannels of the four largest singular values (panel c) reveal the presence of the gold nanoparticles. dg | Time-gated reflection eigenchannels. Panel d shows representative trajectories of multiple-scattered waves with different flight times (labelled 1–4) within a scattering medium. Panel e shows the corresponding intensity (I) profile of backscattered waves as a function of the time of flight (τ). By coupling light to the time-gated reflection eigenchannels at the flight time indicated as (3), the intensity at the selected time-gating window increases (the red curve). The blue dotted curve is the intensity profile for the uncontrolled wave. Images of the wave propagation calculated by the finite-difference time-domain method (where Zs is the depth of the target object) show the differences in wave propagation for a random input (panel f) and eigenchannel coupling (panel g). hk | Smart optical coherence tomography (OCT) for deep optical imaging. Panel h shows the time-gated reflection matrix R(ro,ri) (bottom) measured for a scattering layer covering three ZnO beads with diameters of 5 μm (top), where ri and ro are the incident-focusing and output-focusing points, respectively. Panel i shows the single-scattered matrix RS obtained by applying a multiple-scattered filter to the raw matrix R with a virtual pinhole mask with a size (lc) of 5 μm. Panel j shows an en face OCT image deduced from the diagonal of R. The images reconstructed using the first three eigenchannels of RS, |UVi| (panel k, top), yield the image of the three beads in the presence of a scattering layer (bottom). l,m | Panel l shows an en face OCT image of a USAF target under 800-μm-thick rat intestinal tissue. Panel m shows the corresponding eigenchannel image of the resolution target obtained from the first 250 eigenchannels of the filtered matrix. Panels ac adapted with permission from ref.136, APS. Panels dg adapted from ref.138, Springer Nature Limited. Panels hm adapted with permission from ref.139, AAAS.

To enhance the sensitivity in the presence of strong MS noise, a time-gated reflection matrix R(τ0) can be considered10,137, in which a large fraction of background MS noise is eliminated by time gating (Fig. 7d,e). This approach was used to demonstrate the extraction of time-gated reflection eigenchannels associated with target objects and the enhancement of light energy delivered to the target138. Unlike the wave trajectories for the uncontrolled incident wave (Fig. 7f), most of the waves coupled through the eigenchannels were guided to the target, which increased reflection from the target (Fig. 7g). This observation verifies that eigenchannel coupling controls \({| {{\mathcal{E}}}_{{\rm{TM}}}| }^{2}\) MS waves that interact with the target such that they are focused on the target.

In another method for deep optical imaging, referred to as smart OCT, eigenchannels of a time-gated reflection matrix are used for image reconstruction139. For ZnO particles covered by a 12.25ls-thick scattering layer, R(τ0) was obtained in the position basis (Fig. 7h). To reveal the target objects, an MS filter was devised to remove matrix elements far from the diagonal (Fig. 7i). The en face OCT image obtained from the diagonal elements of R(τ0) shows only speckle noise (Fig. 7j), whereas the three particles were clearly resolved in the image reconstructed using the first three eigenchannels of the filtered matrix (Fig. 7k). To demonstrate the feasibility of imaging an extended target, a USAF target behind thick biological tissue was tested. Compared with the conventional en face OCT image (Fig. 7l), the three bars in the image reconstructed using 250 eigenchannels (Fig. 7m) are more clearly resolved.

The image reconstruction techniques based on eigenchannels discussed so far have proved to be powerful tools for ultra-deep optical imaging. MS waves that interact with target objects are deterministically used in image reconstruction. Therefore, the imaging depth can exceed that of methods based on SS waves, such as confocal microscopy, multiphoton microscopy and OCT. Theoretically, the imaging depth of the eigenchannel-based methods is expected to reach 22ls compared with 12ls for conventional OCT11,139. Moreover, these methods also have a better spatial resolution than those of fully stochastic approaches. However, neither the input nor output eigenchannels can represent the real electric field at the plane of the object. Furthermore, image reconstruction does not keep track of the momentum change induced by the target object. Therefore, the spatial resolution is lower than the optical diffraction limit, especially in the presence of sample-induced aberrations. Future research is expected to harness the benefits of eigenchannels in controlling MS waves for high-resolution optical imaging.

Future perspectives

A surge in research related to deep optical imaging has been witnessed since the first demonstration of focusing MS waves to a sharp spot using wavefront shaping. Many approaches for using MS waves have emerged based on the mesoscopic physics of wave propagation, and those covered in this Review have provided opportunities that were not previously conceivable. In particular, the deterministic use of MS waves for optimal image formation has raised the prospect of overcoming the imaging depth limit of conventional high-resolution optical microscopy. Because high-resolution optical imaging relies mainly on SS waves, its depth limit is set by the efficiency of the single-scattering gating methods that distinguish SS signal waves from strong MS noise. Although this depth limit varies from system to system, it is typically ~10ls. The deterministic use of MS waves converts some of the waves considered to be noise into a signal, thus increasing the effective signal-to-noise ratio. This approach will ultimately lead to an imaging depth that is greater than the single-scattering gating limit and one that may even exceed the fundamental limit set by the dynamic range of the detector, which is ~18–20ls.

At present, however, most studies have been proof-of-concept and are subject to assumptions and constraints that are not fully compatible with in vivo imaging, and/or the improvement relative to conventional imaging has been marginal. Thus, the proposed methodologies have not yet reached a level at which MS waves are used fully deterministically to form a diffraction-limited optical image under standard imaging conditions. For these newly proposed methodologies to mature, the aforementioned assumptions and constraints need to be accounted for. Indeed, there are many ongoing studies that are addressing the limitations of each modality. Specifically, researchers working on deep imaging based on memory effects are combining their efforts to find ways to effectively generate a focus within a scattering medium and to extend the working range of the memory effect140,141. Studies are also underway to address the degradation of the TM due to the bending and twisting of multimode fibres, which will enable multimode-fibre ultrathin endoscopes to create images from curved pathways90,91,92,93. Optical-scale imaging using acousto-optic interactions and the selective detection of single optical speckles is advancing towards realistic conditions in which the speckle grain size reaches half of the wavelength121,124. High-resolution imaging based on a time-gated reflection matrix is also starting to gain control of MS waves in reconstructing the object image126,127,139. Additionally, improvements in the sensitivity and speed of cameras and the increase in the number of pixels and control speed of the SLM will help to resolve some of the practical issues.

The formation of an ideal image using MS waves is an underdetermined problem, and advances in computational tools, such as convolutional neural networks, may help in finding a solution by taking advantage of prior knowledge from mesoscopic physics. Deep learning based on neural networks has been actively used in recent years for image denoising, spatial and spectral deconvolution, super-resolution imaging, scattering noise reduction, image registration and tomographic reconstruction142,143,144. In the context of MS waves, neural networks have been used to train the input–output response for scattering media and multimode optical fibres145,146 and to reconstruct refractive index maps for spatially confined objects from the MS waves of the objects themselves147,148. It remains to be seen whether these approaches can be extended to selectively train MS waves that interact with the target object to form a near-diffraction-limited image under the strong MS noise induced by a large-volume scattering medium. We expect that these efforts will improve the practicality of deep optical imaging in complex scattering media and make it possible to investigate physical and biological systems in a more detailed but less invasive manner than before.

References

  1. 1.

    Luker, G. D. & Luker, K. E. Optical imaging: current applications and future directions. J. Nucl. Med. 49, 1–4 (2008).

  2. 2.

    Wilt, B. A. et al. Advances in light microscopy for neuroscience. Annu. Rev. Neurosci. 32, 435–506 (2009).

  3. 3.

    Hadjipanayis, C. G., Jiang, H., Roberts, D. W. & Yang, L. Current and future clinical applications for optical imaging of cancer: from intraoperative surgical guidance to cancer screening. Semin. Oncol. 38, 109–118 (2011).

  4. 4.

    Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

  5. 5.

    Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–796 (2006).

  6. 6.

    Klar, T. A., Jakobs, S., Dyba, M., Egner, A. & Hell, S. W. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc. Natl Acad. Sci. USA 97, 8206–8210 (2000).

  7. 7.

    Hell, S. W. Far-Field Optical nanoscopy. Science 316, 1153–1158 (2007).

  8. 8.

    Tyson, R. K. Principles of Adaptive Optics (CRC Press, 2015).

  9. 9.

    Liu, F., Yoo, K. M. & Alfano, R. R. Transmitted photon intensity through biological tissues within various time windows. Opt. Lett. 19, 740–742 (1994).

  10. 10.

    Kang, S. et al. Imaging deep within a scattering medium using collective accumulation of single-scattered waves. Nat. Photonics 9, 253–258 (2015). The first report of the measurement of a wide-field time-gated reflection matrix.

  11. 11.

    Badon, A., Boccara, A. C., Lerosey, G., Fink, M. & Aubry, A. Multiple scattering limit in optical microscopy. Opt. Express 25, 28914–28934 (2017).

  12. 12.

    Dunsby, C. & French, P. M. W. Techniques for depth-resolved imaging through turbid media including coherence-gated imaging. J. Phys. D. Appl. Phys. 36, R207–R227 (2003). An early study on the development of single-scattering gating methods for deep optical imaging.

  13. 13.

    Minsky, M. Microscopy apparatus. US Patent 3013467 (1961).

  14. 14.

    Huang, D. et al. Optical coherence tomography. Science. 254, 1178–1181 (1991).

  15. 15.

    Bouma, B. E. & Tearney, G. J. (eds) Handbook of Optical Coherence Tomography (Marcel Dekker, 2002).

  16. 16.

    Pawley, J. (ed.) Handbook of Biological Confocal Microscopy (Springer, 2010).

  17. 17.

    Yao, G. & Wang, L. V. Theoretical and experimental studies of ultrasound-modulated optical tomography in biological tissue. Appl. Opt. 39, 659–664 (2000).

  18. 18.

    Wang, L. V. Ultrasound-mediated biophotonic imaging: a review of acousto-optical tomography and photo-acoustic tomography. Dis. Markers 19, 123–138 (2004).

  19. 19.

    Izatt, J. A., Hee, M. R., Owen, G. M., Swanson, E. A. & Fujimoto, J. G. Optical coherence microscopy in scattering media. Opt. Lett. 19, 590–592 (1994).

  20. 20.

    Beaurepaire, E., Boccara, A. C., Lebec, M., Blanchot, L. & Saint-Jalmes, H. Full-field optical coherence microscopy. Opt. Lett. 23, 244–246 (1998).

  21. 21.

    Oh, W. Y. et al. Ultrahigh-resolution full-field optical coherence microscopy using InGaAs camera. Opt. Express 14, 726–735 (2006).

  22. 22.

    Denk, W., Strickler, J. H. & Webb, W. W. Two-photon laser scanning fluorescence microscopy. Science 248, 73–76 (1990).

  23. 23.

    Hell, S. W. et al. Three-photon excitation in fluorescence microscopy. J. Biomed. Opt. 1, 71–74 (1996).

  24. 24.

    Zipfel, W. R. et al. Live tissue intrinsic emission microscopy using multiphoton-excited native fluorescence and second harmonic generation. Proc. Natl Acad. Sci. USA 100, 7075–7080 (2003).

  25. 25.

    Barad, Y., Eisenberg, H., Horowitz, M. & Silberberg, Y. Nonlinear scanning laser microscopy by third harmonic generation. Appl. Phys. Lett. 70, 922–924 (1997).

  26. 26.

    Papagiakoumou, E. et al. Functional patterned multiphoton excitation deep inside scattering tissue. Nat. Photonics 7, 274–278 (2013).

  27. 27.

    Rowlands, C. J. et al. Wide-field three-photon excitation in biological samples. Light. Sci. Appl. 6, e16255 (2016).

  28. 28.

    Adrià, E.-M. et al. Wide-field multiphoton imaging through scattering media without correction. Sci. Adv. 4, eaau1338 (2018).

  29. 29.

    Booth, M. J. Adaptive optical microscopy: the ongoing quest for a perfect image. Light. Sci. Appl. 3, e165 (2014).

  30. 30.

    Jang, M., Ko, H., Lee, W. K., Lee, J.-S. & Choi, W. Coherent space-gated microscopy: a step towards deep-tissue phase imaging of biological cells. Preprint at arXiv https://arxiv.org/abs/1811.02755 (2018).

  31. 31.

    Durduran, T. et al. Diffuse optical measurement of blood flow, blood oxygenation, and metabolism in a human brain during sensorimotor cortex activation. Opt. Lett. 29, 1766–1768 (2004).

  32. 32.

    Ntziachristos, V., Yodh, A. G., Schnall, M. & Chance, B. Concurrent MRI and diffuse optical tomography of breast after indocyanine green enhancement. Proc. Natl Acad. Sci. USA 97, 2767–2772 (2000).

  33. 33.

    Wang, L. V. & Hu, S. Photoacoustic tomography: in vivo imaging from organelles to organs. Science 335, 1458–1462 (2012).

  34. 34.

    Vellekoop, I. M. & Mosk, A. P. Focusing coherent light through opaque strongly scattering media. Opt. Lett. 32, 2309–2311 (2007). The first demonstration of optical-wave focusing through a scattering layer.

  35. 35.

    Yaqoob, Z., Psaltis, D., Feld, M. S. & Yang, C. Optical phase conjugation for turbidity suppression in biological samples. Nat. Photonics 2, 110–115 (2008).

  36. 36.

    Dorokhov, O. N. On the coexistence of localized and extended electronic states in the metallic phase. Solid. State Commun. 51, 381–384 (1984).

  37. 37.

    Feng, S., Kane, C., Lee, P. A. & Stone, A. D. Correlations and fluctuations of coherent wave transmission through disordered media. Phys. Rev. Lett. 61, 834–837 (1988).

  38. 38.

    Freund, I., Rosenbluh, M. & Feng, S. Memory effects in propagation of optical waves through disordered media. Phys. Rev. Lett. 61, 2328–2331 (1988).

  39. 39.

    Beenakker, C. W. J. Random-matrix theory of quantum transport. Rev. Mod. Phys. 69, 731–808 (1997).

  40. 40.

    Mosk, A. P., Lagendijk, A., Lerosey, G. & Fink, M. Controlling waves in space and time for imaging and focusing in complex media. Nat. Photonics 6, 283–292 (2012).

  41. 41.

    Rotter, S. & Gigan, S. Light fields in complex media: mesoscopic scattering meets wave control. Rev. Mod. Phys. 89, 015005 (2017).

  42. 42.

    Fink, M. Time reversal of ultrasonic fields. I. Basic principles. IEEE Trans. Ultrason. Ferroelectr. Freq. Control. 39, 555–566 (1992).

  43. 43.

    Derode, A., Roux, P. & Fink, M. Robust acoustic time reversal with high-order multiple scattering. Phys. Rev. Lett. 75, 4206–4209 (1995).

  44. 44.

    Fink, M. Time reversed acoustics. Phys. Today 50, 34–40 (1997).

  45. 45.

    Lerosey, G. et al. Time reversal of electromagnetic waves. Phys. Rev. Lett. 92, 193904 (2004).

  46. 46.

    Lerosey, G., de Rosny, J., Tourin, A., Derode, A. & Fink, M. Time reversal of wideband microwaves. Appl. Phys. Lett. 88, 154101 (2006).

  47. 47.

    Vellekoop, I. M. & Mosk, A. P. Universal optimal transmission of light through disordered materials. Phys. Rev. Lett. 101, 120601 (2008).

  48. 48.

    Kim, M., Choi, W., Yoon, C., Kim, G. H. & Choi, W. Relation between transmission eigenchannels and single-channel optimizing modes in a disordered medium. Opt. Lett. 38, 2994–2996 (2013).

  49. 49.

    Katz, O., Small, E. & Silberberg, Y. Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nat. Photonics 6, 549–553 (2012).

  50. 50.

    Judkewitz, B., Horstmeyer, R., Vellekoop, I. M., Papadopoulos, I. N. & Yang, C. Translation correlations in anisotropically scattering media. Nat. Phys. 11, 684–689 (2015).

  51. 51.

    Vellekoop, I. M., Lagendijk, A. & Mosk, A. P. Exploiting disorder for perfect focusing. Nat. Photonics 4, 320–322 (2010).

  52. 52.

    van Putten, E. G. et al. Scattering lens resolves sub-100 nm structures with visible light. Phys. Rev. Lett. 106, 193905 (2011).

  53. 53.

    Park, J.-H. et al. Subwavelength light focusing using random nanoparticles. Nat. Photonics 7, 454–458 (2013).

  54. 54.

    Choi, W. et al. Control of randomly scattered surface plasmon polaritons for multiple-input and multiple-output plasmonic switching devices. Nat. Commun. 8, 14636 (2017).

  55. 55.

    Gjonaj, B. et al. Focusing and scanning microscopy with propagating surface plasmons. Phys. Rev. Lett. 110, 266804 (2013).

  56. 56.

    Seo, E. et al. Far-field control of focusing plasmonic waves through disordered nanoholes. Opt. Lett. 39, 5838–5841 (2014).

  57. 57.

    Vellekoop, I. M. Feedback-based wavefront shaping. Opt. Express 23, 12189–12206 (2015).

  58. 58.

    Wang, D. et al. Focusing through dynamic tissue with millisecond digital optical phase conjugation. Optica 2, 728–735 (2015).

  59. 59.

    Conkey, D. B., Caravaca-Aguirre, M. A. & Piestun, R. High-speed scattering medium characterization with application to focusing light through turbid media. Opt. Express 20, 1733–1740 (2012).

  60. 60.

    Blochet, B., Bourdieu, L. & Gigan, S. Focusing light through dynamical samples using fast continuous wavefront optimization. Opt. Lett. 42, 4994–4997 (2017).

  61. 61.

    Liu, Y., Ma, C., Shen, Y., Shi, J. & Wang, L. V. Focusing light inside dynamic scattering media with millisecond digital optical phase conjugation. Optica 4, 280–288 (2017).

  62. 62.

    Feldkhun, D., Tzang, O., Wagner, K. H. & Piestun, R. Focusing and scanning through scattering media in microseconds. Optica 6, 72–75 (2019).

  63. 63.

    Ji, N., Milkie, D. E. & Betzig, E. Adaptive optics via pupil segmentation for high-resolution imaging in biological tissues. Nat. Methods 7, 141–147 (2009).

  64. 64.

    Park, J.-H., Sun, W. & Cui, M. High-resolution in vivo imaging of mouse brain through the intact skull. Proc. Natl Acad. Sci. USA 112, 9236–9241 (2015).

  65. 65.

    Frostig, H. et al. Focusing light by wavefront shaping through disorder and nonlinearity. Optica 4, 1073–1079 (2017).

  66. 66.

    Ruan, H. et al. Focusing light inside scattering media with magnetic-particle-guided wavefront shaping. Optica 4, 1337–1343 (2017).

  67. 67.

    Florentin, R. et al. Shaping the light amplified in a multimode fiber. Light Sci. Appl. 6, e16208 (2017).

  68. 68.

    Shen, Y., Liu, Y., Ma, C. & Wang, L. V. Focusing light through biological tissue and tissue-mimicking phantoms up to 9.6 cm in thickness with digital optical phase conjugation. J. Biomed. Opt. 21, 085001 (2016).

  69. 69.

    Popoff, S., Lerosey, G., Fink, M., Boccara, A. C. & Gigan, S. Image transmission through an opaque material. Nat. Commun. 1, 81 (2010).

  70. 70.

    Popoff, S. M. et al. Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media. Phys. Rev. Lett. 104, 100601 (2010). The first report of the measurement of a transmission matrix in optics.

  71. 71.

    Choi, Y. et al. Overcoming the diffraction limit using multiple light scattering in a highly disordered medium. Phys. Rev. Lett. 107, 023902 (2011).

  72. 72.

    Choi, Y. et al. Synthetic aperture microscopy for high resolution imaging through a turbid medium. Opt. Lett. 36, 4263–4265 (2011).

  73. 73.

    van Putten, E. & Mosk, A. The information age in optics: measuring the transmission matrix. Physics 3, 22 (2010).

  74. 74.

    Choi, Y. et al. Scanner-free and wide-field endoscopic imaging by using a single multimode optical fiber. Phys. Rev. Lett. 109, 203901 (2012).

  75. 75.

    Choi, Y., Yoon, C., Kim, M., Choi, W. & Choi, W. Optical imaging with the use of a scattering lens. IEEE J. Sel. Top. Quantum Electron. 20, 6800213 (2014).

  76. 76.

    Yoon, C. et al. Removal of back-reflection noise at ultrathin imaging probes by the single-core illumination and wide-field detection. Sci. Rep. 7, 6524 (2017).

  77. 77.

    Bianchi, S. & Di Leonardo, R. A multi-mode fiber probe for holographic micromanipulation and microscopy. Lab. Chip 12, 635–639 (2012). The first report of imaging through a multimode fibre.

  78. 78.

    Bianchi, S. et al. Focusing and imaging with increased numerical apertures through multimode fibers with micro-fabricated optics. Opt. Lett. 38, 4935–4938 (2013).

  79. 79.

    Čižmár, T. & Dholakia, K. Exploiting multimode waveguides for pure fibre-based imaging. Nat. Commun. 3, 1027 (2012).

  80. 80.

    Čižmár, T. & Dholakia, K. Shaping the light transmission through a multimode optical fibre: complex transformation analysis and applications in biophotonics. Opt. Express 19, 18871–18884 (2011).

  81. 81.

    Kim, D. et al. Toward a miniature endomicroscope: pixelation-free and diffraction-limited imaging through a fiber bundle. Opt. Lett. 39, 1921–1924 (2014).

  82. 82.

    Ohayon, S., Caravaca-Aguirre, A., Piestun, R. & DiCarlo, J. J. Minimally invasive multimode optical fiber microendoscope for deep brain fluorescence imaging. Biomed. Opt. Express 9, 1492–1509 (2018).

  83. 83.

    Caravaca-Aguirre, A. M. & Piestun, R. Single multimode fiber endoscope. Opt. Express 25, 1656–1665 (2017).

  84. 84.

    Turtaev, S. et al. High-fidelity multimode fibre-based endoscopy for deep brain in vivo imaging. Light. Sci. Appl. 7, 92 (2018).

  85. 85.

    Loterie, D. et al. Digital confocal microscopy through a multimode fiber. Opt. Express 23, 23845–23858 (2015).

  86. 86.

    Morales-Delgado, E. E., Psaltis, D. & Moser, C. Two-photon imaging through a multimode fiber. Opt. Express 23, 32158–32170 (2015).

  87. 87.

    Drémeau, A. et al. Reference-less measurement of the transmission matrix of a highly scattering material using a DMD and phase retrieval techniques. Opt. Express 23, 11898–11911 (2015).

  88. 88.

    del Hougne, P., Rajaei, B., Daudet, L. & Lerosey, G. Intensity-only measurement of partially uncontrollable transmission matrix: demonstration with wave-field shaping in a microwave cavity. Opt. Express 24, 18631–18641 (2016).

  89. 89.

    Porat, A. et al. Widefield lensless imaging through a fiber bundle via speckle correlations. Opt. Express 24, 16835–16855 (2016).

  90. 90.

    Farahi, S., Ziegler, D., Papadopoulos, I. N., Psaltis, D. & Moser, C. Dynamic bending compensation while focusing through a multimode fiber. Opt. Express 21, 22504–22514 (2013).

  91. 91.

    Loterie, D., Psaltis, D. & Moser, C. Bend translation in multimode fiber imaging. Opt. Express 25, 6263–6273 (2017).

  92. 92.

    Plöschner, M., Tyc, T. & Čižmár, T. Seeing through chaos in multimode fibres. Nat. Photonics 9, 529–535 (2015).

  93. 93.

    Boonzajer Flaes, D. E. et al. Robustness of light-transport processes to bending deformations in graded-index multimode waveguides. Phys. Rev. Lett. 120, 233901 (2018).

  94. 94.

    Freund, I. Looking through walls and around corners. Phys. A 168, 49–65 (1990).

  95. 95.

    Akkermans, E. & Montambaux, G. Mesoscopic Physics of Electrons and Photons (Cambridge Univ. Press, 2007).

  96. 96.

    Osnabrugge, G., Horstmeyer, R., Papadopoulos, I. N., Judkewitz, B. & Vellekoop, I. M. The generalized optical memory effect. Optica 4, 886–892 (2017).

  97. 97.

    Bertolotti, J. et al. Non-invasive imaging through opaque scattering layers. Nature 491, 232–234 (2012). The first study to use the optical memory effect for epi-fluorescence imaging.

  98. 98.

    Katz, O., Heidmann, P., Fink, M. & Gigan, S. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nat. Photonics 8, 784–790 (2014).

  99. 99.

    Yang, X., Pu, Y. & Psaltis, D. Imaging blood cells through scattering biological tissue using speckle scanning microscopy. Opt. Express 22, 3405–3413 (2014).

  100. 100.

    Schott, S., Bertolotti, J., Léger, J.-F., Bourdieu, L. & Gigan, S. Characterization of the angular memory effect of scattered light in biological tissues. Opt. Express 23, 13505–13516 (2015).

  101. 101.

    Amitonova, L. V., Mosk, A. P. & Pinkse, P. W. H. Rotational memory effect of a multimode fiber. Opt. Express 23, 20569–20575 (2015).

  102. 102.

    Stasio, N., Moser, C. & Psaltis, D. Calibration-free imaging through a multicore fiber using speckle scanning microscopy. Opt. Lett. 41, 3078–3081 (2016).

  103. 103.

    Fienup, J. R. Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758–2769 (1982).

  104. 104.

    Fienup, J. R. Reconstruction of an object from the modulus of its Fourier transform. Opt. Lett. 3, 27–29 (2008).

  105. 105.

    Papadopoulos, I. N., Jouhanneau, J. S., Poulet, J. F. A. & Judkewitz, B. Scattering compensation by focus scanning holographic aberration probing (F-SHARP). Nat. Photonics 11, 116–123 (2017).

  106. 106.

    Feng, G. et al. Imaging neuronal subsets in transgenic mice expressing multiple spectral variants of GFP. Neuron 28, 41–51 (2000).

  107. 107.

    Szabo, T. L. Diagnostic Ultrasound Imaging: Inside Out (Elsevier, 2004).

  108. 108.

    Elson, D. S., Li, R., Dunsby, C., Eckersley, R. & Tang, M. X. Ultrasound-mediated optical tomography: a review of current methods. Interface Focus. 1, 632–648 (2011).

  109. 109.

    Yao, G. & Wang, L. V. in Biomedical Optical Spectroscopy and Diagnostics (Optical Society of America, 2000).

  110. 110.

    Wang, L. V. Prospects of photoacoustic tomography. Med. Phys. 35, 5758–5767 (2008).

  111. 111.

    Wang, L. V. (ed.) Photoacoustic Imaging and Spectroscopy (CRC press, 2017).

  112. 112.

    Culjat, M. O., Goldenberg, D., Tewari, P. & Singh, R. S. A review of tissue substitutes for ultrasound imaging. Ultrasound Med. Biol. 36, 861–873 (2010).

  113. 113.

    Horstmeyer, R., Ruan, H. & Yang, C. Guidestar-assisted wavefront-shaping methods for focusing light into biological tissue. Nat. Photonics 9, 563–571 (2015).

  114. 114.

    Jang, M., Sentenac, A. & Yang, C. Optical phase conjugation (OPC)-assisted isotropic focusing. Opt. Express 21, 8781–8792 (2013).

  115. 115.

    Xu, X., Liu, H. & Wang, L. V. Time-reversed ultrasonically encoded optical focusing into scattering media. Nat. Photonics 5, 154–157 (2011). The first study to combine acousto-optic modulation and optical phase conjugation for optical imaging in a scattering medium.

  116. 116.

    Wang, Y. M., Judkewitz, B., Dimarzio, C. A. & Yang, C. Deep-tissue focal fluorescence imaging with digitally time-reversed ultrasound-encoded light. Nat. Commun. 3, 928 (2012).

  117. 117.

    Si, K., Fiolka, R. & Cui, M. Fluorescence imaging beyond the ballistic regime by ultrasound-pulse-guided digital phase conjugation. Nat. Photonics 6, 657–661 (2012).

  118. 118.

    Si, K., Fiolka, R. & Cui, M. Breaking the spatial resolution barrier via iterative sound-light interaction in deep tissue microscopy. Sci. Rep. 2, 748 (2012).

  119. 119.

    Ruan, H., Jang, M., Judkewitz, B. & Yang, C. Iterative time-reversed ultrasonically encoded light focusing in backscattering mode. Sci. Rep. 4, 7156 (2014).

  120. 120.

    Judkewitz, B., Wang, Y. M., Horstmeyer, R., Mathy, A. & Yang, C. Speckle-scale focusing in the diffusive regime with time reversal of variance-encoded light (TROVE). Nat. Photonics 7, 300–305 (2013).

  121. 121.

    Katz, O., Ramaz, F., Gigan, S. & Fink, M. Controlling light in complex media beyond the acoustic diffraction-limit using the acousto-optic transmission matrix. Nat. Commun. 10, 717 (2019).

  122. 122.

    Izatt, J. A., Swanson, E. A., Fujimoto, J. G., Hee, M. R. & Owen, G. M. Optical coherence microscopy in scattering media. Opt. Lett. 19, 590–592 (2008).

  123. 123.

    Chaigne, T. et al. Controlling light in scattering media non-invasively using the photoacoustic transmission matrix. Nat. Photonics 8, 58–64 (2014).

  124. 124.

    Lai, P., Wang, L., Tay, J. W. & Wang, L. V. Photoacoustically guided wavefront shaping for enhanced optical focusing in scattering media. Nat. Photonics 9, 126–132 (2015).

  125. 125.

    Conkey, D. B. et al. Super-resolution photoacoustic imaging through a scattering wall. Nat. Commun. 6, 7902 (2015).

  126. 126.

    Kang, S. et al. High-resolution adaptive optical imaging within thick scattering media using closed-loop accumulation of single scattering. Nat. Commun. 8, 2157 (2017). A report of a reflection matrix analysis algorithm to correct aberrations without using guide stars.

  127. 127.

    Kim, M. et al. Label-free neuroimaging in vivo using synchronous angular scanning microscopy with single-scattering accumulation algorithm. Nat. Commun. 10, 3152 (2019).

  128. 128.

    Colomb, T. et al. Numerical parametric lens for shifting, magnification, and complete aberration compensation in digital holographic microscopy. J. Opt. Soc. Am. A 23, 3177–3190 (2006).

  129. 129.

    Boppart, S. A. et al. Computational adaptive optics for broadband optical interferometric tomography of biological tissue. Proc. Natl Acad. Sci. USA 109, 933505 (2012).

  130. 130.

    Shemonski, N. D. et al. Computational high-resolution optical imaging of the living human retina. Nat. Photonics 9, 440–443 (2015).

  131. 131.

    Zheng, G., Ou, X., Horstmeyer, R. & Yang, C. Characterization of spatially varying aberrations for wide field-of-view microscopy. Opt. Express 21, 15131–15143 (2013).

  132. 132.

    Pozzi, P. et al. Anisoplanatic adaptive optics in parallelized laser scanning microscopy. Preprint at arXiv https://arxiv.org/abs/1809.07529 (2018).

  133. 133.

    Kim, M. et al. Maximal energy transport through disordered media with the implementation of transmission eigenchannels. Nat. Photonics 6, 581–585 (2012).

  134. 134.

    Prada, C. & Fink, M. Eigenmodes of the time reversal operator: a solution to selective focusing in multiple-target media. Wave Motion 20, 151–163 (1994).

  135. 135.

    Aubry, A. & Derode, A. Random matrix theory applied to acoustic backscattering and imaging in complex media. Phys. Rev. Lett. 102, 084301 (2009).

  136. 136.

    Popoff, S. M. et al. Exploiting the time-reversal operator for adaptive optics, selective focusing, and scattering pattern analysis. Phys. Rev. Lett. 107, 263901 (2011).

  137. 137.

    Choi, Y. et al. Measurement of the time-resolved reflection matrix for enhancing light energy delivery into a scattering medium. Phys. Rev. Lett. 111, 243901 (2013).

  138. 138.

    Jeong, S. et al. Focusing of light energy inside a scattering medium by controlling the time-gated multiple light scattering. Nat. Photonics 12, 277–283 (2018).

  139. 139.

    Badon, A. et al. Smart optical coherence tomography for ultra-deep imaging through highly scattering media. Sci. Adv. 2, e1600370 (2016). A study analysing the time-gated reflection eigenchannels for deep optical imaging.

  140. 140.

    Kadobianskyi, M., Papadopoulos, I. N., Chaigne, T., Horstmeyer, R. & Judkewitz, B. Scattering correlations of time-gated light. Optica 5, 389–394 (2018).

  141. 141.

    Osnabrugge, G., Amitonova, L. V. & Vellekoop, I. M. Blind focusing through strongly scattering media using wavefront shaping with nonlinear feedback. Opt. Express 27, 11673–11688 (2019).

  142. 142.

    McCann, M. T., Jin, K. H. & Unser, M. Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal. Process. Mag. 34, 85–95 (2017).

  143. 143.

    Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

  144. 144.

    Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).

  145. 145.

    Turpin, A., Vishniakou, I. & Seelig, J.d. Light scattering control in transmission and reflection with neural networks. Opt. Express 26, 30911–30929 (2018).

  146. 146.

    Rahmani, B., Loterie, D., Konstantinou, G., Psaltis, D. & Moser, C. Multimode optical fiber transmission with a deep learning network. Light. Sci. Appl. 7, 69 (2018).

  147. 147.

    Kamilov, U. S. et al. Learning approach to optical tomography. Optica 2, 517–522 (2015).

  148. 148.

    Sun, Y., Xia, Z. & Kamilov, U. S. Efficient and accurate inversion of multiple scattering with deep learning. Opt. Express 26, 14678–14688 (2018).

  149. 149.

    Ntziachristos, V. Going deeper than microscopy: the optical imaging. Nat. Methods 7, 603–614 (2010).

Download references

Acknowledgements

This research was supported by the Institute for Basic Science (grant no. IBS-R023-D1), the National Research Foundation of Korea (grant nos. NRF-2019R1C1C1008175 and NRF-2016R1A6A3A11936389), and the Catholic Medical Center Research Foundation in the programme year of 2018.

Author information

The authors contributed equally to all aspects of the article.

Correspondence to Wonshik Choi.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Glossary

Isoplanatic patch

The area over which the wavefront error remains almost the same.

Guide stars

Bright, point-like light sources or scatterers that provide a wavefront reference for measuring and correcting wavefront distortions in adaptive optics systems.

Point optimization

A method or algorithm that optimizes the point spread function by minimizing wavefront errors in adaptive optics systems.

Spatial light modulator

A device that modulates the amplitude, phase or polarization of light waves in space.

Optical memory effect

The phenomenon that speckle patterns of scattered light through thin and diffusive media are invariant to small tilts or shifts in an incident wavefront of light.

Digital micromirror devices

Micromirrors used for high-speed, efficient and reliable spatial-light modulation; originally invented to create video displays in digital projectors.

Epi-detection geometry

An imaging configuration in which an objective is used for both illumination and detection.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yoon, S., Kim, M., Jang, M. et al. Deep optical imaging within complex scattering media. Nat Rev Phys 2, 141–158 (2020). https://doi.org/10.1038/s42254-019-0143-2

Download citation