Article | Open | Published:

# Single-shot real-time femtosecond imaging of temporal focusing

Light: Science & Applicationsvolume 7, Article number: 42 (2018) | Download Citation

## Abstract

While the concept of focusing usually applies to the spatial domain, it is equally applicable to the time domain. Real-time imaging of temporal focusing of single ultrashort laser pulses is of great significance in exploring the physics of the space–time duality and finding diverse applications. The drastic changes in the width and intensity of an ultrashort laser pulse during temporal focusing impose a requirement for femtosecond-level exposure to capture the instantaneous light patterns generated in this exquisite phenomenon. Thus far, established ultrafast imaging techniques either struggle to reach the desired exposure time or require repeatable measurements. We have developed single-shot 10-trillion-frame-per-second compressed ultrafast photography (T-CUP), which passively captures dynamic events with 100-fs frame intervals in a single camera exposure. The synergy between compressed sensing and the Radon transformation empowers T-CUP to significantly reduce the number of projections needed for reconstructing a high-quality three-dimensional spatiotemporal datacube. As the only currently available real-time, passive imaging modality with a femtosecond exposure time, T-CUP was used to record the first-ever movie of non-repeatable temporal focusing of a single ultrashort laser pulse in a dynamic scattering medium. T-CUP’s unprecedented ability to clearly reveal the complex evolution in the shape, intensity, and width of a temporally focused pulse in a single measurement paves the way for single-shot characterization of ultrashort pulses, experimental investigation of nonlinear light-matter interactions, and real-time wavefront engineering for deep-tissue light focusing.

## Introduction

The space–time duality in optics originates from the mathematical equivalence between paraxial diffraction and dispersive propagation1. Remarkably, this duality enables one to translate spatial-domain optical techniques to the temporal domain, which has fostered the development of powerful temporal imaging approaches, such as temporal microscopy, to characterize optical signals2,3. Among the many temporal imaging phenomena, temporal focusing, as a time-domain counterpart of spatial focusing, describes an exquisite optical phenomenon—temporal compression of the duration of a chirped laser pulse to the shortest time possible at a designated location4,5,6. Temporal focusing has been leveraged in the temporal 4f processor7 and the dispersive Fourier transformer8 for analyzing optical waveforms with unprecedented bandwidths. Akin to spatial focusing—confining photons laterally—temporal focusing enables photon confinement in the longitudinal direction. This salient feature has powered depth-sectioning wide-field nonlinear microscopy for neuroimaging9. Recently, temporal focusing has been achieved through static scattering media10, which has sparked interest in deep biomedical imaging. In addition, the strong intensity localization has made it an attractive tool for material processing11, which has led to extensive studies of elusive physics mechanisms of the strong-field interaction with matter12. Considering the stochastic (e.g., time reversal of dynamic speckle patterns to produce temporal focusing in live biological tissue13) and non-repeatable (e.g., micromachining using temporal focusing in glass14) nature of these transient phenomena, visualizing temporal focusing in real time (i.e., at its actual time of occurrence) becomes a prerequisite for investigating and further utilizing them. In addition, since the width and intensity of the laser pulse experiences drastic changes during temporal focusing, a femtosecond-level exposure time is required to clearly resolve the evolving instantaneous spatiotemporal details of this phenomenon. Moreover, the nanometer-to-micrometer spatial scales of these transient events demand ultrafast imaging for blur-free observation [e.g., for imaging a light-speed event, an imaging speed of 1 trillion frames-per-second (Tfps) is required for a spatial resolution of 300 µm]15. Finally, since these events are often self-luminescent, a passive (i.e., receive-only) detector is highly desired for direct recording.

Existing ultrafast imaging techniques, however, are incapable of providing real-time, femtosecond, passive imaging capability. The current mainstream technique used in ultrafast imaging is based on pump–probe measurements16,17. Although having achieved femtosecond temporal resolution and passive detection, these multiple-shot imaging techniques depend on precise repetition of the targeted ultrafast event during temporal or spatial scanning. Hence, in cases where temporal focusing must be recorded in a single measurement, these imaging techniques are inapplicable.

Recently, a number of single-shot ultrafast imaging techniques18,19,20,21,22 have been developed. Among them, active-illumination-based approaches have achieved frame rates at the Tfps level20,21. However, such approaches are incapable of imaging luminescent transient events, so they are precluded from direct imaging of evolving light patterns in temporal focusing. The requirement of active illumination was recently eliminated by a new single-shot ultrafast imaging modality, termed compressed ultrafast photography (CUP)23,24,25. Synergizing compressed sensing and streak imaging, CUP works by first compressively recording a three dimensional (3D, i.e., x,y,t) scene into a two-dimensional (2D) snapshot and then computationally recovering it by solving an optimization problem. The resultant CUP system can passively receive photons scattered or emitted from dynamic scenes at frame rates of up to 100 billion fps. CUP has been applied to a number of applications, including fluorescence lifetime mapping23, real-time imaging of a propagating photonic Mach cone24, and time-of-flight volumetric imaging25. However, in these previous studies, the frame interval (defined as the reciprocal of the frame rate) was 10 ps, which has hindered the use of CUP for imaging spatiotemporal details of temporal focusing in the femtosecond regime.

## Results

### Principle and system of T-CUP

To enable real-time, ultrafast, passive imaging of temporal focusing, here, we have developed single-shot trillion-frame-per-second compressed ultrafast photography (T-CUP), which can image non-repeatable transient events at a frame rate of up to 10 Tfps in a receive-only fashion. The operation of T-CUP consists of data acquisition and image reconstruction (Fig. 1). For the data acquisition, the intensity distribution of a 3D spatiotemporal scene, I[m,n,k], is first imaged with a beam splitter to form two images. The first image is directly recorded by a 2D imaging sensor via spatiotemporal integration (defined as spatial integration over each pixel and temporal integration over the entire exposure time). This process, which forms a time-unsheared view with an optical energy distribution of Eu[m,n], can be expressed by

$$E_{\rm{u}}\left[ {m,n} \right] = \eta \mathop {\sum }\limits_k \left( {h_{\rm{u}} \ast I} \right)\left[ {m,n,k} \right]$$
(1)

where η is a constant, hu represents spatial low-pass filtering imposed by optics in the time-unsheared view, and * denotes the discrete 2D spatial convolution operation. Equation 1 can be regarded as a single-angle Radon transformation operated on I[m, n, k] (detailed in Supplementary Note 1).

The second image is spatially encoded by a pseudo-random binary pattern. Then the spatially encoded scene is relayed to a femtosecond shearing unit, where temporal frames are sheared on one spatial axis. Finally, the spatially encoded, temporally sheared frames are recorded by another 2D imaging sensor via spatiotemporal integration to form a time-sheared view with an optical energy distribution of Es[m,n]. This process can be described by

$$E_{\rm{s}}\left[ {m,n} \right] = \eta \mathop {\sum }\limits_k \left( {h_{\rm{s}} \ast I_{\rm{C}}} \right)\left[ {f_{\rm{D}},g_{\rm{D}} + k,k} \right]$$
(2)

where hs represents spatial low-pass filtering in the time-sheared view. IC[fD,gD+k,k] is the spatially encoded scene. fD and gD are the discrete coordinates transformed from m and n, according to the distortion in the time-sheared view24. Equation 2 can be regarded as the Radon transformation of the spatiotemporal datacube from an oblique angle determined by the shearing speed of the streak camera and pixel size of the sensor (detailed in Supplementary Note 1).

Combining the two views, the data acquisition of T-CUP can be expressed by a linear equation,

$$\left[ {E_{\rm{u}},\alpha E_{\rm{s}}} \right]^T = \left[ {{\boldsymbol{O}}_{\rm{u}},\alpha {\boldsymbol{O}}_{\rm{s}}} \right]^TI$$
(3)

where α is a scalar factor introduced to balance the energy ratio between the two views during measurement, and Ou and Os are the measurement operators for the two views (see Materials and methods and Supplementary Fig. S1). Thus T-CUP records a 3D dynamic scene into two 2D projections in a single exposure.

Image reconstruction of the scene can be done by solving the minimization problem of $$\min _I\left\{ {\frac{1}{2}\Vert \left[ {E_{\rm{u}},\alpha E_{\rm{s}}} \right]^T - \left[ {{\boldsymbol{O}}_{\rm{u}},\alpha {\boldsymbol{O}}_{\rm{s}}} \right]^TI \Vert_2^2 + \rho {\it{\Phi }}\left( I \right)} \right\}$$, where $$\left\| \cdot \right\|$$ denotes the l2 norm, Ф(I) is a regularization function that promotes sparsity in the dynamic scene, and ρ is the regularization parameter (detailed in Supplementary Notes 2). The solution to this minimization problem can be stably and accurately recovered, even with a highly compressed measurement26.

The integration of compressed sensing into the Radon transformation drastically reduces the required number of projections to two. The time-unsheared view, in which the projection is parallel to the time axis, losslessly retains spatial information while discarding all temporal information. The time-sheared view, on the other hand, preserves temporal information by projecting the spatiotemporal datacube from an oblique angle. As a result, these two views, as an optimal combination, enable one to record an optimum amount of information with the minimum number of measurements. However, a direct inversion of the Radon transform is not possible in this case due to the small number of projections and the fact that the linear system (Eq. 3) that needs to be inverted is under-determined. To solve this problem, compressed sensing is used. Leveraging the sparsity of the scene, as well as the random encoding in the time-sheared view as prior information, the compressed-sensing-based reconstruction algorithm uses the regularization-function-guided search to find a unique solution. Our simulation has demonstrated that this compressed-sensing-augmented two-view projection can retrieve a dynamic scene with a high reconstruction quality (Supplementary Fig. S2 and detailed in Supplementary Note 3).

In practice, T-CUP is embodied in an imaging system (Fig. 2 and detailed in Materials and methods) that uses several key devices to realize specific operations. Specifically, a charge-coupled device (CCD) camera performs spatiotemporal integration, a digital micromirror device (DMD) performs spatial encoding, and the time-varying voltage applied to the sweep electrodes in a femtosecond streak camera accomplishes femtosecond shearing. In addition, a compressed-sensing-based two-view reconstruction algorithm recovers the dynamic scene. The T-CUP system can capture a dynamic scene with spatial dimensions of 450 × 150 pixels and a sequence depth (i.e., number of frames per movie) of 350 frames in a single camera exposure. The frame rate of the reconstructed video is determined by v/d, where v is the temporal shearing velocity of the streak camera, and d is the pixel size of the internal CCD along the temporal shearing direction. By varying v, the frame rate can be widely adjusted from 0.5 to 10 Tfps. Thus, with single-shot data capture, a tunable ultrahigh frame rate, and an appreciable sequence depth, the T-CUP system is well suited for imaging single-event ultrafast transient phenomena occurring over a wide range of timescales (the characterization of the spatial and temporal resolutions of T-CUP is detailed in Supplementary Fig. S3 and Supplementary Note 4). The T-CUP temporal resolutions for 0.5, 1, 2.5, and 10 Tfps frame rates have been quantified to be 6.34, 4.53, 1.81, and 0.58 ps, respectively.

### Imaging temporal focusing of a single femtosecond laser pulse using the T-CUP system

A typical temporal focusing setup consists of a diffraction grating and a 4f imaging system (Fig. 3a). The incident laser pulse is first spatially dispersed by the grating and then collected by a collimation lens. Finally, a focusing lens recombines all the frequencies at the focal plane of the lens (Supplementary Fig. S4 and detailed in Supplementary Note 5). Temporal focusing has two major features: first, the shortest pulse width is at the focal plane of the focusing lens4; second, the angular dispersion of the grating creates a pulse front tilt so that the recombined pulse scans across the focal plane5. The pulse front tilt angle can be expressed by $$\gamma = \tan ^{ - 1}\left( {\lambda _{\rm{c}}/Md_{\rm{g}}} \right)$$ (refs. 27,28), where M is the overall magnification ratio, λc is the central wavelength of the ultrashort pulse, and dg is the grating period. The femtosecond pulse that undergoes temporal focusing presents a complex spatiotemporal profile (Supplementary Fig. S4) that can be revealed only in the captured instantaneous light patterns. Even a picosecond-level exposure time would erase these spatiotemporal details via significant temporal blurring. This speed requirement excludes previous CUP systems23,24,25 from visualizing this ultrafast optical phenomenon. In contrast, T-CUP can achieve unprecedented real-time visualization with a single camera exposure.

We imaged the temporal focusing from both the front and the side (Fig. 3a) at 2.5 Tfps. A collimated femtosecond laser pulse (800 nm central wavelength, 50 fs pulse duration, 1 × 3 mm2 spatial beam size) was used to illuminate a 1200 line mm−1 grating. The 4f imaging system had a magnification ratio of M=1/4. In theory, the tilt angle for the pulse front at the temporal focusing plane was 75.4°.

For front-view detection, T-CUP captured the impingement of the tilted laser pulse front sweeping along the y axis of the temporal focusing plane (Fig. 3b and Supplementary Movie S1). The pulse swept a distance of ~0.75 mm over 10 ps, corresponding to a pulse front tilt of ~76°, which closely matches the theoretical prediction.

For side-view detection, weak water vapor was spread as a dynamic scattering medium. T-CUP revealed the full evolution of the pulse propagation across the temporal focusing plane (Fig. 3c, d, Supplementary Fig. S5, and Supplementary Movies S1 and S2): a tilted pulse propagates toward the right. As it approaches the temporal focusing plane, the pulse width continuously reduces, manifesting as an increasing intensity. At the temporal focusing plane, the focus of the pulse sweeps along the y axis at its peak intensity. The evolution after the temporal focusing plane mirrors the preceding process: the pulse width is elongated, and the intensity is continuously weakened. We then quantitatively analyzed the pulse compression effect of temporal focusing. Figure 3e shows the temporal profiles of the laser pulse on the z axis near the temporal focusing plane, demonstrating the sharp temporal focusing of the laser pulse. Figure 3f shows the pulse duration along the z axis near the temporal focusing plane. The full width at half maximum of the temporal profile is reduced from 10.4 ps to 1.9 ps—compressed by a factor of 5.5. It is notable that the measured pulse width is wider than the incident pulse, which is likely due to dispersion by optical elements and scattering, as well as to the temporal broadening caused by the finite temporal resolution of the T-CUP system.

T-CUP is currently the only technology capable of observing temporal focusing in real time. First, the entire process of the imaged temporal focusing event occurred in ~10 ps, which equals the previous state-of-the-art exposure time for a single frame23; hence, it could not be resolved previously. In contrast, T-CUP, using a frame interval of 0.4 ps, clearly resolved the intensity fluctuation, width compression, and structural change of the temporal focusing process. Second, the dynamic scattering induced by the water vapor makes the scattered temporal focusing pulse non-repeatable. In different measurements, the reconstructed results show a difference in spatial shape, compression ratio, and intensity fluctuation. To demonstrate the non-repeatability, another dataset for the sideways detection of temporal focusing is shown in Supplementary Fig. S6.

Although the ultrashort laser pulse was dispersed and converged in space by the 4f imaging system, it is worth noting that the effect of spatial focusing is limited. As the pulse approached the temporal focusing plane, the beam size fluctuated with a normalized standard deviation of 5.6% over a duration of 4.8 ps (Fig. 3d), while the peak on-axis intensity of the pulse increased approximately five-fold (Fig. 3e). Thus the intensity increase is caused dominantly by the temporal focusing.

### Imaging light-speed phenomena in real time in both the visible and near-infrared spectral ranges

Four fundamental optical phenomena, namely, a beam sweeping across a surface, spatial focusing, splitting, and reflection, were imaged by the T-CUP system in real time (Fig. 4). In the beam sweeping experiment, a collimated near-infrared ultrashort laser pulse (800 nm wavelength, 50 fs pulse duration) obliquely impinged on a scattering bar pattern. The T-CUP system was placed perpendicular to the target to collect the scattered photons (Fig. 4a). Imaging at 10 Tfps, the T-CUP system clearly reveals how the pulse front of the ultrashort laser pulse swept across the bar pattern (Fig. 4b and Supplementary Movie S3).

In addition, T-CUP enables real-time video recording of spatial focusing of a single picosecond pulse. This phenomenon has been previously documented by phase-contrast microscopy29 and interferometry30 using conventional pump–probe schemes. In contrast, here, T-CUP was used to capture the scattered light intensity in a single measurement. In the setup, a single laser pulse (532 nm wavelength, 7 ps pulse width) was focused by a 10× objective lens into a weakly scattering aqueous suspension. T-CUP imaged this phenomenon at 2.5 Tfps (Fig. 4c and Supplementary Movie S4). We analyzed the time course of the light intensity at the spatial focus. After normalization, the intensity profile (Fig. 4d) was fitted by a Gaussian function,$$\hat I\left( t \right) = {\rm{exp}}\left[ { - 2\left( {t - t_0} \right)^2/\tau _{\rm{g}}^2} \right]$$, where t0 = 24.76 ps, and τg = 4.94 ps. The fitted result yields a 1/e width of 6.99 ps, closely matching the experimental specifications.

Imaging at 2.5 Tfps, T-CUP also revealed the spatiotemporal details of the beam splitting process of a single laser pulse (Fig. 4e and Supplementary Movie S5). Impinging on a beam splitter, part of the laser pulse was reflected immediately, while the transmitted portion propagated into the beam splitter and appeared on the other side of the beam splitter after a finite time. To quantitatively analyze the time course of the incident and transmitted pulses, we calculated the average light intensities in the two dashed boxes on both sides of the beam splitter (Fig. 4f). The measured temporal separation between the incident and transmitted pulses was 9.6 ps. Given the 2-mm thickness of this float glass beam splitter (refractive index n=1.52 at 532 nm) and the incident angle of ~25°, in theory, the light pulse needs approximately 10 ps to pass through the beam splitter. Thus our measured result agrees well with the theoretical value. It is also noteworthy that the time latency for the reflected and transmitted pulse (9.6 ps) is beyond the imaging capability of previous techniques23. T-CUP’s unprecedented frame rate reveals for the first time the spatiotemporal details of this transient event.

Finally, imaging at 1 Tfps, T-CUP was used to capture the reflection of a laser pulse by two mirrors over a sufficiently long time window (Supplementary Movie S6). In Fig. 4g, the first frame shows that the laser pulse has just entered the field of view (FOV). Subsequent frames show the propagating pulse being reflected by the two mirrors before finally traveling out of the FOV. It is noted that an inhomogeneous distribution of scatterers in the aqueous suspension led to increased scattered light intensity in the frames after 74 ps. For this reason, the pulse visually appears to be larger. However, the pulse width, when quantitatively measured via the cross-sectional full width at half maximum, was comparable to that in the rest of the frames.

## Discussion

### Current limitations

The performance of the streak camera, and not the principle of the technique, hinders further increases in frame rate, as well as other important characteristics, such as the spatial resolution and spectral range. The limited performance of the streak camera also impacts the choice of a single-sheared view in the system design (detailed in Supplementary Note 6). Finally, the imaging duty cycle for the T-CUP is currently limited to 5 × 10–9–10–7 due to the modest sweep frequency (100 fps) and the size of the internal sensor of the streak camera. A precise synchronization is therefore necessary to capture transient events within the time window. A new streak tube design and customized optical components would enable future implementations of a lossless-encoding scheme24, which is anticipated to improve the spatial and temporal resolutions in reconstructed images. In addition, the implementations of dual sweep-electrode pairs31 and an ultra-large-format camera32 are expected to largely increase the duty cycle with the possibility of even realizing continuous streaming.

### Application potential

Single-shot real-time imaging of temporal focusing is expected to immediately benefit the study of nonlinear light–matter interactions. For example, in femtosecond laser 3D micromachining using transparent media (e.g., glass), it was found that temporal focusing can induce an anisotropic fabrication quality33 depending on the translation direction of the sample. Thus far, the underlying mechanism for this nonreciprocal writing effect remains elusive. Recent theoretical investigations have indicated a close relation to the plasma dynamics controlled by the tilted pulse front of the temporal focusing pulses34. The T-CUP system can substitute for the low-speed cameras that are currently employed in imaging the laser–glass interaction35. Specifically, by changing the current zoom imaging system to a 20×, high numerical aperture (NA) objective lens, the microscopic T-CUP system will provide a 10-Tfps frame rate, a 1-µm spatial resolution, and 150-µm FOV at the sample, which is sufficient to simultaneously capture the evolution of a temporally focused pulse and the induced plasma (using a 10×, 0.2-NA objective lens as the focusing lens in Fig. 3a)10. The measured spatiotemporal profiles will be analyzed using the established models36 to investigate how the pulse front tilt and laser pulse energy affect the transient structure, dispersion properties, and spatial density of the induced plasma. The advantages of single-shot and ultrafast imaging will also pave the way for studying the plasma dynamics generated at microscopically heterogeneous locations (e.g., impurities and defects) in these materials.

Single-shot real-time imaging of temporal focusing by T-CUP also opens up new routes for spatiotemporal characterization of optical waveforms. Currently, temporal microscopes are often deployed as ultrafast all-optical oscilloscopes2 to passively analyze optical waveforms with few picosecond temporal resolution37 at a specific spatial point. The resolution quantification and imaging experiments in our work have demonstrated that T-CUP, while achieving a comparable temporal resolution, outperforms these oscilloscopes by adding a passive two-spatial-dimensional imaging ability. Thus the large parallel characterization of T-CUP could enable simultaneous ultrafast optical signal processing at multiple wavelengths for telecommunication38.

In metrology, a spatiotemporal microscope developed from T-CUP could be well suited for characterizing spatiotemporally complex ultrashort pulses39. In many time-resolved high-field laser experiments, the laser systems employed usually have low repetition rates. Therefore, single-shot characterization powered by T-CUP is attractive especially for fast and precise alignment of the setup40 and for imaging samples that are difficult to be repeatedly delivered41.

In biomedicine, T-CUP holds promise for in vivo tissue imaging. Living biological tissue is an example of dynamic scattering media with a millisecond-level speckle decorrelation time42. Thus far, owing to the limited speed of wavefront characterization in existing methods, spatiotemporal focusing beyond the optical diffusion limit has only been realized with static scattering media43,44. In contrast, T-CUP demonstrates single-shot femtosecond imaging of transient light patterns in a dynamic scattering medium (Fig. 3c). By integrating T-CUP with interferometry, it is possible to examine the scattered electric field of a broadband beam, which would assist in the design of phase conjugation of spatiotemporal focusing in living biological tissue. Therefore, our work, as an important step in imaging instrumentation, will open up new routes toward deep-tissue wide-field two-photon microscopy, photodynamic therapy, and optogenetics.

### Summary

By improving the frame rate by two orders of magnitude compared with the previous state-of-the-art23, T-CUP demonstrated that the ever-lasting pursuit of a higher frame rate is far from ending. As the only detection solution thus far available for passively probing dynamic self-luminescent events at femtosecond timescales in real time, T-CUP was used to reveal spatiotemporal details of transient scattering events that were inaccessible using previous systems. The compressed-sensing-augmented projection extended the application of the Radon transformation to probing spatiotemporal datacubes. This general scheme can be potentially implemented in other imaging modalities, such as tomographic phase microscopy45 and time-of-flight volumography46. T-CUP’s unprecedented ability for real-time, wide-field, femtosecond-level imaging from the visible to the near-infrared will pave the way for future microscopic investigations of time-dependent optical and electronic properties of novel materials under transient out-of-equilibrium conditions47. With continuous improvement in streak camera technologies48, future development may enable a 1 quadrillion fps (1015 fps) frame rate with a wider imaging spectral range, allowing direct visualization and exploration of irreversible chemical reactions49 and nanostructure dynamics50.

## Materials and methods

### Summary of the principle of operation of T-CUP

We first derive the expression for the data acquisition of T-CUP in a continuous model. For data acquisition, T-CUP records the intensity distribution of the dynamic scene, I(x, y, t), in two projected views (Supplementary Fig. S1 and detailed in Supplementary Note 1). The first view, termed the time-unsheared view, directly records the dynamic scene with an external CCD camera (Fig. 2). This recording process is expressed as

$$E_{\rm{u}} = {\boldsymbol{TF}}_{\mathbf{u}}I\left( {x,y,t} \right)$$
(4)

where Eu denotes the measured optical energy distribution on the external CCD camera, the linear operator Fu represents the spatial low-pass filtering in the time-unsheared view, and T represents the spatiotemporal integration.

The second view, termed the time-sheared view, records the projected view of the spatiotemporal scene from an oblique angle (Supplementary Fig. S1). Specifically, the dynamic scene is first spatially encoded by a pseudo-random binary mask, followed by femtosecond shearing along one spatial axis by a time-varying voltage applied to a pair of sweep electrodes before the scene is finally spatiotemporally integrated on an internal CCD camera in the streak camera. Mathematically, the optical energy measured by the internal CCD camera, Es, is related to I(x, y, t) by

$$E_{\rm{s}} = {\boldsymbol{TS}}_{\mathbf{f}}{\boldsymbol{DF}}_{\mathbf{s}}{\boldsymbol{C}}I\left( {x,y,t} \right)$$
(5)

where the linear operator C represents spatial encoding, Fs represents spatial low-pass filtering in the time-sheared view, D represents image distortion in the time-sheared view with respect to the time-unsheared view, and Sf represents femtosecond shearing.

With the two-view projection, the data acquisition of T-CUP can be described as

$$E = {\boldsymbol{O}}I$$
(6)

where E=[Eu,αEs]T and O=[TFu,αTSfDFsC]T are the measurement and the linear operators in their concatenated forms, respectively. The scalar factor α is the energy calibration ratio between the external CCD camera and the streak camera.

For image reconstruction, we discretized Eqs. 46 to obtain Eqs. 13 (detailed in Supplementary Note 1). Given the known measurement matrix and leveraging the intrinsic sparsity in the dynamic scene, we estimate that the datacube for the transient scene by solving the inverse problem of Eq. 3. In practice, a two-view reconstruction method, aided by the two-step iterative shrinkage/thresholding algorithm, is implemented to recover the image (detailed in Supplementary Note 2). The T-CUP system greatly improved the reconstruction quality compared with a previously reported CUP system23 (illustrated in Supplementary Fig. S2 and detailed in Supplementary Note 3).

### System configuration

The T-CUP system configuration is shown in Fig. 2. The dynamic scene is first imaged by a zoom imaging system built in-house, which supports tunable demagnification ratios of 2–5×. Following the intermediate image, a 50:50 beam splitter sends the incident light in two directions. The reflected beam is recorded by an external CCD camera (Point Grey, GS3-U3-28S4M-C). The transmitted beam is passed onto a DMD (Texas Instruments, LightCrafter 3000) by a 4f imaging system with a unit magnification ratio. A pseudo-random binary pattern is displayed on the DMD to encode the input image. As a binary-amplitude spatial light modulator, the DMD consists of hundreds of thousands of micromirrors; each mirror can be tilted to either +12° (as “on” pixels) or –12° (as “off” pixels). The light reflected by the “on” pixels is re-collected by the same 4f imaging system. After being reflected by the beam splitter, the spatially encoded dynamic scene is projected onto the entrance port of a femtosecond streak camera (Hamamatsu, C6138). To enable time-resolved measurement in two spatial dimensions, the entrance port is opened to its full width (3 mm). Inside the streak camera, the spatially encoded dynamic scene is first relayed to a photocathode that generates a number of photoelectrons proportional to the light intensity distribution. To temporally shear the spatially encoded dynamic scene, a sweep voltage deflects the photoelectrons to different vertical positions according to their time of flight. The deflected photoelectrons are multiplied by a micro-channel plate and then converted back into light by a phosphor screen. Relayed by output optics, the temporally sheared, spatially encoded dynamic scene is captured by an internal CCD camera (Hamamatsu, ORCA-R2) with 2 × 2 binning (672 × 512 binned pixels, 12.9 × 12.9 μm2 binned pixel size). With two-view recording, the light throughput for the T-CUP system is 62.5%.

Accepted article preview online: 27 June 2018

## References

1. 1.

Kolner, B. H. Space-time duality and the theory of temporal imaging. IEEE J. Quant. Electron. 30, 1951–1963 (1994).

2. 2.

Foster, M. A. et al. Silicon-chip-based ultrafast optical oscilloscope. Nature 456, 81–84 (2008).

3. 3.

Patera, G., Shi, J., Horoshko, D. B. & Kolobov, M. I. Quantum temporal imaging: application of a time lens to quantum optics. J. Opt. 19, 054001 (2017).

4. 4.

Zhu, G. H., van Howe, J., Durst, M., Zipfel, W. & Xu, C. Simultaneous spatial and temporal focusing of femtosecond pulses. Opt. Express 13, 2153–2159 (2005).

5. 5.

Oron, D., Tal, E. & Silberberg, Y. Scanningless depth-resolved microscopy. Opt. Express 13, 1468–1476 (2005).

6. 6.

Papagiakoumou, E. et al. Functional patterned multiphoton excitation deep inside scattering tissue. Nat. Photonics 7, 274–278 (2013).

7. 7.

Salem, R., Foster, M. A. & Gaeta, A. L. Application of space–time duality to ultrahigh-speed optical signal processing. Adv. Opt. Photonics 5, 274–317 (2013).

8. 8.

Goda, K. & Jalali, B. Dispersive Fourier transformation for fast continuous single-shot measurements. Nat. Photonics 7, 102–112 (2013).

9. 9.

Papagiakoumou, E. et al. Scanless two-photon excitation of channelrhodopsin-2. Nat. Methods 7, 848–854 (2010).

10. 10.

Katz, O., Small, E., Bromberg, Y. & Silberberg, Y. Focusing and compression of ultrashort pulses through scattering media. Nat. Photonics 5, 372–377 (2011).

11. 11.

Beresna, M., Gecevičius, M. & Kazansky, P. G. Ultrafast laser direct writing and nanostructuring in transparent materials. Adv. Opt. Photonics 6, 293–339 (2014).

12. 12.

Jing, C. R., Wang, Z. H. & Cheng, Y. Characteristics and applications of spatiotemporally focused femtosecond laser pulses. Appl. Sci. 6, 428 (2016).

13. 13.

Stockbridge, C. et al. Focusing through dynamic scattering media. Opt. Express 20, 15086–15092 (2012).

14. 14.

Kammel, R. et al. Enhancing precision in fs-laser material processing by simultaneous spatial and temporal focusing. Light Sci. Appl. 3, e169 (2014).

15. 15.

Mikami, H., Gao, L. & Goda, K. Ultrafast optical imaging technology: principles and applications of emerging methods. Nanophotonics 5, 98–110 (2016).

16. 16.

Schaffer, C. B., Nishimura, N., Glezer, E. N., Kim, A. M. T. & Mazur, E. Dynamics of femtosecond laser-induced breakdown in water from femtoseconds to microseconds. Opt. Express 10, 196–203 (2002).

17. 17.

Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3, 745 (2012).

18. 18.

Li, Z. Y., Zgadzaj, R., Wang, X. M., Chang, Y. Y. & Downer, M. C. Single-shot tomographic movies of evolving light-velocity objects. Nat. Commun. 5, 3085 (2014).

19. 19.

Goda, K., Tsia, K. & Jalali, B. Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena. Nature 458, 1145–1149 (2009).

20. 20.

Nakagawa, K. et al. Sequentially timed all-optical mapping photography (STAMP). Nat. Photonics 8, 695–700 (2014).

21. 21.

Ehn, A. et al. FRAME: femtosecond videography for atomic and molecular dynamics. Light Sci. Appl. 6, e17045 (2017).

22. 22.

Kubota, T., Komai, K., Yamagiwa, M. & Awatsuji, Y. Moving picture recording and observation of three-dimensional image of femtosecond light pulse propagation. Opt. Express 15, 14348–14354 (2007).

23. 23.

Gao, L., Liang, J. Y., Li, C. Y. & Wang, L. V. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 74–77 (2014).

24. 24.

Liang, J. Y. et al. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Sci. Adv. 3, e1601814 (2017).

25. 25.

Liang, J. Y., Gao, L., Hai, P. F., Li, C. Y. & Wang, L. V. Encrypted three-dimensional dynamic imaging using snapshot time-of-flight compressed ultrafast photography. Sci. Rep. 5, 15504 (2015).

26. 26.

Candès, E. J. The restricted isometry property and its implications for compressed sensing. C. R. Math. 346, 589–592 (2008).

27. 27.

Bor, Z., Racz, B., Szabo, G., Hilbert, M. & Hazim, H. A. Femtosecond pulse front tilt caused by angular dispersion. Opt. Eng. 32, 2501–2504 (1993).

28. 28.

Hebling, J. Derivation of the pulse front tilt caused by angular dispersion. Opt. Quant. Electron. 28, 1759–1763 (1996).

29. 29.

Mermillod-Blondin, A. et al. Time-resolved imaging of laser-induced refractive index changes in transparent media. Rev. Sci. Instrum. 82, 033703 (2011).

30. 30.

Sun, Q. et al. Measurement of the collision time of dense electronic plasma induced by a femtosecond laser in fused silica. Opt. Lett. 30, 320–322 (2005).

31. 31.

Lumpkin, A. H. & Early, J. W. First dual-sweep streak camera measurements of a photoelectric injector drive laser. Nucl. Instrum. Methods Phys. Res. A 318, 389–395 (1992).

32. 32.

Brady, D. J. et al. Multiscale gigapixel photography. Nature 486, 386–389 (2012).

33. 33.

Vitek, D. N. et al. Spatio-temporally focused femtosecond laser pulses for nonreciprocal writing in optically transparent materials. Opt. Express 18, 24673–24678 (2010).

34. 34.

Wang, Z. H. et al. Time-resolved shadowgraphs of transient plasma induced by spatiotemporally focused femtosecond laser pulses in fused silica glass. Opt. Lett. 40, 5726–5729 (2015).

35. 35.

Wang, X. F. et al. High-frame-rate observation of single femtosecond laser pulse propagation in fused silica using an echelon and optical polarigraphy technique. Appl. Opt. 53, 8395–8399 (2014).

36. 36.

Li, G. H. et al. Second harmonic generation in centrosymmetric gas with spatiotemporally focused intense femtosecond laser pulses. Opt. Lett. 39, 961–964 (2014).

37. 37.

Foster, M. A. et al. Ultrafast waveform compression using a time-domain telescope. Nat. Photonics 3, 581–585 (2009).

38. 38.

van Howe, J. & Xu, C. Ultrafast optical signal processing based upon space-time dualities. J. Light Technol. 24, 2649–2662 (2006).

39. 39.

Weiner, A. M. in Ultrafast Optics (ed Boreman, G.) Ch. 3 (John Wiley & Sons, Inc., Hoboken, NJ, 2008).

40. 40.

Durfee, C. G. & Squier, J. A. Breakthroughs in photonics 2014: spatiotemporal focusing: advances and applications. IEEE Photon J. 7, 0700806 (2015).

41. 41.

Poulin, P. R. & Nelson, K. A. Irreversible organic crystalline chemistry monitored in real time. Science 313, 1756–1760 (2006).

42. 42.

Gross, M. et al. Heterodyne detection of multiply scattered monochromatic light with a multipixel detector. Opt. Lett. 30, 1357–1359 (2005).

43. 43.

Mosk, A. P., Lagendijk, A., Lerosey, G. & Fink, M. Controlling waves in space and time for imaging and focusing in complex media. Nat. Photonics 6, 283–292 (2012).

44. 44.

McCabe, D. J. et al. Spatio-temporal focusing of an ultrafast pulse through a multiply scattering medium. Nat. Commun. 2, 447 (2011).

45. 45.

Choi, W. et al. Tomographic phase microscopy. Nat. Methods 4, 717–719 (2007).

46. 46.

Satat, G. et al. Locating and classifying fluorescent tags behind turbid layers using time-resolved inversion. Nat. Commun. 6, 6796 (2015).

47. 47.

Horng, J. et al. Imaging electric field dynamics with graphene optoelectronics. Nat. Commun. 7, 13704 (2016).

48. 48.

Frühling, U. et al. Single-shot terahertz-field-driven X-ray streak camera. Nat. Photonics 3, 523–528 (2009).

49. 49.

Hockett, P., Bisgaard, C. Z., Clarkin, O. J. & Stolow, A. Time-resolved imaging of purely valence-electron dynamics during a chemical reaction. Nat. Phys. 7, 612–615 (2011).

50. 50.

Gorkhover, T. et al. Femtosecond and nanometre visualization of structural dynamics in superheated nanoparticles. Nat. Photonics 10, 93–97 (2016).

## Acknowledgements

The authors thank Dr. Zhengyan Li from the University of Ottawa, Dr. Shian Zhang from East China Normal University, and Dr. Liang Gao from the University of Illinois at Urbana-Champaign for fruitful discussion. The authors also acknowledge Yujia Chen and Chiye Li for experimental assistance and Professor James Ballard for close reading of the manuscript. This work was supported in part by National Institutes of Health grants DP1 EB016986 (NIH Director’s Pioneer Award) and R01 CA186567 (NIH Director’s Transformative Research Award).

## Author information

### Author notes

• Jinyang Liang

Present address: Centre Énergie Matériaux Télécommunications, Institut National de la Recherche Scientifique, 1650 Boulevard Lionel-Boulet, Varennes, QC, J3X1S2, Canada

1. These authors contributed equally: Jinyang Liang, Liren Zhu

### Affiliations

1. #### Caltech Optical Imaging Laboratory, Andrew and Peggy Cherng Department of Medical Engineering, Department of Electrical Engineering, California Institute of Technology, 1200 East California Boulevard, Mail Code 138-78, Pasadena, CA, 91125, USA

• Jinyang Liang
• , Liren Zhu
•  & Lihong V. Wang

### Contributions

J.L. designed and built the system and conducted all the experiments. L.Z. developed the reconstruction algorithm. J.L. and L.Z. analyzed the data and drafted the manuscript. L.V.W. supervised the project. All authors were involved in revising the manuscript.

### Conflict of interest

The authors declare that they have no conflict of interest.

### Corresponding author

Correspondence to Lihong V. Wang.