Single-shot compressed ultrafast photography at one hundred billion frames per second

Abstract

The capture of transient scenes at high imaging speed has been long sought by photographers1,2,3,4, with early examples being the well known recording in 1878 of a horse in motion5 and the 1887 photograph of a supersonic bullet6. However, not until the late twentieth century were breakthroughs achieved in demonstrating ultrahigh-speed imaging (more than 105 frames per second)7. In particular, the introduction of electronic imaging sensors based on the charge-coupled device (CCD) or complementary metal–oxide–semiconductor (CMOS) technology revolutionized high-speed photography, enabling acquisition rates of up to 107 frames per second8. Despite these sensors’ widespread impact, further increasing frame rates using CCD or CMOS technology is fundamentally limited by their on-chip storage and electronic readout speed9. Here we demonstrate a two-dimensional dynamic imaging technique, compressed ultrafast photography (CUP), which can capture non-repetitive time-evolving events at up to 1011 frames per second. Compared with existing ultrafast imaging techniques, CUP has the prominent advantage of measuring an xyt (x, y, spatial coordinates; t, time) scene with a single camera snapshot, thereby allowing observation of transient events with temporal resolution as tens of picoseconds. Furthermore, akin to traditional photography, CUP is receive-only, and so does not need the specialized active illumination required by other single-shot ultrafast imagers2,3. As a result, CUP can image a variety of luminescent—such as fluorescent or bioluminescent—objects. Using CUP, we visualize four fundamental physical phenomena with single laser shots only: laser pulse reflection and refraction, photon racing in two media, and faster-than-light propagation of non-information (that is, motion that appears faster than the speed of light but cannot convey information). Given CUP’s capability, we expect it to find widespread applications in both fundamental and applied sciences, including biomedical research.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: CUP system configuration.
Figure 2: Characterization of CUP’s spatial frequency responses.
Figure 3: CUP of laser pulse reflection, refraction, and racing in different media.
Figure 4: CUP of faster-than-light propagation of non-information.

References

  1. 1

    Fuller, P. W. W. An introduction to high speed photography and photonics. Imaging Sci. J. 57, 293–302 (2009)

  2. 2

    Li, Z., Zgadzaj, R., Wang, X., Chang, Y.-Y. & Downer, M. C. Single-shot tomographic movies of evolving light-velocity objects. Nature Commun. 5, 3085 (2014)

  3. 3

    Nakagawa, K. et al. Sequentially timed all-optical mapping photography (STAMP). Nature Photon. 8, 695–700 (2014)

  4. 4

    Heshmat, B., Satat, G., Barsi, C. & Raskar, R. Single-shot ultrafast imaging using parallax-free alignment with a tilted lenslet array. CLEO Sci. Innov. http://dx.doi.org/10.1364/CLEO_SI.2014.STu3E.7 (2014)

  5. 5

    Munn, O. & Beach, A. A horse’s motion scientifically determined. Sci. Am. 39, 241 (1878)

  6. 6

    Mach, E. & Salcher, P. Photographische Fixirung der durch Projectile in der Luft eingeleiteten Vorgänge. Ann. Phys. 268, 277–291 (1887)

  7. 7

    Nebeker, S. Rotating mirror cameras. High Speed Photogr. Photon. Newsl. 3, 31–37 (1983)

  8. 8

    Kondo, Y. et al. Development of “HyperVision HPV-X” high-speed video camera. Shimadzu Rev. 69, 285–291 (2012)

  9. 9

    El-Desouki, M. et al. CMOS image sensors for high speed applications. Sensors 9, 430–444 (2009)

  10. 10

    Guide. to Streak Camerashttp://www.hamamatsu.com/resources/pdf/sys/e_streakh.pdf (Hamamatsu Corp., Hamamatsu City, Japan, 2002)

  11. 11

    Velten, A., Lawson, E., Bardagjy, A., Bawendi, M. & Raskar, R. Slow art with a trillion frames per second camera. Proc. SIGGRAPH 44 (2011)

  12. 12

    Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Commun. 3, 745 (2012)

  13. 13

    Solli, D. R., Ropers, C., Koonath, P. & Jalali, B. Optical rogue waves. Nature 450, 1054–1057 (2007)

  14. 14

    Eldar, Y. C. & Kutyniok, G. Compressed Sensing: Theory and Applications (Cambridge Univ. Press, 2012)

  15. 15

    Figueiredo, M. A. T., Nowak, R. D. & Wright, S. J. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1, 586–597 (2007)

  16. 16

    Bioucas-Dias, J. M. & Figueiredo, M. A. T. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007)

  17. 17

    Afonso, M. V., Bioucas-Dias, J. M. & Figueiredo, M. A. T. Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)

  18. 18

    Wright, S. J., Nowak, R. D. & Figueiredo, M. A. T. Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)

  19. 19

    Afonso, M. V., Bioucas-Dias, J. M. & Figueiredo, M. A. T. An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Process. 20, 681–695 (2011)

  20. 20

    Einstein, A. Relativity: The Special and General Theory (Penguin, 1920)

  21. 21

    Agrawal, S. & Vishwanath, S. Secrecy using compressive sensing. Proc. Inform. Theory Workshop, IEEE, 563–567 (2011)

  22. 22

    Orsdemir, A., Altun, H. O., Sharma, G. & Bocko, M. F. On the security and robustness of encryption via compressed sensing. Proc. Military Commun. Conf., IEEE, 1–7 (2008)

  23. 23

    Davis, D. L. Secure compressed imaging. US patent 5,907. 619 (1999)

  24. 24

    Borst, J. W. & Visser, A. J. W. G. Fluorescence lifetime imaging microscopy in life sciences. Meas. Sci. Technol. 21, 102002 (2010)

  25. 25

    Figueiredo, M. A. T., Nowak, R. D. & Wright, S. J. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE Sel. Top. Signal Process. 1, 586–597 (2007)

  26. 26

    Bioucas-Dias, J. M. & Figueiredo, M. A. T. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007)

  27. 27

    Afonso, M. V., Bioucas-Dias, J. M. & Figueiredo, M. A. T. Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)

  28. 28

    Wright, S. J., Nowak, R. D. & Figueiredo, M. A. T. Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)

  29. 29

    Afonso, M. V., Bioucas-Dias, J. M. & Figueiredo, M. A. T. An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Process. 20, 681–695 (2011)

  30. 30

    Selanger, K. A., Falnes, J. & Sikkeland, T. Fluorescence lifetime studies of rhodamine 6G in methanol. J. Phys. Chem. 81, 1960–1963 (1977)

  31. 31

    Rentzepis, P. M., Topp, M. R., Jones, R. P. & Jortner, J. Picosecond emission spectroscopy of homogeneously broadened, electronically excited molecular states. Phys. Rev. Lett. 25, 1742–1745 (1970).

Download references

Acknowledgements

We thank N. Hagen for discussions and J. Ballard for a close reading of the manuscript. We also acknowledge Texas Instruments for providing the DLP device. This work was supported in part by National Institutes of Health grants DP1 EB016986 (NIH Director’s Pioneer Award) and R01 CA186567 (NIH Director’s Transformative Research Award).

Author information

L.G. built the system, performed the experiments, analysed the data and prepared the manuscript. J.L. performed some of the experiments, analysed the data and prepared the manuscript. C.L. prepared the sample and performed some of the experiments. L.V.W. contributed to the conceptual system, experimental design and manuscript preparation.

Correspondence to Lihong V. Wang.

Ethics declarations

Competing interests

L.V.W. has a financial interest in Microphotoacoustics, Inc. and Endra, Inc., which, however, did not support this work.

Extended data figures and tables

Extended Data Figure 1 CUP image formation model.

x, y, spatial coordinates; t, time; m, n, k, matrix indices; Im,n,k, input dynamic scene element; Cm,n, coded mask matrix element; Cm,n−kIm,n−k,k, encoded and sheared scene element; Em,n, image element energy measured by a 2D detector array; tmax, maximum recording time. See Methods for details.

Extended Data Figure 2

A temporally undispersed CCD image of the coded mask, which encodes the uniformly illuminated field with a pseudo-random binary pattern. The position of the mask and definition of the x′–y′ plane are shown in Fig. 1.

Extended Data Figure 3 Multicolour CUP.

a, Custom-built spectral separation unit. b, Representative temporal frames of a pulsed-laser-pumped fluorescence emission process. The pulsed pump laser and fluorescence emission are pseudo-coloured based on their peak emission wavelengths. To explicitly indicate the spatiotemporal pattern of this event, the CUP-reconstructed frames are overlaid with a static background image captured by a monochromatic CCD camera. All temporal frames of this event are provided in Supplementary Video 6. c, Time-lapse pump laser and fluorescence emission intensities averaged within the dashed box in b. The temporal responses of pump laser excitation and fluorescence decay are fitted to a Gaussian function and an exponential function, respectively. The recovered fluorescence lifetime of Rhodamine 6G is 3.8 ns. d, Event function describing the pulsed laser fluorescence excitation. e, Event function describing the fluorescence emission. f, Measured temporal PSF, with a full width at half maximum (FWHM) of 80 ps. Owing to reconstruction artefacts, the PSF has a side lobe and a shoulder extending over a range of 280 ps. g, Simulated temporal responses of event functions d and e after being convolved with the temporal PSF. The maxima of these two time-lapse signals are stretched by 200 ps. Scale bar, 10 mm.

Supplementary information

Light sweeping across a stripe pattern with varied spatial frequencies

Light sweeping across a stripe pattern with varied spatial frequencies. (MOV 107 kb)

Laser pulse reflection from a mirror

Laser pulse reflection from a mirror. (MOV 22 kb)

Laser pulse refraction at an air-resin interface

Laser pulse refraction at an air-resin interface (MOV 17 kb)

Laser pulse racing in different media

Laser pulse racing in different media. (MOV 23 kb)

Apparent faster-than-light phenomenon

Apparent faster-than-light phenomenon. (MOV 107 kb)

Pulsed-laser-pumped fluorescence emission

Pulsed-laser-pumped fluorescence emission. (MOV 679 kb)

PowerPoint slides

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gao, L., Liang, J., Li, C. et al. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 74–77 (2014). https://doi.org/10.1038/nature14005

Download citation

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.