Abstract
Emerging single-photon-sensitive sensors produce picosecond-accurate time-stamped photon counts. Applying advanced inverse methods to process these data has resulted in unprecedented imaging capabilities, such as non-line-of-sight (NLOS) imaging. Rather than imaging photons that travel along direct paths from a source to an object and back to the detector, NLOS methods analyse photons that travel along indirect light paths, scattered from multiple surfaces, to estimate 3D images of scenes outside the direct line of sight of a camera, hidden by a wall or other obstacles. We review the transient imaging techniques that underlie many NLOS imaging approaches, discuss methods for reconstructing hidden scenes from time-resolved measurements, describe some other methods for NLOS imaging that do not require transient imaging and discuss the future of ‘seeing around corners’.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$99.00 per year
only $8.25 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Kirmani, A., Hutchison, T., Davis, J. & Raskar, R. Looking around the corner using transient imaging. Proc. IEEE Int. Conf. Comput. Vis. 2009, 159–166 (2009).
Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3, 745 (2012). Report of full 3D NLOS imaging.
O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338 (2018). Report applying light-cone transform and producing high-quality 3D reconstructions over a large NLOS area.
Liu, X. et al. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572, 620–623 (2019). Virtual-wave reconstruction approach for fast and detailed NLOS scene reconstruction.
Lindell, D. B., Wetzstein, G. & O’Toole, M. Wave-based non-line-of-sight imaging using fast f–k migration. ACM Trans. Graph. 38, 116 (2019). A fast and accurate wave-optics method for 3D NLOS imaging at interactive frame-rates.
Bertolotti, J. et al. Non-invasive imaging through opaque scattering layers. Nature 491, 232–234 (2012).
Katz, O., Heidmann, P., Fink, M. & Gigan, S. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nat. Photonics 8, 784–790 (2014). Correlation technique for diffuse and NLOS imaging that does not rely on time-of-flight measurements.
Lindell, D. B., Wetzstein, G. & Koltun, V. Acoustic non-line-of-sight imaging. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2019, 6773–6782 (2019).
Bouman, K. L. et al. Turning corners into cameras: principles and methods. Proc. IEEE Int. Conf. Comput. Vis. 2017, 2289–2297 (2017).
Saunders, C., Murray-Bruce, J. & Goyal, V. K. Computational periscopy with an ordinary digital camera. Nature 565, 472–475 (2019). A passive NLOS approach that relies on lighting in the scene and uses ordinary cameras.
Boger-Lombard, J. & Katz, O. Non line-of-sight localization by passive optical time-of-flight. Nat. Commun. 10, 3343 (2019).
Maeda, T., Wang, Y., Raskar, R. & Kadambi, A. Thermal non-line-of-sight imaging. Proc. IEEE Int. Conf. Comput. Photogr. 2019, 1–11 (2019).
Kaga, M. et al. Thermal non-line-of-sight imaging from specular and diffuse reflections. IPSJ Trans. Comp. Vis. Appl. 11, 8 (2019).
Faccio, D. & Velten, A. A trillion frames per second: the techniques and applications of light-in-flight photography. Rep. Prog. Phys. 81, 105901 (2018). Review paper on transient imaging.
Hariharan, P. Basics of Holography (Cambridge Univ. Press, 2011).
Abramson, N. Light-in-flight recording by holography. Opt. Lett. 3, 121 (1978).
Abramson, N. Light-in-flight recording: high-speed holographic motion pictures of ultrafast phenomena. Appl. Opt. 22, 215–232 (1983).
Abramson, N. Light-in-flight recording. 4: Visualizing optical relativistic phenomena. Appl. Opt. 24, 3323–3329 (1985).
Abramson, N. Light in Flight or the Holodiagram: The Columbi Egg of Optics (SPIE, 1998).
Gkioulekas, I., Levin, A., Durand, F. & Zickler, T. Micron-scale light transport decomposition using interferometry. ACM Trans. Graph. 34, 37 (2015).
Kadambi, A. et al. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 1–167 (2013).
Heide, F., Hullin, M. B., Gregson, J. & Heidrich, W. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. 32, 45:1–45:10 (2013).
Peters, C., Klein, J., Hullin, M. B. & Klein, R. Solving trigonometric moment problems for fast transient imaging. ACM Trans. Graph. 34, 220 (2015).
Jarabo, A., Masia, B., Marco, J. & Gutierrez, D. Recent advances in transient imaging: a computer graphics and vision perspective. Vis. Inform. 1, 65–79 (2017).
Velten, A. et al. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 44:1–44:8 (2013).
Gao, L., Liang, J., Li, C. & Wang, L. V. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 74–77 (2014).
Mikami, H., Gao, L. & Goda, K. Ultrafast optical imaging technology: principles and applications of emerging methods. Nanophotonics 5, 98–110 (2016).
Zhu, L. et al. Space- and intensity-constrained reconstruction for compressed ultrafast photography. Optica 3, 694–697 (2016).
Laurenzis, M. & Velten, A. Nonline-of-sight laser gated viewing of scattered photons. Opt. Eng. 53, 53–53–7 (2014).
Jarabo, A. et al. Relativistic effects for time-resolved light transport. Comput. Graph. Forum 34, 1–12 (2015).
Laurenzis, M., Klein, J. & Bacher, E. Relativistic effects in imaging of light in flight with arbitrary paths. Opt. Lett. 41, 2001–2004 (2016).
Clerici, M. et al. Observation of image pair creation and annihilation from superluminal scattering sources. Sci. Adv. 2, e1501691 (2016).
Strutt, J. W. (Baron Rayleigh). Theory of Sound (MacMillan, 1896).
Becker, W. Advanced Time-Correlated Single Photon Counting Techniques (Springer, 2005).
Niclass, C., Gersbach, M., Henderson, R., Grant, L. & Charbon, E. A single photon avalanche diode implemented in 130-nm CMOS technology. IEEE J. Sel. Top. Quantum Electron. 13, 863–869 (2007).
Richardson, J. et al. A 32×32 50 ps resolution 10 bit time to digital converter array in 130 nm CMOS for time correlated imaging. Proc. IEEE Custom Integr. Circuits Conf. 2009, 77–80 (2009).
Richardson, J. A., Webster, E. A. G., Grant, L. A. & Henderson, R. K. Scaleable single-photon avalanche diode structures in nanometer CMOS technology. IEEE Trans. Electron. Devices 58, 2028–2035 (2011).
Gersbach, M. et al. A time-resolved, low-noise single-photon image sensor fabricated in deep-submicron CMOS technology. IEEE J. Solid-State Circuits 47, 1394–1407 (2012).
Bronzi, D. et al. 100 000 frames/s 64 × 32 single-photon detector array for 2-D imaging and 3-D ranging. IEEE J. Sel. Top. Quantum Electron. 20, 354–363 (2014).
Kramer, B. et al. A SPAD array detector for spectrally and lifetime resolved microscopy (Poster). 17th Int. Workshop Single Mol. Spectrosc. Ultrasensitive Anal. Life Sci. 69 (2011).
Cammi, C., Gulinatti, A., Rech, I., Panzeri, F. & Ghioni, M. Spad array module for multi-dimensional photon timing applications. J. Mod. Opt. 59, 131–139 (2012).
Zappa, F. & Tosi, A. MiSPIA: microelectronic single-photon 3D imaging arrays for low-light high-speed safety and security applications. Proc. SPIE 8727, 87270L (2013).
Veerappan, C. et al. A 160 × 28 single-photon image sensor with on-pixel 55 ps 10 bit time-to-digital converter. Proc. IEEE Int. Solid-State Circuits Conf. 2011, 312–314 (2011).
Villa, F., Lussana, R., Tamborini, D., Tosi, A. & Zappa, F. High-fill-factor 60 × 1 SPAD array with 60 subnanosecond integrated TDCs. IEEE Photonics Technol. Lett. 27, 1261–1264 (2015).
Burri, S., Homulle, H., Bruschini, C. & Charbon, E. LinoSPAD: a time-resolved 256 × 1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899, 98990D (2016).
Abbas, T. A. et al. Backside illuminated SPAD image sensor with 7.83 μm pitch in 3D-stacked CMOS technology. Proc. IEEE Int. Electron Devices Meet. 2016, 8.1.1–8.1.4 (2016).
Itzler, M., Jiang, X., Ben-Michael, R., Nyman, B. & Slomkowski, K. Geiger-mode APD single photon detectors. Proc. Opt. Fiber Commun. Conf. 2008, 1–3 (2008).
Itzler, M., Jiang, X., Ben-Michael, R., Nyman, B. & Slomkowski, K. Single photon avalanche photodiodes for near-infrared photon counting. SPIE Proc. 6900, 69001E (2008).
Itzler, M. A. et al. Single photon avalanche diodes (SPADs) for 1.5 μm photon counting applications. J. Mod. Opt. 54, 283–304 (2007).
Gariepy, G. et al. Single-photon sensitive light-in-flight imaging. Nat. Commun. 6, 6021 (2015).
Musarra, G. et al. Non-line-of-sight 3D imaging with a single-pixel camera. Phys. Rev. Appl. 12, 011002 (2019).
Chan, S., Warburton, R., Gariepy, G., Leach, J. & Faccio, D. Non-line-of-sight tracking of people at long range. Opt. Express 25, 10109 (2017).
Lindell, D. B., O’Toole, M. & Wetzstein, G. Towards transient imaging at interactive rates with single-photon detectors. Proc. IEEE Int. Conf. Comput. Photogr. 2018, 1–8 (2018).
Pawlikowska, A. M., Halimi, A., Lamb, R. A. & Buller, G. S. Single-photon three-dimensional imaging at up to 10 kilometers range. Opt. Express 25, 11919 (2017).
Gariepy, G., Tonolini, F., Henderson, R., Leach, J. & Faccio, D. Detection and tracking of moving objects hidden from view. Nat. Photonics 10, 23–26 (2016). Real-time tracking of a moving NLOS object.
Buttafava, M., Zeman, J., Tosi, A., Eliceiri, K. & Velten, A. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 20997–21011 (2015).
O’Toole, M. et al. Reconstructing transient images from single-photon sensors. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2017, 2289–2297 (2017).
Tsai, C.-Y., Kutulakos, K. N., Narasimhan, S. G. & Sankaranarayanan, A. C. The geometry of first-returning photons for non-line-of-sight imaging. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2017, 7216–7224 (2017).
Pediredla, A. K., Buttafava, M., Tosi, A., Cossairt, O. & Veeraraghavan, A. Reconstructing rooms using photon echoes: a plane based model and reconstruction algorithm for looking around the corner. Proc. IEEE Int. Conf. Comput. Photogr. 2017, 1–12 (2017).
Starshynov, I., Ghafur, O., Fitches, J. & Faccio, D. Coherent control of light for non-line-of-sight imaging. Preprint at arXiv https://arxiv.org/abs/1908.04094 (2019).
Pediredla, A., Dave, A. & Veeraraghavan, A. Snlos: Non-line-of-sight scanning through temporal focusing. Proc. EEE Int. Conf. Comput. Photogr. 2019, 1–13 (2019).
Tasinkevych, J. & Trots, I. Circular radon transform inversion technique in synthetic aperture ultrasound imaging: an ultrasound phantom evaluation. Arch. Acoust. 39, 569–582 (2014).
Moon, S. On the determination of a function from an elliptical radon transform. J. Math. Anal. Appl. 416, 724–734 (2014).
Gupta, O., Willwacher, T., Velten, A., Veeraraghavan, A. & Raskar, R. Reconstruction of hidden 3D shapes using diffuse reflections. Opt. Express 20, 19096–19108 (2012).
Buttafava, M., Boso, G., Ruggeri, A., Mora, A. D. & Tosi, A. Time-gated single-photon detection module with 110 ps transition time and up to 80 MHz repetition rate. Rev. Sci. Instrum. 85, 083114 (2014).
Laurenzis, M. & Velten, A. Feature selection and back-projection algorithms for nonline-of-sight laser-gated viewing. J. Electron. Imaging 23, 063003 (2014).
Arellano, V., Gutierrez, D. & Jarabo, A. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 11574–11583 (2017).
Kak, A. C., Slaney, M. & Wang, G. Principles of computerized tomographic imaging. Med. Phys. 29, 107–107 (2002).
Wu, D. et al. Frequency analysis of transient light transport with applications in bare sensor imaging. Proc. 12th Eur. Conf. Comput. Vis. 7572, 542–555 (2012).
Heide, F., Xiao, L., Heidrich, W. & Hullin, M. B. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2014, 3222–3229 (2014). A low-cost approach to NLOS imaging over short distances.
O’Toole, M., Lindell, D. B. & Wetzstein, G. Real-time non-line-of-sight imaging. In ACM SIGGRAPH Emerging Technologies 14 (ACM, 2018).
Heide, F. et al. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 22 (2019).
Thrampoulidis, C. et al. Exploiting occlusion in non-line-of-sight active imaging. IEEE Trans. Computational Imaging 4, 419–431 (2018).
Xu, F. et al. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. Opt. Express 26, 9945–9962 (2018).
Seidel, S. W. et al. Corner occluder computational periscopy: estimating a hidden scene from a single photograph. Proc. IEEE Int. Conf. Comput. Photogr. 2019, 1–9 (2019).
Baradad, M. et al. Inferring light fields from shadows. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2018, 6267–6275 (2018).
Xin, S. et al. A theory of Fermat paths for non-line-of-sight shape reconstruction. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2019, 6800–6809 (IEEE, 2019).
Iseringhausen, J. & Hullin, M. B. Non-line-of-sight reconstruction using efficient transient rendering. Preprint at arXiv https://arxiv.org/abs/1809.08044 (2018).
Tsai, C.-Y., Sankaranarayanan, A. C. & Gkioulekas, I. Beyond volumetric albedo — a surface optimization framework for non-line-of-sight imaging. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2019, 1545–1555 (2019).
Young, S., Lindell, D. & Wetzstein, G. Non-line-of-sight surface reconstruction using the directional light-cone transform. In IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2020).
Reza, S. A., La Manna, M. & Velten, A. A physical light transport model for non-line-of-sight imaging applications. Preprint at arXiv https://arxiv.org/abs/1802.01823 (2018).
Teichman, J. A. Phasor field waves: a mathematical treatment. Opt. Express 27, 27500–27506 (2019).
Reza, S. A., Manna, M. L., Bauer, S. & Velten, A. Phasor field waves: experimental demonstrations of wave-like properties. Opt. Express 27, 32587–32608 (2019).
Dove, J. & Shapiro, J. H. Paraxial theory of phasor-field imaging. Opt. Express 27, 18016–18037 (2019).
Stolt, R. H. Migration by Fourier transform. Geophysics 43, 23–48 (1978).
Margrave, G. F. & Lamoureux, M. P. Numerical Methods of Exploration Seismology: with Algorithms in MATLAB (Cambridge Univ. Press, 2018).
Callow, H. J. Signal Processing for Synthetic Aperture Sonar Image Enhancement. Thesis, Univ. Canterbury (2003).
Sheriff, R. W. Synthetic aperture beamforming with automatic phase compensation for high frequency sonars. Proc. IEEE Symp. Auton. Underwater Veh. Technol. 1992, 236–245 (1992).
Garcia, D. et al. Stolt’s fk migration for plane wave ultrasound imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control. 60, 1853–1867 (2013).
Cafforio, C., Prati, C. & Rocca, F. SAR data focusing using seismic migration techniques. IEEE Trans. Aerosp. Electron. Syst. 27, 194–207 (1991).
Tancik, M., Swedish, T., Satat, G. & Raskar, R. Data-driven non-line-of-sight imaging with a traditional camera. In Imaging Appl. Opt. (Optical Society of America, 2018).
Chen, W., Daneau, S., Mannan, F. & Heide, F. Steady-state non-line-of-sight imaging. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2019, 6790–6799 (2019).
Caramazza, P. et al. Neural network identification of people hidden from view with a single-pixel, single-photon detector. Sci. Rep. 8, 11945 (2018).
Pandharkar, R. et al. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2011, 265–272 (2011).
Chan, S., Warburton, R., Gariepy, G., Leach, J. & Faccio, D. Real-time tracking of hidden objects with single-pixel detectors. Electron. Lett. 53, 1005–1008 (2017).
Metzler, C. A., Lindell, D. B. & Wetzstein, G. Keyhole imaging: Non-line-of-sight imaging and tracking of moving objects along a single optical path at long standoff distances. Preprint at arXiv https://arxiv.org/abs/1912.06727 (2019).
Born, M. & Wolf, E. Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light (Elsevier, 2013).
Kadambi, A., Zhao, H., Shi, B. & Raskar, R. Occluded imaging with time-of-flight sensors. ACM Trans. Graph. 35, 15 (2016).
Pediredla, A. K., Matsuda, N., Cossairt, O. & Veeraraghavan, A. Linear systems approach to identifying performance bounds in indirect imaging. Proc. IEEE Int. Conf. Acoust. Speech Signal Process. 2017, 6235–6239 (2017).
Liu, X., Bauer, S. & Velten, A. Analysis of feature visibility in non-line-of-sight measurements. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2019, 10140–10148 (2019).
Klein, J., Peters, C., Martín, J., Laurenzis, M. & Hullin, M. B. Tracking objects outside the line of sight using 2D intensity images. Sci. Rep. 6, 32491 (2016).
Torralba, A. & Freeman, W. T. Accidental pinhole and pinspeck cameras: revealing the scene outside the picture. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2012, 374–381 (2012).
Batarseh, M. et al. Passive sensing around the corner using spatial coherence. Nat. Commun. 9, 3629 (2018).
Metzler, C. A. et al. Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging. Optica 7, 63–71 (2020).
Willomitzer, F., Li, F., Balaji, M. M., Rangarajan, P. & Cossairt, O. High resolution non-line-of-sight imaging with superheterodyne remote digital holography. In Imaging Appl. Opt. CM2A.2 (Optical Society of America, 2019).
Rangarajan, P., Willomitzer, F., Cossairt, O. & Christensen, M. P. Spatially resolved indirect imaging of objects beyond the line of sight. Proc. SPIE 11135, 124–131 (2019).
Brooks, J. & Faccio, D. A single-shot non-line-of-sight range-finder. Sensors 19, 4820 (2019).
Nkwari, P. K. M., Sinha, S. & Ferreira, H. C. Through-the-wall radar imaging: a review. IETE Tech. Rev. 35, 631–639 (2018).
Amin, M. G. Through-the-wall RADAR Imaging (CRC, 2011).
Sume, A. et al. Radar detection of moving targets behind corners. IEEE Trans. Geosci. Remote. Sens. 49, 2259–2267 (2011).
Nag, S., Barnes, M. A., Payment, T. & Holladay, G. Ultrawideband through-wall radar for detecting the motion of people in real time. Proc. SPIE 4744, 48–57 (2002).
Ralston, T., Charvat, G. & Peabody, J. Real-time through-wall imaging using an ultrawideband multiple-input multiple-output (MIMO) phased array radar system. IEEE Int. Symp. Phased Array Syst. Technol. 2010, 551–558 (2010).
Zhao, M. et al. Through-wall human pose estimation using radio signals. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2018, 7356–7365 (2018).
Acknowledgements
G.W. and A.V. acknowledge financial support from DARPA REVEAL (HR0011-16-C-0025). G.W. is supported by NSF CAREER Award (IIS 1553333), PECASE (ARO, W911NF19-1-0120) and the Visual Computing Center CCF grant (KAUST Office of Sponsored Research). D.F. is supported by the Royal Academy of Engineering under the Chairs in Emerging Technologies scheme and by the EPSRC (UK, grant no. EP/T00097X/1).
Author information
Authors and Affiliations
Contributions
All authors contributed to all aspects of manuscript preparation, revision and editing.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information
Nature Reviews Physics thanks E. Charbon, K. Kutulakos and S. Gigan for their contribution to the peer review of this work.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Faccio, D., Velten, A. & Wetzstein, G. Non-line-of-sight imaging. Nat Rev Phys 2, 318–327 (2020). https://doi.org/10.1038/s42254-020-0174-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42254-020-0174-8
This article is cited by
-
Scintillation event imaging with a single photon avalanche diode camera
Communications Engineering (2024)
-
Determining intrinsic sensitivity and the role of multiple scattering in speckle metrology
Nature Reviews Physics (2024)
-
Two-edge-resolved three-dimensional non-line-of-sight imaging with an ordinary camera
Nature Communications (2024)
-
Pilot-assisted beam and correlation to recover information through scattering media [Invited]
Applied Physics B (2024)
-
Toward quantum unmanned systems
Science China Technological Sciences (2024)