Non-line-of-sight imaging allows objects to be observed when partially or fully occluded from direct view, by analysing indirect diffuse reflections off a secondary relay surface. Despite many potential applications1,2,3,4,5,6,7,8,9, existing methods lack practical usability because of limitations including the assumption of single scattering only, ideal diffuse reflectance and lack of occlusions within the hidden scene. By contrast, line-of-sight imaging systems do not impose any assumptions about the imaged scene, despite relying on the mathematically simple processes of linear diffractive wave propagation. Here we show that the problem of non-line-of-sight imaging can also be formulated as one of diffractive wave propagation, by introducing a virtual wave field that we term the phasor field. Non-line-of-sight scenes can be imaged from raw time-of-flight data by applying the mathematical operators that model wave propagation in a conventional line-of-sight imaging system. Our method yields a new class of imaging algorithms that mimic the capabilities of line-of-sight cameras. To demonstrate our technique, we derive three imaging algorithms, modelled after three different line-of-sight systems. These algorithms rely on solving a wave diffraction integral, namely the Rayleigh–Sommerfeld diffraction integral. Fast solutions to Rayleigh–Sommerfeld diffraction and its approximations are readily available, benefiting our method. We demonstrate non-line-of-sight imaging of complex scenes with strong multiple scattering and ambient light, arbitrary materials, large depth range and occlusions. Our method handles these challenging cases without explicitly inverting a light-transport model. We believe that our approach will help to unlock the potential of non-line-of-sight imaging and promote the development of relevant applications not restricted to laboratory conditions.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Nature Communications Open Access 14 July 2022
Nature Communications Open Access 09 June 2022
Nature Communications Open Access 16 May 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The measured data and the phasor-field NLOS code supporting the findings of this study are available in the figshare repository https://doi.org/10.6084/m9.figshare.8084987. Additional data and code are available from the corresponding authors upon request.
Our data and reconstruction code can be found in the figshare repository https://doi.org/10.6084/m9.figshare.8084987.
Kirmani, A., Hutchison, T., Davis, J. & Raskar, R. Looking around the corner using ultrafast transient imaging. Int. J. Comput. Vis. 95, 13–28 (2011).
Gupta, O., Willwacher, T., Velten, A., Veeraraghavan, A. & Raskar, R. Reconstruction of hidden 3D shapes using diffuse reflections. Opt. Express 20, 19096–19108 (2012).
Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3, 745 (2012).
Katz, O., Small, E. & Silberberg, Y. Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nat. Photon. 6, 549–553 (2012).
Heide, F., Xiao, L., Heidrich, W. & Hullin, M. B. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 3222–3229 (IEEE, 2014).
Laurenzis, M. & Velten, A. Nonline-of-sight laser gated viewing of scattered photons. Opt. Eng. 53, 023102 (2014).
Buttafava, M., Zeman, J., Tosi, A., Eliceiri, K. & Velten, A. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 20997–21011 (2015).
Arellano, V., Gutierrez, D. & Jarabo, A. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 11574–11583 (2017).
O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338–341 (2018).
Jarabo, A., Masia, B., Marco, J. & Gutierrez, D. Recent advances in transient imaging: a computer graphics and vision perspective. Visual Informatics 1, 65–79 (2017).
Velten, A. et al. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 44 (2013).
Gupta, M., Nayar, S. K., Hullin, M. B. & Martin, J. Phasor imaging: a generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. 34, 156 (2015).
O’Toole, M. et al. Reconstructing transient images from single-photon sensors. In 2017 IEEE Int. Conf. Computational Photography (CVPR), 1539–1547 (IEEE, 2017).
Gkioulekas, I., Levin, A., Durand, F. & Zickler, T. Micron-scale light transport decomposition using interferometry. ACM Trans. Graph. 34, 37 (2015).
Xin, S. et al. A theory of Fermat paths for non-line-of-sight shape reconstruction. In IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR), 6800–6809 (IEEE, 2019).
Tsai, C., Sankaranarayanan, A. & Gkioulekas, I. Beyond volumetric albedo a surface optimization framework for non-line-of-sight imaging. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 1545–1555 (IEEE, 2019).
Liu, X., Bauer, S. & Velten, A. Analysis of feature visibility in non-line-of-sight measurements. In IEEE Intl Conf. Computer Vision and Pattern Recognition (CVPR) 10140–10148 (IEEE, 2019).
Wu, R. et al. Adaptive polarization-difference transient imaging for depth estimation in scattering media. Opt. Lett. 43, 1299–1302 (2018).
Laurenzis, M. & Velten, A. Feature selection and back-projection algorithms for nonline-of-sight laser-gated viewing. J. Electron. Imaging 23, 063003 (2014).
Heide, F. et al. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 22 (2019).
Kadambi, A., Zhao, H., Shi, B. & Raskar, R. Occluded imaging with time-of-flight sensors. ACM Trans. Graph. 35, 15 (2016).
Shen, F. & Wang, A. Fast-Fourier-transform based numerical integration method for the Rayleigh–Sommerfeld diffraction formula. Appl. Opt. 45, 1102–1110 (2006).
Sen, P. et al. Dual photography. ACM Trans. Graph. 24, 745–755 (2005).
O’Toole, M. et al. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. 33, 87 (2014).
Goodman, J. Introduction to Fourier Optics 3rd edn (Roberts, 2005).
Jarabo, A. et al. A framework for transient rendering. ACM Trans. Graph. 33, 177 (2014).
Galindo, M. et al. A dataset for benchmarking time-resolved non-line-of-sight imaging. In IEEE Intl Conf Computational Photography https://graphics.unizar.es/nlos (IEEE, 2019).
Ward, G. J. Measuring and modeling anisotropic reflection. Comput. Graph. 26, 265–272 (1992).
This work was funded by DARPA through the DARPA REVEAL project (HR0011-16-C-0025), the NASA Innovative Advanced Concepts (NIAC) Program (NNX15AQ29G), the Air Force Office of Scientific Research (AFOSR) Young Investigator Program (FA9550-15-1-0208), the Office of Naval Research (ONR, N00014-15-1-2652), the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation programme (project CHAMELEON, grant no. 682080), the Spanish Ministerio de Economía y Competitividad (project TIN2016-78753-P) and the BBVA Foundation (Leonardo Grant for Researchers and Cultural Creators). We thank J. Teichman for insights and discussions in developing the phasor-field model. We also acknowledge M. Buttafava, A. Tosi and A. Ingle for help with the gated SPAD detector, and B. Masia, S. Malpica and M. Galindo for careful reading of the manuscript.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Peer review information Nature thanks Jeffrey H. Shapiro, Ashok Veeraraghavan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Extended data figures and tables
Capture hardware used for the results shown in this Letter.
a, Raw data for one of the laser positions xp. Shown is the number of photons per second accumulated in each time bin (that is, the collected histogram divided by the integration time in seconds). Time bins are 4 ps wide. As expected, all three curves appear to follow the same mean, but there is a larger variance for lower exposure times. The raw data thus become noisier as exposure time decreases. The effects on the reconstruction are minor, as Extended Data Fig. 4 shows. Tacq, acquisition time. b, Example dataset from ref. 9 for comparison.
a, Base-10 logarithm of the photon counts in all time bins. Pos index, laser position index; the 24,000 laser positions on the wall are labelled with these consecutive numbers. b–d, After removal of the first 833 time bins in each dataset, the plots show: the photon counts for the laser position that received the largest total number of photons in the dataset (b); the counts for the laser position that received the median number of photon counts (c); and the counts for the laser position that contains the time bin with the global maximum count in the entire set (d).
Result for the synthetic bookshelf scene. a, Without interreflections. b, Including high-order interreflections. The quality of the results is very similar. c, Primary data (streak images) from the same scene without (top), and with interreflections (middle). The synthetic data clearly show how the presence of interreflections adds, as expected, low-frequency information resembling echoes of light. The bottom image shows primary data captured from the real office scene in Fig. 2. It follows the same behaviour as the middle image, revealing the presence of strong interreflections. Colours refer to numerical values from Matlab’s ‘fire colormap’, in arbitrary units.
a, Hidden bookshelf. b, Imaging results with increasingly higher exposure times; even at 50 ms, there is no significant loss in quality. Top row, image using only the pulsed laser as illumination source. Bottom row, on adding a large amount of ambient light (same conditions as the photograph in a), the quality remains constant. c, Difference between the 50-ms- and 1,000-ms-exposure captures for the lights-off case.
Reconstruction of the office scene using very short capture times. a, Photograph of the captured scene. b, From left to right, reconstructions for data captured with 10 ms, 5 ms and 1 ms exposure time per laser. The total capture time was about 4 min, 2 min and 24 s, respectively.
Photon counts in the raw data for our office scene for 10 ms (top row), 5 ms (centre row) and 1 ms (bottom row) exposure times per laser position. After removing the first 833 time bins in each dataset, the columns show: the photon counts for the laser position that received the largest total number of photons in the dataset (left); the counts for the laser position that received the median number of photon counts (centre); and the laser position that contains the time bin with the global maximum count in the entire set (right).
Reconstruction of the office scene using very short capture times of 1 ms per laser (24 s in total). a, Filtered backprojection using the Laplacian filter. b, LOG-filtered backprojection. c, Our method.
a, Geometry of our experimental set-up. b, From left to right, imaging results for the Lambertian targets (roughness 1) and increasingly specular surfaces (roughness 0.4 and roughness 0.2). The reconstructed irradiance is essentially the same for all cases.
From left to right: confocal NLOS deconvolution, filtered (LOG) backprojection (FBP) and our proposed method. A large improvement in reconstruction quality for the simple scenes included in the dataset (isolated objects with no interreflections) is not to be expected, as existing methods already deliver reconstructions approaching their resolution limits. Nevertheless, our method achieves improved contrast and cleaner contours, owing to better handling of multiply scattered light.
From left to right: confocal NLOS deconvolution, FBP and our proposed method. Top row represents a non-retroreflective object; bottom row represents a retroreflective object captured in sunlight. In the presence of noisy data, FBP fails. Confocal NLOS includes a Wiener filter that needs to be explicitly estimated. Our phasor-field virtual wave method yields better results automatically. This is particularly important in complex scenes with interreflections, where the background is not uniform across the scene, and the noise level cannot be reliably estimated.
This file contains the details discussion of the methods used in the online paper including supplementary discussion and derivations supporting the main manuscript.
Additional results illustrate the transient virtual camera which can reveal the multi-bounce signal and virtual photo camera refocusing example using operators mentioned in the main text.
About this article
Cite this article
Liu, X., Guillén, I., La Manna, M. et al. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572, 620–623 (2019). https://doi.org/10.1038/s41586-019-1461-3
This article is cited by
Nature Communications (2022)
Nature Communications (2022)
Nature Communications (2022)
Nature Photonics (2022)
Fast non-line-of-sight imaging with high-resolution and wide field of view using synthetic wavelength holography
Nature Communications (2021)