Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Non-line-of-sight imaging using phasor-field virtual wave optics

Abstract

Non-line-of-sight imaging allows objects to be observed when partially or fully occluded from direct view, by analysing indirect diffuse reflections off a secondary relay surface. Despite many potential applications1,2,3,4,5,6,7,8,9, existing methods lack practical usability because of limitations including the assumption of single scattering only, ideal diffuse reflectance and lack of occlusions within the hidden scene. By contrast, line-of-sight imaging systems do not impose any assumptions about the imaged scene, despite relying on the mathematically simple processes of linear diffractive wave propagation. Here we show that the problem of non-line-of-sight imaging can also be formulated as one of diffractive wave propagation, by introducing a virtual wave field that we term the phasor field. Non-line-of-sight scenes can be imaged from raw time-of-flight data by applying the mathematical operators that model wave propagation in a conventional line-of-sight imaging system. Our method yields a new class of imaging algorithms that mimic the capabilities of line-of-sight cameras. To demonstrate our technique, we derive three imaging algorithms, modelled after three different line-of-sight systems. These algorithms rely on solving a wave diffraction integral, namely the Rayleigh–Sommerfeld diffraction integral. Fast solutions to Rayleigh–Sommerfeld diffraction and its approximations are readily available, benefiting our method. We demonstrate non-line-of-sight imaging of complex scenes with strong multiple scattering and ambient light, arbitrary materials, large depth range and occlusions. Our method handles these challenging cases without explicitly inverting a light-transport model. We believe that our approach will help to unlock the potential of non-line-of-sight imaging and promote the development of relevant applications not restricted to laboratory conditions.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: NLOS as a virtual LOS imaging system.
Fig. 2: Reconstructions of a complex NLOS scene.
Fig. 3: Robustness of our technique.
Fig. 4: Additional NLOS imaging applications of our method.

Data availability

The measured data and the phasor-field NLOS code supporting the findings of this study are available in the figshare repository https://doi.org/10.6084/m9.figshare.8084987. Additional data and code are available from the corresponding authors upon request.

Code availability

Our data and reconstruction code can be found in the figshare repository https://doi.org/10.6084/m9.figshare.8084987.

References

  1. 1.

    Kirmani, A., Hutchison, T., Davis, J. & Raskar, R. Looking around the corner using ultrafast transient imaging. Int. J. Comput. Vis. 95, 13–28 (2011).

    Article  Google Scholar 

  2. 2.

    Gupta, O., Willwacher, T., Velten, A., Veeraraghavan, A. & Raskar, R. Reconstruction of hidden 3D shapes using diffuse reflections. Opt. Express 20, 19096–19108 (2012).

    ADS  Article  Google Scholar 

  3. 3.

    Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3, 745 (2012).

    ADS  Article  Google Scholar 

  4. 4.

    Katz, O., Small, E. & Silberberg, Y. Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nat. Photon. 6, 549–553 (2012).

    ADS  CAS  Article  Google Scholar 

  5. 5.

    Heide, F., Xiao, L., Heidrich, W. & Hullin, M. B. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 3222–3229 (IEEE, 2014).

  6. 6.

    Laurenzis, M. & Velten, A. Nonline-of-sight laser gated viewing of scattered photons. Opt. Eng. 53, 023102 (2014).

    ADS  Article  Google Scholar 

  7. 7.

    Buttafava, M., Zeman, J., Tosi, A., Eliceiri, K. & Velten, A. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 20997–21011 (2015).

    ADS  CAS  Article  Google Scholar 

  8. 8.

    Arellano, V., Gutierrez, D. & Jarabo, A. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 11574–11583 (2017).

    ADS  Article  Google Scholar 

  9. 9.

    O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338–341 (2018).

    ADS  Article  Google Scholar 

  10. 10.

    Jarabo, A., Masia, B., Marco, J. & Gutierrez, D. Recent advances in transient imaging: a computer graphics and vision perspective. Visual Informatics 1, 65–79 (2017).

    Article  Google Scholar 

  11. 11.

    Velten, A. et al. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 44 (2013).

    Article  Google Scholar 

  12. 12.

    Gupta, M., Nayar, S. K., Hullin, M. B. & Martin, J. Phasor imaging: a generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. 34, 156 (2015).

    Article  Google Scholar 

  13. 13.

    O’Toole, M. et al. Reconstructing transient images from single-photon sensors. In 2017 IEEE Int. Conf. Computational Photography (CVPR), 1539–1547 (IEEE, 2017).

  14. 14.

    Gkioulekas, I., Levin, A., Durand, F. & Zickler, T. Micron-scale light transport decomposition using interferometry. ACM Trans. Graph. 34, 37 (2015).

    Article  Google Scholar 

  15. 15.

    Xin, S. et al. A theory of Fermat paths for non-line-of-sight shape reconstruction. In IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR), 6800–6809 (IEEE, 2019).

  16. 16.

    Tsai, C., Sankaranarayanan, A. & Gkioulekas, I. Beyond volumetric albedo a surface optimization framework for non-line-of-sight imaging. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 1545–1555 (IEEE, 2019).

  17. 17.

    Liu, X., Bauer, S. & Velten, A. Analysis of feature visibility in non-line-of-sight measurements. In IEEE Intl Conf. Computer Vision and Pattern Recognition (CVPR) 10140–10148 (IEEE, 2019).

  18. 18.

    Wu, R. et al. Adaptive polarization-difference transient imaging for depth estimation in scattering media. Opt. Lett. 43, 1299–1302 (2018).

    ADS  Article  Google Scholar 

  19. 19.

    Laurenzis, M. & Velten, A. Feature selection and back-projection algorithms for nonline-of-sight laser-gated viewing. J. Electron. Imaging 23, 063003 (2014).

    ADS  Article  Google Scholar 

  20. 20.

    Heide, F. et al. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 22 (2019).

    Article  Google Scholar 

  21. 21.

    Kadambi, A., Zhao, H., Shi, B. & Raskar, R. Occluded imaging with time-of-flight sensors. ACM Trans. Graph. 35, 15 (2016).

    Article  Google Scholar 

  22. 22.

    Shen, F. & Wang, A. Fast-Fourier-transform based numerical integration method for the Rayleigh–Sommerfeld diffraction formula. Appl. Opt. 45, 1102–1110 (2006).

    ADS  Article  Google Scholar 

  23. 23.

    Sen, P. et al. Dual photography. ACM Trans. Graph. 24, 745–755 (2005).

    Article  Google Scholar 

  24. 24.

    O’Toole, M. et al. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. 33, 87 (2014).

    MATH  Google Scholar 

  25. 25.

    Goodman, J. Introduction to Fourier Optics 3rd edn (Roberts, 2005).

  26. 26.

    Jarabo, A. et al. A framework for transient rendering. ACM Trans. Graph. 33, 177 (2014).

    MATH  Google Scholar 

  27. 27.

    Galindo, M. et al. A dataset for benchmarking time-resolved non-line-of-sight imaging. In IEEE Intl Conf Computational Photography https://graphics.unizar.es/nlos (IEEE, 2019).

  28. 28.

    Ward, G. J. Measuring and modeling anisotropic reflection. Comput. Graph. 26, 265–272 (1992).

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by DARPA through the DARPA REVEAL project (HR0011-16-C-0025), the NASA Innovative Advanced Concepts (NIAC) Program (NNX15AQ29G), the Air Force Office of Scientific Research (AFOSR) Young Investigator Program (FA9550-15-1-0208), the Office of Naval Research (ONR, N00014-15-1-2652), the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation programme (project CHAMELEON, grant no. 682080), the Spanish Ministerio de Economía y Competitividad (project TIN2016-78753-P) and the BBVA Foundation (Leonardo Grant for Researchers and Cultural Creators). We thank J. Teichman for insights and discussions in developing the phasor-field model. We also acknowledge M. Buttafava, A. Tosi and A. Ingle for help with the gated SPAD detector, and B. Masia, S. Malpica and M. Galindo for careful reading of the manuscript.

Author information

Affiliations

Authors

Contributions

X.L., S.A.R., M.L.M. and A.V. conceived the method. X.L., I.G., M.L.M. and J.H.N. implemented the reconstruction. M.L.M., X.L., J.H.N. and T.H.L. built and calibrated the system. I.G., D.G. and A.J. developed the simulation system. A.J., D.G. and A.V. coordinated the project. All authors contributed to writing the paper.

Corresponding author

Correspondence to Andreas Velten.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Peer review information Nature thanks Jeffrey H. Shapiro, Ashok Veeraraghavan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Extended data figures and tables

Extended Data Fig. 1

Capture hardware used for the results shown in this Letter.

Extended Data Fig. 2 Data comparison.

a, Raw data for one of the laser positions xp. Shown is the number of photons per second accumulated in each time bin (that is, the collected histogram divided by the integration time in seconds). Time bins are 4 ps wide. As expected, all three curves appear to follow the same mean, but there is a larger variance for lower exposure times. The raw data thus become noisier as exposure time decreases. The effects on the reconstruction are minor, as Extended Data Fig. 4 shows. Tacq, acquisition time. b, Example dataset from ref. 9 for comparison.

Extended Data Fig. 3 Visualization of the raw data for our long-exposure office scene.

a, Base-10 logarithm of the photon counts in all time bins. Pos index, laser position index; the 24,000 laser positions on the wall are labelled with these consecutive numbers. b–d, After removal of the first 833 time bins in each dataset, the plots show: the photon counts for the laser position that received the largest total number of photons in the dataset (b); the counts for the laser position that received the median number of photon counts (c); and the counts for the laser position that contains the time bin with the global maximum count in the entire set (d).

Extended Data Fig. 4 Robustness to multiple reflections.

Result for the synthetic bookshelf scene. a, Without interreflections. b, Including high-order interreflections. The quality of the results is very similar. c, Primary data (streak images) from the same scene without (top), and with interreflections (middle). The synthetic data clearly show how the presence of interreflections adds, as expected, low-frequency information resembling echoes of light. The bottom image shows primary data captured from the real office scene in Fig. 2. It follows the same behaviour as the middle image, revealing the presence of strong interreflections. Colours refer to numerical values from Matlab’s ‘fire colormap’, in arbitrary units.

Extended Data Fig. 5 Robustness to ambient light and noise.

a, Hidden bookshelf. b, Imaging results with increasingly higher exposure times; even at 50 ms, there is no significant loss in quality. Top row, image using only the pulsed laser as illumination source. Bottom row, on adding a large amount of ambient light (same conditions as the photograph in a), the quality remains constant. c, Difference between the 50-ms- and 1,000-ms-exposure captures for the lights-off case.

Extended Data Fig. 6 Short-exposure reconstructions.

Reconstruction of the office scene using very short capture times. a, Photograph of the captured scene. b, From left to right, reconstructions for data captured with 10 ms, 5 ms and 1 ms exposure time per laser. The total capture time was about 4 min, 2 min and 24 s, respectively.

Extended Data Fig. 7 Short-exposure data.

Photon counts in the raw data for our office scene for 10 ms (top row), 5 ms (centre row) and 1 ms (bottom row) exposure times per laser position. After removing the first 833 time bins in each dataset, the columns show: the photon counts for the laser position that received the largest total number of photons in the dataset (left); the counts for the laser position that received the median number of photon counts (centre); and the laser position that contains the time bin with the global maximum count in the entire set (right).

Extended Data Fig. 8 Comparison to prior methods.

Reconstruction of the office scene using very short capture times of 1 ms per laser (24 s in total). a, Filtered backprojection using the Laplacian filter. b, LOG-filtered backprojection. c, Our method.

Extended Data Fig. 9 Robustness to scene reflectance.

a, Geometry of our experimental set-up. b, From left to right, imaging results for the Lambertian targets (roughness 1) and increasingly specular surfaces (roughness 0.4 and roughness 0.2). The reconstructed irradiance is essentially the same for all cases.

Extended Data Fig. 10 Reconstruction comparison on a public dataset.

From left to right: confocal NLOS deconvolution, filtered (LOG) backprojection (FBP) and our proposed method. A large improvement in reconstruction quality for the simple scenes included in the dataset (isolated objects with no interreflections) is not to be expected, as existing methods already deliver reconstructions approaching their resolution limits. Nevertheless, our method achieves improved contrast and cleaner contours, owing to better handling of multiply scattered light.

Extended Data Fig. 11 Reconstruction comparison (noisy data).

From left to right: confocal NLOS deconvolution, FBP and our proposed method. Top row represents a non-retroreflective object; bottom row represents a retroreflective object captured in sunlight. In the presence of noisy data, FBP fails. Confocal NLOS includes a Wiener filter that needs to be explicitly estimated. Our phasor-field virtual wave method yields better results automatically. This is particularly important in complex scenes with interreflections, where the background is not uniform across the scene, and the noise level cannot be reliably estimated.

Extended Data Table 1 Photon statistics for captured data used in the paper and in Supplementary Information

Supplementary information

Supplementary Information

This file contains the details discussion of the methods used in the online paper including supplementary discussion and derivations supporting the main manuscript.

Video 1

Additional results illustrate the transient virtual camera which can reveal the multi-bounce signal and virtual photo camera refocusing example using operators mentioned in the main text.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Guillén, I., La Manna, M. et al. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572, 620–623 (2019). https://doi.org/10.1038/s41586-019-1461-3

Download citation

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing