Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Computational periscopy with an ordinary digital camera

Nature volume 565, pages 472–475 (2019)Cite this article

Subjects

Abstract

Computing the amounts of light arriving from different directions enables a diffusely reflecting surface to play the part of a mirror in a periscope—that is, perform non-line-of-sight imaging around an obstruction. Because computational periscopy has so far depended on light-travel distances being proportional to the times of flight, it has mostly been performed with expensive, specialized ultrafast optical systems1,2,3,4,5,6,7,8,9,10,11,12. Here we introduce a two-dimensional computational periscopy technique that requires only a single photograph captured with an ordinary digital camera. Our technique recovers the position of an opaque object and the scene behind (but not completely obscured by) the object, when both the object and scene are outside the line of sight of the camera, without requiring controlled or time-varying illumination. Such recovery is based on the visible penumbra of the opaque object having a linear dependence on the hidden scene that can be modelled through ray optics. Non-line-of-sight imaging using inexpensive, ubiquitous equipment may have considerable value in monitoring hazardous environments, navigation and detecting hidden adversaries.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Experimental setup for computational periscopy.
Fig. 2: Reconstruction procedure.
Fig. 3: Computational field of view.
Fig. 4: Reconstructions of different hidden scenes.

Similar content being viewed by others

Data availability

Raw data captured with our digital camera during the experiments presented here are available on GitHub at https://github.com/Computational-Periscopy/Ordinary-Camera.

References

  1. Kirmani, A., Hutchison, T., Davis, J. & Raskar, R. Looking around the corner using transient imaging. In Proc. 2009 IEEE 12th Int. Conf. Computer Vision 159–166 (IEEE, 2009).

  2. Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3, 745 (2012).

    Article  Google Scholar 

  3. Gupta, O., Willwacher, T., Velten, A., Veeraraghavan, A. & Raskar, R. Reconstruction of hidden 3D shapes using diffuse reflections. Opt. Express 20, 19096–19108 (2012).

    Article  ADS  Google Scholar 

  4. Xu, K. et al. Image contrast model of non-line-of-sight imaging based on laser range-gated imaging. Opt. Eng. 53, 061610 (2013).

    Article  ADS  Google Scholar 

  5. Laurenzis, M. & Velten, A. Nonline-of-sight laser gated viewing of scattered photons. Opt. Eng. 53, 023102 (2014).

    Article  ADS  Google Scholar 

  6. Buttafava, M., Zeman, J., Tosi, A., Eliceiri, K. & Velten, A. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 20997–21011 (2015).

    Article  ADS  CAS  Google Scholar 

  7. Gariepy, G., Tonolini, F., Henderson, R., Leach, J. & Faccio, D. Detection and tracking of moving objects hidden from view. Nat. Photon. 10, 23–26 (2016).

    Article  ADS  CAS  Google Scholar 

  8. Klein, J., Laurenzis, M. & Hullin, M. Transient imaging for real-time tracking around a corner. In Proc. SPIE Electro-Optical Remote Sensing X 998802 (International Society for Optics and Photonics, 2016).

  9. Chan, S., Warburton, R. E., Gariepy, G., Leach, J. & Faccio, D. Non-line-of-sight tracking of people at long range. Opt. Express 25, 10109–10117 (2017).

    Article  Google Scholar 

  10. Tsai, C.-y., Kutulakos, K. N., Narasimhan, S. G. & Sankaranarayanan, A. C. The geometry of first-returning photons for non-line-of-sight imaging. In Proc. 2017 IEEE Conf. Computer Vision and Pattern Recognition 7216–7224 (IEEE, 2017).

  11. Heide, F. et al. Non-line-of-sight imaging with partial occluders and surface normals. Preprint at https://arxiv.org/abs/1711.07134 (2018).

  12. O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338–341 (2018).

    Article  ADS  Google Scholar 

  13. Kirmani, A., Jeelani, H., Montazerhodjat, V. & Goyal, V. K. Diffuse imaging: creating optical images with unfocused time-resolved illumination and sensing. IEEE Signal Process. Lett. 19, 31–34 (2012).

    Article  ADS  Google Scholar 

  14. Heide, F., Hullin, M. B., Gregson, J. & Heidrich, W. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. 32, 45 (2013).

    MATH  Google Scholar 

  15. Heide, F., Xiao, L., Heidrich, W. & Hullin, M. B. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. 2014 IEEE Conf. Computer Vision and Pattern Recognition 3222–3229 (IEEE, 2014).

  16. Kadambi, A., Zhao, H., Shi, B. & Raskar, R. Occluded imaging with time-of-flight sensors. ACM Trans. Graph. 35, 15 (2016).

    Article  Google Scholar 

  17. Pawlikowska, A. M., Halimi, A., Lamb, R. A. & Buller, G. S. Single-photon three-dimensional imaging at up to 10 kilometers range. Opt. Express 25, 11919–11931 (2017).

    Article  ADS  CAS  Google Scholar 

  18. Kirmani, A. et al. First-photon imaging. Science 343, 58–61 (2014).

    Article  ADS  CAS  Google Scholar 

  19. Shin, D., Kirmani, A., Goyal, V. K. & Shapiro, J. H. Photon-efficient computational 3D and reflectivity imaging with single-photon detectors. IEEE Trans. Comput. Imaging 1, 112–125 (2015).

    Article  MathSciNet  Google Scholar 

  20. Altmann, Y., Ren, X., McCarthy, A., Buller, G. S. & McLaughlin, S. Lidar waveform-based analysis of depth images constructed using sparse single-photon data. IEEE Trans. Image Process. 25, 1935–1946 (2016).

    Article  ADS  MathSciNet  Google Scholar 

  21. Rapp, J. & Goyal, V. K. A few photons among many: unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Comput. Imaging 3, 445–459 (2017).

    Article  MathSciNet  Google Scholar 

  22. Pediredla, A. K., Buttafava, M., Tosi, A., Cossairt, O. & Veeraraghavan, A. Reconstructing rooms using photon echoes: a plane based model and reconstruction algorithm for looking around the corner. In Proc. 2017 IEEE Int. Conf. Computational Photography 1–12 (IEEE, 2017).

  23. Pandharkar, R. et al. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In Proc. 2011 IEEE Conf. Computer Vision and Pattern Recognition 265–272 (IEEE, 2011).

  24. Naik, N., Zhao, S., Velten, A., Raskar, R. & Bala, K. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graphics 30, 171 (ACM, 2011).

  25. Thrampoulidis, C. et al. Exploiting occlusion in non-line-of-sight active imaging. IEEE Trans. Comput. Imaging 4, 419–431 (2018).

    Article  Google Scholar 

  26. Xu, F. et al. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. Opt. Express 26, 9945–9962 (2018).

    Article  ADS  Google Scholar 

  27. Torralba, A. & Freeman, W. T. Accidental pinhole and pinspeck cameras: revealing the scene outside the picture. Int. J. Comput. Vis. 110, 92–112 (2014).

    Article  Google Scholar 

  28. Bouman, K. L. et al. Turning corners into cameras: Principles and methods. In Proc. 23rd IEEE Int. Conf. Computer Vision, 2270–2278 (IEEE, 2017).

  29. Baradad, M. et al. Inferring light fields from shadows. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 6267–6275 (2018).

  30. Klein, J., Peters, C., Martín, J., Laurenzis, M. & Hullin, M. B. Tracking objects outside the line of sight using 2D intensity images. Sci. Rep. 6, 32491 (2016).

    Article  ADS  CAS  Google Scholar 

  31. Kajiya, J. T. The rendering equation. In Proc. 13th Conf. Computer Graphics and Interactive Techniques Vol. 20 143–150 (ACM, 1986).

  32. Beck, A. & Teboulle, M. A fast iterative shrinkage-thresholding algorithm. SIAM J. Imaging Sci. 2, 183–202 (2009).

    Article  MathSciNet  Google Scholar 

  33. Vetterli, M., Kovačević, J. & Goyal, V. K. Foundations of Signal Processing (Cambridge Univ. Press, Cambridge, 2014).

    Book  Google Scholar 

  34. Golub, G. H. & Van Loan, C. F. Matrix Computations 3rd edn (Johns Hopkins Univ. Press, Baltimore, 1989),

    Google Scholar 

  35. Bednar, J. B. & Watt, T. L. Alpha-trimmed means and their relationship to median filters. IEEE Trans. Acoust. Speech Signal Process. 32, 145–153 (1984).

    Article  Google Scholar 

Download references

Acknowledgements

We thank F. Durand, W. T. Freeman, Y. Ma, J. Rapp, J. H. Shapiro, A. Torralba, F. N. C. Wong and G. W. Wornell for discussions. This work was supported by the Defense Advanced Research Projects Agency (DARPA) REVEAL Program contract number HR0011-16-C-0030.

Reviewer information

Nature thanks M. Laurenzis, M. O’Toole and A. Velten for their contribution to the peer review of this work.

Author information

Author notes

  1. These authors contributed equally: Charles Saunders, John Murray-Bruce

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Boston University, Boston, MA, USA

    Charles Saunders, John Murray-Bruce & Vivek K Goyal

Authors
  1. Charles Saunders
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Murray-Bruce
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vivek K Goyal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

V.K.G. conceptualized the project, obtained funding and supervised the research. C.S. and J.M.-B. developed the methodology, performed the experiments, wrote the software and validated the results. C.S. produced the visualizations. J.M.-B. wrote the original draft. C.S., J.M.-B. and V.K.G. reviewed and edited the paper.

Corresponding author

Correspondence to Vivek K Goyal.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

This file contains supplementary methods, supplementary text, supplementary references (36–45), figures S1-S19, tables S1 and S2 and a caption for the supplementary video.

Video 1

Supplementary Video describes the experimental setup, the computational field of view of the system, and each of the steps involved in the computational estimation of the non-line-of-sight occluder position and scene. Also shown is a moving scene measured at 1 frame per second and reconstructed frame-by-frame, without exploiting assumptions of temporal continuity of the scene.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saunders, C., Murray-Bruce, J. & Goyal, V.K. Computational periscopy with an ordinary digital camera. Nature 565, 472–475 (2019). https://doi.org/10.1038/s41586-018-0868-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-018-0868-6

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics