Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Image sensing with multilayer nonlinear optical neural networks

Nature Photonics volume 17, pages 408–415 (2023)Cite this article

Subjects

Abstract

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object’s position or contour, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm relies on optical systems that—instead of performing imaging—act as encoders that optically compress images into low-dimensional spaces by extracting salient features; however, the performance of these encoders is typically limited by their linearity. Here we report a nonlinear, multilayer optical neural network (ONN) encoder for image sensing based on a commercial image intensifier as an optical-to-optical nonlinear activation function. This nonlinear ONN outperforms similarly sized linear optical encoders across several representative tasks, including machine-vision benchmarks, flow-cytometry image classification and identification of objects in a three-dimensionally printed real scene. For machine-vision tasks, especially those featuring incoherent broadband illumination, our concept allows for a considerable reduction in the requirement of camera resolution and electronic post-processing complexity. In general, image pre-processing with ONNs should enable image-sensing applications that operate accurately with fewer pixels, fewer photons, higher throughput and lower latency.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A multilayer optical-neural-network encoder as a frontend for image sensing.
Fig. 2: Comparison between linear and nonlinear ONN encoders on diverse image classification tasks.
Fig. 3: Nonlinear ONN encoders trained for classification can be reused for diverse image-sensing tasks by training only new digital backends.
Fig. 4: Simulations of performance scaling with deeper nonlinear optical neural network encoders for ten-class cell-organelle classification.

Similar content being viewed by others

Data availability

The demonstration data for data gathering, as well as training data for the all-optical/digital neural networks, are available at https://github.com/mcmahon-lab/Image-sensing-with-multilayer-nonlinear-optical-neural-networks.

Code availability

All of the data generated, and code used, in this work are available at https://doi.org/10.5281/zenodo.6888985.

References

  1. Duarte, M. F. et al. Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25, 83–91 (2008).

    Article  ADS  Google Scholar 

  2. Sterling, P. Principles of Neural Design (MIT Press, 2015).

  3. Gibson, G. M., Johnson, S. D. & Padgett, M. J. Single-pixel imaging 12 years on: a review. Opt. Express 28, 28190–28208 (2020).

    Article  ADS  Google Scholar 

  4. Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 1–10 (2018).

    Article  Google Scholar 

  5. Martel, J. N. P., Mueller, L. K., Carey, S. J., Dudek, P. & Wetzstein, G. Neural sensors: learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42, 1642–1653 (2020).

    Article  Google Scholar 

  6. Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020).

    Article  ADS  Google Scholar 

  7. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    Article  ADS  Google Scholar 

  8. Pad, P. et al. Efficient neural vision systems based on convolutional image acquisition. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12285–12294 (IEEE, 2020).

  9. Zheng, H. et al. Meta-optic accelerators for object classifiers. Sci. Adv. 8, eabo6410 (2022).

    Article  Google Scholar 

  10. Matic, R. M. & Goodman, J. W. Comparison of optical predetection processing and postdetection linear processing for partially coherent image estimation. J. Opt. Soc. Am. A 6, 213–228 (1989).

    Article  ADS  Google Scholar 

  11. Kubala, K., Dowski, E. & Cathey, W. T. Reducing complexity in computational imaging systems. Opt. Express 11, 2102–2108 (2003).

    Article  ADS  Google Scholar 

  12. Stork, D. G. & Robinson, M. D. Theoretical foundations for joint digital-optical analysis of electro-optical imaging systems. Appl. Opt. 47, B64–B75 (2008).

    Article  Google Scholar 

  13. Sitzmann, V. et al. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 1–13 (2018).

    Article  Google Scholar 

  14. Colburn, S., Chu, Y., Shilzerman, E. & Majumdar, A. Optical frontend for a convolutional neural network. Appl. Optics 58, 3179–3186 (2019).

    Article  ADS  Google Scholar 

  15. Kim, K., Konda, P. C., Cooke, C. L., Appel, R. & Horstmeyer, R. Multi-element microscope optimization by a learned sensing network with composite physical layers. Opt. Lett. 45, 5684–5687 (2020).

    Article  ADS  Google Scholar 

  16. Markley, E., Liu, F. L., Kellman, M., Antipa, N. & Waller, L. Physics-based learned diffuser for single-shot 3D imaging. In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems (NeurIPS, 2021).

  17. Vargas, E., Martel, J. N. P., Wetzstein, G. & Arguello, H. Time-multiplexed coded aperture imaging: learned coded aperture and pixel exposures for compressive imaging systems. In Proc. IEEE/CVF International Conference on Computer Vision 2692–2702 (IEEE, 2021).

  18. Liutkus, A. et al. Imaging with nature: compressive imaging using a multiply scattering medium. Sci. Rep. 4, 1–7 (2014).

    Article  Google Scholar 

  19. Li, J. et al. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. 7, eabd7690 (2021).

    Article  ADS  Google Scholar 

  20. Asif, M. S., Ayremlou, A., Sankaranarayanan, A., Veeraraghavan, A. & Baraniuk, R. G. Flatcam: thin, lensless cameras using coded aperture and computation. IEEE Trans. Comput. Imaging 3, 384–397 (2016).

    Article  MathSciNet  Google Scholar 

  21. Baek, S.-H. et al. Single-shot hyperspectral-depth imaging with learned diffractive optics. In Proc. IEEE/CVF International Conference on Computer Vision 2651–2660 (IEEE, 2021).

  22. Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).

    Article  ADS  Google Scholar 

  23. Brady, D. J. et al. Smart cameras. Preprint at https://arxiv.org/abs/2002.04705 (2020).

  24. Burt, P. J. Smart sensing within a pyramid vision machine. Proc. IEEE 76, 1006–1015 (1988).

    Article  Google Scholar 

  25. Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).

    Article  Google Scholar 

  26. Marciniak, C. D. et al. Optimal metrology with programmable quantum sensors. Nature 603, 604–609 (2022).

    Article  ADS  Google Scholar 

  27. Ballard, Z., Brown, C., Madni, A. M. & Ozcan, A. Machine learning and computation-enabled intelligent sensor design. Nat. Mach. Intell. 3, 556–565 (2021).

    Article  Google Scholar 

  28. Shastri, B. J. et al. Photonics for artificial intelligence and neuromorphic computing. Nat. Photon. 15, 102–114 (2021).

    Article  ADS  Google Scholar 

  29. Li, H.-Y. S., Qiao, Y. & Psaltis, D. Optical network for real-time face recognition. Appl. Opt. 32, 5026–5035 (1993).

    Article  ADS  Google Scholar 

  30. Lin, H. W., Tegmark, M. & Rolnick, D. Why does deep and cheap learning work so well? J. Stat. Phys. 168, 1223–1247 (2017).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  31. Wagner, K. & Psaltis, D. Multilayer optical learning networks. Appl. Opt. 26, 5061–5076 (1987).

    Article  ADS  Google Scholar 

  32. Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).

    Article  ADS  Google Scholar 

  33. Fard, M. M. P. et al. Experimental realization of arbitrary activation functions for optical neural networks. Opt. Express 28, 12138–12148 (2020).

    Article  ADS  Google Scholar 

  34. Ryou, A. et al. Free-space optical neural network based on thermal atomic nonlinearity. Photon. Res. 9, B128–B134 (2021).

    Article  Google Scholar 

  35. Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    Article  ADS  Google Scholar 

  36. Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 607, 501–506 (2022).

    Article  ADS  Google Scholar 

  37. Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon. 15, 367–373 (2021).

    Article  ADS  MathSciNet  Google Scholar 

  38. Bernstein, L. et al. Single-shot optical neural network. Preprint at https://arxiv.org/abs/2205.09103 (2022).

  39. Wang, T. et al. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 13, 1–8 (2022).

    ADS  Google Scholar 

  40. Zemel, J. N. Sensors V 6—Optical Sensors—A Comprehensive Survey (John Wiley & Sons, 1991).

  41. Jongejan, J., Rowley, H., Kawashima, T., Kim, J. & Fox-Gieg, N. The Quick, Draw! AI Experiment https://quickdraw.withgoogle.com/ (2016).

  42. Schraivogel, D. et al. High-speed fluorescence image–enabled cell sorting. Science 375, 315–320 (2022).

    Article  ADS  Google Scholar 

  43. Li, Y. et al. Deep cytometry: deep learning with real-time inference in cell sorting and flow cytometry. Sci. Rep. 9, 1–12 (2019).

    ADS  Google Scholar 

  44. Lee, K. C. M., Guck, J., Goda, K. & Tsia, K. K. Toward deep biophysical cytometry: prospects and challenges. Trends Biotechnol. 39, 1249–1262 (2021).

    Article  Google Scholar 

  45. Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  46. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  47. Li, G. H.et al. All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning. Nanophotonics https://doi.org/10.1515/nanoph-2022-0137 (2022).

  48. Guo, Q. et al. Femtojoule femtosecond all-optical switching in lithium niobate nanophotonics. Nat. Photon. 16, 625–631 (2022).

    Article  ADS  MathSciNet  Google Scholar 

  49. Nahmias, M. A., Shastri, B. J., Tait, A. N. & Prucnal, P. R. A leaky integrate-and-fire laser neuron for ultrafast cognitive computing. IEEE J. Sel. Top. Quantum Electron. 19, 1–12 (2013).

    Article  Google Scholar 

  50. Mirek, R. et al. Neural networks based on ultrafast time-delayed effects in exciton polaritons. Phys. Rev. Appl. 17, 054037 (2022).

    Article  ADS  Google Scholar 

  51. Bandyopadhyay, S. et al. Single chip photonic deep neural network with accelerated training. Preprint at https://arxiv.org/abs/2208.01623 (2022).

  52. Xu, X. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021).

    Article  ADS  Google Scholar 

  53. Makarenko, M. et al. Real-time hyperspectral imaging in hardware via trained metasurface encoders. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12692–12702 (IEEE, 2022); https://doi.org/10.1109/CVPR52688.2022.01236

  54. Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR, 2019).

  55. Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Optuna: a next-generation hyperparameter optimization framework. In Proc. 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2623–2631 (ACM, 2019).

  56. Heuser, T. et al. Developing a photonic hardware platform for brain-inspired computing based on 5 × 5 VCSEL arrays. J. Phys. Photon. 2, 44002 (2020).

    Article  Google Scholar 

  57. Narayan, A., Berger, B. & Cho, H. Assessing single-cell transcriptomic variability through density-preserving data visualization. Nat. Biotechnol. 39, 765–774 (2021).

    Article  Google Scholar 

Download references

Acknowledgements

We wish to thank NTT Research for their financial and technical support (to T.W., L.G.W., S.-Y.M., T.O. and P.L.M.). Portions of this work were supported by the National Science Foundation (award no. CCF-1918549 to T.W., M.M. Stein and P.L.M.), a Kavli Institute at Cornell instrumentation grant (to T.W. and P.L.M.), and a David and Lucile Packard Foundation Fellowship (to P.L.M.). P.L.M. acknowledges membership of the CIFAR Quantum Information Science Program as an Azrieli Global Scholar. T.W. acknowledges the partial support from Schmidt Futures via an Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship to Cornell University. We acknowledge helpful discussions with A. Senanian, B. Malia, F. Presutti, V. Kremenetski, S. Prabhu, A. Barth, R. Oliver, and D. Schraivogel. We also acknowledge S. Sohoni for help with figure design.

Author information

Author notes

  1. These authors contributed equally: Tianyu Wang, Mandar M. Sohoni.

Authors and Affiliations

  1. School of Applied and Engineering Physics, Cornell University, Ithaca, NY, USA

    Tianyu Wang, Mandar M. Sohoni, Logan G. Wright, Martin M. Stein, Shi-Yuan Ma, Tatsuhiro Onodera, Maxwell G. Anderson & Peter L. McMahon

  2. Physics & Informatics Laboratories, NTT Research Inc., Sunnyvale, CA, USA

    Logan G. Wright & Tatsuhiro Onodera

  3. Kavli Institute at Cornell for Nanoscale Science, Cornell University, Ithaca, NY, USA

    Peter L. McMahon

Authors
  1. Tianyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mandar M. Sohoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Logan G. Wright
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin M. Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shi-Yuan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tatsuhiro Onodera
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Maxwell G. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Peter L. McMahon
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

T.W., L.G.W., M.M. Sohoni and P.L.M. conceived the project and designed the experiments. M.M. Sohoni and T.W. built and performed the experiments on the nonlinear and linear ONN encoders, and analysed the data. T.W. performed the extended cell-organelle simulations. M.M. Stein performed the neural architecture search for QuickDraw reconstruction. S-Y.M. and T.O. aided in simulations of deep optical encoders. M.G.A. assisted with 3D-scene modelling. L.G.W., T.W., M.M. Sohoni and P.L.M. wrote the manuscript. P.L.M. and L.G.W. supervised the project.

Corresponding authors

Correspondence to Tianyu Wang, Mandar M. Sohoni, Logan G. Wright or Peter L. McMahon.

Ethics declarations

Competing interests

T.W., M.M. Sohoni, L.G.W. and P.L.M. are listed as inventors on a US provisional patent application (serial no. 63/392,042) on nonlinear optical neural network pre-processors for imaging and image sensing. The other authors declare no competing interests.

Peer review

Peer review information

Nature Photonics thanks Jacques Carolan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–29, Sections 1–3 and Tables 1–11.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Sohoni, M.M., Wright, L.G. et al. Image sensing with multilayer nonlinear optical neural networks. Nat. Photon. 17, 408–415 (2023). https://doi.org/10.1038/s41566-023-01170-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41566-023-01170-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing