Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A programmable diffractive deep neural network based on a digital-coding metasurface array

Nature Electronics volume 5, pages 113–122 (2022)Cite this article

Subjects

Abstract

The development of artificial intelligence is typically focused on computer algorithms and integrated circuits. Recently, all-optical diffractive deep neural networks have been created that are based on passive structures and can perform complicated functions designed by computer-based neural networks. However, once a passive diffractive deep neural network architecture is fabricated, its function is fixed. Here we report a programmable diffractive deep neural network that is based on a multi-layer digital-coding metasurface array. Each meta-atom on the metasurfaces is integrated with two amplifier chips and acts an active artificial neuron, providing a dynamic modulation range of 35 dB (from −22 dB to 13 dB). We show that the system, which we term a programmable artificial intelligence machine, can handle various deep learning tasks for wave sensing, including image classification, mobile communication coding–decoding and real-time multi-beam focusing. We also develop a reinforcement learning algorithm for on-site learning and a discrete optimization algorithm for digital coding.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A reprogrammable D2NN platform.
Fig. 2: The simulation results for oil-painting recognition.
Fig. 3: Experimental results of image classifications using PAIM.
Fig. 4: Experimental results for the encoder and decoder in the CDMA task using the PAIM.
Fig. 5: Space–time telecommunication system with and without the decoding part of PAIM.
Fig. 6: Experimental results of dynamic multi-beam focusing by the on-site reinforcement learning process using the PAIM.

Similar content being viewed by others

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.

Code availability

All the mathematical algorithms we used are provided in the Methods and Supplementary Information. The codes that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.

References

  1. Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  2. Huang, G., Huang, G. B., Song, S. J. & You, K. Y. Trends in extreme learning machines: a review. Neural Netw. 61, 32–48 (2015).

    MATH  Google Scholar 

  3. Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

    Google Scholar 

  4. Ren, H. R., Li, X. P., Zhang, Q. M. & Gu, M. On-chip noninterference angular momentum multiplexing of broadband light. Science 352, 805–809 (2016).

    Google Scholar 

  5. Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).

    Google Scholar 

  6. Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).

    Google Scholar 

  7. Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    Google Scholar 

  8. Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S. & Ma, Y. Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2009).

    Google Scholar 

  9. Ding, C., Choi, J., Tao, D. & Davis, L. S. Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38, 518–531 (2016).

    Google Scholar 

  10. Vazifeh, M. M., Santi, P., Resta, G., Strogatz, S. H. & Ratti, C. Addressing the minimum fleet problem in on-demand urban mobility. Nature 557, 534–538 (2018).

    Google Scholar 

  11. Li, W. et al. AADS: augmented autonomous driving simulation using data-driven algorithms. Sci. Robot. 4, eaaw0863 (2019).

    Google Scholar 

  12. Palangi, H. et al. Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE ACM Trans. Audio Speech Lang. Process. 24, 694–707 (2016).

    Google Scholar 

  13. Young, T., Hazarika, D., Poria, S. & Cambria, E. Recent trends in deep learning based natural language processing. IEEE Comput. Intell. Mag. 13, 55–75 (2018).

    Google Scholar 

  14. He, J. et al. The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25, 30–36 (2019).

    Google Scholar 

  15. Haque, A., Milstein, A. & Fei-Fei, L. Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585, 193–202 (2020).

    Google Scholar 

  16. Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

    MathSciNet  MATH  Google Scholar 

  17. Khoram, E. et al. Nanophotonic media for artificial neural inference. Photon. Res. 7, 823–827 (2019).

    Google Scholar 

  18. Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).

    Google Scholar 

  19. Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon 15, 367–373 (2021).

    Google Scholar 

  20. Cui, T. J., Qi, M. Q., Wan, X., Zhao, J. & Cheng, Q. Coding metamaterials, digital metamaterials and programmable metamaterials. Light Sci. Appl. 3, e218 (2014).

    Google Scholar 

  21. Cui, T. J., Liu, S. & Zhang, L. Information metamaterials and metasurfaces. J. Mater. Chem. C 5, 3644–3668 (2017).

    Google Scholar 

  22. Pendry, J. B., Luo, Y. & Zhao, R. K. Transforming the optical landscape. Science 348, 521–524 (2015).

    Google Scholar 

  23. Qiu, C. W. & Yang, Y. J. Vortex generation reaches a new plateau. Science 357, 645–645 (2017).

    Google Scholar 

  24. Hu, G. W. et al. Coherent steering of nonlinear chiral valley photons with a synthetic Au–WS2 metasurface. Nat. Photon. 13, 467–472 (2019).

    Google Scholar 

  25. Li, L. et al. Metalens-array-based high-dimensional and multiphoton quantum source. Science 368, 1487–1490 (2020).

    Google Scholar 

  26. Cui, T. J., Li, L., Liu, S., Ma, Q. & Cheng, Q. J. I. Information metamaterial systems. iScience 23, 101403 (2020).

    Google Scholar 

  27. Ma, Q. & Cui, T. J. Information metamaterials: bridging the physical world and digital world. PhotoniX 1, 1–32 (2020).

    Google Scholar 

  28. Zhao, J. et al. Programmable time-domain digital-coding metasurface for non-linear harmonic manipulation and new wireless communication systems. Natl Sci. Rev. 6, 231–238 (2019).

    Google Scholar 

  29. Cui, T. J., Liu, S., Bai, G. D. & Ma, Q. Direct transmission of digital message via programmable coding metasurface. Research 2019, 2584509 (2019).

    Google Scholar 

  30. Zhao, H. et al. Metasurface-assisted massive backscatter wireless communication with commodity Wi-Fi signals. Nat. Commun. 11, 3926–3926 (2020).

    Google Scholar 

  31. Zhang, L. et al. A wireless communication scheme based on space- and frequency-division multiplexing using digital metasurfaces. Nat. Electron. 4, 218–227 (2021).

    Google Scholar 

  32. Li, L. L. et al. Electromagnetic reprogrammable coding-metasurface holograms. Nat. Commun. 8, 197 (2017).

    Google Scholar 

  33. Li, L. et al. Intelligent metasurface imager and recognizer. Light Sci. Appl 8, 97 (2019).

    Google Scholar 

  34. Zhang, L. et al. Space-time-coding digital metasurfaces. Nat. Commun. 9, 4334 (2018).

    Google Scholar 

  35. Ma, Q. et al. Smart metasurface with self-adaptively reprogrammable functions. Light Sci. Appl. 8, 98 (2019).

    Google Scholar 

  36. Zhang, X. G. et al. An optically driven digital metasurface for programming electromagnetic functions. Nat. Electron. 3, 165–171 (2020).

    Google Scholar 

  37. Ma, Q. et al. Controllable and programmable nonreciprocity based on detachable digital coding metasurface. Adv. Opt. Mater. 7, 1901285 (2019).

    Google Scholar 

  38. Chen, L. et al. Space-energy digital-coding metasurface based on an active amplifier. Phys. Rev. Appl. 11, 054051 (2019).

    Google Scholar 

  39. Schmidhuber, J. Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).

    Google Scholar 

  40. Ryou, A. et al. Free-space optical neural network based on thermal atomic nonlinearity. Photon. Res. 9, B128–B134 (2021).

    Google Scholar 

  41. Zhu, J., Zhao, M., Zhang, S. & Zhou, W. Exploring the road to 6G: ABC—foundation for intelligent mobile networks. China Commun. 17, 51–67 (2020).

    Google Scholar 

  42. Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).

    Google Scholar 

  43. Kingma, D. P. & Ba, J. L. Adam: a method for stochastic optimization. in 3rd International Conference on Learning Representations (2015).

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (grants nos. 2017YFA0700201, 2017YFA0700202 and 2017YFA0700203).

Author information

Author notes

  1. These authors contributed equally: Che Liu, Qian Ma.

Authors and Affiliations

  1. Institute of Electromagnetic Space, Southeast University, Nanjing, China

    Che Liu, Qian Ma, Zhang Jie Luo, Qiao Ru Hong, Qiang Xiao, Hao Chi Zhang, Wen Ming Yu, Qiang Cheng & Tie Jun Cui

  2. State Key Laboratory of Millimeter Wave, Southeast University, Nanjing, China

    Che Liu, Qian Ma, Zhang Jie Luo, Qiao Ru Hong, Qiang Xiao, Hao Chi Zhang, Wen Ming Yu, Qiang Cheng & Tie Jun Cui

  3. Center of Intelligent Metamaterials, Pazhou Laboratory, Guangzhou, China

    Che Liu, Qian Ma, Zhang Jie Luo, Qiang Xiao, Hao Chi Zhang, Long Miao, Qiang Cheng, Lianlin Li & Tie Jun Cui

  4. Jiangsu Cyber-Space Science & Technology Co., Ltd, Nanjing, China

    Long Miao & Wen Ming Yu

  5. School of Electronic Engineering and Computer Sciences, Peking University, Beijing, China

    Lianlin Li

Authors
  1. Che Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qian Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhang Jie Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiao Ru Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hao Chi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Long Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wen Ming Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Qiang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lianlin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Tie Jun Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

T.J.C., C.L., Q.M. and L.L. conceived the research. C.L. and Q.M. designed the PAIM devices and relevant algorithms. C.L., Q.M. and Z.J.L. contributed to the experiments. T.J.C., C.L. and Q.M. prepared the manuscript. T.J.C. initiated and supervised the research. All authors contributed to the data analysis and writing of the manuscript, which was reviewed by all authors.

Corresponding author

Correspondence to Tie Jun Cui.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Electronics thanks Arka Majumdar and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information

Supplementary Notes 1–12, Figs. 1–12, references 44–46.

Supplementary Video 1

The video recording the experimental results of the space–time telecommunication system with and without the decoding part of PAIM.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, C., Ma, Q., Luo, Z.J. et al. A programmable diffractive deep neural network based on a digital-coding metasurface array. Nat Electron 5, 113–122 (2022). https://doi.org/10.1038/s41928-022-00719-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-022-00719-9

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics