The development of artificial intelligence is typically focused on computer algorithms and integrated circuits. Recently, all-optical diffractive deep neural networks have been created that are based on passive structures and can perform complicated functions designed by computer-based neural networks. However, once a passive diffractive deep neural network architecture is fabricated, its function is fixed. Here we report a programmable diffractive deep neural network that is based on a multi-layer digital-coding metasurface array. Each meta-atom on the metasurfaces is integrated with two amplifier chips and acts an active artificial neuron, providing a dynamic modulation range of 35 dB (from −22 dB to 13 dB). We show that the system, which we term a programmable artificial intelligence machine, can handle various deep learning tasks for wave sensing, including image classification, mobile communication coding–decoding and real-time multi-beam focusing. We also develop a reinforcement learning algorithm for on-site learning and a discrete optimization algorithm for digital coding.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
PhotoniX Open Access 14 October 2022
Light: Science & Applications Open Access 06 July 2022
Nano-Micro Letters Open Access 14 June 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $9.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Huang, G., Huang, G. B., Song, S. J. & You, K. Y. Trends in extreme learning machines: a review. Neural Netw. 61, 32–48 (2015).
Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
Ren, H. R., Li, X. P., Zhang, Q. M. & Gu, M. On-chip noninterference angular momentum multiplexing of broadband light. Science 352, 805–809 (2016).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).
Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S. & Ma, Y. Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2009).
Ding, C., Choi, J., Tao, D. & Davis, L. S. Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38, 518–531 (2016).
Vazifeh, M. M., Santi, P., Resta, G., Strogatz, S. H. & Ratti, C. Addressing the minimum fleet problem in on-demand urban mobility. Nature 557, 534–538 (2018).
Li, W. et al. AADS: augmented autonomous driving simulation using data-driven algorithms. Sci. Robot. 4, eaaw0863 (2019).
Palangi, H. et al. Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE ACM Trans. Audio Speech Lang. Process. 24, 694–707 (2016).
Young, T., Hazarika, D., Poria, S. & Cambria, E. Recent trends in deep learning based natural language processing. IEEE Comput. Intell. Mag. 13, 55–75 (2018).
He, J. et al. The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25, 30–36 (2019).
Haque, A., Milstein, A. & Fei-Fei, L. Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585, 193–202 (2020).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
Khoram, E. et al. Nanophotonic media for artificial neural inference. Photon. Res. 7, 823–827 (2019).
Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).
Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon 15, 367–373 (2021).
Cui, T. J., Qi, M. Q., Wan, X., Zhao, J. & Cheng, Q. Coding metamaterials, digital metamaterials and programmable metamaterials. Light Sci. Appl. 3, e218 (2014).
Cui, T. J., Liu, S. & Zhang, L. Information metamaterials and metasurfaces. J. Mater. Chem. C 5, 3644–3668 (2017).
Pendry, J. B., Luo, Y. & Zhao, R. K. Transforming the optical landscape. Science 348, 521–524 (2015).
Qiu, C. W. & Yang, Y. J. Vortex generation reaches a new plateau. Science 357, 645–645 (2017).
Hu, G. W. et al. Coherent steering of nonlinear chiral valley photons with a synthetic Au–WS2 metasurface. Nat. Photon. 13, 467–472 (2019).
Li, L. et al. Metalens-array-based high-dimensional and multiphoton quantum source. Science 368, 1487–1490 (2020).
Cui, T. J., Li, L., Liu, S., Ma, Q. & Cheng, Q. J. I. Information metamaterial systems. iScience 23, 101403 (2020).
Ma, Q. & Cui, T. J. Information metamaterials: bridging the physical world and digital world. PhotoniX 1, 1–32 (2020).
Zhao, J. et al. Programmable time-domain digital-coding metasurface for non-linear harmonic manipulation and new wireless communication systems. Natl Sci. Rev. 6, 231–238 (2019).
Cui, T. J., Liu, S., Bai, G. D. & Ma, Q. Direct transmission of digital message via programmable coding metasurface. Research 2019, 2584509 (2019).
Zhao, H. et al. Metasurface-assisted massive backscatter wireless communication with commodity Wi-Fi signals. Nat. Commun. 11, 3926–3926 (2020).
Zhang, L. et al. A wireless communication scheme based on space- and frequency-division multiplexing using digital metasurfaces. Nat. Electron. 4, 218–227 (2021).
Li, L. L. et al. Electromagnetic reprogrammable coding-metasurface holograms. Nat. Commun. 8, 197 (2017).
Li, L. et al. Intelligent metasurface imager and recognizer. Light Sci. Appl 8, 97 (2019).
Zhang, L. et al. Space-time-coding digital metasurfaces. Nat. Commun. 9, 4334 (2018).
Ma, Q. et al. Smart metasurface with self-adaptively reprogrammable functions. Light Sci. Appl. 8, 98 (2019).
Zhang, X. G. et al. An optically driven digital metasurface for programming electromagnetic functions. Nat. Electron. 3, 165–171 (2020).
Ma, Q. et al. Controllable and programmable nonreciprocity based on detachable digital coding metasurface. Adv. Opt. Mater. 7, 1901285 (2019).
Chen, L. et al. Space-energy digital-coding metasurface based on an active amplifier. Phys. Rev. Appl. 11, 054051 (2019).
Schmidhuber, J. Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).
Ryou, A. et al. Free-space optical neural network based on thermal atomic nonlinearity. Photon. Res. 9, B128–B134 (2021).
Zhu, J., Zhao, M., Zhang, S. & Zhou, W. Exploring the road to 6G: ABC—foundation for intelligent mobile networks. China Commun. 17, 51–67 (2020).
Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).
Kingma, D. P. & Ba, J. L. Adam: a method for stochastic optimization. in 3rd International Conference on Learning Representations (2015).
This work was supported by the National Key Research and Development Program of China (grants nos. 2017YFA0700201, 2017YFA0700202 and 2017YFA0700203).
The authors declare no competing interests.
Peer review information
Nature Electronics thanks Arka Majumdar and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Liu, C., Ma, Q., Luo, Z.J. et al. A programmable diffractive deep neural network based on a digital-coding metasurface array. Nat Electron 5, 113–122 (2022). https://doi.org/10.1038/s41928-022-00719-9
This article is cited by
Nature Electronics (2022)
Light: Science & Applications (2022)
Nano-Micro Letters (2022)