Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Nature Photonics volume 15, pages 367–373 (2021)Cite this article

Subjects

Abstract

There is an ever-growing demand for artificial intelligence. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. There has been long-term interest in optically constructing the most widely used artificial-intelligence architecture, that is, artificial neural networks, to achieve brain-inspired information processing at the speed of light. However, owing to restrictions in design flexibility and the accumulation of system errors, existing processor architectures are not reconfigurable and have limited model complexity and experimental performance. Here, we propose the reconfigurable diffractive processing unit, an optoelectronic fused computing architecture based on the diffraction of light, which can support different neural networks and achieve a high model complexity with millions of neurons. Along with the developed adaptive training approach to circumvent system errors, we achieved excellent experimental accuracies for high-speed image and video recognition over benchmark datasets and a computing performance superior to that of cutting-edge electronic computing platforms.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Reconfigurable diffractive optoelectronic processor.
Fig. 2: Adaptive training of the optoelectronic D2NN for handwritten digit classification.
Fig. 3: High-accuracy MNIST and Fashion-MNIST classification with the D-NIN-1.
Fig. 4: Human action recognition with the D-RNN.

Similar content being viewed by others

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.

Code availability

All relevant code is available from the corresponding author upon reasonable request.

References

  1. Moore, G. E. Cramming more components onto integrated circuits. Proc. IEEE 86, 82–85 (1998).

    Article  Google Scholar 

  2. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (eds Bartlett, P. et al.) 1097–1105 (Curran Associates, 2013).

  3. Zhang, C. et al. Optimizing FPGA-based accelerator design for deep convolutional neural networks. In Proc. 2015 ACM/SIGDA International Symposium on Field-programmable Gate Arrays 161–170 (Association for Computing Machinery, 2015).

  4. Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    Article  ADS  Google Scholar 

  5. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  ADS  Google Scholar 

  6. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  7. Graves, A., Mohamed, A.-r. & Hinton, G. Speech recognition with deep recurrent neural networks. In Proc. International Conference on Acoustics, Speech and Signal Processing 6645–6649 (IEEE, 2013).

  8. Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).

    Article  ADS  Google Scholar 

  9. Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).

    Article  Google Scholar 

  10. Waldrop, M. M. The chips are down for Moore’s law. Nature 530, 144–147 (2016).

    Article  ADS  Google Scholar 

  11. Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).

    Article  ADS  Google Scholar 

  12. Prucnal, P. R. & Shastri, B. J. Neuromorphic Photonics (CRC, 2017).

    Book  Google Scholar 

  13. Zhang, Q., Yu, H., Barbiero, M., Wang, B. & Gu, M. Artificial neural networks enabled by nanophotonics. Light Sci. Appl. 8, 42 (2019).

    Article  ADS  Google Scholar 

  14. Ferrera, M. et al. On-chip CMOS-compatible all-optical integrator. Nat. Commun. 1, 29 (2010).

    Article  ADS  Google Scholar 

  15. Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).

    Article  ADS  Google Scholar 

  16. Estakhri, N. M., Edwards, B. & Engheta, N. Inverse-designed metastructures that solve equations. Science 363, 1333–1338 (2019).

    Article  ADS  MathSciNet  Google Scholar 

  17. Xu, X.-Y. et al. A scalable photonic computer solving the subset sum problem. Sci. Adv. 6, eaay5853 (2020).

    Article  ADS  Google Scholar 

  18. Liu, W. et al. A fully reconfigurable photonic integrated signal processor. Nat. Photon. 10, 190–195 (2016).

    Article  ADS  Google Scholar 

  19. Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).

    Article  ADS  Google Scholar 

  20. Shainline, J. M., Buckley, S. M., Mirin, R. P. & Nam, S. W. Superconducting optoelectronic circuits for neuromorphic computing. Phys. Rev. Appl. 7, 034013 (2016).

    Article  ADS  Google Scholar 

  21. Van der Sande, G., Brunner, D. & Soriano, M. C. Advances in photonic reservoir computing. Nanophotonics 6, 561–576 (2017).

    Article  Google Scholar 

  22. Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).

    Google Scholar 

  23. Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    Article  ADS  Google Scholar 

  24. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    Article  ADS  Google Scholar 

  25. Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–447 (2017).

    Article  ADS  Google Scholar 

  26. Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).

    Article  ADS  Google Scholar 

  27. Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

    Article  ADS  MathSciNet  Google Scholar 

  28. Tao, Y. et al. Fourier-space diffractive deep neural network. Phys. Rev. Lett. 123, 023901 (2019).

    Article  ADS  Google Scholar 

  29. Zhou, T. et al. In situ optical backpropagation training of diffractive optical neural networks. Photonics Res. 8, 940–953 (2020).

    Article  Google Scholar 

  30. Dou, H. et al. Residual D2NN: training diffractive deep neural networks via learnable light shortcuts. Opt. Lett. 45, 2688–2691 (2020).

    Article  ADS  Google Scholar 

  31. Antonik, P., Marsal, N., Brunner, D. & Rontani, D. Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell. 1, 530–537 (2019).

    Article  Google Scholar 

  32. Rafayelyan, M., Dong, J., Tan, Y., Krzakala, F. & Gigan, S. Large-scale optical reservoir computing for spatiotemporal chaotic systems prediction. Phys. Rev. X 10, 041037 (2020).

    Google Scholar 

  33. Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).

    Article  ADS  Google Scholar 

  34. Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).

    Article  ADS  Google Scholar 

  35. Maktoobi, S. et al. Diffractive coupling for photonic networks: how big can we go? IEEE J. Sel. Top. Quantum Electron. 26, 1–8 (2019).

    Article  Google Scholar 

  36. Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).

    Article  ADS  Google Scholar 

  37. Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light: Sci. Appl. 8, 112 (2019).

    Article  ADS  Google Scholar 

  38. Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).

    Article  ADS  Google Scholar 

  39. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Article  Google Scholar 

  40. Xiao, H., Rasul, K., & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).

  41. Blank, M., Gorelick, L., Shechtman, E., Irani, M. & Basri, R. Actions as space-time shapes. In Proc. Tenth International Conference on Computer Vision 1395–1402 (IEEE, 2005).

  42. Schuldt, C., Laptev, I. & Caputo, B. Recognizing human actions: a local SVM approach. In Proc. 17th International Conference on Pattern Recognition 32–36 (IEEE, 2004).

  43. Jhuang, H., Serre, T., Wolf, L. & Poggio, T. A biologically inspired system for action recognition. In Proc. 11th International Conference on Computer Vision 1–8 (IEEE, 2007).

  44. Shu, N., Tang, Q. & Liu, H. A bio-inspired approach modeling spiking neural networks of visual cortex for human action recognition. In 2014 International Joint Conference on Neural Networks 3450–3457 (IEEE, 2014).

  45. Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).

    Article  Google Scholar 

  46. Wang, J. et al. Terabit free-space data transmission employing orbital angular momentum multiplexing. Nat. Photon. 6, 488–496 (2012).

    Article  ADS  Google Scholar 

  47. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  48. Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).

    Article  ADS  Google Scholar 

  49. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Computer Vision – ECCV 2018 (eds Ferrari, V. et al.) 833–851 (Springer, 2018).

  50. Larger, L. et al. High-speed photonic reservoir computing using a time-delay-based architecture: million words per second classification. Phys. Rev. X 7, 011015 (2017).

    Google Scholar 

Download references

Acknowledgements

We thank T. Zhu and T. Yan for assistance with the software for the prototype system. This work is supported by the Beijing Municipal Science and Technology Commission (No. Z181100003118014), the National Key Research and Development Program of China (No. 2020AAA0130000), the National Natural Science Foundation of China (No. 62088102 and No. 61860206003) and the Tsinghua University Initiative Scientific Research Program.

Author information

Authors and Affiliations

  1. Department of Automation, Tsinghua University, Beijing, China

    Tiankuang Zhou, Xing Lin, Jiamin Wu, Yitong Chen, Hao Xie, Yipeng Li, Jingtao Fan & Qionghai Dai

  2. Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China

    Tiankuang Zhou, Xing Lin, Jiamin Wu, Yitong Chen, Hao Xie, Yipeng Li, Jingtao Fan, Lu Fang & Qionghai Dai

  3. Department of Electronic Engineering, Tsinghua University, Beijing, China

    Tiankuang Zhou & Lu Fang

  4. Graduate School at Shenzhen, Tsinghua University, Shenzhen, China

    Tiankuang Zhou

  5. Beijing Innovation Center for Future Chips, Tsinghua University, Beijing, China

    Xing Lin & Huaqiang Wu

  6. Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China

    Huaqiang Wu, Lu Fang & Qionghai Dai

  7. Institute of Microelectronics, Tsinghua University, Beijing, China

    Huaqiang Wu

Authors
  1. Tiankuang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xing Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiamin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yitong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yipeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jingtao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Huaqiang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lu Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Qionghai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Q.D., X.L. and L.F. initiated and supervised the project. X.L. and T.Z. conceived the research and method. X.L. designed the simulations and experiments. T.Z. conducted the experiments. T.Z. and Y.C. performed the simulations and processed the data. T.Z., J.W., Y.C. and H.X. built the experimental system. X.L., T.Z., J.W., H.W., H.X., J.F., Y.L. and L.F. analysed the results. X.L., T.Z., L.F. and Q.D. prepared the manuscript with input from all authors. All authors discussed the research.

Corresponding authors

Correspondence to Xing Lin, Lu Fang or Qionghai Dai.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Photonics thanks Nathan Youngblood and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–10, discussion and Tables 1–4.

Supplementary Video 1

Experimental demonstration by using a camera to capture the handwritten digits as the inputs for the system.

Supplementary Video 2

Experimental results of D2NN on the MNIST database.

Supplementary Video 3

Experimental results of D-NIN-1 on the MNIST database.

Supplementary Video 4

Experimental results of D-RNN on the Weizmann database.

Supplementary Video 5

Experimental results of D-RNN on the KTH database.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, T., Lin, X., Wu, J. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics 15, 367–373 (2021). https://doi.org/10.1038/s41566-021-00796-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41566-021-00796-w

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing