Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Abstract

There is an ever-growing demand for artificial intelligence. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. There has been long-term interest in optically constructing the most widely used artificial-intelligence architecture, that is, artificial neural networks, to achieve brain-inspired information processing at the speed of light. However, owing to restrictions in design flexibility and the accumulation of system errors, existing processor architectures are not reconfigurable and have limited model complexity and experimental performance. Here, we propose the reconfigurable diffractive processing unit, an optoelectronic fused computing architecture based on the diffraction of light, which can support different neural networks and achieve a high model complexity with millions of neurons. Along with the developed adaptive training approach to circumvent system errors, we achieved excellent experimental accuracies for high-speed image and video recognition over benchmark datasets and a computing performance superior to that of cutting-edge electronic computing platforms.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Reconfigurable diffractive optoelectronic processor.
Fig. 2: Adaptive training of the optoelectronic D2NN for handwritten digit classification.
Fig. 3: High-accuracy MNIST and Fashion-MNIST classification with the D-NIN-1.
Fig. 4: Human action recognition with the D-RNN.

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.

Code availability

All relevant code is available from the corresponding author upon reasonable request.

References

  1. 1.

    Moore, G. E. Cramming more components onto integrated circuits. Proc. IEEE 86, 82–85 (1998).

    Article  Google Scholar 

  2. 2.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (eds Bartlett, P. et al.) 1097–1105 (Curran Associates, 2013).

  3. 3.

    Zhang, C. et al. Optimizing FPGA-based accelerator design for deep convolutional neural networks. In Proc. 2015 ACM/SIGDA International Symposium on Field-programmable Gate Arrays 161–170 (Association for Computing Machinery, 2015).

  4. 4.

    Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    ADS  Article  Google Scholar 

  5. 5.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    ADS  Article  Google Scholar 

  6. 6.

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  7. 7.

    Graves, A., Mohamed, A.-r. & Hinton, G. Speech recognition with deep recurrent neural networks. In Proc. International Conference on Acoustics, Speech and Signal Processing 6645–6649 (IEEE, 2013).

  8. 8.

    Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).

    ADS  Article  Google Scholar 

  9. 9.

    Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).

    Article  Google Scholar 

  10. 10.

    Waldrop, M. M. The chips are down for Moore’s law. Nature 530, 144–147 (2016).

    ADS  Article  Google Scholar 

  11. 11.

    Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).

    ADS  Article  Google Scholar 

  12. 12.

    Prucnal, P. R. & Shastri, B. J. Neuromorphic Photonics (CRC, 2017).

    Book  Google Scholar 

  13. 13.

    Zhang, Q., Yu, H., Barbiero, M., Wang, B. & Gu, M. Artificial neural networks enabled by nanophotonics. Light Sci. Appl. 8, 42 (2019).

    ADS  Article  Google Scholar 

  14. 14.

    Ferrera, M. et al. On-chip CMOS-compatible all-optical integrator. Nat. Commun. 1, 29 (2010).

    ADS  Article  Google Scholar 

  15. 15.

    Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).

    ADS  Article  Google Scholar 

  16. 16.

    Estakhri, N. M., Edwards, B. & Engheta, N. Inverse-designed metastructures that solve equations. Science 363, 1333–1338 (2019).

    ADS  MathSciNet  Article  Google Scholar 

  17. 17.

    Xu, X.-Y. et al. A scalable photonic computer solving the subset sum problem. Sci. Adv. 6, eaay5853 (2020).

    ADS  Article  Google Scholar 

  18. 18.

    Liu, W. et al. A fully reconfigurable photonic integrated signal processor. Nat. Photon. 10, 190–195 (2016).

    ADS  Article  Google Scholar 

  19. 19.

    Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).

    ADS  Article  Google Scholar 

  20. 20.

    Shainline, J. M., Buckley, S. M., Mirin, R. P. & Nam, S. W. Superconducting optoelectronic circuits for neuromorphic computing. Phys. Rev. Appl. 7, 034013 (2016).

    ADS  Article  Google Scholar 

  21. 21.

    Van der Sande, G., Brunner, D. & Soriano, M. C. Advances in photonic reservoir computing. Nanophotonics 6, 561–576 (2017).

    Article  Google Scholar 

  22. 22.

    Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).

    Google Scholar 

  23. 23.

    Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    ADS  Article  Google Scholar 

  24. 24.

    Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    ADS  Article  Google Scholar 

  25. 25.

    Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–447 (2017).

    ADS  Article  Google Scholar 

  26. 26.

    Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).

    ADS  Article  Google Scholar 

  27. 27.

    Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

    ADS  MathSciNet  Article  Google Scholar 

  28. 28.

    Tao, Y. et al. Fourier-space diffractive deep neural network. Phys. Rev. Lett. 123, 023901 (2019).

    ADS  Article  Google Scholar 

  29. 29.

    Zhou, T. et al. In situ optical backpropagation training of diffractive optical neural networks. Photonics Res. 8, 940–953 (2020).

    Article  Google Scholar 

  30. 30.

    Dou, H. et al. Residual D2NN: training diffractive deep neural networks via learnable light shortcuts. Opt. Lett. 45, 2688–2691 (2020).

    ADS  Article  Google Scholar 

  31. 31.

    Antonik, P., Marsal, N., Brunner, D. & Rontani, D. Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell. 1, 530–537 (2019).

    Article  Google Scholar 

  32. 32.

    Rafayelyan, M., Dong, J., Tan, Y., Krzakala, F. & Gigan, S. Large-scale optical reservoir computing for spatiotemporal chaotic systems prediction. Phys. Rev. X 10, 041037 (2020).

    Google Scholar 

  33. 33.

    Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).

    ADS  Article  Google Scholar 

  34. 34.

    Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).

    ADS  Article  Google Scholar 

  35. 35.

    Maktoobi, S. et al. Diffractive coupling for photonic networks: how big can we go? IEEE J. Sel. Top. Quantum Electron. 26, 1–8 (2019).

    Article  Google Scholar 

  36. 36.

    Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).

    ADS  Article  Google Scholar 

  37. 37.

    Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light: Sci. Appl. 8, 112 (2019).

    ADS  Article  Google Scholar 

  38. 38.

    Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).

    ADS  Article  Google Scholar 

  39. 39.

    LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Article  Google Scholar 

  40. 40.

    Xiao, H., Rasul, K., & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).

  41. 41.

    Blank, M., Gorelick, L., Shechtman, E., Irani, M. & Basri, R. Actions as space-time shapes. In Proc. Tenth International Conference on Computer Vision 1395–1402 (IEEE, 2005).

  42. 42.

    Schuldt, C., Laptev, I. & Caputo, B. Recognizing human actions: a local SVM approach. In Proc. 17th International Conference on Pattern Recognition 32–36 (IEEE, 2004).

  43. 43.

    Jhuang, H., Serre, T., Wolf, L. & Poggio, T. A biologically inspired system for action recognition. In Proc. 11th International Conference on Computer Vision 1–8 (IEEE, 2007).

  44. 44.

    Shu, N., Tang, Q. & Liu, H. A bio-inspired approach modeling spiking neural networks of visual cortex for human action recognition. In 2014 International Joint Conference on Neural Networks 3450–3457 (IEEE, 2014).

  45. 45.

    Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).

    Article  Google Scholar 

  46. 46.

    Wang, J. et al. Terabit free-space data transmission employing orbital angular momentum multiplexing. Nat. Photon. 6, 488–496 (2012).

    ADS  Article  Google Scholar 

  47. 47.

    Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  48. 48.

    Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).

    ADS  Article  Google Scholar 

  49. 49.

    Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Computer Vision – ECCV 2018 (eds Ferrari, V. et al.) 833–851 (Springer, 2018).

  50. 50.

    Larger, L. et al. High-speed photonic reservoir computing using a time-delay-based architecture: million words per second classification. Phys. Rev. X 7, 011015 (2017).

    Google Scholar 

Download references

Acknowledgements

We thank T. Zhu and T. Yan for assistance with the software for the prototype system. This work is supported by the Beijing Municipal Science and Technology Commission (No. Z181100003118014), the National Key Research and Development Program of China (No. 2020AAA0130000), the National Natural Science Foundation of China (No. 62088102 and No. 61860206003) and the Tsinghua University Initiative Scientific Research Program.

Author information

Affiliations

Authors

Contributions

Q.D., X.L. and L.F. initiated and supervised the project. X.L. and T.Z. conceived the research and method. X.L. designed the simulations and experiments. T.Z. conducted the experiments. T.Z. and Y.C. performed the simulations and processed the data. T.Z., J.W., Y.C. and H.X. built the experimental system. X.L., T.Z., J.W., H.W., H.X., J.F., Y.L. and L.F. analysed the results. X.L., T.Z., L.F. and Q.D. prepared the manuscript with input from all authors. All authors discussed the research.

Corresponding authors

Correspondence to Xing Lin or Lu Fang or Qionghai Dai.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Photonics thanks Nathan Youngblood and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–10, discussion and Tables 1–4.

Supplementary Video 1

Experimental demonstration by using a camera to capture the handwritten digits as the inputs for the system.

Supplementary Video 2

Experimental results of D2NN on the MNIST database.

Supplementary Video 3

Experimental results of D-NIN-1 on the MNIST database.

Supplementary Video 4

Experimental results of D-RNN on the Weizmann database.

Supplementary Video 5

Experimental results of D-RNN on the KTH database.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhou, T., Lin, X., Wu, J. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics 15, 367–373 (2021). https://doi.org/10.1038/s41566-021-00796-w

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing