Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

11 TOPS photonic convolutional accelerator for optical neural networks

Abstract

Convolutional neural networks, inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to provide greatly reduced parametric complexity and to enhance the accuracy of prediction. They are of great interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis1,2,3,4,5,6,7. Optical neural networks offer the promise of dramatically accelerating computing speed using the broad optical bandwidths available. Here we demonstrate a universal optical vector convolutional accelerator operating at more than ten TOPS (trillions (1012) of operations per second, or tera-ops per second), generating convolutions of images with 250,000 pixels—sufficiently large for facial image recognition. We use the same hardware to sequentially form an optical convolutional neural network with ten output neurons, achieving successful recognition of handwritten digit images at 88 per cent accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. This approach is scalable and trainable to much more complex networks for demanding applications such as autonomous vehicles and real-time video recognition.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Operation principle of the TOPS photonic CA.
Fig. 2: Image processing.
Fig. 3: Experimental results of the image processing.
Fig. 4: Experimental schematic of the optical CNN.
Fig. 5: Convolutional layer.
Fig. 6: Experimental and theoretically calculated results for image recognition.

Data availability

The authors declare that the data supporting the findings of this study are available within the paper and its supplementary information files.

Code availability

The authors declare that the algorithm of the demonstrated neural network supporting the findings of this study is available within the paper and its supplementary information files.

References

  1. 1.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    ADS  CAS  Article  Google Scholar 

  2. 2.

    Schalkoff, R. J. Pattern recognition. In Wiley Encyclopedia of Computer Science and Engineering (ed. Wah, B. W.) https://doi.org/10.1002/9780470050118.ecse302 (Wiley, 2007).

  3. 3.

    Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

    ADS  CAS  Article  Google Scholar 

  4. 4.

    Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    ADS  CAS  Article  Google Scholar 

  5. 5.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017).

    Article  Google Scholar 

  6. 6.

    Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).

    ADS  CAS  Article  Google Scholar 

  7. 7.

    Lawrence, S., Giles, C. L., Tsoi, A. C. & Back, A. D. Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8, 98–113 (1997).

    CAS  Article  Google Scholar 

  8. 8.

    Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).

    ADS  CAS  Article  Google Scholar 

  9. 9.

    Larger, L. et al. High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification. Phys. Rev. X 7, 011015 (2017).

    Google Scholar 

  10. 10.

    Peng, H.-T., Nahmias, M. A., de Lima, T. F., Tait, A. N. & Shastri, B. J. Neuromorphic photonic integrated circuits. IEEE J. Sel. Top. Quantum Electron. 24, 6101715 (2018).

    Google Scholar 

  11. 11.

    Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

    ADS  MathSciNet  CAS  Article  Google Scholar 

  12. 12.

    Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    ADS  CAS  Article  Google Scholar 

  13. 13.

    Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 558, 60–67 (2018).

    ADS  CAS  Article  Google Scholar 

  14. 14.

    Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl Acad. Sci. USA 113, 11441–11446 (2016).

    CAS  Article  Google Scholar 

  15. 15.

    Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016).

    ADS  Article  Google Scholar 

  16. 16.

    Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).

    ADS  CAS  Article  Google Scholar 

  17. 17.

    Appeltant, L. et al. Information processing using a single dynamical node as complex system. Nat. Commun. 2, 468 (2011).

    ADS  CAS  Article  Google Scholar 

  18. 18.

    Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).

    ADS  Article  Google Scholar 

  19. 19.

    Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5, 3541 (2014).

    ADS  Article  Google Scholar 

  20. 20.

    Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).

    ADS  Article  Google Scholar 

  21. 21.

    Tait, A. N., Chang, J., Shastri, B. J., Nahmias, M. A. & Prucnal, P. R. Demonstration of WDM weighted addition for principal component analysis. Opt. Express 23, 12758–12765 (2015).

    ADS  Article  Google Scholar 

  22. 22.

    Xu, X. et al. Photonic perceptron based on a Kerr microcomb for high‐speed, scalable, optical neural networks. Laser Photon. Rev. 14, https://doi.org/10.1002/lpor.202000070 (2020).

  23. 23.

    Pasquazi, A. et al. Micro-combs: a novel generation of optical sources. Phys. Rep. 729, 1–81 (2018).

    ADS  MathSciNet  CAS  Article  Google Scholar 

  24. 24.

    Moss, D. J., Morandotti, R., Gaeta, A. L. & Lipson, M. New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics. Nat. Photon. 7, 597–607 (2013).

    ADS  CAS  Article  Google Scholar 

  25. 25.

    Kippenberg, T. J., Gaeta, A. L., Lipson, M. & Gorodetsky, M. L. Dissipative Kerr solitons in optical microresonators. Science 361, eaan8083 (2018).

    Article  Google Scholar 

  26. 26.

    Savchenkov, A. A. et al. Tunable optical frequency comb with a crystalline whispering gallery mode resonator. Phys. Rev. Lett. 101, 093902 (2008).

    ADS  Article  Google Scholar 

  27. 27.

    Spencer, D. T. et al. An optical-frequency synthesizer using integrated photonics. Nature 557, 81–85 (2018).

    ADS  CAS  Article  Google Scholar 

  28. 28.

    Marin-Palomo, P. et al. Microresonator-based solitons for massively parallel coherent optical communications. Nature 546, 274–279 (2017).

    ADS  CAS  Article  Google Scholar 

  29. 29.

    Kues, M. et al. Quantum optical microcombs. Nat. Photon. 13, 170–179 (2019).

    ADS  CAS  Article  Google Scholar 

  30. 30.

    Cole, D. C., Lamb, E. S., Del’Haye, P., Diddams, S. A. & Papp, S. B. Soliton crystals in Kerr resonators. Nat. Photon. 11, 671–676 (2017).

    ADS  CAS  Article  Google Scholar 

  31. 31.

    Stern, B., Ji, X., Okawachi, Y., Gaeta, A. L. & Lipson, M. Battery-operated integrated frequency comb generator. Nature 562, 401–405 (2018).

    ADS  CAS  Article  Google Scholar 

  32. 32.

    Wu, J. et al. RF photonics: an optical microcombs’ perspective. IEEE J. Sel. Top. Quant. Electron. 24, 6101020 (2018).

    ADS  Google Scholar 

  33. 33.

    LeCun, Y., Cortes, C. & Borges, C. J. C. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/

  34. 34.

    Metcalf, A. J. et al. Integrated line-by-line optical pulse shaper for high-fidelity and rapidly reconfigurable RF-filtering. Opt. Express 24, 23925–23940 (2016).

    ADS  CAS  Article  Google Scholar 

  35. 35.

    NVIDIA Corporation. Comparison of Convolution Methods for GPUs. http://ska-sdp.org/publications/released-sdp-memos-2 (2018).

  36. 36.

    Sahin, E., Ooi, K., Png, C. & Tan, D. Large, scalable dispersion engineering using cladding-modulated Bragg gratings on a silicon chip. Appl. Phys. Lett. 110, 161113 (2017).

    ADS  Article  Google Scholar 

  37. 37.

    Roeloffzen, C. G. H. et al. Low-loss Si3N4 TriPleX optical waveguides: technology and applications overview. IEEE J. Sel. Top. Quantum Electron. 24, 4400321 (2018).

    Article  Google Scholar 

  38. 38.

    Wang, C. et al. Integrated lithium niobate electro-optic modulators operating at CMOS-compatible voltages. Nature 562, 101–104 (2018).

    ADS  CAS  Article  Google Scholar 

  39. 39.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    ADS  CAS  Article  Google Scholar 

  40. 40.

    Capper, D. et al. DNA methylation-based classification of central nervous system tumours. Nature 555, 469–474 (2018).

    ADS  CAS  Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Australian Research Council Discovery Projects Program (grant numbers DP150104327, DP190102773 and DP190101576). R.M. acknowledges support by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Strategic, Discovery and Acceleration Grants Schemes, by the MESI PSR-SIIRI Initiative in Quebec, and by the Canada Research Chair Program. B.E.L. was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (grant number XDB24030000). D.G.H. was supported in part by the Australian Research Council (grant number FT104101104). R.M. is affiliated with the Institute of Fundamental and Frontier Sciences (China) as an adjoint faculty member.

Author information

Affiliations

Authors

Contributions

X.X. conceived the idea and designed the project. X.X. and M.T. performed the experiments. X.X. analysed the data, and performed the numerical simulations and the offline training. S.T.C. and B.E.L. designed and fabricated the integrated devices. B.C., J.W., A.B., T.G.N., R.M. and A.M. contributed to the development of the experiment and to the data analysis. X.X. and D.J.M. wrote the manuscript. D.J.M. supervised the research.

Corresponding author

Correspondence to David J. Moss.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature thanks Sylvain Gigan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 VCA, for processing one-dimensional data.

It consists of the experimental setup (right panel), the optical and electronic control and signal flow (left panel). ADC, analogue-to-digital converter. 1D, one-dimensional.

Extended Data Fig. 2 Generation of soliton crystal microcombs.

a, Schematic diagram of the soliton crystal microcomb, generated by pumping an on-chip high-Q (quality factor >1 million) nonlinear micro-ring resonator with a continuous-wave laser. b, Image of the MRR (upper inset) and a scanning electron microscope image of the MRR’s waveguide cross-section (lower inset). c, Measured dispersion Dint of the MRR showing the mode crossing at about 1,552 nm. d, Measured soliton crystal step of the intra-cavity power. e, Optical spectrum of the microcomb when sweeping the pump wavelength. f, Optical spectrum of the generated coherent microcomb at different pump detunings at a fixed power. FSR, free spectral range.

Extended Data Fig. 3 The architecture of the optical CNN.

The architecture includes a convolutional layer, a pooling layer and a fully connected layer.

Extended Data Fig. 4 Fully connected layers.

Architecture and experimental results. The left panel depicts the experimental setup, similar to the convolutional layer. The right panel shows the experimental results for one output neuron, including the shaped comb spectrum (top); the pooled feature maps of the digit 3 and the corresponding input electrical waveform (the grey and red lines illustrate the ideal and experimentally generated waveforms, respectively; middle); and the output waveform of the neuron and sampled intensities (bottom). Conv layer, convolutional layer. CW pump, continuous-wave pump laser.

Supplementary information

Supplementary Information

Information on the operation principle of the photonic convolution accelerator, matrix flattening, network training and digital processing, additional experimental results, a performance comparison with other results in the literature, scaling the networks in performance and speed, and a theoretical evaluation of a scaled network, including Supplementary Figures S1–S30, Tables S1 to S2, and Supplementary References.

Supplementary Information

A Supplementary Presentation. Digital neuromorphic processors typically process one dimensional data streams and so to process matrices, the matrix must first be converted to a vector – effectively “flattened”. How this is done will be determined by the size of the kernel being used to process the data, and this in turn will result in a reduction of the matrix processing speed relative to the vector processing speed – effectively a speed “overhead”. This is a fundamental and generic issue that applies to any processor. This presentation graphically illustrates this issue and includes a presentation of methods designed to eliminate this overhead.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xu, X., Tan, M., Corcoran, B. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021). https://doi.org/10.1038/s41586-020-03063-0

Download citation

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing