Abstract
Convolutional neural networks, inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to provide greatly reduced parametric complexity and to enhance the accuracy of prediction. They are of great interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis1,2,3,4,5,6,7. Optical neural networks offer the promise of dramatically accelerating computing speed using the broad optical bandwidths available. Here we demonstrate a universal optical vector convolutional accelerator operating at more than ten TOPS (trillions (1012) of operations per second, or tera-ops per second), generating convolutions of images with 250,000 pixels—sufficiently large for facial image recognition. We use the same hardware to sequentially form an optical convolutional neural network with ten output neurons, achieving successful recognition of handwritten digit images at 88 per cent accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. This approach is scalable and trainable to much more complex networks for demanding applications such as autonomous vehicles and real-time video recognition.
This is a preview of subscription content, access via your institution
Relevant articles
Open Access articles citing this article.
-
Recent progress on coherent computation based on quantum squeezing
AAPPS Bulletin Open Access 01 March 2023
-
kHz-precision wavemeter based on reconfigurable microsoliton
Nature Communications Open Access 12 January 2023
-
Dispersion-less Kerr solitons in spectrally confined optical cavities
Light: Science & Applications Open Access 09 January 2023
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 per month
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Rent or buy this article
Get just this article for as long as you need it
$39.95
Prices may be subject to local taxes which are calculated during checkout
Data availability
The authors declare that the data supporting the findings of this study are available within the paper and its supplementary information files.
Code availability
The authors declare that the algorithm of the demonstrated neural network supporting the findings of this study is available within the paper and its supplementary information files.
References
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Schalkoff, R. J. Pattern recognition. In Wiley Encyclopedia of Computer Science and Engineering (ed. Wah, B. W.) https://doi.org/10.1002/9780470050118.ecse302 (Wiley, 2007).
Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017).
Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).
Lawrence, S., Giles, C. L., Tsoi, A. C. & Back, A. D. Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8, 98–113 (1997).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).
Larger, L. et al. High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification. Phys. Rev. X 7, 011015 (2017).
Peng, H.-T., Nahmias, M. A., de Lima, T. F., Tait, A. N. & Shastri, B. J. Neuromorphic photonic integrated circuits. IEEE J. Sel. Top. Quantum Electron. 24, 6101715 (2018).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 558, 60–67 (2018).
Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl Acad. Sci. USA 113, 11441–11446 (2016).
Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016).
Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).
Appeltant, L. et al. Information processing using a single dynamical node as complex system. Nat. Commun. 2, 468 (2011).
Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).
Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5, 3541 (2014).
Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
Tait, A. N., Chang, J., Shastri, B. J., Nahmias, M. A. & Prucnal, P. R. Demonstration of WDM weighted addition for principal component analysis. Opt. Express 23, 12758–12765 (2015).
Xu, X. et al. Photonic perceptron based on a Kerr microcomb for high‐speed, scalable, optical neural networks. Laser Photon. Rev. 14, https://doi.org/10.1002/lpor.202000070 (2020).
Pasquazi, A. et al. Micro-combs: a novel generation of optical sources. Phys. Rep. 729, 1–81 (2018).
Moss, D. J., Morandotti, R., Gaeta, A. L. & Lipson, M. New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics. Nat. Photon. 7, 597–607 (2013).
Kippenberg, T. J., Gaeta, A. L., Lipson, M. & Gorodetsky, M. L. Dissipative Kerr solitons in optical microresonators. Science 361, eaan8083 (2018).
Savchenkov, A. A. et al. Tunable optical frequency comb with a crystalline whispering gallery mode resonator. Phys. Rev. Lett. 101, 093902 (2008).
Spencer, D. T. et al. An optical-frequency synthesizer using integrated photonics. Nature 557, 81–85 (2018).
Marin-Palomo, P. et al. Microresonator-based solitons for massively parallel coherent optical communications. Nature 546, 274–279 (2017).
Kues, M. et al. Quantum optical microcombs. Nat. Photon. 13, 170–179 (2019).
Cole, D. C., Lamb, E. S., Del’Haye, P., Diddams, S. A. & Papp, S. B. Soliton crystals in Kerr resonators. Nat. Photon. 11, 671–676 (2017).
Stern, B., Ji, X., Okawachi, Y., Gaeta, A. L. & Lipson, M. Battery-operated integrated frequency comb generator. Nature 562, 401–405 (2018).
Wu, J. et al. RF photonics: an optical microcombs’ perspective. IEEE J. Sel. Top. Quant. Electron. 24, 6101020 (2018).
LeCun, Y., Cortes, C. & Borges, C. J. C. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/
Metcalf, A. J. et al. Integrated line-by-line optical pulse shaper for high-fidelity and rapidly reconfigurable RF-filtering. Opt. Express 24, 23925–23940 (2016).
NVIDIA Corporation. Comparison of Convolution Methods for GPUs. http://ska-sdp.org/publications/released-sdp-memos-2 (2018).
Sahin, E., Ooi, K., Png, C. & Tan, D. Large, scalable dispersion engineering using cladding-modulated Bragg gratings on a silicon chip. Appl. Phys. Lett. 110, 161113 (2017).
Roeloffzen, C. G. H. et al. Low-loss Si3N4 TriPleX optical waveguides: technology and applications overview. IEEE J. Sel. Top. Quantum Electron. 24, 4400321 (2018).
Wang, C. et al. Integrated lithium niobate electro-optic modulators operating at CMOS-compatible voltages. Nature 562, 101–104 (2018).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Capper, D. et al. DNA methylation-based classification of central nervous system tumours. Nature 555, 469–474 (2018).
Acknowledgements
This work was supported by the Australian Research Council Discovery Projects Program (grant numbers DP150104327, DP190102773 and DP190101576). R.M. acknowledges support by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Strategic, Discovery and Acceleration Grants Schemes, by the MESI PSR-SIIRI Initiative in Quebec, and by the Canada Research Chair Program. B.E.L. was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (grant number XDB24030000). D.G.H. was supported in part by the Australian Research Council (grant number FT104101104). R.M. is affiliated with the Institute of Fundamental and Frontier Sciences (China) as an adjoint faculty member.
Author information
Authors and Affiliations
Contributions
X.X. conceived the idea and designed the project. X.X. and M.T. performed the experiments. X.X. analysed the data, and performed the numerical simulations and the offline training. S.T.C. and B.E.L. designed and fabricated the integrated devices. B.C., J.W., A.B., T.G.N., R.M. and A.M. contributed to the development of the experiment and to the data analysis. X.X. and D.J.M. wrote the manuscript. D.J.M. supervised the research.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Nature thanks Sylvain Gigan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
Extended Data Fig. 1 VCA, for processing one-dimensional data.
It consists of the experimental setup (right panel), the optical and electronic control and signal flow (left panel). ADC, analogue-to-digital converter. 1D, one-dimensional.
Extended Data Fig. 2 Generation of soliton crystal microcombs.
a, Schematic diagram of the soliton crystal microcomb, generated by pumping an on-chip high-Q (quality factor >1 million) nonlinear micro-ring resonator with a continuous-wave laser. b, Image of the MRR (upper inset) and a scanning electron microscope image of the MRR’s waveguide cross-section (lower inset). c, Measured dispersion Dint of the MRR showing the mode crossing at about 1,552 nm. d, Measured soliton crystal step of the intra-cavity power. e, Optical spectrum of the microcomb when sweeping the pump wavelength. f, Optical spectrum of the generated coherent microcomb at different pump detunings at a fixed power. FSR, free spectral range.
Extended Data Fig. 3 The architecture of the optical CNN.
The architecture includes a convolutional layer, a pooling layer and a fully connected layer.
Extended Data Fig. 4 Fully connected layers.
Architecture and experimental results. The left panel depicts the experimental setup, similar to the convolutional layer. The right panel shows the experimental results for one output neuron, including the shaped comb spectrum (top); the pooled feature maps of the digit 3 and the corresponding input electrical waveform (the grey and red lines illustrate the ideal and experimentally generated waveforms, respectively; middle); and the output waveform of the neuron and sampled intensities (bottom). Conv layer, convolutional layer. CW pump, continuous-wave pump laser.
Supplementary information
Supplementary Information
Information on the operation principle of the photonic convolution accelerator, matrix flattening, network training and digital processing, additional experimental results, a performance comparison with other results in the literature, scaling the networks in performance and speed, and a theoretical evaluation of a scaled network, including Supplementary Figures S1–S30, Tables S1 to S2, and Supplementary References.
Supplementary Information
A Supplementary Presentation. Digital neuromorphic processors typically process one dimensional data streams and so to process matrices, the matrix must first be converted to a vector – effectively “flattened”. How this is done will be determined by the size of the kernel being used to process the data, and this in turn will result in a reduction of the matrix processing speed relative to the vector processing speed – effectively a speed “overhead”. This is a fundamental and generic issue that applies to any processor. This presentation graphically illustrates this issue and includes a presentation of methods designed to eliminate this overhead.
Rights and permissions
About this article
Cite this article
Xu, X., Tan, M., Corcoran, B. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021). https://doi.org/10.1038/s41586-020-03063-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41586-020-03063-0
This article is cited by
-
Prospects and applications of on-chip lasers
eLight (2023)
-
Image sensing with multilayer nonlinear optical neural networks
Nature Photonics (2023)
-
kHz-precision wavemeter based on reconfigurable microsoliton
Nature Communications (2023)
-
Graphene oxide for photonics, electronics and optoelectronics
Nature Reviews Chemistry (2023)
-
Microcomb-based integrated photonic processing unit
Nature Communications (2023)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
