Spike-based neuromorphic hardware holds promise for more energy-efficient implementations of deep neural networks (DNNs) than standard hardware such as GPUs. But this requires us to understand how DNNs can be emulated in an event-based sparse firing regime, as otherwise the energy advantage is lost. In particular, DNNs that solve sequence processing tasks typically employ long short-term memory units that are hard to emulate with few spikes. We show that a facet of many biological neurons, slow after-hyperpolarizing currents after each spike, provides an efficient solution. After-hyperpolarizing currents can easily be implemented in neuromorphic hardware that supports multi-compartment neuron models, such as Intel’s Loihi chip. Filter approximation theory explains why after-hyperpolarizing neurons can emulate the function of long short-term memory units. This yields a highly energy-efficient approach to time-series classification. Furthermore, it provides the basis for an energy-efficient implementation of an important class of large DNNs that extract relations between words and sentences in order to answer questions about the text.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Scientific Reports Open Access 17 August 2023
Nature Communications Open Access 06 April 2023
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
The Loihi source code is freely available from Github at https://github.com/intel-nrc-ecosystem/models/tree/master/nxsdk_modules_ncl/lsnn.
Davies, M. et al. Advancing neuromorphic computing with Loihi: a survey of results and outlook. Proc. IEEE 109, 911–934 (2021).
Benda, J. & Herz, A. V. M. A universal model for spike-frequency adaptation. Neur. Comput. 15, 2523–2564 (2003).
Gutkin, B. & Zeldenrust, F. Spike frequency adaptation. Scholarpedia 9, 30643, revision 14332 https://doi.org/10.4249/scholarpedia.30643 (2014).
Allen Institute Brain Atlas: Cell Feature Search (Allen Institute, accessed 3 August 2021); https://celltypes.brain-map.org/data
Davies, M. et al. Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82–99 (2018).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neur. Comput. 9, 1735–1780 (1997).
Shrestha, A. et al. A spike-based long short-term memory on a neurosynaptic processor. In 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) 631–637 (IEEE, 2017).
Akopyan, F. et al. Truenorth: design and tool flow of a 65 mW 1 million neuron programmable neurosynaptic chip. IEEE Trans. Comput. Aid. Des. Integr. Circ. Syst. 34, 1537–1557 (2015).
Lotfi Rezaabad, A. & Vishwanath, S. Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems 2020 3 (Association for Computing Machinery, 2020); https://doi.org/10.1145/3407197.3407211
Maass, W., Natschläger, T. & Markram, H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neur. Comput. 14, 2531–2560 (2002).
Furber, S. B., Galluppi, F., Temple, S. & Plana, L. A. The SpiNNaker project. Proc. IEEE 102, 652–665 (2014).
Bellec, G., Salaj, D., Subramoney, A., Legenstein, R. & Maass, W. Long short-term memory and learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems Vol. 31 (eds Bengio, S. et al.) 795–805 (Curran Associates, Inc., 2018).
LeCun, Y., Cortes, C. & Burges, C. MNIST Database of Handwritten Digits (ATT Labs, 2010); http://yann.lecun.com/exdb/mnist
Bellec, G., Kappel, D., Maass, W. & Legenstein, R. Deep rewiring: training very sparse deep networks. In International Conference on Learning Representations (2018).
Santoro, A. et al. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (Ed. Guyon, I., Von Luxburg, U., et al.) 30, 4967–4976 (NIPS, 2017).
Weston, J. et al. Towards AI-complete question answering: a set of prerequisite toy tasks. Preprint at https://arxiv.org/abs/1502.05698 (2015).
Bellec, G. et al. A solution to the learning dilemma for recurrent networks of spiking neurons. Nat. Commun. 11, 3625 (2020).
Scherr, F., Stöckl, C. & Maass, W. One-shot learning with spiking neural networks. Preprint at bioRxiv https://doi.org/10.1101/2020.06.17.156513 (2020).
Billeh, Y. N. et al. Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron 106, 388–403 (2020).
Zenke, F. & Vogels, T. P. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neur. Comput. 33, 899–925 (2021).
Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl. Acad. Sci. USA 113, 11441–11446 (2016).
Shrestha, S. B. & Orchard, G. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems Vol. 31 (eds Bengio, S. et al.) (Curran Associates, Inc., 2018).
Neftci, E. O., Mostafa, H. & Zenke, F. Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Process. Mag. 36, 51–63 (2019).
Zenke, F. & Ganguli, S. SuperSpike: supervised learning in multilayer spiking neural networks. Neur. Comput. 30, 1514–1541 (2018).
Zhu, X., Zhao, B., Ma, D. & Tang, H. An efficient learning algorithm for direct training deep spiking neural networks. IEEE Trans. Cogn. Dev. Syst. (2021).
Florey, D. (2020, December 9). Neuromorphic Software Overview. Neuromorphic Software Overview - INRC Public - Confluence. Retrieved April 26, 2022, from https://intel-ncl.atlassian.net/wiki/spaces/INRC/pages/524354/Neuromorphic+Software+Overview
This research/project was supported by the Human Brain Project (grant agreement number 785907 and 945539, both to W.M.) of the European Union and a grant from Intel (to W.M.). Special thanks go to G. Bellec and D. Salaj for their insightful comments and ideas when carrying out this work.
P.P. and A.W. are currently employed by Intel Labs, developers of the Loihi neuromorphic system. W.M. and A.R. are members of the Intel Neuromorphic Research Community and W.M. has received research funding from Intel for related work.
Peer review information
Nature Machine Intelligence thanks Jianhua Yang, Tara Hamilton and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rao, A., Plank, P., Wild, A. et al. A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware. Nat Mach Intell 4, 467–479 (2022). https://doi.org/10.1038/s42256-022-00480-w
This article is cited by
Scientific Reports (2023)
Nature Communications (2023)
Three-dimensional memristive Morris–Lecar model with magnetic induction effects and its FPGA implementation
Cognitive Neurodynamics (2023)
Cognitive Neurodynamics (2023)