Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware

Nature Machine Intelligence volume 4, pages 467–479 (2022)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Spike-based neuromorphic hardware holds promise for more energy-efficient implementations of deep neural networks (DNNs) than standard hardware such as GPUs. But this requires us to understand how DNNs can be emulated in an event-based sparse firing regime, as otherwise the energy advantage is lost. In particular, DNNs that solve sequence processing tasks typically employ long short-term memory units that are hard to emulate with few spikes. We show that a facet of many biological neurons, slow after-hyperpolarizing currents after each spike, provides an efficient solution. After-hyperpolarizing currents can easily be implemented in neuromorphic hardware that supports multi-compartment neuron models, such as Intel’s Loihi chip. Filter approximation theory explains why after-hyperpolarizing neurons can emulate the function of long short-term memory units. This yields a highly energy-efficient approach to time-series classification. Furthermore, it provides the basis for an energy-efficient implementation of an important class of large DNNs that extract relations between words and sentences in order to answer questions about the text.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematics and dynamics of a two-compartment LIF neuron model with AHP currents.
Fig. 2: PSPR, gradient transmission and energy consumption of AHP-SNNs for the sMNIST task.
Fig. 3: Spiking RelNet implementation with very sparse firing.
Fig. 4: Voltage regularization and its capability to enforce a sparse firing regime in combination with spike rate regularization.
Fig. 5: Spiking RelNet placement and optimization on Loihi.

Similar content being viewed by others

Data availability

The MNIST dataset13 is freely available at http://yann.lecun.com/exdb/mnist. The bAbI dataset16 is freely available at https://research.fb.com/downloads/babi.

Code availability

The Loihi source code is freely available from Github at https://github.com/intel-nrc-ecosystem/models/tree/master/nxsdk_modules_ncl/lsnn.

References

  1. Davies, M. et al. Advancing neuromorphic computing with Loihi: a survey of results and outlook. Proc. IEEE 109, 911–934 (2021).

    Article  Google Scholar 

  2. Benda, J. & Herz, A. V. M. A universal model for spike-frequency adaptation. Neur. Comput. 15, 2523–2564 (2003).

    Article  Google Scholar 

  3. Gutkin, B. & Zeldenrust, F. Spike frequency adaptation. Scholarpedia 9, 30643, revision 14332 https://doi.org/10.4249/scholarpedia.30643 (2014).

  4. Allen Institute Brain Atlas: Cell Feature Search (Allen Institute, accessed 3 August 2021); https://celltypes.brain-map.org/data

  5. Davies, M. et al. Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82–99 (2018).

    Article  Google Scholar 

  6. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neur. Comput. 9, 1735–1780 (1997).

    Article  Google Scholar 

  7. Shrestha, A. et al. A spike-based long short-term memory on a neurosynaptic processor. In 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) 631–637 (IEEE, 2017).

  8. Akopyan, F. et al. Truenorth: design and tool flow of a 65 mW 1 million neuron programmable neurosynaptic chip. IEEE Trans. Comput. Aid. Des. Integr. Circ. Syst. 34, 1537–1557 (2015).

    Article  Google Scholar 

  9. Lotfi Rezaabad, A. & Vishwanath, S. Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems 2020 3 (Association for Computing Machinery, 2020); https://doi.org/10.1145/3407197.3407211

  10. Maass, W., Natschläger, T. & Markram, H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neur. Comput. 14, 2531–2560 (2002).

    Article  Google Scholar 

  11. Furber, S. B., Galluppi, F., Temple, S. & Plana, L. A. The SpiNNaker project. Proc. IEEE 102, 652–665 (2014).

    Article  Google Scholar 

  12. Bellec, G., Salaj, D., Subramoney, A., Legenstein, R. & Maass, W. Long short-term memory and learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems Vol. 31 (eds Bengio, S. et al.) 795–805 (Curran Associates, Inc., 2018).

  13. LeCun, Y., Cortes, C. & Burges, C. MNIST Database of Handwritten Digits (ATT Labs, 2010); http://yann.lecun.com/exdb/mnist

  14. Bellec, G., Kappel, D., Maass, W. & Legenstein, R. Deep rewiring: training very sparse deep networks. In International Conference on Learning Representations (2018).

  15. Santoro, A. et al. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (Ed. Guyon, I., Von Luxburg, U., et al.) 30, 4967–4976 (NIPS, 2017).

  16. Weston, J. et al. Towards AI-complete question answering: a set of prerequisite toy tasks. Preprint at https://arxiv.org/abs/1502.05698 (2015).

  17. Bellec, G. et al. A solution to the learning dilemma for recurrent networks of spiking neurons. Nat. Commun. 11, 3625 (2020).

    Article  Google Scholar 

  18. Scherr, F., Stöckl, C. & Maass, W. One-shot learning with spiking neural networks. Preprint at bioRxiv https://doi.org/10.1101/2020.06.17.156513 (2020).

  19. Billeh, Y. N. et al. Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron 106, 388–403 (2020).

    Article  Google Scholar 

  20. Zenke, F. & Vogels, T. P. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neur. Comput. 33, 899–925 (2021).

    Article  MathSciNet  Google Scholar 

  21. Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl. Acad. Sci. USA 113, 11441–11446 (2016).

    Article  Google Scholar 

  22. Shrestha, S. B. & Orchard, G. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems Vol. 31 (eds Bengio, S. et al.) (Curran Associates, Inc., 2018).

  23. Neftci, E. O., Mostafa, H. & Zenke, F. Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Process. Mag. 36, 51–63 (2019).

    Article  Google Scholar 

  24. Zenke, F. & Ganguli, S. SuperSpike: supervised learning in multilayer spiking neural networks. Neur. Comput. 30, 1514–1541 (2018).

    Article  MathSciNet  Google Scholar 

  25. Zhu, X., Zhao, B., Ma, D. & Tang, H. An efficient learning algorithm for direct training deep spiking neural networks. IEEE Trans. Cogn. Dev. Syst. (2021).

  26. Florey, D. (2020, December 9). Neuromorphic Software Overview. Neuromorphic Software Overview - INRC Public - Confluence. Retrieved April 26, 2022, from https://intel-ncl.atlassian.net/wiki/spaces/INRC/pages/524354/Neuromorphic+Software+Overview

Download references

Acknowledgements

This research/project was supported by the Human Brain Project (grant agreement number 785907 and 945539, both to W.M.) of the European Union and a grant from Intel (to W.M.). Special thanks go to G. Bellec and D. Salaj for their insightful comments and ideas when carrying out this work.

Author information

Author notes

  1. These authors contributed equally: Arjun Rao and Philipp Plank.

Authors and Affiliations

  1. Institute of Theoretical Computer Science, Graz University of Technology, Graz, Austria

    Arjun Rao, Philipp Plank & Wolfgang Maass

  2. Intel Labs, Intel Corporation, Neubiberg, Germany

    Philipp Plank

  3. Intel Labs, Intel Corporation, Hillsboro, OR, USA

    Andreas Wild

Authors
  1. Arjun Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philipp Plank
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andreas Wild
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wolfgang Maass
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.R., P.P. and W.M. contributed to the design and planning of the experiments. A.R. and P.P. carried out the experiments. A.R., P.P., A.W. and W.M. participated in the analysis of the experimental data. A.R., P.P., A.W. and W.M. wrote the manuscript.

Corresponding author

Correspondence to Wolfgang Maass.

Ethics declarations

Competing interests

P.P. and A.W. are currently employed by Intel Labs, developers of the Loihi neuromorphic system. W.M. and A.R. are members of the Intel Neuromorphic Research Community and W.M. has received research funding from Intel for related work.

Peer review

Peer review information

Nature Machine Intelligence thanks Jianhua Yang, Tara Hamilton and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information

Supplementary Figs. 1 and 2, Tables 1–5 and details of the benchmarking, tasks and parameters.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rao, A., Plank, P., Wild, A. et al. A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware. Nat Mach Intell 4, 467–479 (2022). https://doi.org/10.1038/s42256-022-00480-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00480-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing