In-memory hyperdimensional computing

Abstract

Hyperdimensional computing is an emerging computational framework that takes inspiration from attributes of neuronal circuits including hyperdimensionality, fully distributed holographic representation and (pseudo)randomness. When employed for machine learning tasks, such as learning and classification, the framework involves manipulation and comparison of large patterns within memory. A key attribute of hyperdimensional computing is its robustness to the imperfections associated with the computational substrates on which it is implemented. It is therefore particularly amenable to emerging non-von Neumann approaches such as in-memory computing, where the physical attributes of nanoscale memristive devices are exploited to perform computation. Here, we report a complete in-memory hyperdimensional computing system in which all operations are implemented on two memristive crossbar engines together with peripheral digital complementary metal–oxide–semiconductor (CMOS) circuits. Our approach can achieve a near-optimum trade-off between design complexity and classification accuracy based on three prototypical hyperdimensional computing-related learning tasks: language classification, news classification and hand gesture recognition from electromyography signals. Experiments using 760,000 phase-change memory devices performing analog in-memory computing achieve comparable accuracies to software implementations.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: The concept of in-memory HDC.
Fig. 2: AM search.
Fig. 3: In-memory n-gram encoding based on 2-minterm.
Fig. 4: The complete in-memory HDC system.

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.

References

  1. 1.

    Kanerva, P. Sparse Distributed Memory (MIT Press, 1988).

  2. 2.

    Kanerva, P. Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1, 139–159 (2009).

    Article  Google Scholar 

  3. 3.

    Kanerva, P., Kristoferson, J. & Holst, A. Random indexing of text samples for latent semantic analysis. In Proceedings of the Annual Meeting of the Cognitive Science Society Vol. 22 (Cognitive Science Society, 2000).

  4. 4.

    Rahimi, A., Kanerva, P., Benini, L. & Rabaey, J. M. Efficient biosignal processing using hyperdimensional computing: network templates for combined learning and classification of ExG signals. Proc. IEEE 107, 123–143 (2019).

    Article  Google Scholar 

  5. 5.

    Burrello, A., Cavigelli, L., Schindler, K., Benini, L. & Rahimi, A. Laelaps: an energy-efficient seizure detection algorithm from long-term human iEEG recordings without false alarms. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE) 752–757 (IEEE, 2019).

  6. 6.

    Räsänen, O. J. & Saarinen, J. P. Sequence prediction with sparse distributed hyperdimensional coding applied to the analysis of mobile phone use patterns. IEEE Trans. Neural Netw. Learn. Syst. 27, 1878–1889 (2015).

    MathSciNet  Article  Google Scholar 

  7. 7.

    Kleyko, D. & Osipov, E. Brain-like classifier of temporal patterns. In Proceedings of the International Conference on Computer and Information Sciences (ICCOINS) 1–6 (IEEE, 2014).

  8. 8.

    Kleyko, D., Osipov, E., Papakonstantinou, N. & Vyatkin, V. Hyperdimensional computing in industrial systems: the use-case of distributed fault isolation in a power plant. IEEE Access 6, 30766–30777 (2018).

    Article  Google Scholar 

  9. 9.

    Chang, E., Rahimi, A., Benini, L. & Wu, A. A. Hyperdimensional computing-based multimodality emotion recognition with physiological signals. In Proceedings of the IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) 137–141 (IEEE, 2019).

  10. 10.

    Mitrokhin, A., Sutor, P., Fermüller, C. & Aloimonos, Y. Learning sensorimotor control with neuromorphic sensors: toward hyperdimensional active perception. Sci. Robot. 4, eaaw6736 (2019).

    Article  Google Scholar 

  11. 11.

    Montagna, F., Rahimi, A., Benatti, S., Rossi, D. & Benini, L. PULP-HD: accelerating brain-inspired high-dimensional computing on a parallel ultra-low power platform. In Proceedings of the 55th Annual Design Automation Conference DAC 2018, 111:1–111:6 (ACM, 2018).

  12. 12.

    Emruli, B., Gayler, R. W. & Sandin, F. Analogical mapping and inference with binary spatter codes and sparse distributed memory. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) 1–8 (IEEE, 2013).

  13. 13.

    Kleyko, D., Osipov, E., Gayler, R. W., Khan, A. I. & Dyer, A. G. Imitation of honey bees’ concept learning processes using vector symbolic architectures. Biol. Inspired Cogn. Architectures 14, 57–72 (2015).

    Article  Google Scholar 

  14. 14.

    Slipchenko, S. V. & Rachkovskij, D. A. Analogical mapping using similarity of binary distributed representations. Inf. Theories Appl. 16, 269–290 (2009).

    Google Scholar 

  15. 15.

    Bandaragoda, T. et al. Trajectory clustering of road traffic in urban environments using incremental machine learning in combination with hyperdimensional computing. In Proceedings of the IEEE Intelligent Transportation Systems Conference (ITSC) 1664–1670 (IEEE, 2019).

  16. 16.

    Osipov, E., Kleyko, D. & Legalov, A. Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing. In Proceedings of the Annual Conference of the IEEE Industrial Electronics Society 3276–3281 (IEEE, 2017).

  17. 17.

    Kleyko, D., Frady, E. P. & Osipov, E. Integer echo state networks: hyperdimensional reservoir computing. Preprint at https://arxiv.org/pdf/1706.00280.pdf (2017).

  18. 18.

    Rahimi, A. et al. High-dimensional computing as a nanoscalable paradigm. IEEE Trans. Circuits Syst. I Regular Papers 64, 2508–2521 (2017).

    Article  Google Scholar 

  19. 19.

    Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013).

    Article  Google Scholar 

  20. 20.

    Sebastian, A. et al. Temporal correlation detection using computational phase-change memory. Nat. Commun. 8, 1115 (2017).

    Article  Google Scholar 

  21. 21.

    Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).

    Google Scholar 

  22. 22.

    Ielmini, D. & Wong, H.-S. P. In-memory computing with resistive switching devices. Nat. Electron. 1, 333–343 (2018).

    Article  Google Scholar 

  23. 23.

    Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R. & Eleftheriou, E. Memory devices and applications for in-memory computing. Nat. Nanotechnol. https://doi.org/10.1038/s41565-020-0655-z (2020).

  24. 24.

    Li, H. et al. Hyperdimensional computing with 3D VRRAM in-memory kernels: device-architecture co-design for energy-efficient, error-resilient language recognition. In Proceedings of the IEEE International Electron Devices Meeting (IEDM) 16.1.1–16.1.4 (IEEE, 2016).

  25. 25.

    Li, H., Wu, T. F., Mitra, S. & Wong, H. S. P. Device-architecture co-design for hyperdimensional computing with 3D vertical resistive switching random access memory (3D VRRAM). In Proceedings of the International Symposium on VLSI Technology, Systems and Application (VLSI-TSA) 1–2 (IEEE, 2017).

  26. 26.

    Wu, T. F. et al. Brain-inspired computing exploiting carbon nanotube FETs and resistive RAM: hyperdimensional computing case study. In Proceedings of the International Solid State Circuits Conference (ISSCC) 492–494 (IEEE, 2018).

  27. 27.

    Kanerva, P. Binary spatter-coding of ordered k-tuples. In Proceedings of the International Conference on Artificial Neural Networks (ICANN), Vol. 1112, 869–873 (Lecture Notes in Computer Science, Springer, 1996).

  28. 28.

    Joshi, A., Halseth, J. T. & Kanerva, P. Language geometry using random indexing. In Proceedings of the International Symposium on Quantum Interaction 265–274 (Springer, 2016).

  29. 29.

    Chua, L. Resistance switching memories are memristors. Appl. Phys. A 102, 765–783 (2011).

    Article  Google Scholar 

  30. 30.

    Wong, H.-S. P. & Salahuddin, S. Memory leads the way to better computing. Nat. Nanotechnol. 10, 191–194 (2015).

    Article  Google Scholar 

  31. 31.

    Borghetti, J. et al. ‘Memristive’ switches enable ‘stateful’ logic operations via material implication. Nature 464, 873–876 (2010).

    Article  Google Scholar 

  32. 32.

    Kvatinsky, S. et al. Magic—memristor-aided logic. IEEE Trans. Circuits Syst II Express Briefs 61, 895–899 (2014).

    Article  Google Scholar 

  33. 33.

    Shen, W. et al. Stateful logic operations in one-transistor-one-resistor resistive random access memory array. Electron Device Lett. 40, 1538–1541 (2019).

    Article  Google Scholar 

  34. 34.

    Wong, H.-S. P. et al. Phase change memory. Proc. IEEE 98, 2201–2227 (2010).

    Article  Google Scholar 

  35. 35.

    Burr, G. W. et al. Recent progress in phase-change memory technology. IEEE J. Emerging Selected Topics Circuits Syst. 6, 146–162 (2016).

    Article  Google Scholar 

  36. 36.

    Kuzum, D., Jeyasingh, R. G., Lee, B. & Wong, H.-S. P. Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing. Nano Lett. 12, 2179–2186 (2011).

    Article  Google Scholar 

  37. 37.

    Tuma, T., Pantazi, A., Le Gallo, M., Sebastian, A. & Eleftheriou, E. Stochastic phase-change neurons. Nat. Nanotechnol. 11, 693–699 (2016).

    Article  Google Scholar 

  38. 38.

    Boybat, I. et al. Neuromorphic computing with multi-memristive synapses. Nat. Commun. 9, 2514 (2018).

    Article  Google Scholar 

  39. 39.

    Sebastian, A. et al. Tutorial: brain-inspired computing using phase-change memory devices. J. Appl. Phys. 124, 111101 (2018).

    Article  Google Scholar 

  40. 40.

    Joshi, V. et al. Accurate deep neural network inference using computational phase-change memory. Nat. Commun. https://doi.org/10.1038/s41467-020-16108-9 (2020).

  41. 41.

    Hosseini, P., Sebastian, A., Papandreou, N., Wright, C. D. & Bhaskaran, H. Accumulation-based computing using phase-change memories with FET access devices. Electron Device Lett. 36, 975–977 (2015).

    Article  Google Scholar 

  42. 42.

    Le Gallo, M. et al. Mixed-precision in-memory computing. Nat. Electron. 1, 246–253 (2018).

    Article  Google Scholar 

  43. 43.

    Xiong, F., Liao, A. D., Estrada, D. & Pop, E. Low-power switching of phase-change materials with carbon nanotube electrodes. Science 332, 568–570 (2011).

    Article  Google Scholar 

  44. 44.

    Waser, R. & Aono, M. in Nanoscience and Technology: a Collection of Reviews from Nature Journals 158–165 (World Scientific, 2010).

  45. 45.

    Kent, A. D. & Worledge, D. C. A new spin on magnetic memories. Nat. Nanotechnol. 10, 187–191 (2015).

    Article  Google Scholar 

  46. 46.

    Close, G. et al. Device, circuit and system-level analysis of noise in multi-bit phase-change memory. In Proceedings of the International Electron Devices Meeting (IEDM) 29.5.1–29.5.4 (IEEE, 2010).

  47. 47.

    Breitwisch, M. et al. Novel lithography-independent pore phase change memory. In Proceedings of the Symposium on VLSI Technology 100–101 (IEEE, 2007).

  48. 48.

    Rahimi, A., Kanerva, P. & Rabaey, J. M. A robust and energy-efficient classifier using brain-inspired hyperdimensional computing. In Proceedings of the 2016 International Symposium on Low Power Electronics and Design ISLPED 2016, 64–69 (ACM, 2016).

  49. 49.

    Quasthoff, U., Richter, M. & Biemann, C. Corpus portal for search in monolingual corpora. In Proceedings of the International Conference on Language Resources and Evaluation (LREC) 1799–1802 (ELRA, 2006).

  50. 50.

    Koehn, P. Europarl: a parallel corpus for statistical machine translation. In Proceedings of the MT Summit Vol. 5, 79–86 (AAMT, 2005).

  51. 51.

    Mimaroglu, D. S. Some Text Datasets (Univ. Massachusetts, accessed 9 March 2018); https://www.cs.umb.edu/smimarog/textmining/datasets/

  52. 52.

    Rahimi, A., Benatti, S., Kanerva, P., Benini, L. & Rabaey, J. M. Hyperdimensional biosignal processing: a case study for EMG-based hand gesture recognition. In Proceedings of the 2016 IEEE International Conference on Rebooting Computing (ICRC) 1–8 (IEEE, 2016).

  53. 53.

    Chandoke, N., Chitkara, N. & Grover, A. Comparative analysis of sense amplifiers for SRAM in 65 nm CMOS technology. In Proceedings of the International Conference on Electrical, Computer and Communication Technologies (ICECCT), 1–7 (IEEE, 2015).

Download references

Acknowledgements

This work was supported in part by the European Research Council through the European Union’s Horizon 2020 Research and Innovation Programme under grant no. 682675 and in part by the European Union’s Horizon 2020 Research and Innovation Programme through the project MNEMOSENE under grant no. 780215.

Author information

Affiliations

Authors

Contributions

All authors collectively conceived the idea of in-memory hyperdimensional computing. G.K. performed the experiments and analysed the results under the supervision of M.L.G., A.R. and A.S. G.K., M.L.G., A.R. and A.S. wrote the manuscript with input from all authors.

Corresponding authors

Correspondence to Abbas Rahimi or Abu Sebastian.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Table 1 Architecture configurations and hyperparameters used for the tree different tasks
Extended Data Table 2 Parameters for PCM crossbars energy and area estimation

Supplementary information

Supplementary Information

Supplementary Notes 1–6.

Supplementary Video 1

Experimental demonstration of language recognition with in-memory hyperdimensional computing implemented in the phase-change memory hardware platform.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Karunaratne, G., Le Gallo, M., Cherubini, G. et al. In-memory hyperdimensional computing. Nat Electron 3, 327–337 (2020). https://doi.org/10.1038/s41928-020-0410-3

Download citation

Further reading

Search

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing