Abstract
Resistive memory technologies could be used to create intelligent systems that learn locally at the edge. However, current approaches typically use learning algorithms that cannot be reconciled with the intrinsic non-idealities of resistive memory, particularly cycle-to-cycle variability. Here, we report a machine learning scheme that exploits memristor variability to implement Markov chain Monte Carlo sampling in a fabricated array of 16,384 devices configured as a Bayesian machine learning model. We apply the approach experimentally to carry out malignant tissue recognition and heart arrhythmia detection tasks, and, using a calibrated simulator, address the cartpole reinforcement learning task. Our approach demonstrates robustness to device degradation at ten million endurance cycles, and, based on circuit and system-level simulations, the total energy required to train the models is estimated to be on the order of microjoules, which is notably lower than in complementary metal–oxide–semiconductor (CMOS)-based approaches.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Code availability
All software programs used in the presentation of the Article are freely available upon request.
References
Shi, W., Cao, J., Zhang, Q., Li, Y. & Xu, L. Edge computing: vision and challenges. IEEE Internet Things J. 3, 637–646 (2016).
Edge AI Chipsets: Technology Outlook and Use Cases. Technical Report (ABI Research, 2019).
von Neumann, J. First draft of a report on the EDVAC. IEEE Ann. Hist. Comput. 15, 27–75 (1993).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Strubell, E., Ganesh, A. & Mccallum, A. Energy and policy considerations for deep learning in NLP. In Proc. 57th Annual Meeting of the Association for Computational Linguistics (ACL) 3645–3650 (ACL, 2019).
Li, D., Chen, X., Becchi, M. & Zong, Z. Evaluating the energy efficiency of deep convolutional neural networks on CPUs and GPUs. In 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom) 477–484 (IEEE, 2016).
Chua, L. Memristor—the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519 (1971).
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2014).
Wong, H. P. et al. Phase change memory. Proc. IEEE 98, 2201–2227 (2010).
Chappert, C., Fert, A. & Dau, F. The emergence of spin electronics in data storage. Nat. Mater. 6, 813–823 (2007).
Liu, Q. et al. Resistive switching: real-time observation on dynamic growth/dissolution of conductive filaments in oxide-electrolyte-based ReRAM. Adv. Mater. 24, 1774–1774 (2012).
Beck, A., Bednorz, J. G., Gerber, C., Rossel, C. & Widmer, D. Reproducible switching effect in thin oxide films for memory applications. Appl. Phys. Lett. 77, 139–141 (2000).
Garbin, D. et al. HfO2-based OxRAM devices as synapses for convolutional neural networks. IEEE Trans. Electron Dev. 62, 2494–2501 (2015).
Gokmen, T., Onen, M. & Haensch, W. Training deep convolutional neural networks with resistive cross-point devices. Front. Neurosci. 11, 538 (2017).
Boybat, I. et al. Neuromorphic computing with multi-memristive synapses. Nat. Commun. 9, 2514 (2017).
Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 558, 60–67 (2018).
Nandakumar, S. R. et al. Mixed-precision architecture based on computational memory for training deep neural networks. In Proc. 2018 IEEE International Symposium on Circuits and Systems (ISCAS) 1–5 (IEEE, 2018).
Li, C. et al. Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. Nat. Commun. 9, 2385 (2018).
Wang, Z. et al. Reinforcement learning with analogue memristor arrays. Nat. Electron. 2, 115–124 (2019).
Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).
Burr, G. W. et al. Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses) using phase-change memory as the synaptic weight element. IEEE Trans. Electron Dev. 62, 3498–3507 (2015).
Sebastian, A., Krebs, D., Gallo, M. L., Pozidis, H. & Eleftheriou, E. A collective relaxation model for resistance drift in phase change memory cells. In Proc. 2015 IEEE International Reliability Physics Symposium MY.5.1–MY.5.6 (IEEE, 2015).
Ambrogio, S. et al. Statistical fluctuations in HfOx resistive-switching memory: Part 1—set/reset variability. IEEE Trans. Electron Dev. 61, 2912–2919 (2014).
Sidler, S. et al. Large-scale neural networks implemented with non-volatile memory as the synaptic weight element: impact of conductance response. In Proc. 2016 46th European Solid-State Device Research Conference (ESSDERC) 440–443 (IEEE, 2016).
Agarwal, S. et al. Resistive memory device requirements for a neural algorithm accelerator. In Proc. 2016 International Joint Conference on Neural Networks (IJCNN) 929–938 (IEEE, 2016).
Querlioz, D., Bichler, O., Dollfus, P. & Gamrat, C. Immunity to device variations in a spiking neural network with memristive nanodevices. IEEE Trans. Nanotechnol. 12, 288–295 (2013).
Serb, A. et al. Unsupervised learning in probabilistic neural networks with multi-state metal–oxide memristive synapses. Nat. Commun. 7, 12611 (2016).
Dalgaty, T. et al. Hybrid neuromorphic circuits exploiting non-conventional properties of RRAM for massively parallel local plasticity mechanisms. APL Mater. 7, 081125 (2019).
Balatti, S., Ambrogio, S., Wang, Z. & Ielmini, D. True random number generation by variability of resistive switching in oxide-based devices. IEEE J. Emerg. Select. Top. Circuits Syst. 5, 214–221 (2015).
Vodenicarevic, D. et al. Low-energy truly random number generation with superparamagnetic tunnel junctions for unconventional computing. Phys. Rev. Appl. 8, 054045 (2017).
Faria, R., Camsari, K. Y. & Datta, S. Implementing Bayesian networks with embedded stochastic MRAM. AIP Adv. 8, 045101 (2018).
Mizrahi, A. et al. Neural-like computing with populations of superparamagnetic basis functions. Nat. Commun. 9, 1533 (2018).
Camsari, K. Y., Faria, R., Sutton, B. M. & Datta, S. Stochastic p-bits for invertible logic. Phys. Rev. X 7, 031014 (2017).
Borders, W. A. et al. Integer factorization using stochastic magnetic tunnel junctions. Nature 573, 390–393 (2019).
Hastings, W. K. Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57, 97–109 (1970).
Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).
Neal, R. M. Bayesian Learning for Neural Networks Vol. 118 (Springer Science & Business Media, 2012).
Grossi, A. et al. Resistive RAM endurance: array-level characterization and correction techniques targeting deep learning applications. IEEE Trans. Electron Dev. 66, 1281–1288 (2019).
Ielmini, D. Modeling the universal set/reset characteristics of bipolar RRAM by field- and temperature-driven filament growth. IEEE Trans. Electron Dev. 58, 4309–4317 (2011).
Wolberg, W. H. & Mangasarian, O. L. Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc. Natl Acad. Sci. USA 87, 9193–9196 (1990).
Moody, G. B. & Mark, R. G. The impact of the MIT-BIH arrhythmia database. IEEE Eng. Med. Biol. Mag. 20, 45–50 (2001).
Sutton, R. S. & Barto, A. G. Introduction to Reinforcement Learning (MIT Press, 1998).
Hoffman, M., Doucet, A., Freitas, N. D. & Jasra, A. Trans-dimensional MCMC for Bayesian policy learning. In Proc. 20th International Conference on Neural Information Processing Systems, NIPS’07 665–672 (Curran Associates, 2007).
Barto, A. G., Sutton, R. S. & Anderson, C. W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. SMC-13, 834–846 (1983).
Berdan, R. et al. In-memory reinforcement learning with moderately-stochastic conductance switching of ferroelectric tunnel junctions. In Proc. 2019 Symposium on VLSI Technology T22–T23 (IEEE, 2019).
Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
Pourret, O., Naïm, P. & Marcot, B. Bayesian Networks: A Practical Guide to Applications (Wiley, 2008).
Maclaurin, D. & Adams, R. P. Firefly Monte Carlo: exact MCMC with subsets of data. In Proc. Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI’14 543–552 (AUAI Press, 2014).
Korattikara, A., Chen, Y. & Welling, M. Austerity in MCMC land: cutting the Metropolis–Hastings budget. In Proc. 31st International Conference on Machine Learning 181–189 (PMLR, 2014).
Hoffman, M. D. & Gelman, A. The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res. 15, 1593–1623 (2014).
Liu, H. & Setiono, R. Chi2: feature selection and discretization of numeric attributes. In Proc. 7th IEEE International Conference on Tools with Artificial Intelligence 388–391 (IEEE, 1995).
Acknowledgements
We acknowledge funding support from the French ANR via Carnot funding as well as the H2020 MeM-Scales project (871371) and the European Research Council (grant NANOINFER, no. 715872). In addition, we thank E. Esmanhotto, J. Sandrini and C. Cagli (CEA-Leti) for help with the measurement set-up, J. F. Nodin (CEA-Leti) for providing the images in Fig. 3d and to S. Mitra (Stanford University), M. Payvand (ETH Zurich), A. Valentian, M. Solinas-Angel, E. Nowak (CEA-Leti), J. Diard (CNRS, Université Grenoble Alpes), P. Bessiére and J. Droulez (CNRS, Sorbonne Université), J. Grollier (CNRS, Thales) and J.-M. Portal (Aix-Marseille Université) for discussing various aspects of the Article.
Author information
Authors and Affiliations
Contributions
T.D. developed the concept of RRAM-based MCMC sampling. N.C. built the computer-in-the-loop test set-up with the resistive memory array. T.D. and N.C. performed the computer-in-the-loop experiments with the resistive memory array. T.D. implemented the behavioural simulator, performed measurements on the array, which were used to calibrate the simulation and performed the benchmarking. K.-E.H., C.T. and D.Q. performed the design and energy analysis of the full system implementation. T.D., D.Q. and E.V. developed ideas and wrote the Article together.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Figs. 1–17, Table 1 and Note 1.
Supplementary Video 1
Video of the cartpole task during one iteration of training through RRAM-based MCMC.
Supplementary Video 2
Video of the cartpole task during one iteration of testing using the trained model.
Rights and permissions
About this article
Cite this article
Dalgaty, T., Castellani, N., Turck, C. et al. In situ learning using intrinsic memristor variability via Markov chain Monte Carlo sampling. Nat Electron 4, 151–161 (2021). https://doi.org/10.1038/s41928-020-00523-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-020-00523-3
This article is cited by
-
Scaling neuromorphic systems with 3D technologies
Nature Electronics (2024)
-
Mosaic: in-memory computing and routing for small-world spike-based neuromorphic systems
Nature Communications (2024)
-
A life in electrical engineering, glancing at biology
Nature Reviews Electrical Engineering (2024)
-
Memristor-based hardware accelerators for artificial intelligence
Nature Reviews Electrical Engineering (2024)
-
Thermally stable threshold selector based on CuAg alloy for energy-efficient memory and neuromorphic computing applications
Nature Communications (2023)