Resistive memory technologies could be used to create intelligent systems that learn locally at the edge. However, current approaches typically use learning algorithms that cannot be reconciled with the intrinsic non-idealities of resistive memory, particularly cycle-to-cycle variability. Here, we report a machine learning scheme that exploits memristor variability to implement Markov chain Monte Carlo sampling in a fabricated array of 16,384 devices configured as a Bayesian machine learning model. We apply the approach experimentally to carry out malignant tissue recognition and heart arrhythmia detection tasks, and, using a calibrated simulator, address the cartpole reinforcement learning task. Our approach demonstrates robustness to device degradation at ten million endurance cycles, and, based on circuit and system-level simulations, the total energy required to train the models is estimated to be on the order of microjoules, which is notably lower than in complementary metal–oxide–semiconductor (CMOS)-based approaches.
This is a preview of subscription content
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
All software programs used in the presentation of the Article are freely available upon request.
Shi, W., Cao, J., Zhang, Q., Li, Y. & Xu, L. Edge computing: vision and challenges. IEEE Internet Things J. 3, 637–646 (2016).
Edge AI Chipsets: Technology Outlook and Use Cases. Technical Report (ABI Research, 2019).
von Neumann, J. First draft of a report on the EDVAC. IEEE Ann. Hist. Comput. 15, 27–75 (1993).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Strubell, E., Ganesh, A. & Mccallum, A. Energy and policy considerations for deep learning in NLP. In Proc. 57th Annual Meeting of the Association for Computational Linguistics (ACL) 3645–3650 (ACL, 2019).
Li, D., Chen, X., Becchi, M. & Zong, Z. Evaluating the energy efficiency of deep convolutional neural networks on CPUs and GPUs. In 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom) 477–484 (IEEE, 2016).
Chua, L. Memristor—the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519 (1971).
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2014).
Wong, H. P. et al. Phase change memory. Proc. IEEE 98, 2201–2227 (2010).
Chappert, C., Fert, A. & Dau, F. The emergence of spin electronics in data storage. Nat. Mater. 6, 813–823 (2007).
Liu, Q. et al. Resistive switching: real-time observation on dynamic growth/dissolution of conductive filaments in oxide-electrolyte-based ReRAM. Adv. Mater. 24, 1774–1774 (2012).
Beck, A., Bednorz, J. G., Gerber, C., Rossel, C. & Widmer, D. Reproducible switching effect in thin oxide films for memory applications. Appl. Phys. Lett. 77, 139–141 (2000).
Garbin, D. et al. HfO2-based OxRAM devices as synapses for convolutional neural networks. IEEE Trans. Electron Dev. 62, 2494–2501 (2015).
Gokmen, T., Onen, M. & Haensch, W. Training deep convolutional neural networks with resistive cross-point devices. Front. Neurosci. 11, 538 (2017).
Boybat, I. et al. Neuromorphic computing with multi-memristive synapses. Nat. Commun. 9, 2514 (2017).
Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 558, 60–67 (2018).
Nandakumar, S. R. et al. Mixed-precision architecture based on computational memory for training deep neural networks. In Proc. 2018 IEEE International Symposium on Circuits and Systems (ISCAS) 1–5 (IEEE, 2018).
Li, C. et al. Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. Nat. Commun. 9, 2385 (2018).
Wang, Z. et al. Reinforcement learning with analogue memristor arrays. Nat. Electron. 2, 115–124 (2019).
Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).
Burr, G. W. et al. Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses) using phase-change memory as the synaptic weight element. IEEE Trans. Electron Dev. 62, 3498–3507 (2015).
Sebastian, A., Krebs, D., Gallo, M. L., Pozidis, H. & Eleftheriou, E. A collective relaxation model for resistance drift in phase change memory cells. In Proc. 2015 IEEE International Reliability Physics Symposium MY.5.1–MY.5.6 (IEEE, 2015).
Ambrogio, S. et al. Statistical fluctuations in HfOx resistive-switching memory: Part 1—set/reset variability. IEEE Trans. Electron Dev. 61, 2912–2919 (2014).
Sidler, S. et al. Large-scale neural networks implemented with non-volatile memory as the synaptic weight element: impact of conductance response. In Proc. 2016 46th European Solid-State Device Research Conference (ESSDERC) 440–443 (IEEE, 2016).
Agarwal, S. et al. Resistive memory device requirements for a neural algorithm accelerator. In Proc. 2016 International Joint Conference on Neural Networks (IJCNN) 929–938 (IEEE, 2016).
Querlioz, D., Bichler, O., Dollfus, P. & Gamrat, C. Immunity to device variations in a spiking neural network with memristive nanodevices. IEEE Trans. Nanotechnol. 12, 288–295 (2013).
Serb, A. et al. Unsupervised learning in probabilistic neural networks with multi-state metal–oxide memristive synapses. Nat. Commun. 7, 12611 (2016).
Dalgaty, T. et al. Hybrid neuromorphic circuits exploiting non-conventional properties of RRAM for massively parallel local plasticity mechanisms. APL Mater. 7, 081125 (2019).
Balatti, S., Ambrogio, S., Wang, Z. & Ielmini, D. True random number generation by variability of resistive switching in oxide-based devices. IEEE J. Emerg. Select. Top. Circuits Syst. 5, 214–221 (2015).
Vodenicarevic, D. et al. Low-energy truly random number generation with superparamagnetic tunnel junctions for unconventional computing. Phys. Rev. Appl. 8, 054045 (2017).
Faria, R., Camsari, K. Y. & Datta, S. Implementing Bayesian networks with embedded stochastic MRAM. AIP Adv. 8, 045101 (2018).
Mizrahi, A. et al. Neural-like computing with populations of superparamagnetic basis functions. Nat. Commun. 9, 1533 (2018).
Camsari, K. Y., Faria, R., Sutton, B. M. & Datta, S. Stochastic p-bits for invertible logic. Phys. Rev. X 7, 031014 (2017).
Borders, W. A. et al. Integer factorization using stochastic magnetic tunnel junctions. Nature 573, 390–393 (2019).
Hastings, W. K. Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57, 97–109 (1970).
Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).
Neal, R. M. Bayesian Learning for Neural Networks Vol. 118 (Springer Science & Business Media, 2012).
Grossi, A. et al. Resistive RAM endurance: array-level characterization and correction techniques targeting deep learning applications. IEEE Trans. Electron Dev. 66, 1281–1288 (2019).
Ielmini, D. Modeling the universal set/reset characteristics of bipolar RRAM by field- and temperature-driven filament growth. IEEE Trans. Electron Dev. 58, 4309–4317 (2011).
Wolberg, W. H. & Mangasarian, O. L. Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc. Natl Acad. Sci. USA 87, 9193–9196 (1990).
Moody, G. B. & Mark, R. G. The impact of the MIT-BIH arrhythmia database. IEEE Eng. Med. Biol. Mag. 20, 45–50 (2001).
Sutton, R. S. & Barto, A. G. Introduction to Reinforcement Learning (MIT Press, 1998).
Hoffman, M., Doucet, A., Freitas, N. D. & Jasra, A. Trans-dimensional MCMC for Bayesian policy learning. In Proc. 20th International Conference on Neural Information Processing Systems, NIPS’07 665–672 (Curran Associates, 2007).
Barto, A. G., Sutton, R. S. & Anderson, C. W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. SMC-13, 834–846 (1983).
Berdan, R. et al. In-memory reinforcement learning with moderately-stochastic conductance switching of ferroelectric tunnel junctions. In Proc. 2019 Symposium on VLSI Technology T22–T23 (IEEE, 2019).
Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
Pourret, O., Naïm, P. & Marcot, B. Bayesian Networks: A Practical Guide to Applications (Wiley, 2008).
Maclaurin, D. & Adams, R. P. Firefly Monte Carlo: exact MCMC with subsets of data. In Proc. Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI’14 543–552 (AUAI Press, 2014).
Korattikara, A., Chen, Y. & Welling, M. Austerity in MCMC land: cutting the Metropolis–Hastings budget. In Proc. 31st International Conference on Machine Learning 181–189 (PMLR, 2014).
Hoffman, M. D. & Gelman, A. The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res. 15, 1593–1623 (2014).
Liu, H. & Setiono, R. Chi2: feature selection and discretization of numeric attributes. In Proc. 7th IEEE International Conference on Tools with Artificial Intelligence 388–391 (IEEE, 1995).
We acknowledge funding support from the French ANR via Carnot funding as well as the H2020 MeM-Scales project (871371) and the European Research Council (grant NANOINFER, no. 715872). In addition, we thank E. Esmanhotto, J. Sandrini and C. Cagli (CEA-Leti) for help with the measurement set-up, J. F. Nodin (CEA-Leti) for providing the images in Fig. 3d and to S. Mitra (Stanford University), M. Payvand (ETH Zurich), A. Valentian, M. Solinas-Angel, E. Nowak (CEA-Leti), J. Diard (CNRS, Université Grenoble Alpes), P. Bessiére and J. Droulez (CNRS, Sorbonne Université), J. Grollier (CNRS, Thales) and J.-M. Portal (Aix-Marseille Université) for discussing various aspects of the Article.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Dalgaty, T., Castellani, N., Turck, C. et al. In situ learning using intrinsic memristor variability via Markov chain Monte Carlo sampling. Nat Electron 4, 151–161 (2021). https://doi.org/10.1038/s41928-020-00523-3