Time-series analysis including forecasting is essential in a range of fields from finance to engineering. However, long-term forecasting is difficult, particularly for cases where the underlying models and parameters are complex and unknown. Neural networks can effectively process features in temporal units and are attractive for such purposes. Reservoir computing, in particular, can offer efficient temporal processing of recurrent neural networks with a low training cost, and is thus well suited to time-series analysis and forecasting tasks. Here, we report a reservoir computing hardware system based on dynamic tungsten oxide (WOx) memristors that can efficiently process temporal data. The internal short-term memory effects of the WOx memristors allow the memristor-based reservoir to nonlinearly map temporal inputs into reservoir states, where the projected features can be readily processed by a linear readout function. We use the system to experimentally demonstrate two standard benchmarking tasks: isolated spoken-digit recognition with partial inputs, and chaotic system forecasting. A high classification accuracy of 99.2% is obtained for spoken-digit recognition, and autonomous chaotic time-series forecasting has been demonstrated over the long term.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data that support the plots within this paper and other findings of this study are available from the corresponding author on reasonable request.
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982).
Werbos, P. J. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 1550–1560 (1990).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).
Verstraeten, D., Schrauwen, B. & Stroobandt, D. Reservoir-based techniques for speech recognition. In 2006 International Joint Conference on Neural Networks (IJCNN) 1050–1053 (IEEE, 2006).
Triefenbach, F., Jalalvand, A., Schrauwen, B. & Martens, J.-P. Phoneme recognition with large hierarchical reservoirs. Adv. Neural Inf. Process. Syst. 23, 2307–2315 (2010).
Jaeger, H. & Haas, H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004).
Pathak, J., Hunt, B., Girvan, M., Lu, Z. & Ott, E. Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett. 120, 024102 (2018).
Ilies, I. et al. Stepping Forward Through Echoes of the Past: Forecasting with Echo State Networks Technical Report http://www.neural-forecasting-competition.com/downloads/NN3/methods/27-NN3_Herbert_Jaeger_report.pdf (Jacobs University Bremen, 2007).
Sacchi, R., Ozturk, M. C., Principe, J. C., Carneiro, A. A. & Da Silva, I. N. Water inflow forecasting using the echo state network: a Brazilian case study. In 2007 International Joint Conference on Neural Networks (IJCNN) 2403–2408 (IEEE, 2007).
Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun. 8, 2204 (2017).
Sillin, H. O. et al. A theoretical and experimental study of neuromorphic atomic switch networks for reservoir computing. Nanotechnology 24, 384004 (2013).
Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5, 3541 (2014).
Torrejon, J. et al. Neuromorphic computing with nanoscale spintronic oscillators. Nature 547, 428–431 (2017).
Chua, L. O. Memristor—the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519 (1971).
Waser, R. & Aono, M. Nanoionics-based resistive switching memories. Nat. Mater. 6, 833–840 (2007).
Strukov, D. B., Snider, G. S., Stewart, D. R. & Williams, R. S. The missing memristor found. Nature 453, 80–83 (2008).
Pershin, Y. V. & Di Ventra, M. Neuromorphic, digital, and quantum computation with memory circuit elements. Proc. IEEE 100, 2071–2080 (2011).
Jo, S. H. et al. Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 1297–1301 (2010).
Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013).
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).
Sheridan, P. M. et al. Sparse coding with memristor networks. Nat. Nanotechnol. 12, 784–789 (2017).
Appeltant, L. et al. Information processing using a single dynamical node as complex system. Nat. Commun. 2, 468 (2011).
Larger, L. et al. Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. Opt. Express 20, 3241–3249 (2012).
Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
Antonik, P., Haelterman, M. & Massar, S. Brain-inspired photonic signal processor for generating periodic patterns and emulating chaotic systems. Phys. Rev. Appl. 7, 054014 (2017).
Box, G. E. & Pierce, D. A. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. J. Am. Stat. Assoc. 65, 1509–1526 (1970).
Said, S. E. & Dickey, D. A. Testing for unit roots in autoregressive-moving average models of unknown order. Biometrika 71, 599–607 (1984).
Kim, K. J. Financial time series forecasting using support vector machines. Neurocomputing 55, 307–319 (2003).
Kuremoto, T., Kimura, S., Kobayashi, K. & Obayashi, M. Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing 137, 47–56 (2014).
Zhang, G. P. An investigation of neural networks for linear time-series forecasting. Comput. Oper. Res. 28, 1183–1202 (2001).
Connor, J. T., Martin, R. D. & Atlas, L. E. Recurrent neural networks and robust time series prediction. IEEE Trans. Neural Netw. 5, 240–254 (1994).
Assaad, M., Boné, R. & Cardot, H. A new boosting algorithm for improved time-series forecasting with recurrent neural networks. Inf. Fusion 9, 41–55 (2008).
Mirikitani, D. T. & Nikolaev, N. Recursive bayesian recurrent neural networks for time-series modeling. IEEE Trans. Neural Netw. 21, 262–274 (2009).
Yang, Y. et al. Observation of conducting filament growth in nanoscale resistive memories. Nat. Commun. 3, 732 (2012).
Chang, T., Jo, S. H. & Lu, W. Short-term memory to long-term memory transition in a nanoscale memristor. ACS Nano 5, 7669–7676 (2011).
Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nat. Mater. 16, 101–108 (2017).
Texas Instruments-Developed 46-Word Speaker-Dependent Isolated Word Corpus (TI46) NIST Speech Disc 7-1.1 (Texas Instruments, 1991).
Lyon, R. F. A computational model of filtering, detection, and compression in the cochlea. Proc. IEEE-ICASSP'82 7, 1282–1285 (1982).
Bennett, C. H., Querlioz, D. & Klein, J. O. Spatio-temporal learning with arrays of analog nanosynapses. In 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) 125–130 (IEEE, 2017).
Frazier, C. & Kockelman, K. M. Chaos theory and transportation systems: instructive example. Transp. Res. Rec. 1897, 9–17 (2004).
Mackey, M. C. & Glass, L. Oscillation and chaos in physiological control systems. Science 197, 287–289 (1977).
Farmer, J. D. Chaotic attractors of an infinite-dimensional dynamical system. Physica D 4, 366–393 (1982).
Du, C. Metal Oxide Memristors with Internal Dynamics for Neuromorphic Applications (University of Michigan, 2017).
Gallicchio, C., Micheli, A. & Pedrelli, L. Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017).
Triefenbach, F., Jalalvand, A., Demuynck, K. & Martens, J. P. Acoustic modeling with hierarchical reservoirs. IEEE Trans. Audio Speech Lang. Process. 21, 2439–2450 (2013).
Gallicchio, C., Micheli, A. & Silvestri, L. Local Lyapunov exponents of deep echo state networks. Neurocomputing 298, 34–45 (2018).
Cai, F. et al. A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. Nat. Electron. 2, 290–299 (2019).
Tanaka, G. et al. Recent advances in physical reservoir computing: a review. Neural Netw. 115, 100–123 (2019).
We acknowledge inspiring discussions with M. Zidan. This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) through award HR0011-13-2-0015, the National Science Foundation (NSF) through grant CCF-1617315, and the Applications Driving Architectures (ADA) Research Centre, a JUMP Centre cosponsored by SRC and DARPA.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moon, J., Ma, W., Shin, J.H. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat Electron 2, 480–487 (2019). https://doi.org/10.1038/s41928-019-0313-3
Nature Communications (2021)
In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks
Nature Materials (2021)
Scientific Reports (2021)
Nature Communications (2021)