Abstract
Time-series analysis including forecasting is essential in a range of fields from finance to engineering. However, long-term forecasting is difficult, particularly for cases where the underlying models and parameters are complex and unknown. Neural networks can effectively process features in temporal units and are attractive for such purposes. Reservoir computing, in particular, can offer efficient temporal processing of recurrent neural networks with a low training cost, and is thus well suited to time-series analysis and forecasting tasks. Here, we report a reservoir computing hardware system based on dynamic tungsten oxide (WOx) memristors that can efficiently process temporal data. The internal short-term memory effects of the WOx memristors allow the memristor-based reservoir to nonlinearly map temporal inputs into reservoir states, where the projected features can be readily processed by a linear readout function. We use the system to experimentally demonstrate two standard benchmarking tasks: isolated spoken-digit recognition with partial inputs, and chaotic system forecasting. A high classification accuracy of 99.2% is obtained for spoken-digit recognition, and autonomous chaotic time-series forecasting has been demonstrated over the long term.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the plots within this paper and other findings of this study are available from the corresponding author on reasonable request.
References
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982).
Werbos, P. J. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 1550–1560 (1990).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).
Verstraeten, D., Schrauwen, B. & Stroobandt, D. Reservoir-based techniques for speech recognition. In 2006 International Joint Conference on Neural Networks (IJCNN) 1050–1053 (IEEE, 2006).
Triefenbach, F., Jalalvand, A., Schrauwen, B. & Martens, J.-P. Phoneme recognition with large hierarchical reservoirs. Adv. Neural Inf. Process. Syst. 23, 2307–2315 (2010).
Jaeger, H. & Haas, H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004).
Pathak, J., Hunt, B., Girvan, M., Lu, Z. & Ott, E. Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett. 120, 024102 (2018).
Ilies, I. et al. Stepping Forward Through Echoes of the Past: Forecasting with Echo State Networks Technical Report http://www.neural-forecasting-competition.com/downloads/NN3/methods/27-NN3_Herbert_Jaeger_report.pdf (Jacobs University Bremen, 2007).
Sacchi, R., Ozturk, M. C., Principe, J. C., Carneiro, A. A. & Da Silva, I. N. Water inflow forecasting using the echo state network: a Brazilian case study. In 2007 International Joint Conference on Neural Networks (IJCNN) 2403–2408 (IEEE, 2007).
Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun. 8, 2204 (2017).
Sillin, H. O. et al. A theoretical and experimental study of neuromorphic atomic switch networks for reservoir computing. Nanotechnology 24, 384004 (2013).
Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5, 3541 (2014).
Torrejon, J. et al. Neuromorphic computing with nanoscale spintronic oscillators. Nature 547, 428–431 (2017).
Chua, L. O. Memristor—the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519 (1971).
Waser, R. & Aono, M. Nanoionics-based resistive switching memories. Nat. Mater. 6, 833–840 (2007).
Strukov, D. B., Snider, G. S., Stewart, D. R. & Williams, R. S. The missing memristor found. Nature 453, 80–83 (2008).
Pershin, Y. V. & Di Ventra, M. Neuromorphic, digital, and quantum computation with memory circuit elements. Proc. IEEE 100, 2071–2080 (2011).
Jo, S. H. et al. Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 1297–1301 (2010).
Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013).
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).
Sheridan, P. M. et al. Sparse coding with memristor networks. Nat. Nanotechnol. 12, 784–789 (2017).
Appeltant, L. et al. Information processing using a single dynamical node as complex system. Nat. Commun. 2, 468 (2011).
Larger, L. et al. Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. Opt. Express 20, 3241–3249 (2012).
Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
Antonik, P., Haelterman, M. & Massar, S. Brain-inspired photonic signal processor for generating periodic patterns and emulating chaotic systems. Phys. Rev. Appl. 7, 054014 (2017).
Box, G. E. & Pierce, D. A. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. J. Am. Stat. Assoc. 65, 1509–1526 (1970).
Said, S. E. & Dickey, D. A. Testing for unit roots in autoregressive-moving average models of unknown order. Biometrika 71, 599–607 (1984).
Kim, K. J. Financial time series forecasting using support vector machines. Neurocomputing 55, 307–319 (2003).
Kuremoto, T., Kimura, S., Kobayashi, K. & Obayashi, M. Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing 137, 47–56 (2014).
Zhang, G. P. An investigation of neural networks for linear time-series forecasting. Comput. Oper. Res. 28, 1183–1202 (2001).
Connor, J. T., Martin, R. D. & Atlas, L. E. Recurrent neural networks and robust time series prediction. IEEE Trans. Neural Netw. 5, 240–254 (1994).
Assaad, M., Boné, R. & Cardot, H. A new boosting algorithm for improved time-series forecasting with recurrent neural networks. Inf. Fusion 9, 41–55 (2008).
Mirikitani, D. T. & Nikolaev, N. Recursive bayesian recurrent neural networks for time-series modeling. IEEE Trans. Neural Netw. 21, 262–274 (2009).
Yang, Y. et al. Observation of conducting filament growth in nanoscale resistive memories. Nat. Commun. 3, 732 (2012).
Chang, T., Jo, S. H. & Lu, W. Short-term memory to long-term memory transition in a nanoscale memristor. ACS Nano 5, 7669–7676 (2011).
Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nat. Mater. 16, 101–108 (2017).
Texas Instruments-Developed 46-Word Speaker-Dependent Isolated Word Corpus (TI46) NIST Speech Disc 7-1.1 (Texas Instruments, 1991).
Lyon, R. F. A computational model of filtering, detection, and compression in the cochlea. Proc. IEEE-ICASSP'82 7, 1282–1285 (1982).
Bennett, C. H., Querlioz, D. & Klein, J. O. Spatio-temporal learning with arrays of analog nanosynapses. In 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) 125–130 (IEEE, 2017).
Frazier, C. & Kockelman, K. M. Chaos theory and transportation systems: instructive example. Transp. Res. Rec. 1897, 9–17 (2004).
Mackey, M. C. & Glass, L. Oscillation and chaos in physiological control systems. Science 197, 287–289 (1977).
Farmer, J. D. Chaotic attractors of an infinite-dimensional dynamical system. Physica D 4, 366–393 (1982).
Du, C. Metal Oxide Memristors with Internal Dynamics for Neuromorphic Applications (University of Michigan, 2017).
Gallicchio, C., Micheli, A. & Pedrelli, L. Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017).
Triefenbach, F., Jalalvand, A., Demuynck, K. & Martens, J. P. Acoustic modeling with hierarchical reservoirs. IEEE Trans. Audio Speech Lang. Process. 21, 2439–2450 (2013).
Gallicchio, C., Micheli, A. & Silvestri, L. Local Lyapunov exponents of deep echo state networks. Neurocomputing 298, 34–45 (2018).
Cai, F. et al. A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. Nat. Electron. 2, 290–299 (2019).
Tanaka, G. et al. Recent advances in physical reservoir computing: a review. Neural Netw. 115, 100–123 (2019).
Acknowledgements
We acknowledge inspiring discussions with M. Zidan. This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) through award HR0011-13-2-0015, the National Science Foundation (NSF) through grant CCF-1617315, and the Applications Driving Architectures (ADA) Research Centre, a JUMP Centre cosponsored by SRC and DARPA.
Author information
Authors and Affiliations
Contributions
J.M., W.M. and W.D.L. conceived the project and constructed the research frame. J.M., W.M., F.C., C.D. and S.H.L. prepared the memristor arrays and built the hardware and software package. J.M. and W.M. performed the hardware measurements. J.M., W.M., J.H.S. and W.D.L. analysed the experimental data and simulation results. W.D.L. directed the project. All authors discussed the results and implications and commented on the manuscript at all stages.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary information
Supplementary Figs. 1–18, Table 1 and Notes 1–11.
Supplementary Video 1
Time evolution of the autonomously generated output from the memristor RC system, along with the ground truth of the Mackey–Glass time series, plotted in phase space.
Rights and permissions
About this article
Cite this article
Moon, J., Ma, W., Shin, J.H. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat Electron 2, 480–487 (2019). https://doi.org/10.1038/s41928-019-0313-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-019-0313-3
This article is cited by
-
Memristor-based storage system with convolutional autoencoder-based image compression network
Nature Communications (2024)
-
Task-adaptive physical reservoir computing
Nature Materials (2024)
-
Synchronization of non-smooth chaotic systems via an improved reservoir computing
Scientific Reports (2024)
-
Universal scaling between wave speed and size enables nanoscale high-performance reservoir computing based on propagating spin-waves
npj Spintronics (2024)
-
Physical reservoir computing with emerging electronics
Nature Electronics (2024)