Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Temporal data classification and forecasting using a memristor-based reservoir computing system

Nature Electronics volume 2, pages 480–487 (2019)Cite this article

Subjects

Abstract

Time-series analysis including forecasting is essential in a range of fields from finance to engineering. However, long-term forecasting is difficult, particularly for cases where the underlying models and parameters are complex and unknown. Neural networks can effectively process features in temporal units and are attractive for such purposes. Reservoir computing, in particular, can offer efficient temporal processing of recurrent neural networks with a low training cost, and is thus well suited to time-series analysis and forecasting tasks. Here, we report a reservoir computing hardware system based on dynamic tungsten oxide (WOx) memristors that can efficiently process temporal data. The internal short-term memory effects of the WOx memristors allow the memristor-based reservoir to nonlinearly map temporal inputs into reservoir states, where the projected features can be readily processed by a linear readout function. We use the system to experimentally demonstrate two standard benchmarking tasks: isolated spoken-digit recognition with partial inputs, and chaotic system forecasting. A high classification accuracy of 99.2% is obtained for spoken-digit recognition, and autonomous chaotic time-series forecasting has been demonstrated over the long term.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Memristor-based RC system.
Fig. 2: Spoken-digit recognition task implementation.
Fig. 3: Classification using partial inputs.
Fig. 4: Autonomous forecasting of Mackey–Glass time series.
Fig. 5: Long-term forecasting of Mackey–Glass time series with periodic updates.

Similar content being viewed by others

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author on reasonable request.

References

  1. Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982).

    Article  MathSciNet  MATH  Google Scholar 

  2. Werbos, P. J. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 1550–1560 (1990).

    Article  Google Scholar 

  3. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

    Article  Google Scholar 

  4. Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).

    Article  MATH  Google Scholar 

  5. Verstraeten, D., Schrauwen, B. & Stroobandt, D. Reservoir-based techniques for speech recognition. In 2006 International Joint Conference on Neural Networks (IJCNN) 1050–1053 (IEEE, 2006).

  6. Triefenbach, F., Jalalvand, A., Schrauwen, B. & Martens, J.-P. Phoneme recognition with large hierarchical reservoirs. Adv. Neural Inf. Process. Syst. 23, 2307–2315 (2010).

    Google Scholar 

  7. Jaeger, H. & Haas, H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004).

    Article  Google Scholar 

  8. Pathak, J., Hunt, B., Girvan, M., Lu, Z. & Ott, E. Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett. 120, 024102 (2018).

    Article  Google Scholar 

  9. Ilies, I. et al. Stepping Forward Through Echoes of the Past: Forecasting with Echo State Networks Technical Report http://www.neural-forecasting-competition.com/downloads/NN3/methods/27-NN3_Herbert_Jaeger_report.pdf (Jacobs University Bremen, 2007).

  10. Sacchi, R., Ozturk, M. C., Principe, J. C., Carneiro, A. A. & Da Silva, I. N. Water inflow forecasting using the echo state network: a Brazilian case study. In 2007 International Joint Conference on Neural Networks (IJCNN) 2403–2408 (IEEE, 2007).

  11. Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun. 8, 2204 (2017).

    Article  Google Scholar 

  12. Sillin, H. O. et al. A theoretical and experimental study of neuromorphic atomic switch networks for reservoir computing. Nanotechnology 24, 384004 (2013).

    Article  Google Scholar 

  13. Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5, 3541 (2014).

    Article  Google Scholar 

  14. Torrejon, J. et al. Neuromorphic computing with nanoscale spintronic oscillators. Nature 547, 428–431 (2017).

    Article  Google Scholar 

  15. Chua, L. O. Memristor—the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519 (1971).

    Article  Google Scholar 

  16. Waser, R. & Aono, M. Nanoionics-based resistive switching memories. Nat. Mater. 6, 833–840 (2007).

    Article  Google Scholar 

  17. Strukov, D. B., Snider, G. S., Stewart, D. R. & Williams, R. S. The missing memristor found. Nature 453, 80–83 (2008).

    Article  Google Scholar 

  18. Pershin, Y. V. & Di Ventra, M. Neuromorphic, digital, and quantum computation with memory circuit elements. Proc. IEEE 100, 2071–2080 (2011).

    Article  Google Scholar 

  19. Jo, S. H. et al. Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 1297–1301 (2010).

    Article  Google Scholar 

  20. Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013).

    Article  Google Scholar 

  21. Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).

    Article  Google Scholar 

  22. Sheridan, P. M. et al. Sparse coding with memristor networks. Nat. Nanotechnol. 12, 784–789 (2017).

    Article  Google Scholar 

  23. Appeltant, L. et al. Information processing using a single dynamical node as complex system. Nat. Commun. 2, 468 (2011).

    Article  Google Scholar 

  24. Larger, L. et al. Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. Opt. Express 20, 3241–3249 (2012).

    Article  Google Scholar 

  25. Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).

    Article  Google Scholar 

  26. Antonik, P., Haelterman, M. & Massar, S. Brain-inspired photonic signal processor for generating periodic patterns and emulating chaotic systems. Phys. Rev. Appl. 7, 054014 (2017).

    Article  Google Scholar 

  27. Box, G. E. & Pierce, D. A. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. J. Am. Stat. Assoc. 65, 1509–1526 (1970).

    Article  MathSciNet  MATH  Google Scholar 

  28. Said, S. E. & Dickey, D. A. Testing for unit roots in autoregressive-moving average models of unknown order. Biometrika 71, 599–607 (1984).

    Article  MathSciNet  MATH  Google Scholar 

  29. Kim, K. J. Financial time series forecasting using support vector machines. Neurocomputing 55, 307–319 (2003).

    Article  Google Scholar 

  30. Kuremoto, T., Kimura, S., Kobayashi, K. & Obayashi, M. Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing 137, 47–56 (2014).

    Article  Google Scholar 

  31. Zhang, G. P. An investigation of neural networks for linear time-series forecasting. Comput. Oper. Res. 28, 1183–1202 (2001).

    Article  MATH  Google Scholar 

  32. Connor, J. T., Martin, R. D. & Atlas, L. E. Recurrent neural networks and robust time series prediction. IEEE Trans. Neural Netw. 5, 240–254 (1994).

    Article  Google Scholar 

  33. Assaad, M., Boné, R. & Cardot, H. A new boosting algorithm for improved time-series forecasting with recurrent neural networks. Inf. Fusion 9, 41–55 (2008).

    Article  Google Scholar 

  34. Mirikitani, D. T. & Nikolaev, N. Recursive bayesian recurrent neural networks for time-series modeling. IEEE Trans. Neural Netw. 21, 262–274 (2009).

    Article  Google Scholar 

  35. Yang, Y. et al. Observation of conducting filament growth in nanoscale resistive memories. Nat. Commun. 3, 732 (2012).

    Article  Google Scholar 

  36. Chang, T., Jo, S. H. & Lu, W. Short-term memory to long-term memory transition in a nanoscale memristor. ACS Nano 5, 7669–7676 (2011).

    Article  Google Scholar 

  37. Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nat. Mater. 16, 101–108 (2017).

    Article  Google Scholar 

  38. Texas Instruments-Developed 46-Word Speaker-Dependent Isolated Word Corpus (TI46) NIST Speech Disc 7-1.1 (Texas Instruments, 1991).

  39. Lyon, R. F. A computational model of filtering, detection, and compression in the cochlea. Proc. IEEE-ICASSP'82 7, 1282–1285 (1982).

    Google Scholar 

  40. Bennett, C. H., Querlioz, D. & Klein, J. O. Spatio-temporal learning with arrays of analog nanosynapses. In 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) 125–130 (IEEE, 2017).

  41. Frazier, C. & Kockelman, K. M. Chaos theory and transportation systems: instructive example. Transp. Res. Rec. 1897, 9–17 (2004).

    Article  Google Scholar 

  42. Mackey, M. C. & Glass, L. Oscillation and chaos in physiological control systems. Science 197, 287–289 (1977).

    Article  MATH  Google Scholar 

  43. Farmer, J. D. Chaotic attractors of an infinite-dimensional dynamical system. Physica D 4, 366–393 (1982).

    Article  MathSciNet  MATH  Google Scholar 

  44. Du, C. Metal Oxide Memristors with Internal Dynamics for Neuromorphic Applications (University of Michigan, 2017).

  45. Gallicchio, C., Micheli, A. & Pedrelli, L. Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017).

    Article  Google Scholar 

  46. Triefenbach, F., Jalalvand, A., Demuynck, K. & Martens, J. P. Acoustic modeling with hierarchical reservoirs. IEEE Trans. Audio Speech Lang. Process. 21, 2439–2450 (2013).

    Article  Google Scholar 

  47. Gallicchio, C., Micheli, A. & Silvestri, L. Local Lyapunov exponents of deep echo state networks. Neurocomputing 298, 34–45 (2018).

    Article  Google Scholar 

  48. Cai, F. et al. A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. Nat. Electron. 2, 290–299 (2019).

    Article  Google Scholar 

  49. Tanaka, G. et al. Recent advances in physical reservoir computing: a review. Neural Netw. 115, 100–123 (2019).

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge inspiring discussions with M. Zidan. This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) through award HR0011-13-2-0015, the National Science Foundation (NSF) through grant CCF-1617315, and the Applications Driving Architectures (ADA) Research Centre, a JUMP Centre cosponsored by SRC and DARPA.

Author information

Author notes

  1. These authors contributed equally: John Moon, Wen Ma.

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA

    John Moon, Wen Ma, Jong Hoon Shin, Fuxi Cai, Chao Du, Seung Hwan Lee & Wei D. Lu

Authors
  1. John Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jong Hoon Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fuxi Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chao Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Seung Hwan Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wei D. Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.M., W.M. and W.D.L. conceived the project and constructed the research frame. J.M., W.M., F.C., C.D. and S.H.L. prepared the memristor arrays and built the hardware and software package. J.M. and W.M. performed the hardware measurements. J.M., W.M., J.H.S. and W.D.L. analysed the experimental data and simulation results. W.D.L. directed the project. All authors discussed the results and implications and commented on the manuscript at all stages.

Corresponding author

Correspondence to Wei D. Lu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information

Supplementary Figs. 1–18, Table 1 and Notes 1–11.

Supplementary Video 1

Time evolution of the autonomously generated output from the memristor RC system, along with the ground truth of the Mackey–Glass time series, plotted in phase space.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moon, J., Ma, W., Shin, J.H. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat Electron 2, 480–487 (2019). https://doi.org/10.1038/s41928-019-0313-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-019-0313-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing