Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Energy-efficient Mott activation neuron for full-hardware implementation of neural networks

Nature Nanotechnology volume 16, pages 680–687 (2021)Cite this article

Subjects

Abstract

To circumvent the von Neumann bottleneck, substantial progress has been made towards in-memory computing with synaptic devices. However, compact nanodevices implementing non-linear activation functions are required for efficient full-hardware implementation of deep neural networks. Here, we present an energy-efficient and compact Mott activation neuron based on vanadium dioxide and its successful integration with a conductive bridge random access memory (CBRAM) crossbar array in hardware. The Mott activation neuron implements the rectified linear unit function in the analogue domain. The neuron devices consume substantially less energy and occupy two orders of magnitude smaller area than those of analogue complementary metal–oxide semiconductor implementations. The LeNet-5 network with Mott activation neurons achieves 98.38% accuracy on the MNIST dataset, close to the ideal software accuracy. We perform large-scale image edge detection using the Mott activation neurons integrated with a CBRAM crossbar array. Our findings provide a solution towards large-scale, highly parallel and energy-efficient in-memory computing systems for neural networks.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The Mott ReLU device for the hardware implementation of a neural network.
Fig. 2: Switching mechanisms of VO2 gap.
Fig. 3: Electrical characteristics of the Mott ReLU device.
Fig. 4: Network-level implementations.
Fig. 5: System-level benchmarking results.
Fig. 6: Hardware demonstration of the integration of Mott ReLU devices and a synaptic array.

Similar content being viewed by others

Data availability

The data that support the plots and other results of this paper are available from the corresponding author upon request.

Code availability

The software codes used for this study are available from the corresponding author upon request.

References

  1. Wong, H. S. P. et al. Metal–oxide RRAM. Proc. IEEE 100, 1951–1970 (2012).

    Article  CAS  Google Scholar 

  2. Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).

    Article  Google Scholar 

  3. Kang, D.-H. et al. A neuromorphic device implemented on a salmon-DNA electrolyte and its application to artificial neural networks. Adv. Sci. 6, 1901265 (2019).

    Article  Google Scholar 

  4. Ge, R. et al. Atomristor: nonvolatile resistance switching in atomic sheets of transition metal dichalcogenides. Nano Lett. 18, 434–441 (2018).

    Article  CAS  Google Scholar 

  5. van de Burgt, Y. et al. A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing. Nat. Mater. 16, 414–418 (2017).

    Article  CAS  Google Scholar 

  6. Zhao, X. et al. Confining cation injection to enhance CBRAM performance by nanopore graphene layer. Small 13, 1603948 (2017).

    Article  Google Scholar 

  7. Chakrabarti, B. et al. A multiply-add engine with monolithically integrated 3D memristor crossbar/CMOS hybrid circuit. Sci. Rep. 7, 42429 (2017).

    Article  CAS  Google Scholar 

  8. Kim, S. et al. Binarized neural network with silicon nanosheet synaptic transistors for supervised pattern classification. Sci. Rep. 9, 11705 (2019).

    Article  Google Scholar 

  9. Oh, S., Huang, Z., Shi, Y. & Kuzum, D. The impact of resistance drift of phase change memory (PCM) synaptic devices on artificial neural network performance. IEEE Electron Device Lett. 40, 1325–1328 (2019).

    Article  CAS  Google Scholar 

  10. He, K., Zhang, X., Ren, S. & Sun J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, 2016).

  11. Kataeva, I. et al. Towards the development of analog neuromorphic chip prototype with 2.4M integrated memristors. In Proc. IEEE International Symposium on Circuits and Systems (ISCAS) 255–259 (IEEE, 2019).

  12. Gao, B. et al. Ultra-low-energy three-dimensional oxide-based electronic synapses for implementation of robust high-accuracy neuromorphic computation systems. ACS Nano 8, 6998–7004 (2014).

    Article  CAS  Google Scholar 

  13. Yang, T.-J. & Sze, V. Design considerations for efficient deep neural networks on processing-in-memory accelerators. In Proc. IEEE International Electron Devices Meeting (IEDM) 514–517 (IEEE, 2019).

  14. Krestinskaya, O., Salama, K. N. & James, A. P. Learning in memristive neural network architectures using analog backpropagation circuits. IEEE Trans. Circuits Syst. I 66, 719–732 (2019).

    Article  Google Scholar 

  15. Giordano, M. et al. Analog-to-digital conversion with reconfigurable function mapping for neural networks activation function acceleration. IEEE J. Emerg. Sel. Top. Circuits Syst. 9, 367–376 (2019).

    Article  Google Scholar 

  16. Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 558, 60–67 (2018).

    Article  CAS  Google Scholar 

  17. Stefanovich, G., Pergament, A. & Stefanovich, D. Electrical switching and Mott transition in VO2. J. Phys. Condens. Matter 12, 8837–8845 (2000).

    Article  CAS  Google Scholar 

  18. Qazilbash, M. M. et al. Mott transition in VO2 revealed by infrared spectroscopy and nano-imaging. Science 318, 1750–1753 (2007).

    Article  CAS  Google Scholar 

  19. del Valle, J. et al. Subthreshold firing in Mott nanodevices. Nature 569, 388–392 (2019).

    Article  Google Scholar 

  20. Madan, H., Jerry, M., Pogrebnyakov, A., Mayer, T. & Datta, S. Quantitative mapping of phase coexistence in Mott-Peierls insulator during electronic and thermally driven phase transition. ACS Nano 9, 2009–2017 (2015).

    Article  CAS  Google Scholar 

  21. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).

    Article  Google Scholar 

  22. Radu, I. P. et al. Switching mechanism in two-terminal vanadium dioxide devices. Nanotechnology 26, 165202 (2015).

    Article  Google Scholar 

  23. del Valle, J., Salev, P., Kalcheim, Y. & Schuller, I. K. A caloritronics-based Mott neuristor. Sci. Rep. 10, 4292 (2020).

    Article  Google Scholar 

  24. Shi, Y. et al. Neuroinspired unsupervised learning and pruning with subquantum CBRAM arrays. Nat. Commun. 9, 5312 (2018).

    Article  CAS  Google Scholar 

  25. Chen, P.-Y., Peng, X. & Yu, S. NeuroSim: a circuit-level macro model for benchmarking neuro-inspired architectures in online learning. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 37, 3067–3080 (2018).

    Article  Google Scholar 

  26. Shrivakshan, G. & Chandrasekar, C. A comparison of various edge detection techniques used in image processing. Int. J. Comput. Sci. Issues 9, 269–276 (2012).

    Google Scholar 

  27. Zhao, W. & Cao, Y. Predictive technology model for nano-CMOS design exploration. ACM J. Emerg. Technol. Comput. Syst. 3, https://doi.org/10.1145/1229175.1229176 (2007).

  28. Zhao, W. & Cao, Y. New generation of predictive technology model for sub-45 nm early design exploration. IEEE Trans. Electron Devices 53, 2816–2823 (2006).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

This work was supported by Office of Naval Research (N000142012405 and N00014162531), Samsung Electronics, the National Science Foundation (ECCS-1752241, ECCS-2024776 and ECCS-1734940), the National Institutes of Health (R21 EY029466, R21 EB026180 and DP2 EB030992) and Qualcomm Fellowship. The experimental aspects of this work were supported as part of the Quantum Materials for Energy Efficient Neuromorphic Computing (Q-MEEN-C) Energy Frontier Research Center (EFRC), funded by the US Department of Energy, Office of Science, Basic Energy Sciences under award #DE-SC0019273. The fabrication of the devices was performed at the San Diego Nanotechnology Infrastructure (SDNI) of the University of California San Diego, supported by the National Science Foundation (ECCS-1542148).

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, University of California San Diego, La Jolla, CA, USA

    Sangheon Oh, Yuhan Shi, Yichen Lu, Zhisheng Huang & Duygu Kuzum

  2. Department of Physics, University of California San Diego, La Jolla, CA, USA

    Javier del Valle, Pavel Salev, Yoav Kalcheim & Ivan K. Schuller

Authors
  1. Sangheon Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuhan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Javier del Valle
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pavel Salev
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yichen Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhisheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yoav Kalcheim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ivan K. Schuller
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Duygu Kuzum
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.O., Y.S., I.K.S. and D.K. conceived the idea. S.O. developed the SPICE model and performed SPICE and network simulations. S.O., Y.S. and Z.H. performed the system-level benchmarking simulations. P.S., J.d.V. and Y.K. fabricated and measured the electrical characteristics of the Mott ReLU devices under the supervision of I.K.S.; S.O. and Y.L. fabricated the CBRAM array. Y.S. designed the PCB. S.O. and Y.S. performed the measurements for the CBRAM array and Mott ReLU device integration. All the authors discussed the results and contributed to the writing of the manuscript. I.K.S. and D.K. supervised the work.

Corresponding author

Correspondence to Duygu Kuzum.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Nanotechnology thanks Jinfeng Kang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–6, Table 1 and Notes 1–7.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Oh, S., Shi, Y., del Valle, J. et al. Energy-efficient Mott activation neuron for full-hardware implementation of neural networks. Nat. Nanotechnol. 16, 680–687 (2021). https://doi.org/10.1038/s41565-021-00874-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41565-021-00874-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing