Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition

Abstract

With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important1. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Retina-inspired 2D retinomorphic hardware for MDR.
Fig. 2: Photoconductivity properties of 2D retinomorphic devices for MDR.
Fig. 3: Illustration of motion detection based on the 2D retinomorphic hardware.
Fig. 4: 2D retinomorphic hardware implementation for trichromatic trolley MDR.

Data availability

The data that support the findings of this study are available from the corresponding authors on reasonable request. Source data are provided with this paper.

Code availability

The codes used for the simulation are available from the corresponding author on reasonable request.

References

  1. 1.

    Choo, K. D. et al. Energy-efficient low-noise CMOS image sensor with capacitor array-assisted charge-injection SAR ADC for motion-triggered low-power IoT application. In Proc. IEEE International Solid-State Circuits Conference (ISSCC) 96–98 (IEEE, 2019).

  2. 2.

    Chen, T. & Lu, S. Object-level motion detection from moving cameras. IEEE Trans. Circuits Syst. Video Technol. 27, 2333–2343 (2016).

    Article  Google Scholar 

  3. 3.

    Yang, S., Kim, K., Kim, E., Baek, K. & Kim, S. An ultra low power CMOS motion detector. IEEE Trans. Consum. Electron. 55, 2425–2430 (2009).

    Article  Google Scholar 

  4. 4.

    Zhao, B., Zhang, X., Chen, S., Low, K. & Zhuang, H. A 64×64 CMOS image sensor with on-chip moving object detection and localization. IEEE Trans. Circuits Syst. Video Technol. 22, 581–588 (2012).

    Article  Google Scholar 

  5. 5.

    Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).

    Article  Google Scholar 

  6. 6.

    Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).

    CAS  Article  Google Scholar 

  7. 7.

    Boahen, K. A retinomorphic chip with parallel pathways: encoding INCREASING, ON, DECREASING, and OFF visual signals. AICSP 30, 121–135 (2002).

    Google Scholar 

  8. 8.

    Trujillo Herrera, C. & Labram, J. G. A perovskite retinomorphic sensor. Appl. Phys. Lett. 117, 233501 (2020).

    CAS  Article  Google Scholar 

  9. 9.

    Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

    CAS  Article  Google Scholar 

  10. 10.

    Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat. Commun. 11, 5934 (2020).

    CAS  Article  Google Scholar 

  11. 11.

    Wang, S. et al. A MoS2/PTCDA hybrid heterojunction synapse with efficient photoelectric dual modulation and versatility. Adv. Mater. 31, 1806227 (2019).

    Article  Google Scholar 

  12. 12.

    Hou, X. et al. A logic-memory transistor with the integration of visible information sensing-memory-processing. Adv. Sci. 7, 2002072 (2020).

    CAS  Article  Google Scholar 

  13. 13.

    Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    CAS  Article  Google Scholar 

  14. 14.

    Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B. & Delbruck, T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484 (2014).

    Article  Google Scholar 

  15. 15.

    Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).

    CAS  Article  Google Scholar 

  16. 16.

    Masland, R. H. The neuronal organization of the retina. Neuron 76, 266–280 (2012).

    CAS  Article  Google Scholar 

  17. 17.

    Dowling, J. & Boycott, B. B. Organization of the primate retina: electron microscopy. Proc. R. Soc. Lond. Ser. B. Biol. Sci. 166, 80–111 (1966).

    CAS  Article  Google Scholar 

  18. 18.

    Demb, J. B. Cellular mechanisms for direction selectivity in the retina. Neuron 55, 179–186 (2007).

    CAS  Article  Google Scholar 

  19. 19.

    Liu, C. et al. Two-dimensional materials for next-generation computing technologies. Nat. Nanotechnol. 15, 545–557 (2020).

    CAS  Article  Google Scholar 

  20. 20.

    Wang, S. et al. Two-dimensional ferroelectric channel transistors integrating ultra-fast memory and neural computing. Nat. Commun. 12, 53 (2021).

    CAS  Article  Google Scholar 

  21. 21.

    Ölveczky, B. P., Baccus, S. A. & Meister, M. Segregation of object and background motion in the retina. Nature 423, 401–408 (2003).

    Article  Google Scholar 

  22. 22.

    Uyttendaele, M., Eden, A. & Skeliski, R. Eliminating ghosting and exposure artifacts in image mosaics. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) Vol. 2, II–II (IEEE, 2001).

  23. 23.

    Yan, B., Liu, C., Liu, X., Chen, Y. & Li, H. Understanding the trade-offs of device, circuit and application in ReRAM-based neuromorphic computing systems. In Proc. IEEE International Electron Devices Meeting (IEDM) 11.4.1–11.4.4 (IEEE, 2017).

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (grant nos. 61925402, 61851402, 61725505, 11734016 and 62090032), Science and Technology Commission of Shanghai Municipality (grant nos. 19JC1416600 and 21JC1406100) and Shanghai Education Development Foundation and Shanghai Municipal Education Commission Shuguang Program (grant no. 18SG01).

Author information

Affiliations

Authors

Contributions

Z.Z. and S.W. designed and conducted the experiments. P.Z. and W.H. conceived the idea. R.X. supported the characterization of materials. C.L. provided assistance with mechanism analysis and discussion. S.W. and Z.Z. wrote the manuscript and all authors contributed to the revision of the paper.

Corresponding authors

Correspondence to Weida Hu or Peng Zhou.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Nanotechnology thanks Han Zhang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Sections 1–11 containing Supplementary Figs. 1–11 and corresponding discussions.

Source data

Source Data Fig. 2

Source data of Fig. 2.

Source Data Fig. 3

Source data of Fig. 3.

Source Data Fig. 4

Source data of Fig. 4.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Wang, S., Liu, C. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. (2021). https://doi.org/10.1038/s41565-021-01003-1

Download citation

Search

Quick links

Find nanotechnology articles, nanomaterial data and patents all in one place. Visit Nano by Nature Research