Abstract
With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important1. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding authors on reasonable request. Source data are provided with this paper.
Code availability
The codes used for the simulation are available from the corresponding author on reasonable request.
References
Choo, K. D. et al. Energy-efficient low-noise CMOS image sensor with capacitor array-assisted charge-injection SAR ADC for motion-triggered low-power IoT application. In Proc. IEEE International Solid-State Circuits Conference (ISSCC) 96–98 (IEEE, 2019).
Chen, T. & Lu, S. Object-level motion detection from moving cameras. IEEE Trans. Circuits Syst. Video Technol. 27, 2333–2343 (2016).
Yang, S., Kim, K., Kim, E., Baek, K. & Kim, S. An ultra low power CMOS motion detector. IEEE Trans. Consum. Electron. 55, 2425–2430 (2009).
Zhao, B., Zhang, X., Chen, S., Low, K. & Zhuang, H. A 64×64 CMOS image sensor with on-chip moving object detection and localization. IEEE Trans. Circuits Syst. Video Technol. 22, 581–588 (2012).
Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).
Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).
Boahen, K. A retinomorphic chip with parallel pathways: encoding INCREASING, ON, DECREASING, and OFF visual signals. AICSP 30, 121–135 (2002).
Trujillo Herrera, C. & Labram, J. G. A perovskite retinomorphic sensor. Appl. Phys. Lett. 117, 233501 (2020).
Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat. Commun. 11, 5934 (2020).
Wang, S. et al. A MoS2/PTCDA hybrid heterojunction synapse with efficient photoelectric dual modulation and versatility. Adv. Mater. 31, 1806227 (2019).
Hou, X. et al. A logic-memory transistor with the integration of visible information sensing-memory-processing. Adv. Sci. 7, 2002072 (2020).
Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).
Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B. & Delbruck, T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484 (2014).
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).
Masland, R. H. The neuronal organization of the retina. Neuron 76, 266–280 (2012).
Dowling, J. & Boycott, B. B. Organization of the primate retina: electron microscopy. Proc. R. Soc. Lond. Ser. B. Biol. Sci. 166, 80–111 (1966).
Demb, J. B. Cellular mechanisms for direction selectivity in the retina. Neuron 55, 179–186 (2007).
Liu, C. et al. Two-dimensional materials for next-generation computing technologies. Nat. Nanotechnol. 15, 545–557 (2020).
Wang, S. et al. Two-dimensional ferroelectric channel transistors integrating ultra-fast memory and neural computing. Nat. Commun. 12, 53 (2021).
Ölveczky, B. P., Baccus, S. A. & Meister, M. Segregation of object and background motion in the retina. Nature 423, 401–408 (2003).
Uyttendaele, M., Eden, A. & Skeliski, R. Eliminating ghosting and exposure artifacts in image mosaics. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) Vol. 2, II–II (IEEE, 2001).
Yan, B., Liu, C., Liu, X., Chen, Y. & Li, H. Understanding the trade-offs of device, circuit and application in ReRAM-based neuromorphic computing systems. In Proc. IEEE International Electron Devices Meeting (IEDM) 11.4.1–11.4.4 (IEEE, 2017).
Acknowledgements
This work was supported by the National Natural Science Foundation of China (grant nos. 61925402, 61851402, 61725505, 11734016 and 62090032), Science and Technology Commission of Shanghai Municipality (grant nos. 19JC1416600 and 21JC1406100) and Shanghai Education Development Foundation and Shanghai Municipal Education Commission Shuguang Program (grant no. 18SG01).
Author information
Authors and Affiliations
Contributions
Z.Z. and S.W. designed and conducted the experiments. P.Z. and W.H. conceived the idea. R.X. supported the characterization of materials. C.L. provided assistance with mechanism analysis and discussion. S.W. and Z.Z. wrote the manuscript and all authors contributed to the revision of the paper.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Nature Nanotechnology thanks Han Zhang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Sections 1–11 containing Supplementary Figs. 1–11 and corresponding discussions.
Source data
Source Data Fig. 2
Source data of Fig. 2.
Source Data Fig. 3
Source data of Fig. 3.
Source Data Fig. 4
Source data of Fig. 4.
Rights and permissions
About this article
Cite this article
Zhang, Z., Wang, S., Liu, C. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022). https://doi.org/10.1038/s41565-021-01003-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41565-021-01003-1
This article is cited by
-
Non-volatile rippled-assisted optoelectronic array for all-day motion detection and recognition
Nature Communications (2024)
-
High performance artificial visual perception and recognition with a plasmon-enhanced 2D material neural network
Nature Communications (2024)
-
Photogating-assisted tunneling boosts the responsivity and speed of heterogeneous WSe2/Ta2NiSe5 photodetectors
Nature Communications (2024)
-
In-sensor dynamic computing for intelligent machine vision
Nature Electronics (2024)
-
Bioinspired sensing-memory-computing integrated vision systems: biomimetic mechanisms, design principles, and applications
Science China Information Sciences (2024)