Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Ultrafast machine vision with 2D material neural network image sensors

## Abstract

Machine vision technology has taken huge leaps in recent years, and is now becoming an integral part of various intelligent systems, including autonomous vehicles and robotics. Usually, visual information is captured by a frame-based camera, converted into a digital format and processed afterwards using a machine-learning algorithm such as an artificial neural network (ANN)1. The large amount of (mostly redundant) data passed through the entire signal chain, however, results in low frame rates and high power consumption. Various visual data preprocessing techniques have thus been developed2,3,4,5,6,7 to increase the efficiency of the subsequent signal processing in an ANN. Here we demonstrate that an image sensor can itself constitute an ANN that can simultaneously sense and process optical images without latency. Our device is based on a reconfigurable two-dimensional (2D) semiconductor8,9 photodiode10,11,12 array, and the synaptic weights of the network are stored in a continuously tunable photoresponsivity matrix. We demonstrate both supervised and unsupervised learning and train the sensor to classify and encode images that are optically projected onto the chip with a throughput of 20 million bins per second.

## Access options

from\$8.99

All prices are NET prices.

## Data availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

## References

1. 1.

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

2. 2.

Mead, C. A. & Mahowald, M. A. A silicon model of early visual processing. Neural Netw. 1, 91–97 (1988).

3. 3.

Lichtsteiner, P., Posch, C. & Delbruck, T. A. 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits 43, 566–576 (2008).

4. 4.

Cottini, N., Gottardi, M., Massari, N., Passerone, R. & Smilansky, Z. A. 33 μW 64×64 pixel vision sensor embedding robust dynamic background subtraction for EVENT detection and scene interpretation. IEEE J. Solid-State Circuits 48, 850–863 (2013).

5. 5.

Kyuma, K. et al. Artificial retinas—fast, versatile image processors. Nature 372, 197–198 (1994).

6. 6.

Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B. & Delbruck, T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484 (2014).

7. 7.

Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

8. 8.

Manzeli, S., Ovchinnikov, D., Pasquier, D., Yazyev, O. V. & Kis, A. 2D transition metal dichalcogenides. Nat. Rev. Mater. 2, 17033 (2017).

9. 9.

Mueller, T. & Malic, E. Exciton physics and device application of two-dimensional transition metal dichalcogenide semiconductors. npj 2D Mater. Appl. 2, 29 (2018).

10. 10.

Pospischil, A., Furchi, M. M. & Mueller, T. Solar-energy conversion and light emission in an atomic monolayer p–n diode. Nat. Nanotechnol. 9, 257–261 (2014).

11. 11.

Baugher, B. W., Churchill, H. O., Yang, Y. & Jarillo-Herrero, P. Optoelectronic devices based on electrically tunable p–n diodes in a monolayer dichalcogenide. Nat. Nanotechnol. 9, 262–267 (2014).

12. 12.

Ross, J. S. et al. Electrically tunable excitonic light-emitting diodes based on monolayer WSe2 p–n junctions. Nat. Nanotechnol. 9, 268–272 (2014).

13. 13.

Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).

14. 14.

Sheridan, P. M. et al. Sparse coding with memristor networks. Nat. Nanotechnol. 12, 784–789 (2017).

15. 15.

Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nat. Electron. 1, 52–59 (2018).

16. 16.

Kim, K. H. et al. A functional hybrid memristor crossbar-array/CMOS system for data storage and neuromorphic applications. Nano Lett. 12, 389–395 (2012).

17. 17.

Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).

18. 18.

Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

19. 19.

Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).

20. 20.

Psaltis, D., Brady, D., Gu, X. G. & Lin, S. Holography in artificial neural networks. Nature 343, 325–330 (1990).

21. 21.

Kolb, H. How the retina works: much of the construction of an image takes place in the retina itself through the use of specialized neural circuits. Am. Sci. 91, 28–35 (2003).

22. 22.

Jeong, K.-H., Kim, J. & Lee, L. P. Biologically inspired artificial compound eyes. Science 312, 557–561 (2006).

23. 23.

Choi, C. et al. Human eye-inspired soft optoelectronic device using high-density MoS2-graphene curved image sensor array. Nat. Commun. 8, 1664 (2017).

24. 24.

Schwarte, R. et al. New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device (PMD). In Proc. SPIE Sensors, Sensor Systems, and Sensor Data Processing Vol. 3100, 245–253 (SPIE, 1997).

25. 25.

Sugeta, T., Urisu, T., Sakata, S. & Mizushima, Y. Metal-semiconductor-metal photodetector for high-speed optoelectronic circuits. Jpn. J. Appl. Phys. 19, 459 (1980).

26. 26.

Wachter, S., Polyushkin, D. K., Bethge, O. & Mueller, T. A microprocessor based on a two-dimensional semiconductor. Nat. Commun. 8, 14948 (2017).

27. 27.

Goossens, S. et al. Broadband image sensor array based on graphene–CMOS integration. Nat. Photon. 11, 366–371 (2017).

28. 28.

Bishop, C. M. Training with noise is equivalent to Tikhonov regularization. Neural Comput. 7, 108–116 (1995).

29. 29.

Bengio, Y. in Neural Networks: Tricks of the Trade Vol. 7700 (eds Montavon G. et al.) 437–478 (Springer, 2012).

30. 30.

Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

31. 31.

Massicotte, M. et al. Dissociation of two-dimensional excitons in monolayer WSe2. Nat. Commun. 9, 1633 (2018).

32. 32.

Li, D. et al. Two-dimensional non-volatile programmable p–n junctions. Nat. Nanotechnol. 12, 901–906 (2017).

33. 33.

Lv, L. et al. Reconfigurable two-dimensional optoelectronic devices enabled by local ferroelectric polarization. Nat. Commun. 10, 3331 (2019).

34. 34.

Bertolazzi, S., Krasnozhon, D. & Kis, A. Nonvolatile memory cells based on MoS2/graphene heterostructures. ACS Nano 7, 3246–3252 (2013).

35. 35.

Salakhutdinov, R. & Hinton, G. Semantic hashing. Int. J. Approx. Reason. 50, 969–978 (2009).

36. 36.

Castellanos-Gomez, A. et al. Deterministic transfer of two-dimensional materials by all-dry viscoelastic stamping. 2D Mater. 1, 011002 (2014).

37. 37.

Gerchberg, R. W. & Saxton, W. O. A practical algorithm for the determination of the phase from image and diffraction plane pictures. Optik 35, 237–246 (1972).

## Acknowledgements

We thank B. Limbacher for discussions and A. Kleinl, M. Paur and F. Dona for technical assistance. We acknowledge financial support by the Austrian Science Fund FWF (START Y 539-N16) and the European Union (grant agreement number 785219 Graphene Flagship and Marie Sklodowska-Curie Individual Fellowship OPTOvanderWAALS, grant ID 791536).

## Author information

Authors

### Contributions

T.M. conceived the experiment. L.M. designed and built the experimental setup, programmed the machine-learning algorithms, carried out the measurements and analysed the data. J.S. fabricated the ANN vision sensor. S.W. and D.K.P. contributed to the sample fabrication. A.J.M.-M. fabricated and characterized the floating-gate detector. L.M., J.S. and T.M. prepared the manuscript. All authors discussed the results and commented on the manuscript.

### Corresponding authors

Correspondence to Lukas Mennel or Thomas Mueller.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Peer review information Nature thanks Yang Chai, Frank Koppens and Sangyoun Lee for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Extended data figures and tables

### Extended Data Fig. 1 Photodiode array uniformity.

Gate tunability of the responsivities of all 27 photodetectors. One of the detector elements (pixel 7, subpixel 2) did not show any response to light (due to a broken electrical wire), which, however, had no crucial influence on the overall system performance.

### Extended Data Fig. 2 Photodiode characteristics.

a, Current–voltage characteristic curve under dark (blue) and illuminated (green) conditions. The series resistance Rs and shunt resistance Rsh are ~106 Ω and 109 Ω, respectively. For zero-bias operation, we estimate a noise-equivalent power of NEP = Ith/R ≈ 10−13 W Hz−1/2, where R ≈ 60 mA W−1 is the (maximum) responsivity and $${I}_{{\rm{th}}}=\sqrt{4{k}_{{\rm{B}}}T\Delta f/{R}_{{\rm{sh}}}}$$ the thermal noise, where kB is the Boltzmann constant, Δf is the bandwidth and T is the temperature. b, Dependence of the short-circuit photocurrent on the light intensity for different split-gate voltages. Importantly, the response is linear (I P), as assumed in equation (1).

### Extended Data Fig. 3

Circuit of the ANN photodiode array.

### Extended Data Fig. 4 Experimental setup.

a, Experimental setup for training the classifier and the autoencoder. CW, continuous wave. b, Experimental setup for time-resolved measurements. TIA, transimpedance amplifier. A pulse generator triggers the pulsed laser as well as the oscilloscope. c, Photograph of the optical setup (for schematic see Fig. 2d). d, Flow chart of the training algorithm. The blue shaded boxes are interactions with the ANN photodiode array.

### Extended Data Fig. 5 Classifier training.

Photoresponsivity values of all 27 photodetectors with σ = 0.3 training data. a, b, Initial (a) and epoch 30 (b) responsivity values. The weights for the σ = 0.2 and σ = 0.4 training data are similar. c, Measured currents over all epochs for a specific projected letter and at all three noise levels. d, Histogram of the initial and final responsivity values for the three different noise levels.

### Extended Data Fig. 6 Comparison with computer simulation.

Classifier training of the analogue vision sensor (solid lines) and simulation of the system on a computer (dashed lines) for different data noise levels σ. The same ANN architecture, input data, effective learning rate and starting weights have been used. The same accuracy and loss are eventually reached after training. The slightly slower convergence of the analogue implementation compared with the simulation reflects the nonidealities (defective subpixel, device-to-device variations) of the former. Further discussion on the impact of nonidealities is provided in Extended Data Fig. 10.

### Extended Data Fig. 7 Training datasets.

a, b, Dataset of 30 epochs of classifier (a) and autoencoder training (b) with a test data noise level of σ = 0.4 and σ = 0.15 respectively.

### Extended Data Fig. 8 Autoencoder photoresponsivities/weights.

a, b, Initial (a) and epoch 30 (b) encoder photoresponsivity values (left) and decoder weights (right).

### Extended Data Fig. 9 Floating-split-gate photodiode with memory.

a, Schematic of the floating gate photodiode. The addition of 2-nm-thick Au layers, sandwiched between Al2O3 and hexagonal boron nitride (hBN), enables the storage of electric charge when a gate voltage is applied to the device, acting as a floating-gate memory. b, Electronic characteristic curves of the photodiode operated in p–n, n–p and p–p configurations. c, The ability of the device to ‘remember’ the previous configuration can be verified from the time-resolved photocurrent measurement. The measurement is performed as follows: the back-gate voltages are set to VG1 = +5 V and VG2 = −5 V and are then disconnected, that is, there is no longer an applied gate voltage and the only electric field is that generated by the charge stored on the floating electrodes. The short-circuit photocurrent is then measured upon optical illumination. The light is then switched off, at ~1,100 s, with a corresponding drop of the photocurrent to zero. After ~1,600 s, the light is switched on again, causing the current to reach its initial value, and then a smaller value when the intensity of the light is reduced (~1,700 s). After ~2,300 s, the opposite voltage configuration is applied to the back gates (VG1 = −5 V and VG2 = +5 V), inducing a polarity inversion that also remains permanent. Now, a positive photocurrent (red line) is obtained.

### Extended Data Fig. 10 Robustness of the network.

a, Detector uniformity, extracted from Extended Data Fig. 1. The fitted Gaussian probability distribution has a standard deviation of σ = 0.205 (40 mA W−1 V−1). b, Monte Carlo simulation of a vision sensor with detector responsivities of a given standard deviation. (The photodetectors of the actual device have a measured photoresponsivity standard deviation of 0.205.) Trained on the MNIST database of handwritten digits, the classifier has 784 pixels and 10 subpixels per pixel. For each data point, 50 random photoresponsivity variations were evaluated. c, Accuracy dependence on the number of (randomly chosen) defective subpixels. The same ANN and Monte Carlo simulation scheme as in b were used. For each data point, 50 random sets of modified photoresponsivities were evaluated.

## Rights and permissions

Reprints and Permissions

Mennel, L., Symonowicz, J., Wachter, S. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020). https://doi.org/10.1038/s41586-020-2038-x

• Accepted:

• Published:

• Issue Date:

• ### Highlighting photonics: looking into the next decade

• Zhigang Chen
•  & Mordechai Segev

eLight (2021)

• ### Bioinspired multisensory neural network with crossmodal integration and recognition

• Hongwei Tan
• , Yifan Zhou
• , Quanzheng Tao
• , Johanna Rosen
•  & Sebastiaan van Dijken

Nature Communications (2021)

• ### Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

• Tiankuang Zhou
• , Xing Lin
• , Jiamin Wu
• , Yitong Chen
• , Hao Xie
• , Yipeng Li
• , Jingtao Fan
• , Huaqiang Wu
• , Lu Fang
•  & Qionghai Dai

Nature Photonics (2021)

• ### A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems

• Qian-Bing Zhu
• , Bo Li
• , Dan-Dan Yang
• , Chi Liu
• , Shun Feng
• , Mao-Lin Chen
• , Yun Sun
• , Ya-Nan Tian
• , Xin Su
• , Xiao-Mu Wang
• , Song Qiu
• , Qing-Wen Li
• , Xiao-Ming Li
• , Hai-Bo Zeng
• , Hui-Ming Cheng
•  & Dong-Ming Sun

Nature Communications (2021)

• ### Weighing in on photonic-based machine learning for automotive mobility

• Sean Phillip Rodrigues
• , Ziqi Yu
• , Paul Schmalenberg
• , Jae Lee
• , Hideo Iizuka
•  & Ercan M. Dede

Nature Photonics (2021)