Complementary metal–oxide–semiconductor (CMOS) image sensors allow machines to interact with the visual world. In these sensors, image capture in front-end silicon photodiode arrays is separated from back-end image processing. To reduce the energy cost associated with transferring data between the sensing and computing units, in-sensor computing approaches are being developed where images are processed within the photodiode arrays. However, such methods require electrostatically doped photodiodes where photocurrents can be electrically modulated or programmed, and this is challenging in current CMOS image sensors that use chemically doped silicon photodiodes. Here we report in-sensor computing using electrostatically doped silicon photodiodes. We fabricate thousands of dual-gate silicon p–i–n photodiodes, which can be integrated into CMOS image sensors, at the wafer scale. With a 3 × 3 network of the electrostatically doped photodiodes, we demonstrate in-sensor image processing using seven different convolutional filters electrically programmed into the photodiode network.
This is a preview of subscription content, access via your institution
Subscribe to Nature+
Get immediate online access to the entire Nature family of 50+ journals
Subscribe to Journal
Get full journal access for 1 year
only $9.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Source data are provided with this paper. The data that support the other findings of this study are available from the corresponding authors upon reasonable request.
Experimental code is available from the corresponding authors on reasonable request.
Kwon, M. et al. A low-power 65/14 nm stacked CMOS image sensor. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS) 1–4 (IEEE, 2020).
Park, J. et al. 7.9 1/2.74-inch 32Mpixel-prototype CMOS image sensor with 0.64 μm unit pixels separated by full-depth deep-trench isolation. In 2021 IEEE International Solid-State Circuits Conference (ISSCC) Vol. 64, 122–124 (IEEE, 2021).
Chai, Y. In-sensor computing for machine vision. Nature 579, 32–33 (2020).
Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).
Sonka, M., Hlavac, V. & Boyle, R. Image Processing, Analysis and Machine Vision 2nd edn (Springer, 2014).
Zhan, C., Duan, X., Xu, S., Song, Z. & Luo, M. An improved moving object detection algorithm based on frame difference and edge detection. In Fourth International Conference on Image and Graphics (ICIG 2007) 519–523 (IEEE, 2007).
Hussin, R., Juhari, M. R., Kang, N. W., Ismail, R. C. & Kamarudin, A. Digital image processing techniques for object detection from complex background image. Procedia Eng. 41, 340–344 (2012).
Beresnev, P. et al. Automated driving system based on roadway and traffic conditions monitoring. In Proc. 4th International Conference on Vehicle Technology and Intelligent Transport Systems 363–370 (VEHITS, 2018).
Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).
Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).
Wang, C. Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).
Baugher, B. W. H., Churchill, H. O. H., Yang, Y. & Jarillo-Herrero, P. Optoelectronic devices based on electrically tunable p–n diodes in a monolayer dichalcogenide. Nat. Nanotechnol. 9, 262–267 (2014).
Pospischil, A., Furchi, M. M. & Mueller, T. Solar-energy conversion and light emission in an atomic monolayer p–n diode. Nat. Nanotechnol. 9, 257–261 (2014).
Lee, C.-H. et al. Atomically thin p–n junctions with van der Waals heterointerfaces. Nat. Nanotechnol. 9, 676–681 (2014).
Dennis, V. C. et al. 2022 roadmap on neuromorphic computing and engineering. Neuromorph. Comput. Eng. 2, 022501 (2021).
Liao, F., Zhou, F. & Chai, Y. Neuromorphic vision sensors: principle, progress and perspectives. J. Semicond. 42, 013105 (2021).
Sze, S. M., Li, Y. & Ng, K. K. Physics of Semiconductor Devices (John Wiley & Sons, 2021).
Xu, K. Silicon electro-optic micro-modulator fabricated in standard CMOS technology as components for all silicon monolithic integrated optoelectronic systems. J. Micromech. Microeng. 31, 054001 (2021).
Wu, K., Zhang, H., Chen, Y., Luo, Q. & Xu, K. All-silicon microdisplay using efficient hot-carrier electroluminescence in standard 0.18 µm CMOS technology. IEEE Electron Device Lett. 42, 541–544 (2021).
Signal and Image Processing Institute (USC-SIPI) Image Database (Univ. Southern California, accessed 22 April 2021); https://sipi.usc.edu/database/
Yiyang, Z. The design of glass crack detection system based on image preprocessing technology. In 2014 IEEE 7th Joint International Information Technology and Artificial Intelligence Conference 39–42 (IEEE, 2014).
Kanopoulos, N., Vasanthavada, N. & Baker, R. L. Design of an image edge detection filter using the Sobel operator. IEEE J. Solid-State Circuits 23, 358–367 (1988).
Gedraite, E. S. & Hadad, M. Investigation on the effect of a Gaussian blur in image filtering and segmentation. In Proc. ELMAR-2011 393–396 (IEEE, 2011).
Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32, e2002431 (2020).
Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nat. Electron. 1, 52–59 (2018).
Anila, S. & Devarajan, N. Preprocessing technique for face recognition applications under varying illumination conditions. Glob. J. Comput. Sci. Technol. XII, 12–18 (2012).
Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).
Wang, Z. et al. Resistive switching materials for information processing. Nat. Rev. Mater. 5, 173–195 (2020).
Ham, D., Park, H., Hwang, S. & Kim, K. Neuromorphic electronics based on copying and pasting the brain. Nat. Electron. 4, 635–644 (2021).
Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, eabg1455 (2021).
Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5, 84–91 (2022).
We acknowledge support by the Samsung Advanced Institute of Technology (SAIT) under Contract A30216 and by the National Science Foundation (NSF) Science and Technology Center for Integrated Quantum Materials under Contract DMR-1231319. Device fabrication was performed in part at the Harvard Center for Nanoscale Systems (CNS), which is supported by the NSF under Contract 1541959.
The authors declare no competing interests.
Peer review information
Nature Electronics thanks Yang Chai and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Jang, H., Hinton, H., Jung, WB. et al. In-sensor optoelectronic computing using electrostatically doped silicon. Nat Electron 5, 519–525 (2022). https://doi.org/10.1038/s41928-022-00819-6