Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Multichannel meta-imagers for accelerating machine vision

Nature Nanotechnology volume 19, pages 471–478 (2024)Cite this article

Subjects

Abstract

Rapid developments in machine vision technology have impacted a variety of applications, such as medical devices and autonomous driving systems. These achievements, however, typically necessitate digital neural networks with the downside of heavy computational requirements and consequent high energy consumption. As a result, real-time decision-making is hindered when computational resources are not readily accessible. Here we report a meta-imager designed to work together with a digital back end to offload computationally expensive convolution operations into high-speed, low-power optics. In this architecture, metasurfaces enable both angle and polarization multiplexing to create multiple information channels that perform positively and negatively valued convolution operations in a single shot. We use our meta-imager for object classification, achieving 98.6% accuracy in handwritten digits and 88.8% accuracy in fashion images. Owing to its compactness, high speed and low power consumption, our approach could find a wide range of applications in artificial intelligence and machine vision applications.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic of the meta-imager.
Fig. 2: Meta-optic architecture.
Fig. 3: Design of the meta-imager.
Fig. 4: Fabrication and characterization of the meta-imager.
Fig. 5: Classification of MNIST and Fashion MNIST objects.

Similar content being viewed by others

Data availability

The data that support the findings of this study are available in the Article and its Supplementary Information and/or are available from the corresponding author upon reasonable request.

References

  1. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations 1–14 (ICLR, 2015).

  2. Wang, G. et al. Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans. Med. Imaging 37, 1562–1573 (2018).

    PubMed  Google Scholar 

  3. Furui, S., Deng, L., Gales, M., Ney, H. & Tokuda, K. Fundamental technologies in modern speech recognition. IEEE Signal Process Mag. 29, 16–17 (2012).

    Google Scholar 

  4. Sak, H., Senior, A., Rao, K. & Beaufays, F. Fast and accurate recurrent neural network acoustic models for speech recognition. In Proc. Annual Conference of the International Speech Communication Association, INTERSPEECH 1468–1472 (ISCA, 2015).

  5. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

  6. Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    CAS  PubMed  Google Scholar 

  7. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    CAS  PubMed  Google Scholar 

  8. Liu, L. et al. Computing systems for autonomous driving: state of the art and challenges. IEEE Internet Things J. 8, 6469–6486 (2021).

    Google Scholar 

  9. Shi, W. et al. LOEN: lensless opto-electronic neural network empowered machine vision. Light Sci. Appl. 11, 121 (2022).

    CAS  PubMed  PubMed Central  Google Scholar 

  10. Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).

    CAS  Google Scholar 

  11. Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020).

    CAS  PubMed  Google Scholar 

  12. Shastri, B. J. et al. Photonics for artificial intelligence and neuromorphic computing. Nat. Photon. 15, 102–114 (2021).

    CAS  Google Scholar 

  13. Xue, W. & Miller, O. D. High-NA optical edge detection via optimized multilayer films. J. Optics 23, 125004 (2021).

  14. Wang, T. et al. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 13, 123 (2022).

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Wang, T. et al. Image sensing with multilayer nonlinear optical neural networks. Nat. Photon. 17, 8–17 (2023).

    Google Scholar 

  16. Badloe, T., Lee, S. & Rho, J. Computation at the speed of light: metamaterials for all-optical calculations and neural networks. Adv. Photon. 4, 064002 (2022).

  17. Vanderlugt, A. Optical Signal Processing (Wiley, 1993).

  18. Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).

    PubMed  PubMed Central  Google Scholar 

  19. Colburn, S., Chu, Y., Shilzerman, E. & Majumdar, A. Optical frontend for a convolutional neural network. Appl. Opt. 58, 3179 (2019).

    PubMed  Google Scholar 

  20. Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon. 15, 367–373 (2021).

    CAS  Google Scholar 

  21. Chen, Y. H., Krishna, T., Emer, J. S. & Sze, V. Eyeriss: an energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE J. Solid-State Circuits 52, 127–138 (2017).

    Google Scholar 

  22. Neshatpour, K., Homayoun, H. & Sasan, A. ICNN: the iterative convolutional neural network. In ACM Transactions on Embedded Computing Systems 18, 119 (ACM, 2019).

  23. Xu, X. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021).

    CAS  PubMed  Google Scholar 

  24. Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).

    CAS  PubMed  Google Scholar 

  25. Wu, C. et al. Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network. Nat. Commun. 12, 96 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  26. Zhang, H. et al. An optical neural chip for implementing complex-valued neural network. Nat. Commun. 12, 457 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  27. Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 606, 501–506 (2022).

    CAS  PubMed  Google Scholar 

  28. Fu, T. et al. Photonic machine learning with on-chip diffractive optics. Nat. Commun. 14, 70 (2023).

    CAS  PubMed  PubMed Central  Google Scholar 

  29. Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).

    CAS  PubMed  Google Scholar 

  30. Qian, C. et al. Performing optical logic operations by a diffractive neural network. Light Sci. Appl. 9, 59 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  31. Luo, X. et al. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light Sci. Appl. 11, 158 (2022).

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Kwon, H., Arbabi, E., Kamali, S. M., Faraji-Dana, M. S. & Faraon, A. Single-shot quantitative phase gradient microscopy using a system of multifunctional metasurfaces. Nat. Photon. 14, 109–114 (2020).

    CAS  Google Scholar 

  33. Xiong, B. et al. Breaking the limitation of polarization multiplexing in optical metasurfaces with engineered noise. Science 379, 294–299 (2023).

    CAS  PubMed  Google Scholar 

  34. Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).

    CAS  PubMed  Google Scholar 

  35. Kim, J. et al. Scalable manufacturing of high-index atomic layer–polymer hybrid metasurfaces for metaphotonics in the visible. Nat. Mater. 22, 474–481 (2023).

    CAS  PubMed  Google Scholar 

  36. Levanon, N. et al. Angular transmission response of in-plane symmetry-breaking quasi-BIC all-dielectric metasurfaces. ACS Photonics 9, 3642–3648 (2022).

    CAS  Google Scholar 

  37. Nolen, J. R., Overvig, A. C., Cotrufo, M. & Alù, A. Arbitrarily polarized and unidirectional emission from thermal metasurfaces. Preprint at https://arxiv.org/abs/2301.12301 (2023).

  38. Guo, C., Xiao, M., Minkov, M., Shi, Y. & Fan, S. Photonic crystal slab Laplace operator for image differentiation. Optica 5, 251–256 (2018).

    Google Scholar 

  39. Cordaro, A. et al. High-index dielectric metasurfaces performing mathematical operations. Nano Lett. 19, 8418–8423 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  40. Zhou, Y., Zheng, H., Kravchenko, I. I. & Valentine, J. Flat optics for image differentiation. Nat. Photon. 14, 316–323 (2020).

    CAS  Google Scholar 

  41. Fu, W. et al. Ultracompact meta-imagers for arbitrary all-optical convolution. Light Sci. Appl. 11, 62 (2022).

    CAS  PubMed  PubMed Central  Google Scholar 

  42. Wang, H., Guo, C., Zhao, Z. & Fan, S. Compact incoherent image differentiation with nanophotonic structures. ACS Photonics 7, 338–343 (2020).

    CAS  Google Scholar 

  43. Zhang, X., Bai, B., Sun, H. B., Jin, G. & Valentine, J. Incoherent optoelectronic differentiation based on optimized multilayer films. Laser Photon Rev. 16, 2200038 (2022).

    Google Scholar 

  44. Zheng, H. et al. Meta-optic accelerators for object classifiers. Sci. Adv. 8, eabo6410 (2022).

    PubMed  PubMed Central  Google Scholar 

  45. Bernstein, L. et al. Single-shot optical neural network. Sci. Adv. 9, eadg7904 (2023).

    CAS  PubMed  PubMed Central  Google Scholar 

  46. Shen, Z. et al. Monocular metasurface camera for passive single-shot 4D imaging. Nat. Commun. 14, 1035 (2023).

    CAS  PubMed  PubMed Central  Google Scholar 

  47. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2323 (1998).

    Google Scholar 

  48. Zheng, H. et al. Compound meta-optics for complete and loss-less field control. ACS Nano 16, 15100–15107 (2022).

    CAS  PubMed  Google Scholar 

  49. Liu, S. et al. More ConvNets in the 2020s: scaling up kernels beyond 51x51 using sparsity. In 11th International Conference on Learning Representations 1–23 (ICLR, 2023).

  50. Barron, J. T. A general and adaptive robust loss function. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 4326–4334 (IEEE, 2019).

  51. Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In 9th International Conference on Learning Representations 1–22 (ICLR, 2021).

  52. Stillmaker, A. & Baas, B. Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7 nm. Integration 58, 74–81 (2017).

    Google Scholar 

  53. McClung, A., Samudrala, S., Torfeh, M., Mansouree, M. & Arbabi, A. Snapshot spectral imaging with parallel metasystems. Sci. Adv. 6, eabc7646 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  54. Ding, X., Zhang, X., Han, J. & Ding, G. Scaling up your kernels to 31 × 31: revisiting large kernel design in CNNs. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 11953–11965 (IEEE, 2022).

  55. Ding, X. et al. RepVgg: making VGG-style ConvNets great again. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 13728–13737 (IEEE, 2021).

  56. Li, L. et al. Intelligent metasurface imager and recognizer. Light Sci. Appl. 8, 97 (2019).

    PubMed  PubMed Central  Google Scholar 

  57. Zhao, R. et al. Multichannel vectorial holographic display and encryption. Light Sci. Appl. 7, 95 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  58. Kim, I. et al. Pixelated bifunctional metasurface-driven dynamic vectorial holographic color prints for photonic security platform. Nat. Commun. 12, 3614 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  59. Li, L. et al. Metalens-array-based high-dimensional and multiphoton quantum source. Science 368, 1487–1490 (2020).

    CAS  PubMed  Google Scholar 

  60. Hugonin, A. J. P. & Lalanne, P. RETICOLO software for grating analysis. Preprint at https://arxiv.org/abs/2101.00901 (2023).

Download references

Acknowledgements

H.Z. and J.G.V. acknowledge support from DARPA under contract HR001118C0015 and NAVAIR under contract N6893622C0030. X.Z. acknowledges support from ONR under contract N000142112468. Y.H. and Q.L. acknowledge support from NIH under contract R01DK135597. Meta-optic devices were manufactured as part of a user project at the Center for Nanophase Materials Sciences (CNMS), which is a US Department of Energy, Office of Science User Facility, Oak Ridge National Laboratory.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA

    Hanyu Zheng

  2. Department of Computer Science, Vanderbilt University, Nashville, TN, USA

    Quan Liu & Yuankai Huo

  3. Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Ivan I. Kravchenko

  4. Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, USA

    Xiaomeng Zhang & Jason G. Valentine

Authors
  1. Hanyu Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ivan I. Kravchenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaomeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuankai Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jason G. Valentine
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.Z. and J.G.V. developed the idea. H.Z. conducted the optical modelling and system design. Q.L. and H.Z. trained the digital neural network. H.Z. fabricated the samples. I.I.K. performed the silicon growth and electron-beam-lithography for the metasurfaces. H.Z. conducted the experimental measurements. H.Z., Q.L. and X.Z. performed the data analysis. H.Z. and J.G.V. wrote the manuscript with input from all the authors. The project was supervised by Y.H. and J.G.V.

Corresponding author

Correspondence to Jason G. Valentine.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Nanotechnology thanks Junsuk Rho, Tianyu Wang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–22 and Figs. 1–20.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, H., Liu, Q., Kravchenko, I.I. et al. Multichannel meta-imagers for accelerating machine vision. Nat. Nanotechnol. 19, 471–478 (2024). https://doi.org/10.1038/s41565-023-01557-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41565-023-01557-2

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing