Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Flat optics for image differentiation

Nature Photonics volume 14, pages 316–323 (2020)Cite this article

Subjects

Abstract

Image processing has become a critical technology in a variety of science and engineering disciplines. Although most image processing is performed digitally, optical analog processing has the advantages of being low-power and high-speed, but it requires a large volume. Here, we demonstrate flat optics for direct image differentiation, allowing us to significantly shrink the required optical system size. We first demonstrate how the differentiator can be combined with traditional imaging systems such as a commercial optical microscope and camera sensor for edge detection with a numerical aperture up to 0.32. We next demonstrate how the entire processing system can be realized as a monolithic compound flat optic by integrating the differentiator with a metalens. The compound nanophotonic system manifests the advantage of thin form factor as well as the ability to implement complex transfer functions, and could open new opportunities in applications such as biological imaging and computer vision.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

Buy

All prices are NET prices.

Fig. 1: Two-dimensional image differentiation using nanophotonic materials.
Fig. 2: Fabrication and characterization of the nanophotonic spatial differentiator.
Fig. 3: Differentiator resolution characterization.
Fig. 4: Edge detection microscope at visible frequencies.
Fig. 5: Large-scale image differentiator using nanosphere lithography.
Fig. 6: Compound metaoptic.

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.

References

  1. Marr, D. & Hildreth, E. Theory of edge detection. Proc. R. Soc. Lond. B 207, 187–217 (1980).

    Article  ADS  Google Scholar 

  2. Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986).

    Article  Google Scholar 

  3. Hsu, H.-S. & Tsai, W.-H. Moment-preserving edge detection and its application to image data compression. Opt. Eng. 32, 1596 (1993).

    Article  ADS  Google Scholar 

  4. Brosnan, T. & Sun, D.-W. Improving quality inspection of food products by computer vision—a review. J. Food Eng. 61, 3–16 (2004).

    Article  Google Scholar 

  5. Fürhapter, S., Jesacher, A., Bernet, S. & Ritsch-Marte, M. Spiral phase contrast imaging in microscopy. Opt. Express 13, 689–694 (2005).

    Article  ADS  Google Scholar 

  6. Gebäck, T. & Koumoutsakos, P. Edge detection in microscopy images using curvelets. BMC Bioinformatics 10, 75 (2009).

    Article  Google Scholar 

  7. Cardullo, R. A. Fundamentals of image processing in light microscopy. Methods Cell Biol. 72, 217–242 (2003).

    Article  Google Scholar 

  8. Haralick, R. M. & Shapiro, L. G. Computer and robot vision. IEEE Robot. Autom. Mag. 1, 28–48 (1991).

    Google Scholar 

  9. Solli, D. R. & Jalali, B. Analog optical computing. Nat. Photon. 9, 704–706 (2015).

    Article  ADS  Google Scholar 

  10. Yu, N. & Capasso, F. Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).

    Article  ADS  Google Scholar 

  11. Joannopoulos, J. D., Villeneuve, P. R. & Fan, S. Photonic crystals putting a new twist on light. Nature 386, 143–149 (1997).

    Article  ADS  Google Scholar 

  12. Silva, A. et al. Performing mathematical operations with metamaterials. Science 343, 160–163 (2014).

    Article  ADS  MathSciNet  Google Scholar 

  13. Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).

    Article  ADS  Google Scholar 

  14. Bykov, D. A., Doskolovich, L. L., Bezus, E. A. & Soifer, V. A. Optical computation of the Laplace operator using phase-shifted Bragg grating. Opt. Express 22, 25084–25092 (2014).

    Article  ADS  Google Scholar 

  15. Guo, C., Xiao, M., Minkov, M., Shi, Y. & Fan, S. Photonic crystal slab Laplace operator for image differentiation. Optica 5, 251 (2018).

    Article  ADS  Google Scholar 

  16. Cordaro, A. et al. High-index dielectric metasurfaces performing mathematical operations. Nano Lett. 19, 8418–8423 (2019).

    Article  ADS  Google Scholar 

  17. Zhu, T. et al. Generalized spatial differentiation from the spin Hall effect of light and its application in image processing of edge detection. Phys. Rev. Appl. 11, 034043 (2019).

    Article  ADS  Google Scholar 

  18. Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).

    Article  ADS  Google Scholar 

  19. Zhou, J. et al. Optical edge detection based on high-efficiency dielectric metasurface. Proc. Natl Acad. Sci. USA 116, 11137–11140 (2019).

    Article  ADS  Google Scholar 

  20. Bracewell, R. N. The Fourier Transform and its Applications (McGraw Hill, 2000).

  21. Krivenkov, V. I. Guided modes in photonic crystal fibers. Dokl. Phys. 48, 414–417 (2003).

    Article  ADS  Google Scholar 

  22. Fan, S. & Joannopoulos, J. D. Analysis of guided resonances in photonic crystal slabs. Phys. Rev. B 65, 235112 (2002).

    Article  ADS  Google Scholar 

  23. Zhou, W. et al. Progress in 2D photonic crystal Fano resonance photonics. Prog. Quantum Electron. 38, 1–74 (2014).

    Article  ADS  Google Scholar 

  24. Liu, Z. S., Tibuleac, S., Shin, D., Young, P. P. & Magnusson, R. High-efficiency guided-mode resonance filter. Opt. Lett. 23, 1556–1558 (1998).

    Article  ADS  Google Scholar 

  25. Suh, W., Yanik, M. F., Solgaard, O. & Fan, S. Displacement-sensitive photonic crystal structures based on guided resonance in photonic crystal slabs. Appl. Phys. Lett. 82, 1999–2001 (2003).

    Article  ADS  Google Scholar 

  26. Winn, J. N., Fink, Y., Fan, S. & Joannopoulos, J. D. Omnidirectional reflection from a one-dimensional photonic crystal. Opt. Lett. 23, 1573–1575 (1998).

    Article  ADS  Google Scholar 

  27. Hsu, C. W., Zhen, B., Stone, A. D., Joannopoulos, J. D. & Soljacic, M. Bound states in the continuum. Nat. Rev. Mater 1, 16048 (2016).

    Article  ADS  Google Scholar 

  28. Xu, L. et al. Dynamic nonlinear image tuning through magnetic dipole quasi‐BIC ultrathin resonators. Adv. Sci. 6, 1802119 (2019).

    Article  Google Scholar 

  29. Oskooi, A. F. et al. Meep: a flexible free-software package for electromagnetic simulations by the FDTD method. Comput. Phys. Commun. 181, 687–702 (2010).

    Article  ADS  Google Scholar 

  30. Lee, J. et al. Observation and differentiation of unique high-Q optical resonances near zero wave vector in macroscopic photonic crystal slabs. Phys. Rev. Lett. 109, 067401 (2012).

    Article  ADS  Google Scholar 

  31. Moitra, P. et al. Large-scale all-dielectric metamaterial perfect reflectors. ACS Photon. 2, 692–698 (2015).

    Article  Google Scholar 

  32. Zhou, Y. et al. Multilayer noninteracting dielectric metasurfaces for multiwavelength metaoptics. Nano Lett. 18, 7529–7537 (2018).

    Article  ADS  Google Scholar 

  33. Zhou, Y. et al. Multifunctional metaoptics based on bilayer metasurfaces. Light Sci. Appl. 8, 80 (2019).

    Article  ADS  Google Scholar 

  34. Phan, T. et al. High-efficiency, large-area, topology-optimized metasurfaces. Light Sci. Appl. 8, 48 (2019).

    Article  ADS  Google Scholar 

  35. Molesky, S. et al. Outlook for inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).

    Article  ADS  Google Scholar 

  36. Arbabi, A., Horie, Y., Ball, A. J., Bagheri, M. & Faraon, A. Subwavelength-thick lenses with high numerical apertures and large efficiency based on high-contrast transmitarrays. Nat. Commun. 6, 7069 (2015).

    Article  ADS  Google Scholar 

  37. Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).

    Article  ADS  Google Scholar 

  38. Lin, Z., Groever, B., Capasso, F., Rodriguez, A. W. & Lončar, M. Topology-optimized multilayered metaoptics. Phys. Rev. Appl. 9, 044030 (2018).

    Article  ADS  Google Scholar 

  39. Sell, D., Yang, J., Doshay, S., Yang, R. & Fan, J. A. Large-angle, multifunctional metagratings based on freeform multimode geometries. Nano Lett. 17, 3752–3757 (2017).

    Article  ADS  Google Scholar 

Download references

Acknowledgements

We acknowledge support from the Office of Naval Research under award no. N00014-18-1-2563 and DARPA under the NLM programme, award no. HR001118C0015. Part of the fabrication process was conducted at the Center for Nanophase Materials Sciences, which is a DOE Office of Science User Facility. The remainder of the fabrication process took place in the Vanderbilt Institute of Nanoscale Science and Engineering (VINSE) and we thank the staff, particularly K. Heinrich, for their support.

Author information

Authors and Affiliations

  1. Interdisciplinary Materials Science Program, Vanderbilt University, Nashville, TN, USA

    You Zhou

  2. Department of Electric Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA

    Hanyu Zheng

  3. Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Ivan I. Kravchenko

  4. Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, USA

    Jason Valentine

Authors
  1. You Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanyu Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ivan I. Kravchenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jason Valentine
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.Z. and J.V. developed the idea. Y.Z. conducted the modelling and theoretical analysis. Y.Z. and H.Z. fabricated the samples with small die size (less than 1 mm2) and H.Z. fabricated the samples based on self-assembled masks. I.I.K. provided the substrates and fabricated the larger die size samples not based on self-assembled masks. Y.Z. performed all of the experimental measurements and data analysis, with assistance from H.Z. Y.Z. and J.V. wrote the manuscript with input from all of the authors. The project was supervised by J.V.

Corresponding author

Correspondence to Jason Valentine.

Ethics declarations

Competing interests

Y.Z., H.Z. and J.V. have submitted a patent application for this work, assigned to Vanderbilt University.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Discussion and Figs. 1–8.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Zheng, H., Kravchenko, I.I. et al. Flat optics for image differentiation. Nat. Photonics 14, 316–323 (2020). https://doi.org/10.1038/s41566-020-0591-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41566-020-0591-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing