Image processing has become a critical technology in a variety of science and engineering disciplines. Although most image processing is performed digitally, optical analog processing has the advantages of being low-power and high-speed, but it requires a large volume. Here, we demonstrate flat optics for direct image differentiation, allowing us to significantly shrink the required optical system size. We first demonstrate how the differentiator can be combined with traditional imaging systems such as a commercial optical microscope and camera sensor for edge detection with a numerical aperture up to 0.32. We next demonstrate how the entire processing system can be realized as a monolithic compound flat optic by integrating the differentiator with a metalens. The compound nanophotonic system manifests the advantage of thin form factor as well as the ability to implement complex transfer functions, and could open new opportunities in applications such as biological imaging and computer vision.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Scientific Reports Open Access 12 October 2022
Nature Communications Open Access 21 April 2022
Light: Science & Applications Open Access 19 April 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.
Marr, D. & Hildreth, E. Theory of edge detection. Proc. R. Soc. Lond. B 207, 187–217 (1980).
Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986).
Hsu, H.-S. & Tsai, W.-H. Moment-preserving edge detection and its application to image data compression. Opt. Eng. 32, 1596 (1993).
Brosnan, T. & Sun, D.-W. Improving quality inspection of food products by computer vision—a review. J. Food Eng. 61, 3–16 (2004).
Fürhapter, S., Jesacher, A., Bernet, S. & Ritsch-Marte, M. Spiral phase contrast imaging in microscopy. Opt. Express 13, 689–694 (2005).
Gebäck, T. & Koumoutsakos, P. Edge detection in microscopy images using curvelets. BMC Bioinformatics 10, 75 (2009).
Cardullo, R. A. Fundamentals of image processing in light microscopy. Methods Cell Biol. 72, 217–242 (2003).
Haralick, R. M. & Shapiro, L. G. Computer and robot vision. IEEE Robot. Autom. Mag. 1, 28–48 (1991).
Solli, D. R. & Jalali, B. Analog optical computing. Nat. Photon. 9, 704–706 (2015).
Yu, N. & Capasso, F. Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).
Joannopoulos, J. D., Villeneuve, P. R. & Fan, S. Photonic crystals putting a new twist on light. Nature 386, 143–149 (1997).
Silva, A. et al. Performing mathematical operations with metamaterials. Science 343, 160–163 (2014).
Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).
Bykov, D. A., Doskolovich, L. L., Bezus, E. A. & Soifer, V. A. Optical computation of the Laplace operator using phase-shifted Bragg grating. Opt. Express 22, 25084–25092 (2014).
Guo, C., Xiao, M., Minkov, M., Shi, Y. & Fan, S. Photonic crystal slab Laplace operator for image differentiation. Optica 5, 251 (2018).
Cordaro, A. et al. High-index dielectric metasurfaces performing mathematical operations. Nano Lett. 19, 8418–8423 (2019).
Zhu, T. et al. Generalized spatial differentiation from the spin Hall effect of light and its application in image processing of edge detection. Phys. Rev. Appl. 11, 034043 (2019).
Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).
Zhou, J. et al. Optical edge detection based on high-efficiency dielectric metasurface. Proc. Natl Acad. Sci. USA 116, 11137–11140 (2019).
Bracewell, R. N. The Fourier Transform and its Applications (McGraw Hill, 2000).
Krivenkov, V. I. Guided modes in photonic crystal fibers. Dokl. Phys. 48, 414–417 (2003).
Fan, S. & Joannopoulos, J. D. Analysis of guided resonances in photonic crystal slabs. Phys. Rev. B 65, 235112 (2002).
Zhou, W. et al. Progress in 2D photonic crystal Fano resonance photonics. Prog. Quantum Electron. 38, 1–74 (2014).
Liu, Z. S., Tibuleac, S., Shin, D., Young, P. P. & Magnusson, R. High-efficiency guided-mode resonance filter. Opt. Lett. 23, 1556–1558 (1998).
Suh, W., Yanik, M. F., Solgaard, O. & Fan, S. Displacement-sensitive photonic crystal structures based on guided resonance in photonic crystal slabs. Appl. Phys. Lett. 82, 1999–2001 (2003).
Winn, J. N., Fink, Y., Fan, S. & Joannopoulos, J. D. Omnidirectional reflection from a one-dimensional photonic crystal. Opt. Lett. 23, 1573–1575 (1998).
Hsu, C. W., Zhen, B., Stone, A. D., Joannopoulos, J. D. & Soljacic, M. Bound states in the continuum. Nat. Rev. Mater 1, 16048 (2016).
Xu, L. et al. Dynamic nonlinear image tuning through magnetic dipole quasi‐BIC ultrathin resonators. Adv. Sci. 6, 1802119 (2019).
Oskooi, A. F. et al. Meep: a flexible free-software package for electromagnetic simulations by the FDTD method. Comput. Phys. Commun. 181, 687–702 (2010).
Lee, J. et al. Observation and differentiation of unique high-Q optical resonances near zero wave vector in macroscopic photonic crystal slabs. Phys. Rev. Lett. 109, 067401 (2012).
Moitra, P. et al. Large-scale all-dielectric metamaterial perfect reflectors. ACS Photon. 2, 692–698 (2015).
Zhou, Y. et al. Multilayer noninteracting dielectric metasurfaces for multiwavelength metaoptics. Nano Lett. 18, 7529–7537 (2018).
Zhou, Y. et al. Multifunctional metaoptics based on bilayer metasurfaces. Light Sci. Appl. 8, 80 (2019).
Phan, T. et al. High-efficiency, large-area, topology-optimized metasurfaces. Light Sci. Appl. 8, 48 (2019).
Molesky, S. et al. Outlook for inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).
Arbabi, A., Horie, Y., Ball, A. J., Bagheri, M. & Faraon, A. Subwavelength-thick lenses with high numerical apertures and large efficiency based on high-contrast transmitarrays. Nat. Commun. 6, 7069 (2015).
Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).
Lin, Z., Groever, B., Capasso, F., Rodriguez, A. W. & Lončar, M. Topology-optimized multilayered metaoptics. Phys. Rev. Appl. 9, 044030 (2018).
Sell, D., Yang, J., Doshay, S., Yang, R. & Fan, J. A. Large-angle, multifunctional metagratings based on freeform multimode geometries. Nano Lett. 17, 3752–3757 (2017).
We acknowledge support from the Office of Naval Research under award no. N00014-18-1-2563 and DARPA under the NLM programme, award no. HR001118C0015. Part of the fabrication process was conducted at the Center for Nanophase Materials Sciences, which is a DOE Office of Science User Facility. The remainder of the fabrication process took place in the Vanderbilt Institute of Nanoscale Science and Engineering (VINSE) and we thank the staff, particularly K. Heinrich, for their support.
Y.Z., H.Z. and J.V. have submitted a patent application for this work, assigned to Vanderbilt University.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zhou, Y., Zheng, H., Kravchenko, I.I. et al. Flat optics for image differentiation. Nat. Photonics 14, 316–323 (2020). https://doi.org/10.1038/s41566-020-0591-3
This article is cited by
Nature Photonics (2023)
Nature Nanotechnology (2023)
Nature Photonics (2023)
Nature Photonics (2023)
Nature Communications (2022)