Flat optics for image differentiation

Abstract

Image processing has become a critical technology in a variety of science and engineering disciplines. Although most image processing is performed digitally, optical analog processing has the advantages of being low-power and high-speed, but it requires a large volume. Here, we demonstrate flat optics for direct image differentiation, allowing us to significantly shrink the required optical system size. We first demonstrate how the differentiator can be combined with traditional imaging systems such as a commercial optical microscope and camera sensor for edge detection with a numerical aperture up to 0.32. We next demonstrate how the entire processing system can be realized as a monolithic compound flat optic by integrating the differentiator with a metalens. The compound nanophotonic system manifests the advantage of thin form factor as well as the ability to implement complex transfer functions, and could open new opportunities in applications such as biological imaging and computer vision.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Two-dimensional image differentiation using nanophotonic materials.
Fig. 2: Fabrication and characterization of the nanophotonic spatial differentiator.
Fig. 3: Differentiator resolution characterization.
Fig. 4: Edge detection microscope at visible frequencies.
Fig. 5: Large-scale image differentiator using nanosphere lithography.
Fig. 6: Compound metaoptic.

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.

References

  1. 1.

    Marr, D. & Hildreth, E. Theory of edge detection. Proc. R. Soc. Lond. B 207, 187–217 (1980).

    ADS  Article  Google Scholar 

  2. 2.

    Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986).

    Article  Google Scholar 

  3. 3.

    Hsu, H.-S. & Tsai, W.-H. Moment-preserving edge detection and its application to image data compression. Opt. Eng. 32, 1596 (1993).

    ADS  Article  Google Scholar 

  4. 4.

    Brosnan, T. & Sun, D.-W. Improving quality inspection of food products by computer vision—a review. J. Food Eng. 61, 3–16 (2004).

    Article  Google Scholar 

  5. 5.

    Fürhapter, S., Jesacher, A., Bernet, S. & Ritsch-Marte, M. Spiral phase contrast imaging in microscopy. Opt. Express 13, 689–694 (2005).

    ADS  Article  Google Scholar 

  6. 6.

    Gebäck, T. & Koumoutsakos, P. Edge detection in microscopy images using curvelets. BMC Bioinformatics 10, 75 (2009).

    Article  Google Scholar 

  7. 7.

    Cardullo, R. A. Fundamentals of image processing in light microscopy. Methods Cell Biol. 72, 217–242 (2003).

    Article  Google Scholar 

  8. 8.

    Haralick, R. M. & Shapiro, L. G. Computer and robot vision. IEEE Robot. Autom. Mag. 1, 28–48 (1991).

    Google Scholar 

  9. 9.

    Solli, D. R. & Jalali, B. Analog optical computing. Nat. Photon. 9, 704–706 (2015).

    ADS  Article  Google Scholar 

  10. 10.

    Yu, N. & Capasso, F. Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).

    ADS  Article  Google Scholar 

  11. 11.

    Joannopoulos, J. D., Villeneuve, P. R. & Fan, S. Photonic crystals putting a new twist on light. Nature 386, 143–149 (1997).

    ADS  Article  Google Scholar 

  12. 12.

    Silva, A. et al. Performing mathematical operations with metamaterials. Science 343, 160–163 (2014).

    ADS  MathSciNet  Article  Google Scholar 

  13. 13.

    Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).

    ADS  Article  Google Scholar 

  14. 14.

    Bykov, D. A., Doskolovich, L. L., Bezus, E. A. & Soifer, V. A. Optical computation of the Laplace operator using phase-shifted Bragg grating. Opt. Express 22, 25084–25092 (2014).

    ADS  Article  Google Scholar 

  15. 15.

    Guo, C., Xiao, M., Minkov, M., Shi, Y. & Fan, S. Photonic crystal slab Laplace operator for image differentiation. Optica 5, 251 (2018).

    ADS  Article  Google Scholar 

  16. 16.

    Cordaro, A. et al. High-index dielectric metasurfaces performing mathematical operations. Nano Lett. 19, 8418–8423 (2019).

    ADS  Article  Google Scholar 

  17. 17.

    Zhu, T. et al. Generalized spatial differentiation from the spin Hall effect of light and its application in image processing of edge detection. Phys. Rev. Appl. 11, 034043 (2019).

    ADS  Article  Google Scholar 

  18. 18.

    Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).

    ADS  Article  Google Scholar 

  19. 19.

    Zhou, J. et al. Optical edge detection based on high-efficiency dielectric metasurface. Proc. Natl Acad. Sci. USA 116, 11137–11140 (2019).

    ADS  Article  Google Scholar 

  20. 20.

    Bracewell, R. N. The Fourier Transform and its Applications (McGraw Hill, 2000).

  21. 21.

    Krivenkov, V. I. Guided modes in photonic crystal fibers. Dokl. Phys. 48, 414–417 (2003).

    ADS  Article  Google Scholar 

  22. 22.

    Fan, S. & Joannopoulos, J. D. Analysis of guided resonances in photonic crystal slabs. Phys. Rev. B 65, 235112 (2002).

    ADS  Article  Google Scholar 

  23. 23.

    Zhou, W. et al. Progress in 2D photonic crystal Fano resonance photonics. Prog. Quantum Electron. 38, 1–74 (2014).

    ADS  Article  Google Scholar 

  24. 24.

    Liu, Z. S., Tibuleac, S., Shin, D., Young, P. P. & Magnusson, R. High-efficiency guided-mode resonance filter. Opt. Lett. 23, 1556–1558 (1998).

    ADS  Article  Google Scholar 

  25. 25.

    Suh, W., Yanik, M. F., Solgaard, O. & Fan, S. Displacement-sensitive photonic crystal structures based on guided resonance in photonic crystal slabs. Appl. Phys. Lett. 82, 1999–2001 (2003).

    ADS  Article  Google Scholar 

  26. 26.

    Winn, J. N., Fink, Y., Fan, S. & Joannopoulos, J. D. Omnidirectional reflection from a one-dimensional photonic crystal. Opt. Lett. 23, 1573–1575 (1998).

    ADS  Article  Google Scholar 

  27. 27.

    Hsu, C. W., Zhen, B., Stone, A. D., Joannopoulos, J. D. & Soljacic, M. Bound states in the continuum. Nat. Rev. Mater 1, 16048 (2016).

    ADS  Article  Google Scholar 

  28. 28.

    Xu, L. et al. Dynamic nonlinear image tuning through magnetic dipole quasi‐BIC ultrathin resonators. Adv. Sci. 6, 1802119 (2019).

    Article  Google Scholar 

  29. 29.

    Oskooi, A. F. et al. Meep: a flexible free-software package for electromagnetic simulations by the FDTD method. Comput. Phys. Commun. 181, 687–702 (2010).

    ADS  Article  Google Scholar 

  30. 30.

    Lee, J. et al. Observation and differentiation of unique high-Q optical resonances near zero wave vector in macroscopic photonic crystal slabs. Phys. Rev. Lett. 109, 067401 (2012).

    ADS  Article  Google Scholar 

  31. 31.

    Moitra, P. et al. Large-scale all-dielectric metamaterial perfect reflectors. ACS Photon. 2, 692–698 (2015).

    Article  Google Scholar 

  32. 32.

    Zhou, Y. et al. Multilayer noninteracting dielectric metasurfaces for multiwavelength metaoptics. Nano Lett. 18, 7529–7537 (2018).

    ADS  Article  Google Scholar 

  33. 33.

    Zhou, Y. et al. Multifunctional metaoptics based on bilayer metasurfaces. Light Sci. Appl. 8, 80 (2019).

    ADS  Article  Google Scholar 

  34. 34.

    Phan, T. et al. High-efficiency, large-area, topology-optimized metasurfaces. Light Sci. Appl. 8, 48 (2019).

    ADS  Article  Google Scholar 

  35. 35.

    Molesky, S. et al. Outlook for inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).

    ADS  Article  Google Scholar 

  36. 36.

    Arbabi, A., Horie, Y., Ball, A. J., Bagheri, M. & Faraon, A. Subwavelength-thick lenses with high numerical apertures and large efficiency based on high-contrast transmitarrays. Nat. Commun. 6, 7069 (2015).

    ADS  Article  Google Scholar 

  37. 37.

    Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352, 1190–1194 (2016).

    ADS  Article  Google Scholar 

  38. 38.

    Lin, Z., Groever, B., Capasso, F., Rodriguez, A. W. & Lončar, M. Topology-optimized multilayered metaoptics. Phys. Rev. Appl. 9, 044030 (2018).

    ADS  Article  Google Scholar 

  39. 39.

    Sell, D., Yang, J., Doshay, S., Yang, R. & Fan, J. A. Large-angle, multifunctional metagratings based on freeform multimode geometries. Nano Lett. 17, 3752–3757 (2017).

    ADS  Article  Google Scholar 

Download references

Acknowledgements

We acknowledge support from the Office of Naval Research under award no. N00014-18-1-2563 and DARPA under the NLM programme, award no. HR001118C0015. Part of the fabrication process was conducted at the Center for Nanophase Materials Sciences, which is a DOE Office of Science User Facility. The remainder of the fabrication process took place in the Vanderbilt Institute of Nanoscale Science and Engineering (VINSE) and we thank the staff, particularly K. Heinrich, for their support.

Author information

Affiliations

Authors

Contributions

Y.Z. and J.V. developed the idea. Y.Z. conducted the modelling and theoretical analysis. Y.Z. and H.Z. fabricated the samples with small die size (less than 1 mm2) and H.Z. fabricated the samples based on self-assembled masks. I.I.K. provided the substrates and fabricated the larger die size samples not based on self-assembled masks. Y.Z. performed all of the experimental measurements and data analysis, with assistance from H.Z. Y.Z. and J.V. wrote the manuscript with input from all of the authors. The project was supervised by J.V.

Corresponding author

Correspondence to Jason Valentine.

Ethics declarations

Competing interests

Y.Z., H.Z. and J.V. have submitted a patent application for this work, assigned to Vanderbilt University.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Discussion and Figs. 1–8.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Zheng, H., Kravchenko, I.I. et al. Flat optics for image differentiation. Nat. Photonics 14, 316–323 (2020). https://doi.org/10.1038/s41566-020-0591-3

Download citation

Further reading