Abstract
Structured light (SL)based depthsensing technology illuminates the objects with an array of dots, and backscattered light is monitored to extract threedimensional information. Conventionally, diffractive optical elements have been used to form laser dot array, however, the fieldofview (FOV) and diffraction efficiency are limited due to their micronscale pixel size. Here, we propose a metasurfaceenhanced SLbased depthsensing platform that scatters highdensity ~10 K dot array over the 180° FOV by manipulating light at subwavelengthscale. As a proofofconcept, we place face masks one on the beam axis and the other 50° apart from axis within distance of 1 m and estimate the depth information using a stereo matching algorithm. Furthermore, we demonstrate the replication of the metasurface using the nanoparticleembeddedresin (nanoPER) imprinting method which enables highthroughput manufacturing of the metasurfaces on any arbitrary substrates. Such a fullspace diffractive metasurface may afford ultracompact depth perception platform for face recognition and automotive robot vision applications.
Similar content being viewed by others
Introduction
Laserbased threedimensional (3D) surface imaging technology has exhibited potential in a variety of sensor applications such as augmented/virtual reality (AR/VR), autonomous driving, robot vision, and face recognition in mobile devices with the advances in highspeed computing platforms^{1}. The depth information of the 3D objects is estimated by illuminating targets with the controlled laser beam through transmitters and by monitoring backscattered light from them at the receivers. Depending on the depth calculation methods at the detectors, the 3D imaging systems are classified into time of flight (TOF) and structured light (SL) types. Using a pulsed laser to image 3D objects, the depth information is extracted from the TOF, which is the measured time delay between the illuminated pulse light and the backscattered signal. To achieve a wide field of view (FOV), the TOF must be sequentially measured at each point while rotating and scanning with the laser^{2,3}. Laser scanning systems are realized by mechanically rotating mirrors, or microelectromechanical systems (MEMS). However, mechanical rotating of mirrors demands highpower due to the inertia of the bulky components, resulting in a decreased frame rate of object acquisition. MEMS can reduce the power consumption but oscillatingbased operation limits the scannable scene into onedimensional (1D) and is vulnerable to vibrations and shocks. Recently, electrical beam steering using transparent conductive oxides (TCOs)^{4,5,6} and liquid crystals (LCs)^{7,8} have been proposed, complying with the physical mechanism of the solidstate optical phased array antennas, but still suffering from small FOV.
In contrast, SLbased 3D imaging systems use specially designed twodimensional (2D) light patterns to project the object, efficiently enlarging the FOV without any actual scanning of light. When the surface of a 3D object is nonplanar, it distorts the projected SL pattern, and the surface profile can be calculated from distorted light pattern using a variety of algorithms^{9}. Such a 2D operation can increase the frame rate of object acquisition with less computational load by facilitating the simultaneous imaging of multiple objects. Thus far, diffractive optical elements (DOEs) or spatial light modulators (SLMs) are used to generate 2D light patterns. The conventional microsized DOEs should be etched to multiple depths to modulate the phase with multiple steps which induce challenges in terms of fabrication. In addition, both DOEs and SLMs have large pixel sizes on the order of microns, resulting in low efficiency and uniformity of the diffracted beam arrays, especially at large angles. The uniformity of diffracted beams from SLM can be resolved by calculating phase profile of SLM using vectorial Debye approximation^{10} which considers the depolarization effect when focusing light with a high numerical aperture objective^{11,12}. However, the requirement of bulky objective lenses is still challenging for miniaturization of SL illuminating system. Therefore, the demands for nextgeneration, compact, and lightweight illumination methods of SL imaging systems are increasing, in both the academic and industrial fields, and nanophotonics is a promising candidate to overcome the challenges of current SLbased 3D imaging systems^{13}.
Metasurfaces, 2D arrangements of designed nanostructures named metaatoms, dominantly inherit their properties from the composing building blocks, i.e., resonant and waveguiding effects^{14,15}. They have been used to demonstrate extraordinary light modulation at the nanoscale, which cannot be observed using conventional optical devices^{16,17,18}. With intense studies of lightmatter interactions in subwavelengthscale structures^{19,20,21,22}, metasurfaces have demonstrated exceptional functionalities in a variety of applications such as imaging^{23,24,25}, color filters^{26,27,28}, holographic displays^{29,30}, polarizing elements^{31,32}, and beam shapers^{33,34}. In terms of metasurface for 3D imaging systems, the subwavelength pitch pixels of metasurface can improve the FOV and diffraction efficiency compared to conventional DOEs and SLMs by modulating the phase at a subwavelength resolution. Using metasurfaces, SL projection with a 120° FOV has been developed using vectorial electromagnetic diffraction theory^{35}. The intensities of the diffracted beams are optimized using an interiorpoint method but have a limited number of diffracted beams, up to \(\pm\)4 diffraction orders. Moreover, 441 random point generating metasurfaces have been demonstrated, underlining the scalable nanofabrication of metasurfaces on 12inch glass wafer using immersion lithography^{36}. However, the largest scanning angle is limited to 15°. The metasurfaces projecting dots of light into diffraction angles close to 90°, giving a 180° FOV, have been demonstrated in the reflective and transmissive spaces^{37,38}. The diffraction efficiencies to both reflective and transmissive spaces are same (27%) and were experimentally verified by a blazed grating and beam splitter. As a result, the 4044 projected dots are generated from the pixelated metasurfaces. Furthermore, such metasurfacebased SL imaging with a large FOV can be integrated with onchip light sources such as vertical cavity surfaceemitting lasers (VCSELs), facilitated by the flatness features of the metasurfaces^{39,40,41}.
Here, we propose a metasurfacebased SL projecting device with ~10 K of highdensity dots of light or ~100 of parallel lines of light into the full space, realizing an extreme FOV of 180° (Fig. 1). The metasurfaces are composed of periodic supercells designed to diffract incident laser light to highdensity number of diffraction orders with uniform intensity. Then, the supercells are periodically arranged with different periodicities along the x and ydirections with consideration of the interference effects induced from the arrangement of supercells. The final diffraction patterns are analyzed by the convolution theorem regarding the phaseonly distribution of a single supercell as the kernel function of the convolution. By understanding the total optical response with respect to the diffraction and interference effects^{42,43,44}, multiple illumination types of the metasurfacebased SL projector, i.e., 2D dot arrays, 1D dot arrays, and 2D parallel line arrays of light are numerically and experimentally demonstrated covering a 180° FOV. As a proofofconcept, we place face masks within the range of 1 m with wide viewing angle up to 60° with respect to optical axis and illuminate the highdensity dot arrays generated from the proposed metasurfaces. The depth information of the 3D face masks is extracted from stereo matching algorithm using two cameras. Furthermore, we demonstrate a prototype of the metasurfacebased depth sensor for compact and lightweight AR glasses using a nanoparticleembeddedresin (nanoPER)based scalable imprinting fabrication method, facilitating direct printing of metasurface to a curved surface of glasses. Such a metasurfacebased SL imaging platform can realize imaging of 3D objects over a full FOV with highdensity dot arrays in an ergonomically and commercially viable form factor.
Results
Design principle of the fullspace diffractive metasurface
The diffraction pattern of periodic scatterers can be analyzed by two effects: the diffraction effect of single scatterer and interference effects induced by the periodic arrangement of the scatterers. This viewpoint is widely adopted to understand the optical response of periodic structures, e.g., diffraction from many slits^{45}, Xray diffractometry^{46}, and lightemitting metasurfaces^{42}. The interaction between neighboring scatterers is assumed to be negligible to comply with Huygens’ principle, where each point of the wavefront is considered a secondary source of propagation. The proposed 2D fullspace diffractive metasurface is composed of periodically arranged supercells, where each supercell is composed of n \(\times\) n metaatoms with pixel pitch P, and N \(\times\) N supercells forming the entire metasurface. According to the Nyquist sampling theorem in signal processing, the sampling period should be shorter than half of the signal period to sufficiently resolve the high frequency components. Analogously, the metasurface with pixel pitch of P can resolve spatial frequency \({f}_{x,{y}}\) up to \(\pm \frac{1}{2P}\), and when the light diffracts at a large angle to the optical axis, it carries high spatial frequency reaching maximum of \(\pm \frac{1}{\lambda }\) (Fig. 2a). Therefore, \(\pm \frac{1}{\lambda } < \pm \frac{1}{2P}\) gives the upper limit to pixel pitch as \(2P < \lambda\). A further effect of the pixel pitch on the diffraction behavior also in terms of the diffraction grating equation is discussed in Supplementary Note 1. Figure 2a shows the required spatial frequency region, where all propagating waves are located in the radius of \({\lambda }^{1}\) have unity amplitude, and evanescent waves located in the \({{f}_{x}}^{2}+{{f}_{y}}^{2}\ge {\lambda }^{2}\) regions are excluded. From the initial target, the phase profile of the single supercell (Fig. 2b) is retrieved using an iterative discrete 2D Fourier transform, i.e., Gerchberg–Saxton (GS) algorithm.
Next, we need to consider interference effects induced by the periodic arrangement of the designed supercells. Figure 2c shows the periodic arrangement itself in both the x, and ydirections in the space domain, which is described by a 2D Dirac comb function. Here, the spacing of the comb function is the same as the width of the supercell in the both x, and ydirections which is the product of the number of metaatoms n and pixel pitch P. The entire metasurface is composed of periodically arranged supercells and can be described by the convolution of single supercell and a 2D Dirac comb function resulting in the final phase profile \(\varphi\) (Fig. 2f), where the single supercell acts as the kernel function of the convolution. According to the convolution theorem, the Fourier transform of two convolved functions is simply expressed as a multiplication of the Fourier transform of each function. Therefore, the diffraction pattern of the entire metasurface (Fig. 2g) is represented in the spatial frequency domain as a product of the single supercell diffraction pattern (Fig. 2a) and the Fourier transform of 2D Dirac comb function (Fig. 2d). As the Fourier transformed 2D Dirac comb has a spacing of \(\frac{1}{{nP}}\), the maximum signal frequency of \(\pm \frac{1}{\lambda }\) is sampled with \(\frac{1}{{nP}}\). Accordingly, the number of diffraction orders is described as \(\pm \frac{{nP}}{\lambda }\), which is \(\pm\)47 for n = 100, P = 300 nm, and \(\lambda\) = 633 nm. Using a larger supercell with a larger n, the number of diffraction orders can be increased. It should be noted that the intensity of the diffracted beam decreases at large propagating angles because the number of sampling points decreases at higher frequency components. The decrease in intensity of higherorder diffraction becomes larger as pixel pitch increases, in other words decreasing the uniformity of intensity distribution (Supplementary Fig. 1). The propagation angle of the m^{th} order diffracted beam \({\theta }_{z}^{m}\) with respect to the optical axis is described as \({{{\sin }}}^{2}{\theta }_{z}^{m}={{{\sin }}}^{2}{\theta }_{x}^{m}+{{{\sin }}}^{2}{\theta }_{y}^{m}\), where angles about the x, and yaxis are expressed as
where m represents the diffraction order and the largest diffraction angle \({\theta }_{x,{y}}^{m=\pm 47}\) is 82.6°. It should be noted that the uniformly sampled spatial frequency, as shown in Fig. 2g, does not ensure a uniformly spaced propagation angle, since \({{\sin }}\theta\) can be approximated to \(\theta\) only at small angles. Therefore, the spacing between diffracted beams at higher orders becomes larger, as does the size of the beam. The angle cone of each diffracted beam \({\varOmega }_{{N\; x},{y}}^{m}\) is described as \({\varOmega }_{{N\; x},{y}}^{m}={{\arcsin }}\left(\frac{m\lambda }{{nP}}+\frac{\lambda }{2{NnP}}\right){{\arcsin }}\left(\frac{m\lambda }{{nP}}\frac{\lambda }{2{NnP}}\right)\), where N is number of repeated supercells (Supplementary Fig. 2). In addition, the propagation angle is dependent on the operation wavelength; in other words, the diffraction angle becomes larger at the longer wavelength (Supplementary Fig. 3).
To implement the final phase \(\varphi\), the metaatoms that make up the metasurface should be optimized to modulate the phase of the transmitted light with uniform and high transmission efficiency. For this purpose, we use a rectangular metaatom made of hydrogenated amorphous silicon (aSi:H) with high refractive index (n = 2.8) and low extinction coefficient (k = 0.001) at the wavelength of 633 nm (Supplementary Fig. 4). Under circularly polarized (CP) light incidence, which is represented as \({{{{{{\bf{E}}}}}}}_{{{{{{\rm{in}}}}}}}=[{{1} \atop {\pm} {{{{{\rm{i}}}}}}}]\), the output electric field \({{{{{{\bf{E}}}}}}}_{{{{{{\rm{out}}}}}}}\), transmitted through the metaatom exhibiting strong birefringence along the long and short axes can be described as
where \({t}_{l}\) and \({t}_{s}\) denote the complex transmission coefficients along the long and short axis of the metaatom, respectively, and \(\theta \left(x,{y}\right)\) is inplane rotation angle of the metaatom, as shown in inset of Fig. 2e. From the second term of Eq. (2), the number of oscillations along two axes of nanopillar differs by half of wavelength (Supplementary Fig. 5), manifested by oppositely converted handedness of CP light. According to the physical mechanism of geometric phase, the phase of the converted CP light is modulated by twice of inplane rotation angle, where + and − sign represents the right circularly polarized (RCP) light, and left circularly polarized (LCP) light incidence, respectively. Therefore, a high polarization conversion efficiency (CE), \({\left\frac{{t}_{l}{t}_{s}}{2}\right}^{2}\), is required and is optimized by rigorous coupledwave analysis (RCWA) as a function of structure parameters, i.e., height, length, and width of the rectangular metaatom as shown in Fig. 2e. A high CE of 88% is obtained from a metaatom with L = 250 nm, and W = 110 nm with H = 475 nm. It should be noted that in general, the opposite sign of modulated phase depending on handedness of CP light leads to the unwanted conjugated orders of diffraction. However, in this work, the diffracted beams are designed to propagate in every diffraction order symmetrically to each other (Fig. 2g), in that, it is operable under both RCP and LCP illuminance. As a result, the proposed fullspace diffractive metasurface is independent of the incident polarization state, simplifying the optical setup without additional optical components such as polarizer and waveplate.
Characterization of fullspace diffractive metasurface
The final phase distribution obtained in Fig. 2f is realized using the optimized metaatom, and corresponding subwavelength modulation of light properties enables an extremely large FOV about 180° (Fig. 2h). We also demonstrate a large FOV of our fullspace diffractive metasurfaces with a hemisphere screen and Fourier microscope which allows imaging of the spatial frequency domain (Supplementary Fig. 6). We measure the overall diffraction efficiency (DE) to be 60%, defined as all the transmitted light, including zerothorder beam, normalized to the intensity of the incident laser. The zerothorder efficiency itself is 32% of the incident light. Compared to the simulated results, we attribute discrepancies to the fact that the fabricated metasurface could not fully realize the calculated phase profile due to the coupling between metaatoms and fabrication defects. Coupling between neighboring metaatoms can be alleviated by strongly confining light in the metaatoms or considering the coupling during the design process of the phase profile. For example, high refractive index titanium dioxide (TiO_{2}) materialbased metasurfaces can strongly confine the electromagnetic waves, reducing the unwanted zerothorder beam^{47}. On the other hand, vectorial diffraction theories such as finitedifference timedomain (FDTD) and Fourier modal method (FMM) can be directly used to calculate the phase profile, which considers the coupling between unit cell in design process^{35}. The tilted sidewall profile of fabricated metaatoms reduces the DE compared to the calculated CE of 88%. To correct a sidewall profile as rightangled, the etching processes should be carefully optimized to fully exploit the calculated CE^{48}.
We numerically and experimentally analyze the intensity uniformity of diffracted beams using 1D fullspace diffractive metasurface as there are too many points for 2D fullspace diffractive metasurface. We simulate a 1D metasurface with the designed structural parameters and material properties using COMSOL Multiphysics, and the electric field distribution in the near field of metasurface is extracted and used as the source of farfield diffraction up to 5 mm from the metasurfaces. Note that this electric field distribution is the fullwave result considering all lightmatter interactions at the nanometer scale. After light transmits through the metasurface, it propagates through free space. Therefore, we calculate the amplitude of the electric field distribution in the x–z plane using diffraction theory. As the diffracted beams from our metasurface propagate at large angles with respect to the optical axis, the Fresnel or Fraunhofer diffraction theory which uses the paraxial approximation is not appropriate. Therefore, we use Rayleigh–Sommerfeld diffraction theory to visualize wideangle diffracted light pattern, computed by the fast Fourier transformbased direct integral method^{49}. As a result, the amplitude of the electric field is achieved in the x–z plane (Fig. 3a). The enlarged far field amplitude distribution shows the highdensity diffracted beam paths (Fig. 3b) and the enlarged near field result shows the interference effects between the neighboring supercells (Fig. 3c). Finally, the intensity distribution is obtained over every diffraction order, note that for each diffracted beam, the intensity is numerically integrated considering its finite angular width.
For experimental measurement of intensity of diffracted beams, we fabricate another sample which generates fullhorizontalspace dot patterns. To diffract the incident beam into the horizontal direction only, supercells are needed to be designed with different widths along the x and ydirections, and convoluted with different periodicities in each direction, equal to the widths of the supercell in x and y directions, respectively (Supplementary Fig. 7a). The power meter is mounted on the rotating rail measuring every diffraction order spanning from −90° to +90° (Fig. 3d). The number of diffraction orders and the intensities of the measured diffracted beams are plotted with the simulated intensity distributions as shown in Fig. 3e. The total \(\pm\)47 diffraction orders are grouped into propagating angles at intervals of 10°, and the intensities of the simulated and measured diffracted beams at each group are averaged. As analyzed from Eq. (1), due to the larger angular spacing at high diffraction orders, the number of diffracted beams decreases. The uniformity of the intensity distribution is calculated by comparing the simulation results and the measured values. We obtain an RMSE of 27.48% from the 1D fullspace diffractive metasurface, where \({{{{{\rm{RMSE}}}}}}=\sqrt{\mathop{\sum }\nolimits_{i=1}^{M}\frac{{({I}_{i}^{{{{{{\rm{Sim}}}}}}}{I}_{i}^{{{{{{\rm{Exp}}}}}}})}^{2}}{M}}\). Here, \({I}_{i}^{{{{{{\rm{Sim}}}}}}}\) and \({I}_{i}^{{{{{{\rm{Exp}}}}}}}\) are the simulated and measured ith order diffraction intensity, respectively, and M is the total number of the diffraction order. Furthermore, we compare the performance metrics of our fullspace diffractive metasurface with stateoftheart commercial DOE products (Supplementary Note 2).
Performance of the proposed metasurfacebased fullspace SL imaging
To demonstrate the prospective application of our metasurfaces as a depth estimating module in 3D sensing technology, we showcase an experimental example of 3D depth reconstruction of a face mask using fullspace dot patterns. We use a stereo system with two cameras to calculate the depth of the projected dot points (Fig. 4a). Before the actual measurement, the cameras are calibrated using a standard checkerboard pattern^{50} (Supplementary Fig. 8). The intrinsic parameters, e.g., lens focal length, and the extrinsic parameters, e.g., the relative location and orientation of each camera, are obtained by detecting the image projection of the pattern. These parameters allow us to map realworld distances to pixels which, in principle, is determined by the relation of similar triangles, \(\frac{{x}_{1}{x}_{2}}{f}=\frac{B}{Z}\) in a linear camera model (Fig. 4b). Here, Z is the depth and \({x}_{1}\), \({x}_{2}\) are the coordinate points in each image plane which correspond to the same point in the object scene. f is the focal length of the camera and B is the distance between the two cameras. The objective is to estimate Z, which is inversely proportional to the disparity, \({x}_{1}{x}_{2}\). In the optical setup shown in Fig. 4a, as laser light passes through the metasurface, it projects a set of dots over a wide FOV. The scattered dot points on the surface of the object are captured by each camera.
The depth estimation takes three steps. (1) 2D coordinate extraction of the dots from the two captured images. (2) Matching each dot in camera 1 and its corresponding dot in camera 2 (Fig. 4b). (3) Calculating the depth using the stereo camera trigonometry with the coordinate information of the matched dot pairs. In step 1, the captured images are Gaussianblurred and binarized and then the resultant contours are detected to determine the coordinate of each dot. In step 2, to match each corresponding dot pair, we use a point set registration method called coherent point drift (CPD) (Supplementary Note 3)^{51}. Due to its probabilistic modeling approach, CPD is found to be robust against outliers and missing points and can preserve the topological structure of the point sets. Note that the CPD algorithm has to compute the inversion of a M \(\times\) M matrix per iteration and therefore has a computation complexity of \(O({M}^{3})\) in the case of a nonrigid transformation, where M represents the number of points. In step 3, based on the matched point pairs and the obtained camera parameters, the depth of each dot is calculated using the camera trigonometry relation.
Based on the stereo matching algorithm and by taking advantage of fullspace diffractive metasurfaces, a wide FOV 3D sensing is demonstrated. Here, we use a 6 mWpower laser source. Due to the limited power of the laser and the power dispersion of the highdensity diffraction orders, the beam intensity drops as the distance and the viewing angle from the metasurface get larger. In addition, because the diffracted beam spreads out in full hemisphere, the spacing between the dots expands as it gets further away from the metasurface, degrading the resolution for faraway objects. Here, the depolarization effect at largeangle diffraction is also considered which distorts the beam shape into elliptical shape (Supplementary Note 4). The vectorial Debye theory can be considered to alleviate distortion, which originates from the deviation of the phase distribution between the metasurface plane and spherical wavefront^{10,11,12}. Therefore, we characterize the system with maximum axial working distance as 1 m and maximum viewing angle as 60° from optical axis. Although the system is not suitable for objects placed in ranges above these values, such limitations can be overcome using a higherpower laser and designing the metasurface with larger periodicity of the supercells.
We reconstruct the depth of two objects; one placed normally with respect to the laser beam axis, and the other placed at 50° from the axis (Fig. 4c), which shows the wide FOV characteristic of our system while maintaining its depth reconstruction quality. In both cases, the reconstruction of the 3D points in Fig. 4d shows a wellpreserved shape of the original face mask with absolute depth error of 8.5% and 1.9% for the object 1 and 2, respectively. In addition, from the reconstruction results, we calculate the relative difference between the maximum and minimum depth point which is 8.4 cm for object 1 and 6.5 cm for object 2. Considering the fact that the actual depth difference (distance from the nearest to the farthest point of face mask) of both objects is 8 cm, the estimation results are fairly accurate. In case of object 2, the relative depth difference is much shorter. This is because of the shaded, sparse projection of dots which is inevitable for objects placed in large FOV, where the dots cannot be projected at shaded, deep surface.
The time taken for the stereo matching algorithm to reconstruct one fullframe depth map was 0.2436 s and 0.3466 s for object 1 and object 2 respectively. This corresponds to roughly 3 to 4 fps processing speed, which is fairly slow compared to the minimum of 30 fps in realtime operating devices. Because of the inherent computational complexity of the CPD algorithm, the calculation time is largely dependent on the number of points. Therefore, the speed of the depth reconstruction process will be slower if there are a greater number of points. Such limitations can be alleviated by adopting other faster algorithms^{52}. We did not perform any continuous depth reconstruction task for moving objects in this work due to a lack of the required experimental setup. However, in principle, as long as the number of processing point pairs is comparable to 300 (which was the maximum number of captured points in the experiments), it is possible to operate our system in the scale of 1 fps speed with support from highspeed steaming cameras and an integrated software framework.
Prototype demonstration with scalable nanofabrication
We demonstrate a prototype of the metasurfacebased SL imaging device using a scalable nanoPER imprinting fabrication. Unlike conventional electron beam lithography (EBL) based metasurface fabrication, nanoimprinting can be easily used for highthroughput replication of metasurfaces on any arbitrary (e.g., flat, curved, flexible) substrates. Here, we design a metasurface to generate parallel line patterns for depth extraction. These line patterns are widely used in SL 3D scanning technology^{9}. When a single strip of line is projected onto the object scene, the depth information of the object on that line can be reconstructed by intersecting the projection line of the examined image point^{53}. SL enables 3D reconstruction without linebyline scanning by projecting a known parallel set of line pattern onto the object scene and measuring how much it has been altered by the object. The correspondence matching between the elements in the projected and captured patterns requires an encoding of the pattern, and hence, numerous encoding algorithms using timemultiplexing^{54,55,56,57}, lineshifting^{58}, and dynamic scenes^{56,59} have been studied. The proposed fullspace diffractive metasurface can be designed to project patterns needed in these algorithms. By applying the convolution theorem with differently designed kernel functions, i.e., the phase profile of the supercell, an arbitrary period and widths of the projected lines can be achieved by arranging the supercells with different periodicities as demonstrated in Supplementary Fig. 7b.
After designing the metasurface, we used soft mold with nanoPER to transfer the pattern of the metasurface to a curved surface of a pair of safety glasses. As shown in Fig. 5a, the master stamp is fabricated using EBL (Fig. 5b) and is then coated with a soft mold composed of hard polydimethylsiloxane (hPDMS) followed by polydimethylsiloxane (PDMS). The ultraviolet (UV) curable nanoPER is then spincoated on the soft mold (Fig. 5c). Here, TiO_{2} nanoparticles are used due to their low extinction coefficient at 633 nm (Supplementary Fig. 4). However, due to the lower refractive index of TiO_{2} nanoPER compared to aSi:H at the operation wavelength, the height of nanoPERbased metasurface is designed as 900 nm as shown in tilted view of master mold (Fig. 5b), which is higher than 475 nm height of aSi:H based metasurface. After putting the nanoPER coated soft mold on the curved surface of glasses, the nanoPER is cured under an UV light source. Finally, the replication of the metasurface is completed on the curved surface after releasing the cured resin (Fig. 5d). Note that the release process of the mold is enhanced by increasing surface tension of the target curved surface using oxygen (O_{2}) plasma and decreasing the surface tension of the soft mold using fluorosurfactant coating.
By illuminating laser light to the nanoPERbased metasurface printed on glasses (Fig. 5e), 2D line array pattern is experimentally demonstrated to cover the fullspace as shown in Fig. 5f. Here, the size of printed metasurface is 510 μm by 510 μm. Note that, due to the nonlinearity between spatial frequency and propagation angle described as Eq. (1), line pattern on the flat observation plane shows curved shape at large angle with respect to optical axis. Although we do not provide an explicit demonstration of depth estimation using these patterns, we highlight the versatility of our proposed metasurface with scalable nanofabrication methods in that it can be developed to generate the different patterns required for existing 3D reconstruction algorithms.
Discussion
In conclusion, we numerically simulated and analyzed the optical response of a metasurface for 3D depth reconstruction, and experimentally validated the fullspace diffracted features of the proposed metasurface. Regarding the supercell as the kernel function of the convolution, the diffraction pattern of the entire metasurfaces composed of periodic supercells was analyzed using the convolution theorem, realizing the number of desired diffracted SL patterns needed for 3D object recognition. As a result, a flat metasurfacebased SL imaging system has been realized with highdensity order of diffraction covering a 180° FOV. In a singleshot of projected dots, the depth information of 3D objects placed at both normal and wide angles was extracted by matching the monitored points at two different positions using a stereo matching algorithm. The demonstrated stereo system can also adopt other unique, optimized patterns with designed metasurfaces, for further enhancement of their robustness and accuracy^{60}. Moreover, we demonstrate a prototype of AR glasses by printing SL projecting metasurface on a curved surface of glasses using the nanoPER imprinting method which enables highthroughput manufacturing of the metasurfaces on any arbitrary substrates.
In terms of operating distance, the proposed metasurfacebased SL imaging has limited range compared to single laser scanning, due to the power dispersion through the spreading of light to the multiple diffraction orders. In addition, the spacing between diffracted beams is increased at farther distance from metasurface and at larger angle from optical axis, resulting in limited resolution of imaging due to sparse dot patterns on the object. In this work, the incident power to the metasurface was 6 mW at the operation wavelength of 633 nm, generating ~10 K dots. We achieved operating range of 1 m from metasurface, and 60° from optical axis, i.e., 120° FOV, even though the diffracted SL patterns covered 180° FOV. The operating range can be enhanced using high power incident laser or higher density of dot patterns, which can be achieved by increasing the number of metaatoms comprising supercell. We experimentally demonstrate the operation of metasurfacebased SL imaging reaching long distances with a high power (100 mW) laser (Supplementary Fig. 10). However, considering the safety issues, the invisible nearinfrared (NIR) light source at 1550 nm can be considered, where this range of light is strongly absorbed at the cornea and lens, therefore do not reach to the retina. To do so, silicon nanoparticles^{61}, which have a high refractive index and low extinction coefficient at the NIR can be used instead of the TiO_{2} nanoparticles, enabling high power operation at longer distances. Furthermore, such a flat passive metasurfacebased SL imaging does not require the application of a linebyline voltage bias compared to the TCO^{6} and LC^{7,8} based active beam steering approaches, alleviating the requirement of extremely complex wiring configurations. Also, by taking advantage of the flat feature of the proposed metasurfaces, light engines such as VCSELs can be integrated, realizing onchip illuminating devices for ultracompact and lightweight 3D imaging systems.
Methods
Metasurface fabrication
Metasurface was fabricated on a glass substrate. A 475 nm thick aSi:H layer was deposited using plasma enhanced chemical vapor deposition (PECVD, BMR Technology HiDepSC) with silane (SiH_{4}) and hydrogen (H_{2}) gases at a flow rate of 10 sccm and 75 sccm, respectively, at the optimized 25 mTorr pressure and 200 °C. The designed pattern of the metasurface was transferred to positive tone photoresist of polymethyl methacrylate (950 PMMA A2, MicroChem) using a standard EBL process (ELIONIX, ELS7800; acceleration voltage: 80 kV, beam current: 100 pA). After the development process using MIBK/IPA 1:3 developer, 40 nm thickness chromium (Cr) was deposited using electron beam evaporator (KVT, KVEENS4004) and liftoffed. Using Cr patterns as an etching mask, the metasurface pattern was transferred to the aSi:H layer by dry etching process (DMS, silicon/metal hybrid etcher). Finally, the remaining Cr mask was removed using Cr etchant (CR7).
Synthesis of TiO_{2} nanoparticle embedded resin
The TiO_{2} nanoparticles were dispersed into MIBK (DTTIOA30MIBK (N30), Ditto technology), monomer (dipentaerythritol penta/hexa acrylate, SigmaAldrich), photoinitiator (1Hydroxycyclohexyl phenyl ketone, SigmaAldrich), and MIBK solvent (MIBK, Duksan general science). The weight ratio was selected as 4, 0.7, and 0.3 wt% for TiO_{2} nanoparticles, monomer, and photoinitiator, respectively.
Measurements
To characterize the scattering properties of the fullspace diffractive metasurface, we demonstrated an optical setup as shown in Fig. 4a. The incident light from the solidstate He–Ne laser (HNL210L, Thorlabs), with wavelength of 633 nm and power of 6 mW, was cropped by pinhole aperture to fit into the metasurface. When the light passes through the metasurface, the highdensity dot patterns are generated and projected onto the fullspace. The scattered points from the surface of the object were captured by two cameras, which were positioned in parallel with a slight coordinate difference.
Depth information extraction
Stereo cameras were calibrated using MATLAB camera calibrator application. Images of the standard checkerboard pattern in 15–20 different angle and positions were taken by two cameras respectively. Then, effective camera parameters are retrieved by detecting and matching the points on the pattern. Higherror images were excluded to improve calibration. The depth of each dot was calculated using the Image Processing and Computer Vision APIs in MATLAB.
Data availability
The data that support the findings of the study are available from the corresponding author upon reasonable request.
Code availability
The code used for the metasurface design is available from the corresponding author upon reasonable request.
References
LiDAR drives forwards. Nat. Photon. 12, 441 (2018).
Lee, J., Kim, Y.J., Lee, K., Lee, S. & Kim, S.W. Timeofflight measurement with femtosecond light pulses. Nat. Photon. 4, 716–720 (2010).
Coddington, I., Swann, W. C., Nenadovic, L. & Newbury, N. R. Rapid and precise absolute distance measurements at long range. Nat. Photon. 3, 351–356 (2009).
Huang, Y.W. et al. Gatetunable conducting oxide metasurfaces. Nano Lett. 16, 5319–5325 (2016).
Kafaie Shirmanesh, G., Sokhoyan, R., Pala, R. A. & Atwater, H. A. Dualgated active metasurface at 1550 nm with wide (>300°) phase tunability. Nano Lett. 18, 2957–2963 (2018).
Park, J. et al. Allsolidstate spatial light modulator with independent phase and amplitude control for threedimensional LiDAR applications. Nat. Nanotechnol. 16, 69–76 (2021).
Li, S.Q. et al. Phaseonly transmissive spatial light modulator based on tunable dielectric metasurface. Science 364, 1087–1090 (2019).
Mansha, S. et al. High resolution multispectral spatial light modulators based on tunable Fabry–Perot nanocavities. Light Sci. Appl. 11, 141 (2022).
Geng, J. Structuredlight 3D surface imaging: a tutorial. Adv. Opt. Photonics 3, 128–160 (2011).
Gu, M. Advanced Optical Imaging Theory (Springer, 1999).
Lin, H., Jia, B. & Gu, M. Dynamic generation of Debye diffractionlimited multifocal arrays for direct laser printing nanofabrication. Opt. Lett. 36, 406–408 (2011).
Ren, H., Lin, H., Li, X. & Gu, M. Threedimensional parallel recording with a Debye diffractionlimited and aberrationfree volumetric multifocal array. Opt. Lett. 39, 1621–1624 (2014).
Kim, I. et al. Nanophotonics for light detection and ranging technology. Nat. Nanotechnol. 16, 508–524 (2021).
Lin, D., Fan, P., Hasman, E. & Brongersma, M. L. Dielectric gradient metasurface optical elements. Science 345, 298–302 (2014).
Arbabi, A., Horie, Y., Bagheri, M. & Faraon, A. Dielectric metasurfaces for complete control of phase and polarization with subwavelength spatial resolution and high transmission. Nat. Nanotechnol. 10, 937–943 (2015).
Shalaev, V. M. Optical negativeindex metamaterials. Nat. Photon. 1, 41–48 (2007).
Ni, X., Wong, Z. J., Mrejen, M., Wang, Y. & Zhang, X. An ultrathin invisibility skin cloak for visible light. Science 349, 1310–1314 (2015).
Yu, N. et al. Light propagation with phase discontinuities: Generalized laws of reflection and refraction. Science 334, 333–337 (2011).
Yu, N. & Capasso, F. Flat optics with designer metasurfaces. Nat. Mater. 13, 139–150 (2014).
Meinzer, N., Barnes, W. L. & Hooper, I. R. Plasmonic metaatoms and metasurfaces. Nat. Photon. 8, 889–898 (2014).
Koshelev, K. & Kivshar, Y. Dielectric resonant metaphotonics. ACS Photonics 8, 102–112 (2021).
Jahani, S. & Jacob, Z. Alldielectric metamaterials. Nat. Nanotechnol. 11, 23–36 (2016).
Guo, Q. et al. Compact singleshot metalens depth sensors inspired by eyes of jumping spiders. Proc. Natl Acad. Sci. USA 116, 22959–22965 (2019).
Kim, K., Yoon, G., Baek, S., Rho, J. & Lee, H. Facile nanocasting of dielectric metasurfaces with sub100 nm resolution. ACS Appl. Mater. Interfaces 11, 26109–26115 (2019).
Yoon, G., Kim, K., Huh, D., Lee, H. & Rho, J. Singlestep manufacturing of hierarchical dielectric metalens in the visible. Nat. Commun. 11, 2268 (2020).
Jang, J. et al. Spectral modulation through the hybridization of miescatterers and quasiguided mode resonances: realizing full and gradients of structural color. ACS Nano 14, 15317–15326 (2020).
Joo, W.J. et al. Metasurfacedriven OLED displays beyond 10,000 pixels per inch. Science 370, 459–463 (2020).
Badloe, T. et al. Liquid crystalpowered Mie resonators for electrically tunable photorealistic color gradients and dark blacks. Light Sci. Appl. 11, 118 (2022).
Zheng, G. et al. Metasurface holograms reaching 80% efficiency. Nat. Nanotechnol. 10, 308–312 (2015).
Li, Z. et al. Dielectric metaholograms enabled with dual magnetic resonances in visible light. ACS Nano 11, 9382–9389 (2017).
Rubin, N. A. et al. Matrix Fourier optics enables a compact fullStokes polarization camera. Science 365, eaax1839 (2019).
Song, Q. et al. Ptychography retrieval of fully polarized holograms from geometricphase metasurfaces. Nat. Commun. 11, 2651 (2020).
Huang, C. et al. Ultrafast control of vortex microlasers. Science 367, 1018–1021 (2020).
Divitt, S., Zhu, W., Zhang, C., Lezec, H. J. & Agrawal, A. Ultrafast optical pulse shaping using dielectric metasurfaces. Science 364, 890–894 (2019).
Ni, Y. et al. Metasurface for structured light projection over 120° field of view. Nano Lett. 20, 6719–6724 (2020).
Li, N. et al. Largearea pixelated metasurface beam deflector on a 12inch glass wafer for random point generation. Nanophotonics 8, 1855–1861 (2019).
Li, Z. et al. Fullspace cloud of random points with a scrambling metasurface. Light Sci. Appl. 7, 63 (2018).
Chen, K. et al. 2πspace uniformbackscattering metasurfaces enabled with geometric phase and magnetic resonance in visible light. Opt. Express 28, 12331–12341 (2020).
Xie, Y.Y. et al. Metasurfaceintegrated vertical cavity surfaceemitting lasers for programmable directional lasing emissions. Nat. Nanotechnol. 15, 125–130 (2020).
Wen, D., Meng, J., Cadusch, J. J. & Crozier, K. B. VCSELs with onfacet metasurfaces for polarization state generation and detection. Adv. Opt. Mater. 9, 200 (1780).
Wang, Q.H. et al. Onchip generation of structured light based on metasurface optoelectronic integration. Laser Photonics Rev. 15, 2000385 (2021).
Vaskin, A., Kolkowski, R., Koenderink, A. F. & Staude, I. Lightemitting metasurfaces. Nanophotonics 8, 1151–1198 (2019).
Song, X. et al. Selective diffraction with complex amplitude modulation by dielectric metasurfaces. Adv. Opt. Mater. 6, 1701181 (2018).
Wang, H. et al. Offaxis holography with uniform illumination via 3D printed diffractive optical elements. Adv. Opt. Mater. 7, 1900068 (2019).
Goodman, J. W. Introduction to Fourier Optics (McGrawHill, 1996).
Bowen, D. K. & Tanner, B. K. High Resolution Xray Diffractometry and Topography (CRC Press, Taylor & Francis Group, 2019).
Devlin, R. C., Khorasaninejad, M., Chen, W. T., Oh, J. & Capasso, F. Broadband highefficiency dielectric metasurfaces for the visible spectrum. Proc. Natl Acad. Sci. USA 113, 10473–10478 (2016).
Huang, T.Y. et al. A monolithic immersion metalens for imaging solidstate quantum emitters. Nat. Commun. 10, 2392 (2019).
Shen, F. & Wang, A. FastFouriertransform based numerical integration method for the RayleighSommerfeld diffraction formula. Appl. Opt. 45, 1102–1110 (2006).
Scaramuzza, D., Martinelli, A. & Siegwart, R. A Toolbox for easily calibrating omnidirectional cameras. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 5695–5701 (IEEE, 2006).
Myronenko, A. & Song, Xubo Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32, 2262–2275 (2010).
Feng, X. W., Feng, D. Z. & Zhu, Y. Fast coherent point drift. Preprint at https://arxiv.org/abs/2006.06281 (2020).
Klette, R., Koschan, A., & Schlüns, K. Computer vision: Räumliche information aus digitalen bildern (SpringerVerlag, 2013).
Posdamer, J. L. & Altschuler, M. D. Surface measurement by spaceencoded projected beam systems. Comput. Graph. Image Process. 18, 1–17 (1982).
Inokuchi, S., Sato, K. & Matsuda, F. Rangeimaging system for 3D object recognition. In Proceedings of the International Conference on Pattern Recognition (ICPR) 806–808 (1984).
Rusinkiewicz, S., HallHolt, O. & Levoy, M. Realtime 3D model acquisition. ACM Trans. Graph. 21, 438–446 (2002).
Young, M., Beeson, E., Davis, J., Rusinkiewicz, S. & Ramamoorthi, R. Viewpointcoded structured light. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1–8 (IEEE, 2007).
Guehring, J. Dense 3D surface acquisition by structured light using offtheshelf components. In Videometrics and Optical Methods for 3D Shape Measurement Vol. 4309, 220–231 (SPIE, 2000).
Zhang, L., Curless, B. & Seitz, S. M. Rapid shape acquisition using color structured light and multipass dynamic programming. In Proc. First International Symposium on 3D Data Processing Visualization and Transmission 24–36 (IEEE Comput. Soc, 2002).
Baek, S.H. & Heide, F. Polka Lines: Learning structured illumination and reconstruction for active stereo. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5753–5763 (IEEE, 2021).
Yoon, G. et al. Printable nanocomposite metalens for highcontrast nearinfrared imaging. ACS Nano 15, 698–706 (2021).
Acknowledgements
This work was financially supported by the Samsung Research Funding & Incubation Center for Future Technology grant (SRFCIT190152) funded by Samsung Electronics, the POSCOPOSTECHRIST Convergence Research Center program funded by POSCO, and the National Research Foundation (NRF) grant (NRF2022M3C1A3081312) funded by the Ministry of Science and ICT (MSIT) of the Korean government. G.K. acknowledges the POSTECH Alchemist fellowship. Y.K. acknowledges the Hyundai Motor Chung MongKoo fellowship, and the NRF Ph.D. fellowship (NRF2022R1A6A3A13066251) funded by the Ministry of Education of the Korean government. Y.K. and J.Y. acknowledge the NRF International Research & Development fellowships (NRF2022K1A3A1A12080445 and NRF2022K1A3A1A12080092), respectively, funded by the MSIT of the Korean government. I.K. acknowledges the NRF Sejong Science fellowship (NRF2021R1C1C2004291) funded by the MSIT of the Korean government. The authors thank Dr. Junghyun Park and Dr. Seunghoon Han (Samsung Advanced Institute of Technology) for discussions.
Author information
Authors and Affiliations
Contributions
J.R., I.K., and G.K. conceived the initial idea, and developed the concept. G.K. calculated and designed the fullspace diffractive metasurfaces. G.K. and S.K. performed the simulation of diffractive characteristics of proposed metasurfaces. I.K. and G.K. fabricated the metasurface. G.K., Y.K., I.K., and J.P. conducted the optical measurements. J.Y. developed the stereo matching algorithms and performed the depth estimation of 3D objects. Y.K. and J.P. prepared the data sets of 3D objects for depth estimation. J.K. synthesized the TiO_{2} nanoparticles and conducted replication of metasurface. S.W.M., S.K., and T.B. provided theoretical advice. All authors wrote the manuscript, contributed to the discussion and analysis. J.R. guided the entire project.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks Yuanmu Yang, Haoran Ren, and Wei Ting Chen for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kim, G., Kim, Y., Yun, J. et al. Metasurfacedriven fullspace structured light for threedimensional imaging. Nat Commun 13, 5920 (2022). https://doi.org/10.1038/s41467022321172
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467022321172
This article is cited by

Quasicrystal metasurface for dual functionality of holography and diffraction generation
eLight (2024)

Ultrarobust informational metasurfaces based on spatial coherence structures engineering
Light: Science & Applications (2024)

360° structured light with learned metasurfaces
Nature Photonics (2024)

Wide fieldofhearing metalens for aberrationfree sound capture
Nature Communications (2024)

CostEffective and Environmentally Friendly Mass Manufacturing of Optical Metasurfaces Towards Practical Applications and Commercialization
International Journal of Precision Engineering and ManufacturingGreen Technology (2024)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.