Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network

Neuromorphic photonics has recently emerged as a promising hardware accelerator, with significant potential speed and energy advantages over digital electronics for machine learning algorithms, such as neural networks of various types. Integrated photonic networks are particularly powerful in performing analog computing of matrix-vector multiplication (MVM) as they afford unparalleled speed and bandwidth density for data transmission. Incorporating nonvolatile phase-change materials in integrated photonic devices enables indispensable programming and in-memory computing capabilities for on-chip optical computing. Here, we demonstrate a multimode photonic computing core consisting of an array of programable mode converters based on on-waveguide metasurfaces made of phase-change materials. The programmable converters utilize the refractive index change of the phase-change material Ge2Sb2Te5 during phase transition to control the waveguide spatial modes with a very high precision of up to 64 levels in modal contrast. This contrast is used to represent the matrix elements, with 6-bit resolution and both positive and negative values, to perform MVM computation in neural network algorithms. We demonstrate a prototypical optical convolutional neural network that can perform image processing and recognition tasks with high accuracy. With a broad operation bandwidth and a compact device footprint, the demonstrated multimode photonic core is promising toward large-scale photonic neural networks with ultrahigh computation throughputs.

T he unmet gap between the rate of energy efficiency improvement of current digital electronics and the fastgrowing load of computation by emerging applications such as machine learning and artificial intelligence 1,2 has once again brought optical computing into focus [3][4][5][6] . Integrated photonics provides a scalable hardware platform to realize large-scale optical networks on a chip, which affords an enormous bandwidth density that is unreachable for electronics [7][8][9] . To use integrated photonics for optical computing, programmable photonic components and nonlinear elements are indispensable building blocks. Phase-change materials (PCM) recently emerged as an ideal material system to realize optical programmability [10][11][12] . The optical properties of PCMs change dramatically during the phase transition, which can be electrically or optically controlled. Harnessing this has allowed for embodiments of programmable optical switches, couplers, lens, and metamaterials to be demonstrated [13][14][15][16][17][18][19][20][21] . The phase change in the chalcogenide family of Ge-Sb-Te alloys is nonvolatile, requiring no sustaining power supply to retain the programmed state or stored information [19][20][21][22][23][24][25][26] . Their use in programmable photonic devices thus can have a significant advantage in power consumption over electro-optic [27][28][29] or thermo-optic methods [30][31][32] . Photonic devices incorporating those nonvolatile PCMs thus can realize optical memories and perform in-memory computing simply by measuring the transmission of the optical input data through the programmed device [33][34][35] . Proliferating these phase-change photonic devices in a scalable network, prototypes of optical neural networks (ONN) have been proposed and demonstrated [35][36][37][38] .
Here, we report a programmable waveguide mode converter based on a phase-gradient metasurface made of phase-change material Ge 2 Sb 2 Te 5 (GST). This phase-change metasurface mode converter (PMMC) utilizes GST's large refractive index change during its phase transition to control the conversion of the waveguide's two spatial modes (TE 0 and TE 1 modes). The PMMC can be programmed to control the waveguide mode contrast precisely at 64 distinguishable levels, which is used to represent the weight parameters with 6-bit precision in MVM computation. We build a 2 × 2 array of PMMCs and implement them as programmable kernels to realize a multimode optical convolutional neural network (OCNN). By performing image processing tasks such as edge detection and pattern recognition, we demonstrate the OCNN's viability and potential in largescale optical computing.

Results
High-precision programmable phase-change mode converter. The design of the PMMC is based on the principle of a phasegradient metasurface but replacing noble metals with phase-change materials 39 . Fig. 1a shows a 3D schematic of the design, which consists of a linear array of GST nano-antennae directly integrated on a silicon nitride (SiN) waveguide. Each GST nano-antenna scatters the waveguide mode and causes a phase shift Φ, which depends on its geometry (e.g., width), as well as the refractive index of its material (Fig. 2b). A linear array of such nano-antennae with tapering widths thus produces a spatial gradient of the scattering phases dΦ/dx, which is equivalent to a wavevector k g . If the phasegradient metasurface is designed such that k g matches the wavevector difference between two spatial modes of the waveguide: k mode1 −k mode2 , it satisfies the phase-matching condition and facilitates the conversion between the two modes. Such phase-gradient metasurfaces for waveguide mode conversion were realized using noble metals or dielectrics materials and thus lacked tunability. Here, we use GST, which has a large change in its optical properties when a phase transition happens. When the GST is in the amorphous phase (aGST), its refractive index n is~4.7 (representative value in the literature, the same hereafter) 40 . In contrast, when it is turned to the crystalline phase (cGST), n increases to~7.5 with a drastic change of 2.8 over the whole measured spectral range from 1540 nm to 1580 nm (See Supplementary Fig. 1a for more detailed information). This change will significantly modify the scattered phase of each GST nano-antenna (Fig. 2b) so as to modify the metasurface's function. Fig. 1c plots the simulated phase of the scattered fields inside the waveguide by a single nano-antenna of 30nm-thick GST as a function of its width and for aGST and cGST phases. Since cGST has a much larger n, the scattered phase shows a much stronger dependence on the width than the aGST phase. By controlling the geometry of the GST nano-antennae and the interval between adjacent ones in the array, a well-defined phase gradient dΦ/dx is established (see Supplementary Note 2 and 3 for details). The entire metasurface consists of an array of 25 nano-antennae with tapering widths from 510 nm to 84 nm (shaded region in Fig. 1c) and is patterned on a SiN waveguide 1.8 µm wide and 330 nm thick. The waveguide supports two transverse-electric modes: the fundamental TE 0 mode and the first-order TE 1 mode. We design the metasurface, in the cGST phase, to have a uniform dΦ = 2.5°for every dx = 400 nm to satisfy the generalized phase-matching condition, k 0 ðn TE0 À n TE1 Þ ¼ N Á dΦ=dx, where k 0 is the free-space wavevector, n TE0 and n TE1 are the effective index of the TE 0 and TE 1 modes, respectively, and N is the number of interactions between the guided modes and the metasurface. The cGST metasurface thus can efficiently convert the TE 0 mode to the TE 1 mode, as shown by the finite-difference time-domain (FDTD) simulation result in Fig. 1d. When the GST is transitioned to the aGST phase, as shown in Fig. 1c, the dΦ/dx is much reduced and thus insufficient for the phase-matching condition so that mode conversion between TE 0 and TE 1 modes does not occur, which is clearly seen in Fig. 1e. Therefore, the GST phase-gradient metasurface, as designed here, functions as a programmable waveguide mode converter controlled by the tunable material phase of the GST. Fig. 2a-c shows the scanning electron microscope images of the complete PMMC device. The 30 nm thick GST film is deposited by sputtering on Si 3 N 4 on an oxidized silicon substrate. It is then patterned into metasurface with electron beam lithography and plasma etching, and conformally encapsulated with a 218-nm-thick layer of Al 2 O 3 deposited by atomic layer deposition. The photonic circuits of Si 3 N 4 , including multimode waveguides, directional couplers and grating couplers, are patterned with standard processes 17 . A pair of asymmetric directional couplers (Fig. 2c) is designed to function as mode selectors to selectively couple only the TE 1 mode component in the multimode waveguide with the TE 0 mode component in the single-mode waveguide (See Supplementary Note 5 for details). Fig. 2a depicts the measurement and control scheme. To program the PMMC, we use optical pulses to control the phase of the GST film for simplicity 41 . Previously, electrical control using integrated micro-heaters has been demonstrated by a number of groups, including us 17,26,[42][43][44] . When operating the PMMC, an optical signal is input in the TE 0 mode to the PMMC and converted to TE 1 mode with a proportion controlled by the state of the GST metasurface. At the output of the PMMC, the TE 1 component is separated by the mode selector and coupled out at the second port while the TE 0 component remains in and outputs from the multimode waveguide. The output powers of both modes are measured to determine their respective transmission coefficients. Fig. 2d shows the transmission spectrum of the PMMC when the metasuface is set to be either in the fully aGST or cGST phases. The insertion losses of the input and output fibers and grating couplers have been accounted for by calibration measurements. In the aGST phase, the device is in the on-state for the TE 0 mode with a high transmission T on over a broad wavelength range (1540-1580 nm).
The lowest insertion loss is 0.9 dB at 1575 nm wavelength. A small portion (<−10 dB) of the TE 1 mode is generated due to the asymmetric perturbation induced by the metasurface even though the aGST phase has a low refractive index. The situation changes dramatically when the metasurface is transitioned to the cGST phase and converts the TE 0 mode to the TE 1 mode effectively. In this off-state for the TE 0 mode, its transmission T off is <−15 dB over the entire measured bandwidth. The corresponding switching extinction ratio, defined as ΔT=T off ¼ T on À T off À Á =T off , is~16 dB or 4000%, which is more than 10-fold improvement compared to previously reported switch devices using GST 24,43,45 . This large switching ratio stems from the phase engineering approach to effectively use GST's large refractive index change during its phasetransition, as opposed to only using the absorption coefficient change, to facilitate scattering into a different mode that is filtered. The total area of the GST in the metasurface is only 1.3 µm 2 , significantly smaller than that in prior devices, and thus in principle, our device consumes less energy to switch. As expected from energy conservation, the TE 1 mode is switched in the opposite way to the TE 0 mode. From aGST to cGST phase, the TE 1 transmission increases from~−10 dB to~−6.5 dB, with the insertion loss due to cGST's absorption. Another important parameter to quantify a mode converter's performance is the mode purity in the multimode waveguide, defined as β TE0ðTE1Þ ¼ P TE0ðTE1Þ = P TE0 þ P TE1 ð Þ , where P TE0 (P TE1 ) is the power in the TE 0 (TE 1 ) mode. The PMMC shows very high performance in controlling mode purity. As shown in Fig. 2e, when switching the GST from aGST to cGST phase, the PMMC efficiently converts TE 0 mode to TE 1 mode, changing the mode purity from β TE0 > 80% to β TE1 > 85% over a broad bandwidth, showing an excellent agreement with the numerical simulation results.
PMMC photonic kernel. The phase composition of the GST in the metasurface can be continuously tuned by partial phase transition so that the PMMC can be continuously programmed to multiple intermediate levels of phase purity values. We program the PMMC with a sequence of 50-ns-long control pulses to "quench" the GST progressively from the fully cGST phase toward the fully aGST phase. As a result, the TE 1 mode purity β TE1 increases stepwise. Since the mode selector separates the two modes, we can measure their power and calculate the difference to determine the mode contrast Γ ¼ β TE0 À β TE1 , which is used as a programming parameter. Fig. 2f demonstrates the multi-level programmability of the PMMC, in which Γ is sequentially set to 64 distinguishable levels between −0.73 to +0.67 at 1555 nm. Since the theoretical range of Γ is ðÀ1; 1Þ, it is an ideal parameter to represent the elements in the matrix w, with both positive and negative values, in a multiplyaccumulate (MAC) operation: MAC is the constitutional step of matrix-vector multiplication (MVM) in all neural network algorithms. The PMMC allows storing w by programming Γ in the GST metasurface as a nonvolatile memory. In-memory MAC computing can be performed with the PMMC by a measurement of the transmitted power when the input data x is encoded in the power of the input optical signal. The lower inset of Fig. 2f shows the histograms of 20 repeated programming operations to set the PMMC mode contrast at two adjacent levels (levels 30 and 31), respectively. The wellseparated histograms clearly demonstrate the device's programming resolution and accuracy. The demonstrated 64-level programmability of the PMMC-the highest to the best of our knowledge for phase-change photonic devices 24 -corresponds to 6-bit resolution in setting w, which is critical to the training and inference precision of the neural network 46 We harness the PMMC's high-precision programmability and in-memory computing capability to demonstrate an optical convolutional neural network (OCNN) [28][29][30]48 . A typical CNN consists of an input layer and an output layer, which are connected by multiple hidden layers in between. The hidden layers usually consist of a series of convolutional layers followed by pooling layers and fully connected layers at the end. We design a prototype OCNN using a small network of PMMCs to implement patch-kernel matrix multiplication to compute convolution. Fig. 3a illustrates the operation principle of the OCNN for image processing, where an input grayscale image of dimensions n × n is convolved with a kernel of dimensions k × k to compute an activation map of dimension (n-k+1) × (n-k+1), assuming the convolution stride is 1. When operating the OCNN, we group the input image into (n-k+1) 2 patches (the shaded area in the upper panel of Fig. 3a) with the same dimensions as the convolution kernel, k 2 . Each patch corresponds to the receptive field of an element in the activation map accordingly. Thus, a convolution operation requires (n-k+1) 2 × k 2 MAC operations in total, which is a high load of computation and can most benefit from optical computing's speed and energy advantages.
To compute the convolution, (n-k+1) 2 patch matrices of the input image are optically fed into the photonic kernel sequentially while the kernel elements, that is, the PMMCs, are programmed to fixed values. At each timeframe of the computation, the corresponding patch matrix is reshaped into a single column of data with the length k 2 . The data is input into the optical system in k 2 channels as sequences of incoherent optical pulses, whose power amplitude is controlled by a variable optical attenuator (VOA) to encode the value of each pixel value X ij in the greyscale image. The corresponding element W ij of the kernel matrix is programmed as the mode contrast Γ of each PMMC. The resulting transmitted power of TE 0 and TE 1 modes are then summed incoherently using two photodetectors. Their output difference is calculated electronically and used in post-processing steps. As a result, the output will correspond to a time series of patch-kernel MVM with the amplitude encoding the values of the computation results, which is the activation map of convolution. Since the modal contrast Γ of our PMMCs can assume both positive and negative values, it can represent the kernel matrix elements without the need of an additional offset, which otherwise would take additional steps to set in each computation cycle.
Convolutional edge detection with PMMC core. Experimentally, we build a small-scale, four-channel system with four PMMCs to represent a 2 × 2 kernel matrix, as shown in the optical image in Fig. 3b. As a demonstration, we perform the convolution of a 256 × 256 8-bit grayscale image of a cameraman (Fig. 3c) to detect its edge features. As shown in Fig. 3b, the TE 0 mode output coming from all the PMMCs is combined using on-chip Y-junctions, while the TE 1 mode output power is combined off-chip because the same ports are used to program the PMMCs optically. Because combining four incoherent sources using Y-junctions will inherently reduce the power by a factor of 4, we rescale the measured TE 0 mode power by this factor when calculating the power differences between two c Fig. 2 Operation of the programmable metasurface mode converter (PMMC). a Scanning electron microscope (SEM) image of the complete device and the measurement and control schematics. The complete PMMC device consists of an encapsulated GST phase gradient metasurface (red box) and a mode selector (yellow box). The white box appears because of the edge of the 218-nm-thick Al 2 O 3 encapsulating layer. b Zoomed-in SEM image of the phasegradient metasurface on the waveguide before depositing the Al 2 O 3 encapsulation layer for better imaging. c Zoomed-in SEM image of the TE 0 /TE 1 mode selector. d The transmission coefficient (insertion loss) of the devices for TE 0 and TE 1 modes and aGST and cGST phases. The transmission of the TE 0 mode is switched with a high extinction ratio of >16 dB or 4000%. e The mode purity is controlled by the mode converter to >80% for both modes. f The programable mode converter controls the mode contrast Γ at 64 distinct levels, corresponding to 6-bit programming resolution. Upper inset: zoomed-in view of the contrast levels. Lower inset: histograms of 20 programming operations to set the contrast at two adjacent levels (level 30 and 31). The wellseparated histograms demonstrate the programming repeatability and accuracy. ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-020-20365-z modes. To detect vertical and horizontal edges, kernel matrices as in the right column of Fig. 3d, e are used, and so are the PMMCs programmed. Take the vertical edge detection for example, the kernel is set to be À1 1 À1 1 ! so to compute the discrete first-order derivative, X iþ1;j þ X iþ1;jþ1 À X i;j À X i;jþ1 , where i, j are the indices of the input image matrix. Each kernel element W ij is stored as the mode contrast value Γ in the corresponding PMMC, with W ij = 1 (−1) corresponding to the fully aGST (cGST) phase (see Supplementary Note 9 for a more detailed description about the operation procedure). The computed images after convolution without any post-processing are shown in the left column of Fig. 3d, e, for horizontal and vertical edge detection, respectively. The two images are then added to produce the right image in Fig. 3b, which highlights silhouettes of the objects with sharp edges such as the cameraman and the buildings in the original image, while suppressing smooth features such as the sky and the water. The optically computed edge detection image agrees very well with the calculated result using conventional image processing algorithms (see Supplementary Fig. 16). This result verifies the capacity and fidelity of optical convolution performed with the PMMC-based photonic kernel, which is a prerequisite for an OCNN.
OCNN for image recognition. Beyond the convolution layer, the MAC computation performed with optical signals and the PMMC network can also be applied to the pooling (average pooling) and the fully connected layers, where the PMMCs are used as weight banks instead, to realize a complete OCNN. In our experiment, we sequentially reuse the PMMC array in both convolution and fully connected layers to demonstrate an OCNN and perform proof-of-concept imaging recognition tasks of distinguishing handwritten numbers "1" and "2" from the MNIST database. Fig. 4a illustrates the architecture and processes of the OCNN. The 28 × 28 pixels, 8-bit grayscale images of number "1" or "2" are fed into the input layer as optical signals. The data is then convolved with two 2 × 2 photonic kernels K 1 and K 2 to generate two 27 × 27 images of activation maps. After adding a bias b 1 and applying the nonlinear ReLu function, the output images are sent to an average pooling layer with a subsampling factor of 27, which reduces the images a 2 × 1 vector. This vector is then fed into the fully connected layer with a 2 × 2 photonic weight bank K 3 programmed in the PMMC array, added with a bias b 2 and applied the standard sigmoid function. The final output is a vector that gives the identified class of the input image, that is, 1 0 ½ T corresponds to the number "1" and 0 1 ½ T corresponds to the number "2". In this OCNN, the MVM computations such as the convolution and the fully connected layers are all performed optically with the PMMCs, whereas bias and nonlinear functions are realized electronically.
Before using the OCNN, we first train all the parameters in the layers with the standard back-propagation algorithm using the gradient descent method 49 . The training set consists of 11,000 images of the handwritten number "1" or "2" from MNIST training images. The training yields values for each element in the convolutional kernels and the weight bank, as shown in Fig. 4b, d. We then program the PMMC array to represent these elements. In Fig. 4c, we show the raw data of the convolutional activation maps encoded in time series of optical signals, which are the output from the PMMC array after the input image convolves with the photonic kernel K 1 and K 2 . Since each photonic processing layer results in electrical signals output from the photodetectors, electronic post-processing is performed to add bias and apply nonlinear function and pooling. The resultant data is re-coded into optical signals and fed to the next photonic layer. Further experimental details are included in the Supplementary Note 9. We evaluate the system's performance after training on a recognition test set, which consists of 100 randomly chosen "1" or "2" images (55 number "1" and 45 number "2") from the MNIST testing image database. Fig. 4e shows the result that our OCNN correctly identified 91 out of 100 cases (9% error rate), which compares squarely with the result of a computer (10% error rate). The slight difference is mainly caused by the small deviation of the experimentally programmed values in the matrices (K 1 , K 2 and K 3 ) from the trained values, which occurs when the system's conditions drift during operation. This result successfully demonstrates the OCNN's viability and accuracy in performing standard neural network algorithms.

Discussion
In summary, we have demonstrated a compact programmable waveguide mode converter using GST-based phase-gradient metasurface with high programming resolution, efficiency and broadband operation. We have built a photonic kernel based on an array of such PMMC devices and implemented an optical convolutional neural network to perform image processing and recognition tasks. Our results show that phase-change photonic devices, such as the PMMC demonstrated here, can enable robust and flexible programmability and realize a plethora of unique optical functionalities that are scalable for large-scale optical computing and neuromorphic photonics. Although optical computation in this work is performed at a low speed of~1 kHz by using low-speed VOAs to encode data into optical signals, state-of-the-art integrated photonic transmitters and photodetectors can drive the system at a speed of many 10 s of Gbits/ sec 50,51 . Using wavelength division multiplexing (WDM) can further increase the number of parallel computation. The 2 × 2 array prototype system demonstrated in this work performs optical computation incoherently in a broadband. It can be scaled up toward a large network using a photonic crossbar array architecture 52-57 (see Supplementary Note 10 for details of such a design), and compares favorably with other photonic computing schemes using coherent methods 30 or optical resonators 28,58,59 . The feasible size (n × m) of such crossbar arrays will not be limited by the insertion loss of the PMMC (~7 dB for TE 1 mode, Fig. 2d); rather it will be limited by the directional couplers with coupling efficiency of 1/n, as is needed to equally combine signals from n units. Scaling up to a large network thus faces the challenge of diminishing optical power unless with on-chip optical amplification, which is not yet available. Still, an OCNN system using the PMMC device can afford an extremely high areal computing density (defined as MAC operations per time per unit area) because of its compact footprint of~80 × 20 µm 2 (Fig. 2a, including the mode selector). For example, assuming a moderate datarate of 10 Gbits/sec and 4 WDM wavelengths in parallel per channel, the computing density will reach an upperbound value of 25 TOPS/mm 2 (Tera-operations per second per mm 2 ), which is significantly higher than that of digital electronic accelerators such as GPUs and tensor processing units (TPUs) 60,61 . Using silicon instead of silicon nitride can further reduce the device footprint to increase the computing density 62 . Besides MAC operation, the equally important computing processes of applying nonlinear functions and pooling can also be achieved optically by using elements such as nonlinear optical resonators, modulators, and amplifiers 27,28,63 . Alternatively, a hybrid photonic-electronic system may optimally balance energy-efficiency and speed advantages of photonic systems, while realizing flexible non-linearity, connectivity, and training precision using microelectronics 26,64,65 . With these advances and after overcoming the scaling challenge, the photonic neural network accelerator will be very promising for AI in data centers where massive optical interconnects have already been deployed.

Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability
No custom computer code or mathematical algorithm is used to generate the results that are reported in this study. Fig. 4 Building an optical CNN for imaging recognition. a Operation procedure of using the OCNN to recognize handwriting numbers from the MNIST database. The OCNN consists of a convolution layer with two kernels, a pooling and a fully connected layer. The output gives the answer whether the input image is "1" or "2". b The convolution kernel matrices K 1 and K 2 generated by training the OCNN. c. Raw output data of the convolution layer of two kernel matrices. d The weight bank matrix used in the fully connected layer. e The recognition results from the experiment with the OCNN (left) and the calculation with a computer (right) show an excellent agreement.