Introduction

The unmet gap between the rate of energy efficiency improvement of current digital electronics and the fast-growing load of computation by emerging applications such as machine learning and artificial intelligence1,2 has once again brought optical computing into focus3,4,5,6. Integrated photonics provides a scalable hardware platform to realize large-scale optical networks on a chip, which affords an enormous bandwidth density that is unreachable for electronics7,8,9. To use integrated photonics for optical computing, programmable photonic components and nonlinear elements are indispensable building blocks. Phase-change materials (PCM) recently emerged as an ideal material system to realize optical programmability10,11,12. The optical properties of PCMs change dramatically during the phase transition, which can be electrically or optically controlled. Harnessing this has allowed for embodiments of programmable optical switches, couplers, lens, and metamaterials to be demonstrated13,14,15,16,17,18,19,20,21. The phase change in the chalcogenide family of Ge-Sb-Te alloys is nonvolatile, requiring no sustaining power supply to retain the programmed state or stored information19,20,21,22,23,24,25,26. Their use in programmable photonic devices thus can have a significant advantage in power consumption over electro-optic27,28,29 or thermo-optic methods30,31,32. Photonic devices incorporating those nonvolatile PCMs thus can realize optical memories and perform in-memory computing simply by measuring the transmission of the optical input data through the programmed device33,34,35. Proliferating these phase-change photonic devices in a scalable network, prototypes of optical neural networks (ONN) have been proposed and demonstrated35,36,37,38.

Here, we report a programmable waveguide mode converter based on a phase-gradient metasurface made of phase-change material Ge2Sb2Te5 (GST). This phase-change metasurface mode converter (PMMC) utilizes GST’s large refractive index change during its phase transition to control the conversion of the waveguide’s two spatial modes (TE0 and TE1 modes). The PMMC can be programmed to control the waveguide mode contrast precisely at 64 distinguishable levels, which is used to represent the weight parameters with 6-bit precision in MVM computation. We build a 2 × 2 array of PMMCs and implement them as programmable kernels to realize a multimode optical convolutional neural network (OCNN). By performing image processing tasks such as edge detection and pattern recognition, we demonstrate the OCNN’s viability and potential in large-scale optical computing.

Results

High-precision programmable phase-change mode converter

The design of the PMMC is based on the principle of a phase-gradient metasurface but replacing noble metals with phase-change materials39. Fig. 1a shows a 3D schematic of the design, which consists of a linear array of GST nano-antennae directly integrated on a silicon nitride (SiN) waveguide. Each GST nano-antenna scatters the waveguide mode and causes a phase shift Φ, which depends on its geometry (e.g., width), as well as the refractive index of its material (Fig. 2b). A linear array of such nano-antennae with tapering widths thus produces a spatial gradient of the scattering phases dΦ/dx, which is equivalent to a wavevector kg. If the phase-gradient metasurface is designed such that kg matches the wavevector difference between two spatial modes of the waveguide: kmode1−kmode2, it satisfies the phase-matching condition and facilitates the conversion between the two modes. Such phase-gradient metasurfaces for waveguide mode conversion were realized using noble metals or dielectrics materials and thus lacked tunability. Here, we use GST, which has a large change in its optical properties when a phase transition happens. When the GST is in the amorphous phase (aGST), its refractive index n is ~4.7 (representative value in the literature, the same hereafter)40. In contrast, when it is turned to the crystalline phase (cGST), n increases to ~7.5 with a drastic change of 2.8 over the whole measured spectral range from 1540 nm to 1580 nm (See Supplementary Fig. 1a for more detailed information). This change will significantly modify the scattered phase of each GST nano-antenna (Fig. 2b) so as to modify the metasurface’s function. Fig. 1c plots the simulated phase of the scattered fields inside the waveguide by a single nano-antenna of 30-nm-thick GST as a function of its width and for aGST and cGST phases. Since cGST has a much larger n, the scattered phase shows a much stronger dependence on the width than the aGST phase. By controlling the geometry of the GST nano-antennae and the interval between adjacent ones in the array, a well-defined phase gradient dΦ/dx is established (see Supplementary Note 2 and 3 for details). The entire metasurface consists of an array of 25 nano-antennae with tapering widths from 510 nm to 84 nm (shaded region in Fig. 1c) and is patterned on a SiN waveguide 1.8 µm wide and 330 nm thick. The waveguide supports two transverse-electric modes: the fundamental TE0 mode and the first-order TE1 mode. We design the metasurface, in the cGST phase, to have a uniform dΦ = 2.5° for every dx = 400 nm to satisfy the generalized phase-matching condition, \(k_0(n_{{\mathrm{TE0}}} - n_{{\mathrm{TE1}}}) = N \cdot d{\varPhi}/dx\), where k0 is the free-space wavevector, \(n_{{\mathrm{TE0}}}\) and \(n_{{\mathrm{TE1}}}\) are the effective index of the TE0 and TE1 modes, respectively, and N is the number of interactions between the guided modes and the metasurface. The cGST metasurface thus can efficiently convert the TE0 mode to the TE1 mode, as shown by the finite-difference time-domain (FDTD) simulation result in Fig. 1d. When the GST is transitioned to the aGST phase, as shown in Fig. 1c, the dΦ/dx is much reduced and thus insufficient for the phase-matching condition so that mode conversion between TE0 and TE1 modes does not occur, which is clearly seen in Fig. 1e. Therefore, the GST phase-gradient metasurface, as designed here, functions as a programmable waveguide mode converter controlled by the tunable material phase of the GST.

Fig. 1: Design of the phase-gradient metasurface mode converter.
figure 1

a 3D illustration of the devices. b FDTD simulation of the scattered electric field by one nano-antenna when the GST is in aGST (left panel) and cGST (right panel) phases, respectively, showing the distinctive difference. c. The phase of the scattered mode as a function of the GST nano-antenna width for cGST and aGST phases. The shaded region indicates the range of antenna widths that are used in the phase gradient metasurface. Inset: cross-sectional view of the structure. d, e FDTD simulation results showing effective mode conversion from the TE0 mode to the TE1 mode when the GST is in crystalline phase (d), but only a small perturbation when the GST is in amorphous phase (e).

Full size image
Fig. 2: Operation of the programmable metasurface mode converter (PMMC).
figure 2

a Scanning electron microscope (SEM) image of the complete device and the measurement and control schematics. The complete PMMC device consists of an encapsulated GST phase gradient metasurface (red box) and a mode selector (yellow box). The white box appears because of the edge of the 218-nm-thick Al2O3 encapsulating layer. b Zoomed-in SEM image of the phase-gradient metasurface on the waveguide before depositing the Al2O3 encapsulation layer for better imaging. c Zoomed-in SEM image of the TE0/TE1 mode selector. d The transmission coefficient (insertion loss) of the devices for TE0 and TE1 modes and aGST and cGST phases. The transmission of the TE0 mode is switched with a high extinction ratio of >16 dB or 4000%. e The mode purity is controlled by the mode converter to >80% for both modes. f The programable mode converter controls the mode contrast Γ at 64 distinct levels, corresponding to 6-bit programming resolution. Upper inset: zoomed-in view of the contrast levels. Lower inset: histograms of 20 programming operations to set the contrast at two adjacent levels (level 30 and 31). The well-separated histograms demonstrate the programming repeatability and accuracy.

Full size image

Fig. 2a–c shows the scanning electron microscope images of the complete PMMC device. The 30 nm thick GST film is deposited by sputtering on Si3N4 on an oxidized silicon substrate. It is then patterned into metasurface with electron beam lithography and plasma etching, and conformally encapsulated with a 218-nm-thick layer of Al2O3 deposited by atomic layer deposition. The photonic circuits of Si3N4, including multimode waveguides, directional couplers and grating couplers, are patterned with standard processes17. A pair of asymmetric directional couplers (Fig. 2c) is designed to function as mode selectors to selectively couple only the TE1 mode component in the multimode waveguide with the TE0 mode component in the single-mode waveguide (See Supplementary Note 5 for details). Fig. 2a depicts the measurement and control scheme. To program the PMMC, we use optical pulses to control the phase of the GST film for simplicity41. Previously, electrical control using integrated micro-heaters has been demonstrated by a number of groups, including us17,26,42,43,44. When operating the PMMC, an optical signal is input in the TE0 mode to the PMMC and converted to TE1 mode with a proportion controlled by the state of the GST metasurface. At the output of the PMMC, the TE1 component is separated by the mode selector and coupled out at the second port while the TE0 component remains in and outputs from the multimode waveguide. The output powers of both modes are measured to determine their respective transmission coefficients. Fig. 2d shows the transmission spectrum of the PMMC when the metasuface is set to be either in the fully aGST or cGST phases. The insertion losses of the input and output fibers and grating couplers have been accounted for by calibration measurements. In the aGST phase, the device is in the on-state for the TE0 mode with a high transmission Ton over a broad wavelength range (1540–1580 nm). The lowest insertion loss is 0.9 dB at 1575 nm wavelength. A small portion (<−10 dB) of the TE1 mode is generated due to the asymmetric perturbation induced by the metasurface even though the aGST phase has a low refractive index. The situation changes dramatically when the metasurface is transitioned to the cGST phase and converts the TE0 mode to the TE1 mode effectively. In this off-state for the TE0 mode, its transmission Toff is <−15 dB over the entire measured bandwidth. The corresponding switching extinction ratio, defined as \({\Delta}T/T^{{\mathrm{off}}} = \left( {T^{{\mathrm{on}}} - T^{{\mathrm{off}}}} \right)/T^{{\mathrm{off}}}\), is ~16 dB or 4000%, which is more than 10-fold improvement compared to previously reported switch devices using GST24,43,45. This large switching ratio stems from the phase engineering approach to effectively use GST’s large refractive index change during its phase-transition, as opposed to only using the absorption coefficient change, to facilitate scattering into a different mode that is filtered. The total area of the GST in the metasurface is only 1.3 µm2, significantly smaller than that in prior devices, and thus in principle, our device consumes less energy to switch. As expected from energy conservation, the TE1 mode is switched in the opposite way to the TE0 mode. From aGST to cGST phase, the TE1 transmission increases from ~−10 dB to ~−6.5 dB, with the insertion loss due to cGST’s absorption. Another important parameter to quantify a mode converter’s performance is the mode purity in the multimode waveguide, defined as \(\beta _{{\mathrm{TE0(TE1)}}} = P_{{\mathrm{TE0(TE1)}}}/\left( {P_{{\mathrm{TE0}}} + P_{{\mathrm{TE1}}}} \right)\), where PTE0 (PTE1) is the power in the TE0 (TE1) mode. The PMMC shows very high performance in controlling mode purity. As shown in Fig. 2e, when switching the GST from aGST to cGST phase, the PMMC efficiently converts TE0 mode to TE1 mode, changing the mode purity from βTE0 > 80% to βTE1 > 85% over a broad bandwidth, showing an excellent agreement with the numerical simulation results.

PMMC photonic kernel

The phase composition of the GST in the metasurface can be continuously tuned by partial phase transition so that the PMMC can be continuously programmed to multiple intermediate levels of phase purity values. We program the PMMC with a sequence of 50-ns-long control pulses to “quench” the GST progressively from the fully cGST phase toward the fully aGST phase. As a result, the TE1 mode purity βTE1 increases stepwise. Since the mode selector separates the two modes, we can measure their power and calculate the difference to determine the mode contrast \({\varGamma} = \beta _{{\mathrm{TE0}}} - \beta _{{\mathrm{TE1}}}\), which is used as a programming parameter. Fig. 2f demonstrates the multi-level programmability of the PMMC, in which Γ is sequentially set to 64 distinguishable levels between −0.73 to +0.67 at 1555 nm. Since the theoretical range of Γ is \(( - 1,1)\), it is an ideal parameter to represent the elements in the matrix w, with both positive and negative values, in a multiply-accumulate (MAC) operation: x → x ∙ w + b, where b is the bias parameter. MAC is the constitutional step of matrix-vector multiplication (MVM) in all neural network algorithms. The PMMC allows storing w by programming Γ in the GST metasurface as a nonvolatile memory. In-memory MAC computing can be performed with the PMMC by a measurement of the transmitted power when the input data x is encoded in the power of the input optical signal. The lower inset of Fig. 2f shows the histograms of 20 repeated programming operations to set the PMMC mode contrast at two adjacent levels (levels 30 and 31), respectively. The well-separated histograms clearly demonstrate the device’s programming resolution and accuracy. The demonstrated 64-level programmability of the PMMC—the highest to the best of our knowledge for phase-change photonic devices24—corresponds to 6-bit resolution in setting w, which is critical to the training and inference precision of the neural network46,47.

We harness the PMMC’s high-precision programmability and in-memory computing capability to demonstrate an optical convolutional neural network (OCNN)28,29,30,48. A typical CNN consists of an input layer and an output layer, which are connected by multiple hidden layers in between. The hidden layers usually consist of a series of convolutional layers followed by pooling layers and fully connected layers at the end. We design a prototype OCNN using a small network of PMMCs to implement patch-kernel matrix multiplication to compute convolution. Fig. 3a illustrates the operation principle of the OCNN for image processing, where an input grayscale image of dimensions n × n is convolved with a kernel of dimensions k × k to compute an activation map of dimension (n–k+1) × (n–k+1), assuming the convolution stride is 1. When operating the OCNN, we group the input image into (n–k+1)2 patches (the shaded area in the upper panel of Fig. 3a) with the same dimensions as the convolution kernel, k2. Each patch corresponds to the receptive field of an element in the activation map accordingly. Thus, a convolution operation requires (n–k+1)2 × k2 MAC operations in total, which is a high load of computation and can most benefit from optical computing’s speed and energy advantages.

Fig. 3: Using a PMMC array as a photonic computing core for convolutional image processing.
figure 3

a Schematic of optical convolution for image processing. An array of k2 PMMC is programmed to store the kernel matrix. A patch of pixels of an image is encoded as optical pulses and input into k2 optical channels to perform MAC operation with the kernel. The output in TE0 and TE1 are summed incoherently and measured with photodetectors. The activation map is represented by the mode contrast and could be both positive and negative. b Optical microscope image of the photonic core consisting of four PMMCs with four input channels. The TE0 mode outputs are summed on-chip with Y-junctions whereas TE1 mode outputs are summed off-chip. Optical control pulses are input using the same set of grating couplers used for the TE1 mode detection. c The greyscale image of “cameraman” (with permission from its copyright owner Massachusetts Institute of Technology) is used as the input image. d, e Left: the raw image generated by convolution with the kernel matrix for detection of horizontal (d) and vertical (e) edges. Right: the corresponding kernel matrix for edge detection. f Combined image of horizontal and vertical edge detection, highlighting all the sharp edges in the original image.

Full size image

To compute the convolution, (n–k+1)2 patch matrices of the input image are optically fed into the photonic kernel sequentially while the kernel elements, that is, the PMMCs, are programmed to fixed values. At each timeframe of the computation, the corresponding patch matrix is reshaped into a single column of data with the length k2. The data is input into the optical system in k2 channels as sequences of incoherent optical pulses, whose power amplitude is controlled by a variable optical attenuator (VOA) to encode the value of each pixel value Xij in the greyscale image. The corresponding element Wij of the kernel matrix is programmed as the mode contrast Γ of each PMMC. The resulting transmitted power of TE0 and TE1 modes are then summed incoherently using two photodetectors. Their output difference is calculated electronically and used in post-processing steps. As a result, the output will correspond to a time series of patch-kernel MVM with the amplitude encoding the values of the computation results, which is the activation map of convolution. Since the modal contrast Γ of our PMMCs can assume both positive and negative values, it can represent the kernel matrix elements without the need of an additional offset, which otherwise would take additional steps to set in each computation cycle.

Convolutional edge detection with PMMC core

Experimentally, we build a small-scale, four-channel system with four PMMCs to represent a 2 × 2 kernel matrix, as shown in the optical image in Fig. 3b. As a demonstration, we perform the convolution of a 256 × 256 8-bit grayscale image of a cameraman (Fig. 3c) to detect its edge features. As shown in Fig. 3b, the TE0 mode output coming from all the PMMCs is combined using on-chip Y-junctions, while the TE1 mode output power is combined off-chip because the same ports are used to program the PMMCs optically. Because combining four incoherent sources using Y-junctions will inherently reduce the power by a factor of 4, we rescale the measured TE0 mode power by this factor when calculating the power differences between two modes. To detect vertical and horizontal edges, kernel matrices as in the right column of Fig. 3d, e are used, and so are the PMMCs programmed. Take the vertical edge detection for example, the kernel is set to be \(\left[ {\begin{array}{*{20}{c}} { - 1} & 1 \\ { - 1} & 1 \end{array}} \right]\) so to compute the discrete first-order derivative, \(X_{i + 1,j} + X_{i + 1,j + 1} - X_{i,j} - X_{i,j + 1}\), where i, j are the indices of the input image matrix. Each kernel element Wij is stored as the mode contrast value Γ in the corresponding PMMC, with Wij = 1(−1) corresponding to the fully aGST (cGST) phase (see Supplementary Note 9 for a more detailed description about the operation procedure). The computed images after convolution without any post-processing are shown in the left column of Fig. 3d, e, for horizontal and vertical edge detection, respectively. The two images are then added to produce the right image in Fig. 3b, which highlights silhouettes of the objects with sharp edges such as the cameraman and the buildings in the original image, while suppressing smooth features such as the sky and the water. The optically computed edge detection image agrees very well with the calculated result using conventional image processing algorithms (see Supplementary Fig. 16). This result verifies the capacity and fidelity of optical convolution performed with the PMMC-based photonic kernel, which is a prerequisite for an OCNN.

OCNN for image recognition

Beyond the convolution layer, the MAC computation performed with optical signals and the PMMC network can also be applied to the pooling (average pooling) and the fully connected layers, where the PMMCs are used as weight banks instead, to realize a complete OCNN. In our experiment, we sequentially reuse the PMMC array in both convolution and fully connected layers to demonstrate an OCNN and perform proof-of-concept imaging recognition tasks of distinguishing handwritten numbers “1” and “2” from the MNIST database. Fig. 4a illustrates the architecture and processes of the OCNN. The 28 × 28 pixels, 8-bit grayscale images of number “1” or “2” are fed into the input layer as optical signals. The data is then convolved with two 2 × 2 photonic kernels K1 and K2 to generate two 27 × 27 images of activation maps. After adding a bias b1 and applying the nonlinear ReLu function, the output images are sent to an average pooling layer with a subsampling factor of 27, which reduces the images a 2 × 1 vector. This vector is then fed into the fully connected layer with a 2 × 2 photonic weight bank K3 programmed in the PMMC array, added with a bias b2 and applied the standard sigmoid function. The final output is a vector that gives the identified class of the input image, that is, \(\left[ {\begin{array}{*{20}{c}} 1 & 0 \end{array}} \right]^T\) corresponds to the number “1” and \(\left[ {\begin{array}{*{20}{c}} 0 & 1 \end{array}} \right]^T\) corresponds to the number “2”. In this OCNN, the MVM computations such as the convolution and the fully connected layers are all performed optically with the PMMCs, whereas bias and nonlinear functions are realized electronically.

Fig. 4: Building an optical CNN for imaging recognition.
figure 4

a Operation procedure of using the OCNN to recognize handwriting numbers from the MNIST database. The OCNN consists of a convolution layer with two kernels, a pooling and a fully connected layer. The output gives the answer whether the input image is “1” or “2”. b The convolution kernel matrices K1 and K2 generated by training the OCNN. c. Raw output data of the convolution layer of two kernel matrices. d The weight bank matrix used in the fully connected layer. e The recognition results from the experiment with the OCNN (left) and the calculation with a computer (right) show an excellent agreement.

Full size image

Before using the OCNN, we first train all the parameters in the layers with the standard back-propagation algorithm using the gradient descent method49. The training set consists of 11,000 images of the handwritten number “1” or “2” from MNIST training images. The training yields values for each element in the convolutional kernels and the weight bank, as shown in Fig. 4b, d. We then program the PMMC array to represent these elements. In Fig. 4c, we show the raw data of the convolutional activation maps encoded in time series of optical signals, which are the output from the PMMC array after the input image convolves with the photonic kernel K1 and K2. Since each photonic processing layer results in electrical signals output from the photodetectors, electronic post-processing is performed to add bias and apply nonlinear function and pooling. The resultant data is re-coded into optical signals and fed to the next photonic layer. Further experimental details are included in the Supplementary Note 9. We evaluate the system’s performance after training on a recognition test set, which consists of 100 randomly chosen “1” or “2” images (55 number “1” and 45 number “2”) from the MNIST testing image database. Fig. 4e shows the result that our OCNN correctly identified 91 out of 100 cases (9% error rate), which compares squarely with the result of a computer (10% error rate). The slight difference is mainly caused by the small deviation of the experimentally programmed values in the matrices (K1, K2 and K3) from the trained values, which occurs when the system’s conditions drift during operation. This result successfully demonstrates the OCNN’s viability and accuracy in performing standard neural network algorithms.

Discussion

In summary, we have demonstrated a compact programmable waveguide mode converter using GST-based phase-gradient metasurface with high programming resolution, efficiency and broadband operation. We have built a photonic kernel based on an array of such PMMC devices and implemented an optical convolutional neural network to perform image processing and recognition tasks. Our results show that phase-change photonic devices, such as the PMMC demonstrated here, can enable robust and flexible programmability and realize a plethora of unique optical functionalities that are scalable for large-scale optical computing and neuromorphic photonics. Although optical computation in this work is performed at a low speed of ~1 kHz by using low-speed VOAs to encode data into optical signals, state-of-the-art integrated photonic transmitters and photodetectors can drive the system at a speed of many 10 s of Gbits/sec50,51. Using wavelength division multiplexing (WDM) can further increase the number of parallel computation. The 2 × 2 array prototype system demonstrated in this work performs optical computation incoherently in a broadband. It can be scaled up toward a large network using a photonic crossbar array architecture52,53,54,55,56,57 (see Supplementary Note 10 for details of such a design), and compares favorably with other photonic computing schemes using coherent methods30 or optical resonators28,58,59. The feasible size (n × m) of such crossbar arrays will not be limited by the insertion loss of the PMMC (~7 dB for TE1 mode, Fig. 2d); rather it will be limited by the directional couplers with coupling efficiency of 1/n, as is needed to equally combine signals from n units. Scaling up to a large network thus faces the challenge of diminishing optical power unless with on-chip optical amplification, which is not yet available. Still, an OCNN system using the PMMC device can afford an extremely high areal computing density (defined as MAC operations per time per unit area) because of its compact footprint of ~80 × 20 µm2 (Fig. 2a, including the mode selector). For example, assuming a moderate datarate of 10 Gbits/sec and 4 WDM wavelengths in parallel per channel, the computing density will reach an upperbound value of 25 TOPS/mm2 (Tera-operations per second per mm2), which is significantly higher than that of digital electronic accelerators such as GPUs and tensor processing units (TPUs)60,61. Using silicon instead of silicon nitride can further reduce the device footprint to increase the computing density62. Besides MAC operation, the equally important computing processes of applying nonlinear functions and pooling can also be achieved optically by using elements such as nonlinear optical resonators, modulators, and amplifiers27,28,63. Alternatively, a hybrid photonic-electronic system may optimally balance energy-efficiency and speed advantages of photonic systems, while realizing flexible non-linearity, connectivity, and training precision using microelectronics26,64,65. With these advances and after overcoming the scaling challenge, the photonic neural network accelerator will be very promising for AI in data centers where massive optical interconnects have already been deployed.