Abstract
Recurrent neural networks (RNNs) can process contextual information such as time series signals and language. But their tracking of internal states is a limiting factor, motivating research on analog implementations in photonics. While photonic unidirectional feedforward neural networks (NNs) have demonstrated big leaps, bi-directional optical RNNs present a challenge: the need for a short-term memory that (i) programmable and coherently computes optical inputs, (ii) minimizes added noise, and (iii) allows scalability. Here, we experimentally demonstrate an optoacoustic recurrent operator (OREO) which meets (i, ii, iii). OREO contextualizes the information of an optical pulse sequence via acoustic waves. The acoustic waves link different optical pulses, capturing their information and using it to manipulate subsequent operations. OREO’s all-optical control on a pulse-by-pulse basis offers simple reconfigurability and is used to implement a recurrent drop-out and pattern recognition of 27 optical pulse patterns. Finally, we introduce OREO as bi-directional perceptron for new classes of optical NNs.
Similar content being viewed by others
Introduction
Understanding the context of a situation is a powerful ability of the human brain, allowing it to predict possible outcomes and to make intelligent decisions. While humans can access the context of a situation via the short-term memory, machines struggle in contextualizing. Artificial neural networks, one of the most powerful computing architectures, face this problem as well. To overcome this limitation, they can be equipped with recurrent feedback, allowing them to process current inputs based on previous ones. The so-called recurrent neural networks (RNNs) can contextualize, recognize, and predict sequences of information and are applied for numerous applications such as language processing tasks, and for video and image processing1,2,3,4,5. One of the simplest versions of a RNN is the Elman network6, which adds a recurrent operation to each neuron of its fully-connected network, analogous to the neuron’s activation function. With this three-layer network, Elman was already able to understand simple grammatical structure. More complex models have proven themselves as Chinese poets, rap artists, and empathetic listeners7,8,9.
Currently, the scientific community aims to transfer electronic neural networks into the optical domain. The resulting optical neural networks have attracted great interest due to their promises of high processing speed and broad bandwidth, and low dissipative losses10,11,12. Thus, they are considered to pave the way towards energy efficient and highly parallel optical circuits, enhancing the performance and capabilities of future artificial neural networks13,14,15,16,17,18,19.
Although the field of optical neural networks has made great progress in recent years, the field of recurrent optical neural networks is still very limited to concepts based on artificial reservoirs, such as free-space cavities20, delay systems21,22, and microring resonators23. These designs can face several challenging issues. Firstly, the usage of an artificial cavity, e.g. a ring resonator, requires additional tuning of the individual rings due to manufacturing depended properties such as the free spectral range (FSR). This requires additional compensation routines in order to match the FSR witch the chosen temporal dynamic, wavelength and coupling. Secondly, the free-space cavities and delay systems may not be frequency sensitive, preventing them from being applied for resource-efficient multi-frequency data processing. Finally, the cavity’s recurrent weights cannot be varied rapidly, limiting the control of the recurrent process such as the implementation of recurrent dropout on single pulse level in order to regularize the network.
Here, we experimentally demonstrate an optoacoustic recurrent operator (OREO) based on stimulated Brillouin-Mandelstam scattering (SBS) that can unlock recurrent functionalities in existing optical neural network architectures (see Fig. 1a). SBS is an interaction of optical waves with traveling acoustic waves which serve in our system as a latency component due to the slow acoustic velocity. OREO is therefore able to contextualize a time-encoded stream of information by using acoustic waves as a memory to remember previous operations (see Fig. 1b).
In contrast to previously reported approaches20,21,22,23,24, OREO controls its coherent recurrent operation completely optically on pulse level without the need of any artificial reservoir such as a ring resonator or a delay system. Hence, OREO does not rely on complicated manufacturing processes of microstructures. It functions in any optical waveguide, including on-chip devices, as it harvests the physical property of a sound wave25,26,27. With the announcement of the first on-chip EDFA28 a fully integrated design is in reach. While recognizing the outstanding efforts of the scientific community to implement non-reciprocal devices on-chip29,30,31,32,33,34, we would like to highlight the work35 which implements an on-chip circulator free SBS experiment. More details about how OREO could be integrated on-chip can be found in the discussion.
We demonstrate OREO experimentally from different perspectives. Firstly, we show how OREO links different input states of subsequent optical pulses to each other via acoustic waves. Secondly, we present how the all-optical control of OREO can be used to implement a recurrent dropout. Finally, we apply OREO as an acceptor36 to predict up to 27 different patterns carried by a time series of input pulses.
Results
Concept of an optoacoustic recurrent operator
The recurrent operation of OREO is based on the interaction of optical and acoustic waves through SBS, which is one of the most prominent third-order nonlinear effects and describes the coherent coupling of two optical waves, data and control, to an acoustic wave in a material. The dynamic is illustrated in Fig. 1c and follows from the Hamiltonian (1)37,38,39:
using the optoacoustic coupling constant g0, the frequency relation between the optical fields ωD = ωC + Ω, and the wave packet operators aD, aC, b of the data, control and acoustic field, respectively. Similar to the clinking of a wine glass, the acoustic wave b persists beyond its excitation, decaying exponentially with time \(b(t)\propto \exp \left(-t/{\tau }_{{{{{{{{\rm{ac}}}}}}}}}\right)\), where \({\tau }_{{{{{{{{\rm{ac}}}}}}}}}\propto {\Gamma }_{{{{{{{{\rm{ac}}}}}}}}}^{-1}\) is the acoustic lifetime, which depends on the properties of the used waveguide and is for a photonic crystal fiber (PCF) about τac ≈ 10 ns (see Fig. 1c). As a result, an acoustic wave bi can seed subsequent SBS processes j > i. Moreover, the acoustic builds up with each SBS process, which can be described as a superposition of all previous created acoustic waves bi with amplitude b0,i, created at the time ti and carrying a phase φi. Hence, the acoustic wave bN after N SBS interactions:
yields the recurrence in the interaction Hamiltonian:
Equation (3) shows furthermore that programming the field aC controls the acoustic feedback all-optically, enabling a pulse-by-pulse increase or suppression. For instance, setting aC,i=0 corresponds to a recurrent dropout so that aD,i leaves the fiber unchanged.
We experimentally implement OREO in a telecom-fiber apparatus illustrated in Fig. 1d. Here we launch several consecutive rectangular optical input data pulses aD,i and strong counter-propagating optical control pulses aC,i into a PCF. The optical data pulses are shifted up in frequency by Ω/2π ≈ 10.6 GHz compared to the optical control pulses, which is close to the Brillouin frequency of the PCF. When a data and control pulse pair aD,1 and aC,1 meets inside the PCF, they induce SBS, depleting the data pulse and transferring its energy into the acoustic domain. Eventually, an acoustic wave b1 is generated, which persists much longer than the optical interaction (see Fig. 1e). An optoacoustic recurrent operation is performed, when a subsequent data and control pulse pair (aD,2 and aC,2) reaches the acoustic wave b1 before it has decayed. Hence, the deadtime dt until the second pulse pair arrives must be less then the acoustic lifetime. The previously generated acoustic wave connects to the subsequent SBS process between aD,2 and aC,2 and establishes a link of the second data pulse aD,2 to the first data pulse aD,1. In addition, the second SBS process creates a second acoustic wave b2, carrying information of aD,2. Now, the acoustic domain holds information of both data pulses aD,1 and aD,2 (see Fig. 1f). The discussed procedure could now be repeated also for a third pulse pair and, in general, as long as subsequent pules pairs arrive before the acoustic wave decayed completely (see Fig. 1g).
Recurrent operation
In the following, we continue with the previously introduced setting of three pulse pairs (aD,i, aC,i), i = 1, 2, 3. We use this configuration to experimentally study the recurrent feedback, established by the acoustic waves bi. Precisely, three different acoustic links occur. On the one hand, b1 connects the input aD,1 to the output \({a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) and \({a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) and, on the other hand, b2 connects aD,2 to \({a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) (see Fig. 2a). In order to study the different connections, we sweep the input amplitude of either aD,1 or aD,2, while keeping the other data pulses constant. For instance, if the input amplitude of aD,1 is varied, aD,2 and aD,3 are fixed in amplitude. The control pulses aC,i are kept constant over the entire study. For each amplitude step, we measure the area under the curve (AuC) of the output pulses \({a}_{{{{{{{{\rm{D}}}}}}}},{{{{{{{\rm{i}}}}}}}}}^{{\prime} }\). In the subsequent analysis, we normalize the AuC of \({a}_{{{{{{{{\rm{D}}}}}}}},{{{{{{{\rm{i}}}}}}}}}^{{\prime} }\) with the AuC of an input data pulse aD,i with an amplitude scale of 1, representing the highest input value. In order to measure the reference, we launch a data pulse sequence into OREO without counter-propagating control pulses. In order to rule out drifting effects, we measure each amplitude twice in a random order and take the mean value afterwards. Furthermore, the amplitude sweep is performed for three different time delays dt = 2.5, 4.5, and 10 ns, as the acoustic link decays over time. In order to have a reference for the dynamic of the Brillouin process, we use the interaction of a single data-control pulse pair. The data pulse aD gets depleted by the SBS process in this single pulse interaction (SPI). The degree of depletion depends mainly on the power carried by the control pulse25. For OREO, we deplete about 47% of the data pulse. For a deadtime of 2.5 ns, an increase in amplitude of aD,1 raises the output amplitude \({a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) as shown by Fig. 2b. The reason for this dynamic is that aD,2 ↔ aC,2 is seeded by the acoustic wave b1 influencing the degree of depletion. More precisely, the dynamic can be explained with the different acoustic phases of b1 and b2 (see Eq. (2)), which can lead to constructive or destructive interference during the SBS process. The acoustic phase is introduced by detuning the frequency difference between data and control pulses slightly from the Brillouin frequency. For example, if the interaction aD,i ↔ aC,i has created the acoustic wave bi with a specific phase (see Eq. (2)), then a second SBS process j > i would generate dynamically bj on top of bi. Both bi and bj have a different phase which lead to acoustic interference40. As Brillouin scattering describes a dynamic coupling between photons and phonons, the acoustic interference effects the overall “stimulated" dynamic. Hence, if bi and bj interfere destructively, the “stimulated” dynamic is slowed down and, therefore, the depletion of the data pulse aD,j lower. However, if bi and bj add up via constructive interference, then the stimulation is accelerated, increasing the depletion of aD,j. The acoustic interference is also the reason for the decreasing behavior of the link \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) on aD,3 ↔ aC,3. Here, the acoustic wave b1 accelerates the SBS process of aD,3 ↔ aC,3 which leads to an increased depletion of aD,3.
The symbols + and − mark the constructive and destructive nature of the underlying acoustic interference in Fig. 2b, respectively. OREO achieves a maximal dynamic range (Max DR) of 33%. For a deadtime of dt = 4.5 ns we observe a flip in the dynamic as all links switch their behavior from a constructive (+) to destructive (−) acoustic link, and vice versa (depicted in Fig. 2c). In addition, the overall level of depletion is larger in comparison to the SPI-case. For a deadtime of dt = 10 ns (equal to the acoustic lifetime), the dynamic range of the optical connection decreases further as we can see for the connection \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) in Fig. 2d. Ultimately, the effect of the decaying acoustic wave becomes in particular visible for the interaction \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) as \({a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) remains constant over the entire sweep range of aD,1 (see Fig 2d). Note that we marked vanished acoustic links with the •-symbol.
With this, we have shown that OREO connects the information carried by subsequent optical data pulses. The acoustic link is sensitive to the amplitude and deadtime of the involved optical data pulses. As the interaction is continuous, it can be used for digital and analog recurrent tasks. Moreover, the acoustic interference observed with OREO ties in with previous studies based on continuous optical waves and our measurement extends the observation of acoustic interference into a pulsed context40,41. Being able to distinguish nine different amplitude levels, we conclude that OREO can resolve at least 3-bit information. In general, SBS is sensitive to continous amplitude levels, hence higher number of unique symbols can be encoded. The fundamental limitation is the data encoding and detection. In the supplementary material, we study OREO numerically and experimentally in a highly nonlinear fiber (HNLF), using the framework presented in ref. 42. With the HNLF we study the linear response of OREO, which occurs in the case that the frequency difference of data and control matches exactly the Brillouin frequency. OREO controls the recurrent operation completely optically via the control pulses, enabling us to implement use case specific computations. For instance, in a pulse sequence consisting of three data pulses, one could skip the middle pulse aD,2 by dropping the second control pulse, which could be useful for regularization43. In order to demonstrate the recurrent dropout, we excluded the second control pulse aC,2 from the pulse train. Note, that the amplitudes of the other control pulses remain the same. In a next step, we vary the amplitude of data pulses aD,1 and aD,2 in upward and downward direction and check the impact on the subsequent data pulses (see Fig. 2e). Furthermore, we change the deadtime to investigate the influence of the acoustic interference on the interaction \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\).
OREO turns off the links between \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) and \({a}_{{{{{{{{\rm{D}}}}}}}},2}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) as we can see in in Fig. 2f. As marked with the •-symbol, those two links show a constant behavior for the entire amplitude sweep. Only the interaction \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) is active as the control pulses aC,1 and aC,3 establish the required acoustic link. Note, that for the case of \({a}_{{{{{{{{\rm{D}}}}}}}},2}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) the interaction aD,3 ↔ aC,3 is influenced by the acoustic wave generated of the aD,1 ↔ aC,1-interaction (aD,1 is constant). This link can also explain the lower degree of depletion of \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) at dt = 4.5 ns (see Fig. 2g). Here, the aD,1 and aD,3 are already separated by 10 ns, which eliminated almost their acoustic link. At a deadtime of dt = 10 ns, the \({a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) is completely disconnected from aD,1 and aD,2 as can be seen by the constant behavior of \({a}_{{{{{{{{\rm{D}}}}}}}},3}^{{\prime} }\) for both sweeps of aD,1 and aD,2 (see Fig. 2h). Besides, over all measurements, \({a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) is below the reference level (\({a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} } < 1\)), e.g., for the interaction \({a}_{{{{{{{{\rm{D}}}}}}}},1}\to {a}_{{{{{{{{\rm{D}}}}}}}},2}^{{\prime} }\) in Fig. 2f. The increased optical noise floor appears as soon as the EDFA is turned on and could lead to this intrinsic depletion.
Optical pattern recognition
From the beginning on, recurrent operators have been used to recognize patterns6. In the following section, we experimentally employ OREO as an acceptor36 to recognize any pattern that can be created with two different data pulses a and b: aa, ab, ba & bb, where the b-pulse is half the amplitude of the a-pulse. Each pulse is launched with a matching control pulse aC,i into the PCF, where SBS is introduced. The deadtime between two consecutive pulses is 2.5 ns. Figure 3a shows schematically the information flow of the acceptor task. The individual pulses of the pattern ab are launched into OREO each after 2.5 ns with a corresponding control pulse aC,i. The information of the pattern is captured by the acoustic wave and flows in time, connecting the different SBS processes. Eventually, the sequential information captured by OREO is evaluated with a third optical evaluation pulse (Eval). In this way, we map the sequential information onto a single optical pulse which can be used then for classification, realizing experimentally a photonic acceptor. We perform the final classification by measuring the output evaluation pulse (\(Eva{l}^{{\prime} }\)), extracting its AuC in post-processing and then feeding this value into a digital Random Forest classifier44 (RFC). In total, we check all patterns 250-times in a random order and classify the resulting experimentally obtained data set (70% training, 30% testing) with the RFC implemented in scikit-learn-package v1.1.345. Furthermore, we perform the described study twice, once with the SBS-process and once without to isolate OREO’s effect.
When OREO is off, the RFC cannot distinguish the different patterns and shows the same accuracy as a random guess (see Fig. 3b). However with OREO, the RFC is capable of distinguishing the different patterns almost with an accuracy of almost a 100% (see Fig. 3c).
Next, pushing OREO to the acoustic lifetime limit, we evaluate its performance for three different states encoded onto three pulses. The third state c is three quarters of the a state. In total, we test OREO to distinguish every possible permutation of a, b, and c, giving 27 different patterns. This time we launch a fourth data-control pair into the PCF, in order to evaluate OREO’s memory (see Fig. 3d). Note that all four control pulses aC,i carry the same optical energy as in the three pulse configuration. We increased the sample size n per pattern from 250 to 500 measurements in order to decrease statistical errors. Figure 3e shows the corresponding confusion matrix. OREO functions as acceptor and generates distinguishable distributions for the 27 patterns. The RFC achieves an accuracy of 45%, exceeding the accuracy of a simple guess by 11-times. The performance of OREO is currently limited by experimental precision, which is reduced by drifts of the optical pulses over the measurement period. Therefore, we perform a numerical analysis of OREO as an acceptor in the frequency matched case, in order to assess its potential performance. In this simulated experiment, OREO and the RFC achieve an accuracy of 92%. Figure 3f shows the corresponding confusion matrix. In the supplementary material, we describe the numerical analysis and check the impact of the pulse width, deadtime, acoustic lifetime, and experimental precision on OREO’s pattern recognition performance. This analysis indicates that OREOs performance can even be pushed further to an accuracy of 97%. The current implementation of OREO can be seen as a photonic recurrent extreme learning machine with one layer and one neuron. Hence, we compare the result of the abc-study to a digital RNN-classifier containing a single recurrent neuron and a fully-connected network (FCN). This classifier achieves in the same task on average an accuracy of ≈ 55.9 %. The relative difference to OREO is about 10%P, hence OREO’s performance is almost on pair with the digital RNN. However, it should be noted that the digital RNN is trained actively whereas OREO is used as an extreme learning machine. Accordingly, one could improve the accuracy of OREO in the future by training (in-situ) the amplitudes of the control pulses aC,i, for instance, using simultaneous perturbation stochastic approximation46. Further details on the comparison of OREO with an RNN classifier can be found in the supplementary material.
Discussion
The acoustic link employed by OREO enables the processing of time-encoded serial information within a PCF. Its capability to control the recurrent interaction all-optically, gives the concept unique features. The adjustable amplitudes of the control pulses allow OREO’s behavior to be changed at the single pulse level, offering an all-optical degree of freedom to adjust its recurrent operation. Moreover, we have shown that it offers the possibility to exclude data pulses from the recurrent interaction. As a consequence, a single data pulse can propagate through OREO without experiencing any manipulation. This can be used to implement recurrent dropout as regularization for the RNN.
The coherent nature of the underlying SBS process offers OREO not only to compute amplitude information but also phase information. Eventually, OREO could compute quadrature amplitude modulated (QAM) data streams.
Higher memory depths could be achieved with three different approaches. Firstly, a higher pulse density could be used to increase the number of operations that could be performed within the intrinsic acoustic lifetime. This could be achieved by decreasing the pulse width and the deadtime between the pulse pairs. For instance, with a pulse width of 100 ps and a deadtime of 100 ps (the minimal deadtime is dictated by the length of the waveguide), one could induce up to 50 recurrent interactions. Secondly, one could increase the acoustic lifetime to realize a deep recurrent link, for instance by using materials with longer acoustic lifetimes or operating at cryogenic temperatures. Thirdly, an optical refreshment of the acoustic waves could lead to an increase in memory depth47.
Considering the current configuration of the OREO setup, one could realize up to four layers in an OREO-based RNN. In comparison, to best of our knowledge, the maximum number of layers reported in an ONN without transferring information back to the digital domain is so far three46,48. A more detailed discussion on the scalability of an OREO-based RNN is provided by the supplementary material.
Because the SBS process does not significantly change the optical control pulses, an optical recycling scheme could be applied to achieve high computational efficiencies. Computational efficiency is determined by the number of operations (OPS) that OREO can perform with one Joule of power. With an optical recycling scheme this value depends only on the deadtime between the pulse pairs, yielding an efficiency from up to \(\,\approx \,11\frac{{{{{{{{\rm{POPS}}}}}}}}}{{{{{{{{\rm{J}}}}}}}}}\); it could potentially increase the computational efficiency of the method described in Reference22 by three orders of magnitude. A more detailed description of the computational efficiency can be found in the supplementary material. The information bandwidth of an optical signal can be significantly increased by employing different optical frequencies as independent information channels. This has been recently exploited by Sludds et al.49 to implement an high-performance optical deep learning architecture for edge computing. OREO could be added to this scheme as SBS is highly frequency-selective50.
This unique feature of the optoacoustic interaction could also be employed together with an optical multi-frequency matrix operator51,52,53 to realize an multi-frequency recurrent neural network. Therefore, OREO has to be implemented on-chip. Applications of Brillouin scattering for gyroscopes, photonic memories, and microwave photonic devices haven been experimentally demonstrated in on-chip configurations26,54,55,56,57,58. Moreover, different integrated platforms for Brillouin scattering have been discussed extensively59,60,61. Still, a key challenge of using backward Brillouin scattering is the management of the counter propagating data and control pulses. For an in-fiber setup this is usually achieved by using isolators or circulates. The usage of those non-reciprocal devices on-chip is currently more challenging, however, there is extensive study on possible implementations29,30,31,32,33,34. In the context of Brillouin scattering, Liu et al. have demonstrated a complete on-chip backward Brillouin experiment without the need for isolators or circulators35. Furthermore, one could also employ a narrowband filter that filterers either the data or the control light. In the following, we eleborate on possible waveguide platforms, the footprint of an SBS active waveguide and the energy conmsuption of on-chip Brillouin process.
Firstly, SBS can be used in a variety of different integratable platforms including Silica, Silicon and Germanium60. Backward SBS has also been studied in Silicon Nitride62,63 and recently reported in Lithium Niobate on Insulator waveguides64,65. Moreover, waveguides based on chalcogenide soft glass offer a large Brillouin gain60,66,67.
Secondly, The footrpint of on-chip OREO depends on two properties. First, the pulse width τ defines the optimal length Lopt of the Brillouin active waveguide: Lopt ≈ c0τ/neff with the speed of light c0 and the effective refractive index of the waveguide. Secondly, the minimal bending radius of the Brillouin active waveguide. This is an important parameter as it allow to route Lopt as spiral, which reduces the overall footprint significantly. This reduction of footprint has been used, for instance, to build a 50 cm long Brillouin active spiral in Silicon Nitride63 and chalcogenide spiral with a total length of 5.8 cm and a footprint below 1 mm2(compare67). In addition, bending radii of ≤ 200 μm have been demonstrated for chalcogenide waveguides68,69.
Finally, the power consumption of OREO is mainly defined by the optical power of the control pulses Pc. In order to estimate the power consumption of an on-chip OREO, we assume to describe the Brillouin dynamics with an effective Brillouin gain geff = gLeffPc − αL with the intrinsic Brillouin gain g, the optical loss of the waveguide α, the entire length of waveguide L, the the effective interaction length of two pulses \({L}_{{{{{{{{\rm{eff}}}}}}}}}=(1-\exp (-\alpha {L}_{{{{{{{{\rm{opt}}}}}}}}}))/\alpha\). By choosing different platforms one effectively changes the values for g and α. For instance, the PCF used to demonstrate OREO offers a gain of gPCF ≈ 2.5 m−1 W−1 and a loss of αPCF ≈ 2.7 dB km−1 (compare70). In comparison, a chalcogenide waveguide offers a gain of gAsS ≈ 500 m−1 W−1 with a loss of αAsS ≈ 0.5 dB cm−1 (compare60). In this case, one could reduce the required optical power to realize the same geff by 99% to \({P}_{{{{{{{{\rm{c}}}}}}}}}^{AsS}\,\approx \,1.27\,{{{{{{{\rm{mW}}}}}}}}\). Recalling that this is a theoretical improvement and keeping in mind the experimental Brillouin demonstrations on-chip, we would frame this improvement as a long-term goal. Nonetheless, we would like to highlight the demonstration of a hybrid waveguides66 offering a gain of gHybrid ≈ 750 m−1 W−1 and the proposal for sub-wavelength waveguides with a gigantic Brillouin gain gsub > 100000 m−1 W−1 (compare71). In conclusion, on-chip devices could reduce the power consumption of OREO.
In conclusion, we have demonstrated the first optoacoustic recurrent operator (OREO), which connects the information carried by subsequent optical data pulses. Our work combines for the first time the field of traveling acoustic waves and artificial neural networks and paves the way towards SBS-enhanced computing platforms. This new fusion brings context to optical neural networks, but can also enable much more. Typical building-blocks of a neural network, such as nonlinear activation functions and other types of optoacoustic operators are within reach. Especially, the different time scales of optical and acoustic waves open up a whole new playground for the implementation of a variety of computing architectures.
Methods
Experimental setup
To demonstrate OREO, we build the all-fiber setup shown in Supplementary Fig. 1. As a sample, we use a photonic crystal fiber (PCF) with a length of ≈ 40 cm, an average hole diameter of 1.44 μm, an average core diameter of 1.842 nm, a pitch of 1.756 μm, and d/Λ = 0.82. A continuous wave laser at 1550 nm is split into the data and control branch via a 50/50-splitter. An IQ-modulator shifts the data signal by Ω/2π ≈ 10.6 GHz, which is close to the PCF’s Brillouin frequency of ΩPCF /2π ≈ 10.45 GHz. The data signal’s spectrum is cleaned with a subsequent narrow bandpass filter and afterwards amplified by an Erbium-doped fiber amplifier (EDFA). An optical intensity modulator driven by an arbitrary waveform generator (AWG) generates the rectangular optical pulses and, thus, imprints the amplitude-encoded information.
A single data pulse is 1 ns long and separated to an adjacent data pulse by a deadtime dt. The repetition rate of a pulse sequence is ≈ 1 MHz. The pulses are guided to the PCF by an optical circulator and, afterwards, measured with a high-speed photodiode and a 16 GHz Oscilloscope. The optical power of the data pulse is about 1 mW. An additional narrow bandpass filter cleans the signal before detection. In the control branch, optical pulses are generated with the same pulse width and repetition rate as the data branch. Afterwards, the pulsed signal is amplified by an EDFA and filtered by a narrow bandpass filter before launched into a high-power EDFA. The amplified signal is filtered by a 1 nm-width bandpass filter and launched with an average power of about 126 mW into the SBS process.
Training the Random Forest classifier (RFC)
We use a Random Forest classifier (RFC) to classify the area under the curve (AuC) of the output evaluation pulses \(Eva{l}^{{\prime} }\) (see Fig. 3a, d). We initialize our RFC with the scikit-learn python package v1.1.345, setting its maximal depth to 10 and the random state to 2. We found those to be the best configuration to achieve the highest accuracies for both the ab- and the abc-study. The function sklearn.model_selection.train_test_split splits the initial data sets into the test and training one with a random state of 20. We ensure that that the different patterns are equally distributed to the train and test data set with the stratify-keyword of the train_test_split-function. We fit the RFC to the data set using its default fit-function.
Data availability
The data that supports findings of this study are available from the corresponding author upon request.
Code availability
The code that supports findings of this study are available from the corresponding author upon request.
References
Yu, Y., Si, X., Hu, C. & Zhang, J. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures. Neural Comput. 31, 1235–1270 (2019).
Salehinejad, H., Sankar, S., Barfett, J., Colak, E. & Valaee, S. Recent Advances in Recurrent Neural Networks. Preprint at https://arxiv.org/abs/1801.01078 (2018).
Van Den Oord, A., Kalchbrenner, N. & Kavukcuoglu, K. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, 1747–1756 (JMLR.org, 2016).
Mesnil, G. et al. Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding. IEEE/ACM Trans. Audio Speech Lang. Process. 23, 530–539 (2015).
Donahue, J. et al. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. IEEE Trans. Pattern Anal. Mach. Intell. 39, 677–691 (2017).
Elman, J. L. Finding Structure in Time. Cognit. Sci. 14, 179–211 (1990).
Zhang, X. & Lapata, M. Chinese Poetry Generation with Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 670–680 (Association for Computational Linguistics, 2014).
Potash, P., Romanov, A. & Rumshisky, A. GhostWriter: Using an LSTM for Automatic Rap Lyric Generation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1919–1924 (Association for Computational Linguistics, 2015).
Lee, J. & Tashev, I. High-level feature representation using recurrent neural network for speech emotion recognition. In Interspeech 2015, 1537–1540 (ISCA, 2015).
Shastri, B. J. et al. Photonics for artificial intelligence and neuromorphic computing. Nat. Photonics 15, 102–114 (2021).
Bogaerts, W. et al. Programmable photonic circuits. Nature 586, 207–216 (2020).
Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441–446 (2017).
Teğin, U., Yíldírím, M., Oğuz, I., Moser, C. & Psaltis, D. Scalable optical learning operator. Nat. Comput. Sci. 1, 542–549 (2021).
Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132 (2019).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Zhang, H. et al. An optical neural chip for implementing complex-valued neural network. Nat. Commun. 12, 457 (2021).
Chen, Z. et al. Deep learning with coherent VCSEL neural networks. Nat. Photonics 17, 723–730 (2023).
Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756 (2018).
Brunner, D. et al. Tutorial: Photonic neural networks in delay systems. J. Appl. Phys. 124, 152004 (2018).
Mourgias-Alexandris, G. et al. All-Optical WDM Recurrent Neural Networks With Gating. IEEE J. Sel. Top. Quantum Electronics 26, 1–7 (2020).
Tait, A. N. et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci. Rep. 7, 7430 (2017).
Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).
Zhu, Z., Gauthier, D. J. & Boyd, R. W. Stored Light in an Optical Fiber via Stimulated Brillouin Scattering. Science 318, 1748–1750 (2007).
Merklein, M., Stiller, B., Vu, K., Madden, S. J. & Eggleton, B. J. A chip-integrated coherent photonic-phononic memory. Nat. Commun. 8, 574 (2017).
Stiller, B. et al. On-chip multi-stage optical delay based on cascaded Brillouin light storage. Optics Lett. 43, 4321–4324 (2018).
Liu, Y. et al. A photonic integrated circuit-based erbium-doped amplifier. Science 376, 1309–1313 (2022).
Sohn, D. B., Kim, S. & Bahl, G. Time-reversal symmetry breaking with acoustic pumping of nanophotonic circuits. Nat. Photonics 12, 91–97 (2018).
Kittlaus, E. A., Otterstrom, N. T., Kharel, P., Gertler, S. & Rakich, P. T. Non-reciprocal interband Brillouin modulation. Nat. Photonics 12, 613–619 (2018).
Yan, W. et al. Waveguide-integrated high-performance magneto-optical isolators and circulators on silicon nitride platforms. Optica 7, 1555 (2020).
Tian, H. et al. Magnetic-free silicon nitride integrated optical isolator. Nat. Photonics 15, 828–836 (2021).
Kittlaus, E. A. et al. Electrically driven acousto-optics and broadband non-reciprocity in silicon photonics. Nat. Photonics 15, 43–52 (2021).
Yu, M. et al. Integrated electro-optic isolator on thin-film lithium niobate. Nat. Photonics 17, 666–671 (2023).
Liu, Y. et al. Circulator-Free Brillouin Photonic Planar Circuit. Laser Photonics Rev. 15, 2000481 (2021).
Goldberg, Y. Neural network methods in natural language processing. No. 37 (Springer Nature, 2022).
Sipe, J. E. & Steel, M. J. A Hamiltonian treatment of stimulated Brillouin scattering in nanoscale integrated waveguides. N. J. Phys. 18, 045004 (2016).
Zhang, J., Zhu, C., Wolff, C. & Stiller, B. Quantum coherent control in pulsed waveguide optomechanics. Phys. Rev. Res. 5, 013010 (2023).
Eggleton, B. J., Steel, M. J. & Poulton, C. (eds.) Brillouin scattering. Part 1. (Academic Press, 2022).
Okawa, Y. & Hotate, K. Optical coherent control of stimulated Brillouin scattering via acoustic wave interference. Optics Lett. 45, 3406 (2020).
Feng, Y. et al. Coherent control of acoustic phonons by seeded Brillouin scattering in polarization-maintaining fibers. Optics Lett. 44, 2270 (2019).
de Sterke, C. M., Jackson, K. R. & Robert, B. D. Nonlinear coupled-mode equations on a finite interval: a numerical procedure. J. Optical Soc. Am. B 8, 403 (1991).
Semeniuta, S., Severyn, A. & Barth, E. Recurrent Dropout without Memory Loss. Preprint at https://arxiv.org/abs/1603.05118 (2016).
Tin, K. H. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognition, vol. 1, 278–282 (IEEE, 1995).
Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Bandyopadhyay, S. et al. Single chip photonic deep neural network with accelerated training. Preprint at https://arxiv.org/abs/2208.01623 (2022).
Stiller, B. et al. Coherently refreshing hypersonic phonons for light storage. Optica 7, 492 (2020).
Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 606, 501–506 (2022).
Sludds, A. et al. Delocalized photonic deep learning on the internet’s edge. Science 378, 270–276 (2022).
Stiller, B. et al. Cross talk-free coherent multi-wavelength Brillouin interaction. APL Photonics 4, 040802 (2019).
Davis III, R., Chen, Z., Hamerly, R. & Englund, D. Frequency-Encoded Deep Learning with Speed-of-Light Dominated Latency. Priprint at https://arxiv.org/abs/2207.06883 (2022).
Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).
Buddhiraju, S., Dutt, A., Minkov, M., Williamson, I. A. D. & Fan, S. Arbitrary linear transformations for photons in the frequency synthetic dimension. Nat. Commun. 12, 2401 (2021).
Li, J., Suh, M.-G. & Vahala, K. Microresonator Brillouin gyroscope. Optica 4, 346 (2017).
Marpaung, D. et al. Low-power, chip-based stimulated Brillouin scattering microwave photonic filter with ultrahigh selectivity. Optica 2, 76 (2015).
Liu, Y., Choudhary, A., Marpaung, D. & Eggleton, B. J. Chip-Based Brillouin Processing for Phase Control of RF Signals. IEEE J. Quantum Electronics 54, 1–13 (2018).
Choudhary, A., Liu, Y., Marpaung, D. & Eggleton, B. J. On-Chip Brillouin Filtering of RF and Optical Signals. IEEE J. Sel. Top. Quantum Electronics 24, 1–11 (2018).
Garrett, M., Merklein, M. & Eggleton, B. J. Chip-Based Brillouin Processing for Microwave Photonic Phased Array Antennas. IEEE J. Sel. Top. Quantum Electronics 29, 1–20 (2023).
Merklein, M. et al. Stimulated Brillouin Scattering in Photonic Integrated Circuits: Novel Applications and Devices. IEEE J. Sel. Top. Quantum Electronics 22, 336–346 (2016).
Eggleton, B. J., Poulton, C. G., Rakich, P. T., Steel, M. J. & Bahl, G. Brillouin integrated photonics. Nat. Photonics 13, 664–677 (2019).
Merklein, M., Kabakova, I. V., Zarifi, A. & Eggleton, B. J. 100 years of Brillouin scattering: Historical and future perspectives. Appl. Phys. Rev. 9, 041306 (2022).
Gyger, F. et al. Observation of Stimulated Brillouin Scattering in Silicon Nitride Integrated Waveguides. Phys. Rev. Lett. 124, 013902 (2020).
Botter, R. et al. Guided-acoustic stimulated Brillouin scattering in silicon nitride photonic circuits. Sci. Adv. 8, 2196 (2022).
Ye, K. et al. Surface acoustic wave stimulated Brillouin scattering in thin-film lithium niobate waveguides. Preprint at https://arxiv.org/abs/2311.14697 (2023).
Rodrigues, C. C. et al. On-Chip Backward Stimulated Brillouin Scattering in Lithium Niobate Waveguides. Preprint https://arxiv.org/abs/2311.18135 (2023).
Lai, C. K. et al. Hybrid Chalcogenide-Germanosilicate Waveguides for High Performance Stimulated Brillouin Scattering Applications. Adv. Funct. Mater. 32, 2105230 (2022).
Morrison, B. et al. Compact Brillouin devices through hybrid integration on silicon. Optica 4, 847 (2017).
Choudhary, A. et al. Advanced Integrated Microwave Signal Processing With Giant On-Chip Brillouin Gain. J. Lightw. Technol. 35, 846–854 (2017).
Zarifi, A. et al. Highly localized distributed Brillouin scattering response in a photonic integrated circuit. APL Photonics 3, 036101 (2018).
Beugnot, J.-C. et al. Complete experimental characterization of stimulated Brillouin scattering in photonic crystal fiber. Optics Express 15, 15517 (2007).
Rakich, P. T., Reinke, C., Camacho, R., Davids, P. & Wang, Z. Giant Enhancement of Stimulated Brillouin Scattering in the Subwavelength Limit. Phys. Rev. X 2, 011008 (2012).
Acknowledgements
The authors thank Florian Marquardt, Christian Wolff, Changlong Zhu, Jesús Humberto Marines Cabello, Toby Bi, and Mark Schöne for fruitful discussions.
Funding
We acknowledge funding from the Max Planck Society through the Independent Max Planck Research Groups scheme, the DFG project STI 792/1-1, and the Studienstiftung des deutschen Volkes. Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Contributions
Conceptualization: S.B. and B.S. Methodology: S.B., D.E. and B.S. Investigation: S.B. Visualization: S.B., D.E. and B.S. Funding acquisition: B.S. Project administration: B.S. Supervision: B.S. Writing: S.B., D.E. and B.S.
Corresponding author
Ethics declarations
Competing interests
S.B. and B.S. are the authors of a patent filed by the Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. about the technique and the features of the optoacoustic recurrent operator. The application numbers are EP23153328.2 for the European Patent Office and PCT/EP2024/051779 for the World Intellectual Property Organization. The remaining author declare no competing interests.
Peer review
Peer review information
Nature Communications thanks Angelina Totovic and the other, anonymous, reviewers for their contribution to the peer review of this work. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Becker, S., Englund, D. & Stiller, B. An optoacoustic field-programmable perceptron for recurrent neural networks. Nat Commun 15, 3020 (2024). https://doi.org/10.1038/s41467-024-47053-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467-024-47053-6
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.