Thermodynamics of continuous non-Markovian feedback control

Feedback control mechanisms are ubiquitous in science and technology, and play an essential role in regulating physical, biological and engineering systems. The standard second law of thermodynamics does not hold in the presence of measurement and feedback. Most studies so far have extended the second law for discrete, Markovian feedback protocols; however, non-Markovian feedback is omnipresent in processes where the control signal is applied with a non-negligible delay. Here, we experimentally investigate the thermodynamics of continuous, time-delayed feedback control using the motion of an optically levitated, underdamped microparticle. We test the validity of a generalized second law which bounds the energy extracted from the system, and study the breakdown of feedback cooling for very large time delays.


Introduction
The second law of thermodynamics is of fundamental and practical importance [1].On the one hand, it allows one to predict which transformations are possible in Nature.On the other hand, it offers a method for determining the efficiency of a given process by comparing it to its ideal, reversible limit.According to the standard formulation of the second law, no work can be cyclically extracted from a system coupled to a single reservoir at temperature T 0 , that is, the power output has to be negative, Ẇext ≤ 0 [1].However, in the presence of measurement and feedback, this statement of the second law breaks down and positive work can be produced, as exemplified by Maxwell's and Szilard's thought experiments [2,3].
For Markovian feedback protocols a refined version of the second law reads Ẇext ≤ k B T 0 İmar flow , where T 0 is the bath temperature and İmar flow the information flow to the detector, defined as the time variation of the mutual information between a variable and its measured value [4][5][6].This inequality has been experimentally verified with colloidal particles [7,8] and single electrons [9,10].When the information rate İmar flow is positive, more work can be extracted from the system than permitted by the usual second law of thermodynamics.
The fact that a control signal cannot be applied instantaneously implies that feedback circuits inevitably exhibit memory effects and are thus non-Markovian.The Markovian approximation is only valid when the delay, i.e. the time between measurement and feedback, is much smaller than the typical timescales of the system.Delayed feedback is widespread in many areas, from chaotic systems to biology [11][12][13][14][15], emphasizing the crucial need to expand the second law to account for finite memory.However, such generalization is nontrivial.Because of the non-Markovian nature of the feedback, the conventional approach of stochastic thermodynamics [16] cannot be applied and the usual condition of local detailed balance does not hold.As a result, new contributions to

Particle
(T eff ) Fig. 1: Non-Markovian delayed feedback control.The system consists of a particle (white) in contact with a single heat bath (red) and coupled to a feedback reservoir (green).
The feedback loop continuously detects the position xt of the particle and applies a delayed feedback force F fb (xt−t fb ) proportional to the position at time t − t fb .Depending on the time delay t fb , the feedback cools or heats the system to a steady state effective temperature T eff .The blue arrows in the figure correspond to cooling, when the feedback control pumps the entropy Spump out of the system.Energy flows as heat Q from the bath to the particle and is extracted as work W by the feedback circuit (blue arrows).In the case of heating, the energy and entropy flows change direction.In both regimes the generalized second law holds in the form, Ẇext = − Ẇ ≤ kBT0 Ṡpump.
the nonequilibrium entropy production occur, leading to the extended second law for continuous, non-Markovian feedback, Ẇext ≤ k B T 0 Ṡpump , where Ṡpump is the entropy pumping rate which incorporates the effect of the time delay [17][18][19][20].Since Ṡpump ≤ İmar flow , this is the tightest second-law inequality to date [18].Despite the omnipresence of delay in feedback processes, a dedicated experimental investigation of this non-Markovian generalization of the second-law inequality is still lacking.
Here, we report the experimental study of the thermodynamics of continuous, non-Markovian feedback control applied to the underdamped center-of-mass (CM) motion of a levitated microsphere [21].Levitated particles are an ideal experimental platform to explore thermody- namics in small systems [22][23][24][25][26][27].We confirm the validity of the generalized second law for time delays spanning two decades and observe the breakdown of the highquality-factor (high Q) approximation [18].We establish that the efficiency of the feedback is enhanced when non-Markovian effects are included.We further explore the relation between Markovian and non-Markovian feedback by analyzing how the delay affects the correlations between measurement outcome and velocity of the particle.We finally explore the limitations of feedback cooling and the saturation of the effective CM temperature of the system above the bath temperature for very large delay.

Results
Generalized second law.We consider a harmonic oscillator in contact with a heat bath at temperature T 0 , and subjected to a delayed feedback control that acts as an information reservoir (Fig. 1).Delayed feedback control means here that we acquire information about the oscillator position x t at time t and apply a force F fb (t) ∝ x t−t fb to manipulate its motion based on the position measured at time t − t fb .The stochastic dynamics of the oscillator is governed by the underdamped Langevin equation [28], with m the particle mass, Ω 0 its natural frequency and Γ 0 the damping coefficient.The quantity ξ t is a centered Gaussian white noise with ξ(t)ξ(t ) = δ(t − t ).
The linear feedback is applied via the force F fb = −g(mΓ 0 Ω 0 )x t−t fb with feedback gain g > 0 [18].The mechanical quality factor of the resonator is given by Q 0 = Ω 0 /Γ 0 and the feedback damping rate by Γ fb = gΓ 0 .It is convenient to introduce the normalized delay τ = t fb Ω 0 and the position of the oscillator normalized to its standard deviation in equilibrium [17][18][19][20].The dynamics of the particle is then fully characterized by a set of dimensionless parameters (g, Q 0 , τ ) (Methods).Both the harmonic and Brownian terms in Eq. ( 1) are Markovian.Memory effects enter only via the delayed feedback force F fb .
When the feedback force acts to cool the harmonic oscillator, the extracted work W ext = −W is taken to be positive.According to the first law, the energy balance reads ∆U = W − Q.As result, the heat dissipated into the heat bath in the steady state is Q = −W ext .On the other hand, the steady-state entropy balance is [17][18][19][20], where Ṡpump is the entropy pumping rate, an additional contribution to the entropy production that stems from the feedback control and depends on the delay for non-Markovian protocols.The entropy pumping rate is computed by coarse-graining the harmonic and feedback forces over the position variable [17][18][19][20] (Methods).For a harmonic potential and a linear feedback force, the velocity distribution is a Gaussian (Fig. 2b), The non-Markovian feedback control leads to a cooling of the microparticle, corresponding to negative entropy pumping and extracted work rates, when σ 2 v < 1.By contrast, heating occurs for σ 2 v > 1.The validity of inequality (2) can be assessed by comparing the pumping entropy to Markovian bounds.The usual Markovian velocity feedback (VFB) cooling [29,30], with a feedback force proportional to the instantaneous velocity of the particle, F fb ∝ −v, is recovered in the limit of Q 0 τ and for τ = π/2 + 2πn, with n an integer.The entropy pumping rate corresponds in this case to Ṡvfb = g/Q 0 [33], which we identify as the Markovian information flow İmar flow (Methods).The highquality-factor approximation (Q 0 1) allows one to map the non-Markovian dynamics to an effectively Markovian Langevin equation, because the motion of the oscillator is essentially coherent on that timescale [18].The effect of the feedback is then incorporated in a modified damping Γ = Γ 0 (1 + g sin τ ) and mechanical frequency Ω 2 = Ω 2 0 [1 − (Γ fb /Ω 0 ) cos τ ] of the resonator (Supplementary Note 2).The Markovian second law remains valid with these modified parameters.In the high Q approximation, the entropy pumping is given by ṠhighQ = (g/Q 0 ) sin τ [18].The approximation is expected to break down for larger τ , when the Brownian force noise leads to dephasing between the oscillator motion and the feedback signal.This regime can only be correctly described using the generalized second law (2).
Experimental setup and results.In our experiment, we use an optically levitated microparticle to implement the dynamics of Eq. ( 1), that holds for any harmonic system with linear feedback control.A standing wave is formed by two counterpropagating laser beams (λ = 1064 nm) inside a hollow-core photonic crystal fiber (HCPCF) (Fig. 2a and Methods) [21].A silica microsphere (969 nm diameter) is trapped at an intensity maximum of the standing wave.The amplitude of the particle motion is sufficiently small to allow for a harmonic approximation of the potential with frequency Ω 0 /2π = 404 kHz (Supplementary Note 7).The damping coefficient Γ 0 as well as the bath temperature T 0 = 293 K are determined by the surrounding gas.In our setup, the linear dependence of the damping coefficient Γ 0 on the environmental pressure allows simple and systematic tuning of this parameter along with the mechanical quality factor Q 0 = Ω 0 /Γ 0 .The particle motion along the x-axis is detected by interferometric readout of the light scattered by the particle [21].The signal is fed into a delay line that is digitally implemented and the output signal to control the power of a feedback laser.This laser exerts a radiation pressure force in one direction, accelerating the microparticle proportional to the delayed particle position.The overall amplification of the signal sets the proportionality constant, which is given by gΩ 0 Γ 0 , where the gain in our experiment is g = 0.36 (Supplementary Note 5).The whole feedback circuit has a minimal delay of t fb = 2.6µs, i.e., τ = 2.04π.
We first test the extended second-law (2) by varying the delay over two decades.Figure 3a demonstrates the validity of the non-Markovian inequality (2) over all relevant timescales.The non-Markovian entropy pumping rate Ṡpump (green) is a much more precise upper bound to the extracted work rate (black) than the Markovian pumping rate Ṡvfb (horizontal dashed line).In particular, the Markovian result fails to capture the oscillations of the extracted work rate, as well as the heating phases induced by the delay.The high Q approximation (dasheddotted line) correctly describes the oscillatory behavior of Ṡpump for short delays.Yet, it does not account for the oscillation decrease induced by the Brownian force noise for long delays.We already observe significant deviations for a delay of only three oscillation periods with a mechanical quality factor of Q 0 = 55.We have also verified the second law (2) by varying the dissipation via Q 0 (Supplementary Note 9).The generalized second law ( 2) is crucial to properly estimate the performance of the feedback cooling.In analogy to heat engines and refrigerators, one may define the efficiency of work extraction η pump = Ẇext /(k B T 0 Ṡpump ), which characterizes the conversion of information into extracted work [32].As shown in Fig. 3b, the corresponding Markovian efficiencies for velocity feedback (η vfb ) and for the high Q approximation (η highQ ) vastly underestimate the feedback efficiency.We note that pumping efficiency η pump and cooling power Ẇext exhibit a trade-off similar to that of heat engines: one is maximal when the other is minimal, and vice versa.By contrast, the velocity feedback efficiency η vfb exhibits an opposite dependence on τ .
We may gain physical insight on the breakdown of the standard second law seen in Fig. 3 by analyzing the correlations between particle velocity feedback force, as well as the effective temperature of the system.Cooling is efficient when the feedback force counteracts the motion of the oscillator, in other words, when the velocity v t of the oscillator and the feedback force (F ∝ x t−τ ) are anticorrelated.Heating occurs when they are correlated.Figure 4 shows the correlation function between the two quantities, c(τ ) = 1/(σ q σ v ) y t v t P (y t , v t )dy t dv t with y t = q t−τ (Supplementary Note 4).The delay τ has two effects.First, it changes the phase between the mechanical system and the feedback signal deterministically, resulting in the oscillatory behaviour of the correlations, and thus the difference between heating and cooling.Second, it allows for stochastic dephasing of the mechanical motion with respect to the feedback signal, which translates into a reduction of the correlations for increasing delay.These correlations do not vanish, however, but asymptotically approach a finite value.For long delays, the oscillator thermalizes due to the damping.The action of the feedback circuit can then be seen as an independent force noise with the spectrum of a white-noise driven harmonic oscillator.The positive correlations occur as a result of the resonant driving of the mechanical motion by the feedback signal.
Figure 5 displays the ratio of the effective steady-state temperature, T eff = T 0 σ 2 q [18], and the bath temperature T 0 .Figure 5a clearly shows how the action of the feedback is reduced when the mechanical quality factor Q 0 is decreased or the time delay τ increased.For all values of Q 0 , there is a certain delay τ , beyond which cooling is no longer possible (black line).even longer delays, the effective temperature reaches a constant value, coupling g Q 0 (Supplementary Note 3).This is in line with the second law that predicts an asymptotic negative work extraction, W ∞ ext ≈ −g 2 /(2Q 0 ) for very long delays.Figure 5b provides a cut for constant Q 0 = 55 through Fig. 5a.The grey-shaded area on the right shows the region where T eff > T 0 for our gain g = 0.36.Note that this region can be reduced by decreasing the feedback gain.Excellent agreement between theory (red) and data (black) is observed.

Discussion
In summary, we have performed an extensive experimental study of the second law of thermodynamics in the presence of continuous non-Markovian feedback.Our results constitute an important step towards bridging theoretical developments in stochastic information thermodynamics and more technical applications like continuous feedback.Possible generalizations include the consideration of measurement noise [33] and the extension to nonlinear potentials and nonlinear feedback (e.g.parametric cooling of levitated nanoparticles [34]).Nonlinear delayed feedback control has thus already proven to be a more robust and effective method for synchronization compared to its linear counterpart [35].Another avenue of future research is the use of optimal control via Kalman-Bucy filters [20].Intuitively, one might hope that elaborate filtering methods, like Kalman filters, may overcome the impact of delay that we observe in our simple feedback scenario by an optimal prediction of the instantaneous velocity.However, while this will help to reduce the effect of measurement noise, the effect of stochastic Brownian force noise that occurs during the delay is fundamentally unpredictable.We therefore anticipate no improvement compared to the long delay results presented in our work.

Methods
Normalized form of the equation of motion.After time contraction t → tΩ0 and position normalization x → q = x/x th with x th = kBT0/mΩ 2 0 the thermal root mean square amplitude, the Langevin equation can be rewritten in the dimensionless form, with Q0 = Ω0/Γ0 the quality factor, g = Γ fb /Γ0 the feedback gain, τ = t fb Ω0 the normalized delay and ξt representing a centered Gaussian white noise force with ξ(t)ξ(t ) = δ(t−t ).
Experimental setup.The particle is trapped in an intensity maximum of the standing wave and oscillates in a harmonic trap.For a laser power of 400 mW in each trapping beam, the mechanical frequency is Ω0/2π = 404 kHz.The particle CM motion is recorded using a balanced photodiode and roughly 10% of the light transmitted through the HCPCF and that scattered by the particle.The feedback control is implemented via radiation pressure of a second laser beam which is orthogonally polarized and frequency shifted with respect to the trapping laser to avoid interference effects.The feedback force F fb is realized in the following steps.The read-out signal of the CM motion is bandpass filtered (bandwith: 600 kHz, center frequency: Ω0/2π) to get rid of technical noise in the detector signal.Then, the signal can be amplified and delayed by an arbitrary time t fb with a field programmable gate array (FPGA).This signal is then used as modulation input for the AOM.A more detailed description of the experimental setup and the feedback control can be found in Supplementary Note 6 and in Ref. [21].
Thermodynamic quantities.The non-Markovian entropy pumping rate Ṡpump can be computed by coarse-graining the harmonic (h) and feedback (fb) forces over the position variable: Ṡpump = − dv[F fb (v) + F h (v)]∂vP (v), where P (v) is the velocity distribution and Fi(v) = dxFi(x, v)P (x|v) the corresponding coarse-grained forces [17][18][19][20].On the other hand, the Markovian information flow İmar flow in the case of VFB is given by the time variation of the mutual information between the velocity v and the measured value of the velocity y: İmar flow = dvdy∂vJ(v)ln[P y)/P (v)P (y)] with the velocity probability current J(v) [18][19][20].This quantity diverges for error-free feedback [33].We thus identify the Markovian entropy bound with the Markovian limit of Ṡpump → Ṡvfb = g/Q0 [18][19][20].For a harmonic potential and a linear feedback force, the velocity distribution is a Gaussian, The entropy pumping rate is then explicitly Ṡpump = (1 − σ 2 v )/(Q0σ 2 v ) and the work extraction rate reads Ẇext/(kBT0) = (1 − σ 2 v )/Q0 (Supplementary Information).We experimentally obtain the velocity variance needed to verify the generalized second law as follows: for a given time delay τ , a position time trace x(t) is recorded, filtered with a bandwidth of 3Γ0 via post processing, and normalized with the standard deviation of the feedback beam turned off, to find the normalized position q(t).The velocity v = q is then numerically calculated with finite difference approximation and the variance σ 2 v computed.

Fig. 2 :
Fig.2: Experimental setup.a) Two counterpropagating laser beams (red) are coupled into a hollow-core photonic crystal fiber (HCPCF).They form a standing wave that allows for trapping a silica microparticle at an anti-node (inset, white arrow).The fiber ends are placed inside vacuum chambers (vac) to control the pressure inside the HCPCF.The particle position along the fiber axis is detected with an interferometric readout.The feedback loop is implemented delaying the detected signal by a time t fb .To apply the feedback force, a feedback laser (green) is modulated with the delayed signal via an acousto-optic modulator (AOM).b) Phase space distribution P (x, v) of the microparticle derived from the position measurements at T eff = T0 (red circles) and T eff < T0 (blue circles).The respective marginals for the distributions of position and velocity are shown on the top and right panels.

Fig. 3 :
Fig.3: Test of the generalized second law.a) Experimental entropy pumping (green squares) and extracted work (black circles) rates are plotted as a function of time delay for Q0 = 55 and g = 0.36.The shaded area corresponds to the analytical prediction including experimental parameter drifts (Supplementary Note 8).The error bars represent statistical uncertainties.Theoretical predictions for Markovian entropy bounds to the extracted work are shown for velocity feedback ( Ṡvfb , horizontal dashed line) and for the high-quality-factor approximation ( ṠhighQ , dashed-dotted line).The blue (red) region corresponds to cooling (heating) of the particle motion.For all delays, the entropy pumping is larger than the extracted work.At the same time, it represents a tighter bound to the extracted work than the Markovian predictions.b) Feedback efficiencies plotted in the cooling regions only.Experimental pumping efficiency ηpump = Ẇext/(kBT0 Ṡpump) (green squares) of the generalized 2nd law is compared to the experimental Markovian velocity feedback efficiency η vfb = Ẇext/(kBT0 Ṡvfb ) (black circles) where Ṡvfb = g/Q0.Theoretical Markovian predictions for η vfb (dashed) and η highQ = Ẇext/(kBT0 ṠhighQ ) (dashed-dotted) with ṠhighQ = (g/Q0) sin τ are shown for comparison.Neglecting the non-Markovian behavior leads to a strongly underestimated efficiency of the feedback process.

Fig. 4 :
Fig. 4: Correlation and colored noise for long delays.a) Experimental (black) and theoretical (red) correlation functions of the microparticle CM motion as a function of τ for g = 0.36.b,c) Zoom in for short (b) and long (c) delays.d) Experimental probability distribution P (qt−τ , vt) for different values of time delay in panels b) and c) indicated by black arrows.The system is still correlated for very long delays (τ = 88π) due to resonant driving by colored force noise created by the feedback loop.

Fig. 5 :
Fig. 5: Steady-state effective temperature.a) Calculated 2D color map of the ratio T eff /T0 of the microparticle motion as a function of τ and Q0 for g = 0.36.The black solid line represents the border beyond which cooling is no longer possible.Note that this border is gain dependent.b) Experimental (black) and calculated (red) normalized effective temperature for Q0 = 55, corresponding to the dashed horizontal line shown in panel a).The grey shaded area shows the region with T eff > T0 for very long delays.