Squeezed light from an oscillator measured at the rate of oscillation

Sufficiently fast continuous measurements of the position of an oscillator approach measurements projective on position eigenstates. We evidence the transition into the projective regime for a spin oscillator within an ensemble of 2 × 1010 room-temperature atoms by observing correlations between the quadratures of the meter light field. These correlations squeeze the fluctuations of one light quadrature below the vacuum level. When the measurement is slower than the oscillation, we generate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$11.{5}_{-1.5}^{+2.5}\,{{{{{{{\rm{dB}}}}}}}}$$\end{document}11.5−1.5+2.5dB and detect \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$8.{5}_{-0.1}^{+0.1}\,{{{{{{{\rm{dB}}}}}}}}$$\end{document}8.5−0.1+0.1dB of squeezing in a tunable band that is a fraction of the resonance frequency. When the measurement is as fast as the oscillation, we detect 4.7 dB of squeezing that spans more than one decade of frequencies below the resonance. Our results demonstrate a new regime of continuous quantum measurements on material oscillators, and set a new benchmark for the performance of a linear quantum sensor.


I. INTRODUCTION
Projective, or von Neumann, measurements collapse the observed quantum system on eigenstates of a Hermitian operator, while more general measurements, described by positive operator-valued measures, collapse the system on states from an overcomplete set [1].A gradual transition between the two situations can be realized in continuous measurements using meter fields, a canonical example of which is an optical interferometric measurement of the position of a harmonic oscillator [2].Position measurements are associated with mechanical resonators [3], collective atomic spins [4,5], ferromagnetic solid-state media [6], single molecules [7], or density waves in liquids [8], that are linearly probed by traveling optical or microwave fields.The boundary between generalized and von Neumann measurements occurs at a certain value of the measurement rate [9].When the rate is slower than the oscillation, measurements with the meter in the vacuum input state project the oscillator on coherent states.When the rate is faster than the oscillation, measurements project the oscillator on position-squeezed states.
In addition to the oscillator state, the rate of position measurement affects the output state of the meter field [9].The quadratures of the meter are correlated, and their fluctuations can be below the vacuum level [10,11].In the slow measurement regime, the correlations and the associated squeezing exist in a narrow frequency band near the resonance, and have a strong frequency dependence due to the time-averaged response of the oscillator to the measurement backaction.When the measurement is faster than the oscillation, the correlations and squeezing are broadband and frequency-independent at low frequencies, where the oscillator responds to the backaction instantaneously.The detection of squeezing means observing the backaction-driven motion of the oscillator at frequencies much lower than the resonance, which is a necessary condition for position squeezing [9].
The squeezing of the meter light is both a valuable quantum resource and a figure of merit for the purity of the light-oscillator interaction.In the slow regime, we realize a measurement of a collective spin of a roomtemperature atomic ensemble at a rate fifteen times higher than the rate of thermal decoherence.The generated squeezing of the meter light reaches 11.5 +2.5 −1.5 dB at the output of the cell, exceeding the squeezing demonstrated previously using collective atomic spins [12][13][14], optomechanical cavities [15][16][17], levitated nanoparticles [18,19], and compact on-chip sources utilizing material nonlinearity [20], while approaching the results achievable using bulk nonlinear crystals [21].In the fastmeasurement regime, we detect broadband squeezing in a bandwidth of several MHz while keeping the backactionimprecision product [22] within 20 % from the value saturating the Heisenberg uncertainty relation.These results enable new regimes for sensing surpassing the standard quantum limit [23,24], tests of uncertainty relations for past quantum states [25,26], quantum control of material oscillators [27][28][29][30], and links between collective spins and other material systems [14,[31][32][33].

II. MEASUREMENTS OF SPIN OSCILLATORS
Linearly polarized light traveling through an oriented atomic medium (as illustrated in Fig. 1a-b) continuously measures the projection of the total spin on the propagation direction, Ĵz , via polarization rotation.This measurement acts back on the spin via quantum fluctuations of ponderomotive torque.When the input light is in a strong coherent state, and the spin satisfies the Holstein-Primakoff approximation [34], the process can be described in terms of linearly coupled pairs of canonically conjugate position and momentum variables.The canonical variables of the spin, XS and PS , are the normalized projections defined as XS = Ĵz / √ J x and  PSD (shot noise units) Δ/(2π) = 7 GHz FIG. 1. a) An optical probe spatially shaped in a square tophat beam travels through an atomic ensemble with the total spin J in a magnetic field B, and is detected using balanced polarization homodyning.The detected quadrature is selected using the λ/2 and λ/4 waveplates.The total spin is oriented by the repump beam traveling along x.PBS: polarization beam splitter.b) The polarization angle β of the probe as a meter for the spin projection Ĵz. c) A photograph of an anti-spin-relaxation coated cell.The channel with probed atoms is indicated by the blue rectangle.d) The orange curves show power spectral densities (PSD) of homodyne signals recorded at ∆/(2π) = 7 GHz at different quadratures.The trace showing the largest squeezing is highlighted by the blue curve.The black curve is the theoretical prediction based on the global fit including all quadratures (see the SI).The gray curve is the shot-noise level.The red curve is the theoretical optimum-quadrature squeezing spectrum.PS = − Ĵy / √ J x , which satisfy the commutation relation [ XS , PS ] = i.The variables of the light, XL and PL , are the quadratures proportional to the amplitude and phase differences between the circularly polarized components, respectively.Their commutator is [ XL (t), PL (t )] = (i/2)δ(t − t ).The Heisenberg uncertainty principle constrains the two-sided spectral densities of the imprecision in the PL -quadrature measurements, S imp , and the measurement backaction, S BA , as S imp S BA ≥ /2 (see Ref. [22] and the SI).This uncertainty relation is saturated if the detection efficiency is perfect and there is no excess measurement noise.
When the ensemble is probed far-detuned from optical transitions, the total spin couples to the probe via the position-measurement Hamiltonian Ĥint = −2 √ Γ XL XS , and modifies the probe variables according to the input-output relations [14,35] where γ th = (2 n th +1)γ 0 is the thermal decoherence rate.The term ∝ cos(φ) 2 is due to the spin oscillator motion, and the term ∝ sin(2θ) is due to the cross-correlation between XS and Xout L .Negative cross-correlation can squeeze S φ [Ω] below the vacuum level of 1/4.
In a more general situation, the internal dynamics of the collective spin are those of 2F harmonic oscillators, where F is the ground-state angular momentum number of the atomic species.Their annihilation operators, bm are introduced using the multilevel Holstein-Primakoff approximation [36].In Eq. (3), m is the projection quantum number of the single-atom angular momentum on the x axis, |m + 1 j m| j are the jump operators between the states |m j and |m + 1 j of the individual atoms, and ∆N m = N m+1 −N m are the differences in the mean numbers of atoms in the corresponding states.The frequencies of the oscillators are the energy differences between |m j and |m + 1 j , controlled by an external static magnetic field.The oscillator-light interaction is described by the Hamiltonian Ĥint = −2 where the quadratures of the modes satisfy [ Xm , Pm ] = i, Γ m are the measurement rates, and  experiments.The oscillators experience thermal decoherence due to the spontaneous scattering and the collisions of atoms.The thermal occupancy of the intrinsic damping bath is n th = N m /∆N m , experimentally found to be independent of m.
The multimode structure can affect the response of the spin to the measurement backaction at frequencies close to Ω S , while far away from Ω S the spin acts as a single oscillator with XS = m Γ m /Γ Xm that is measured at the total rate Γ = m Γ m and experiences decoherence at the rate γ th = m γ th,m Γ m /Γ, where γ th,m are the individual decoherence rates of the modes.The quantum cooperativities for the individual modes are defined as the ratios of the measurement and decoherence rates.For the total spin, the cooperativity is C q = Γ/γ th .

III. EXPERIMENT
An ensemble of N ≈ 2 × 10 10 cesium-133 atoms at 52 °C is contained in the 1 mm×1 mm×4 cm channel of a glass chip, shown in Fig. 1c.The channel is coated with paraffin to reduce the spin decoherence from wall collisions [37], and is positioned in a homogeneous magnetic field directed along the x axis (Fig. 1a).The ensemble is continuously probed by a y−polarized laser beam propagating in the z direction that has the wavelength 852.3 nm, blue-detuned from the F = 4 → F = 5 transition of the D2 line by ∆/(2π) = 0.7 − 7 GHz.The ensemble is also continuously repumped using circularly polarized light resonant with the F = 3 → F = 2 transition of the D2 line.The combination of spontaneous scattering of probe photons and repumping maintains a steady-state distribution of atoms over the magnetic sublevels of the F = 4 ground state, which has the macroscopic spin orientation along the magnetic field with polarization Ĵx /(N F ) ≈ 0.78.The steady-state populations are independent of the probe power in our regime, and correspond to the occupancy of the thermal bath n th = 0.9 ± 0.1.The resonance frequencies of the oscillators are set by the Larmor frequency and split by 0 − 40 kHz in different regimes by the quadratic Zeeman and tensor Stark effects.The Larmor frequency can be positive or negative depending on the orientation of the magnetic field, setting the signs of the effective oscillator masses.We work in the negative-mass configuration [31], but the effects that we observe, in particular the squeezing levels, do not change upon the reversal of the sign of mass (see the SI).The output light is detected using balanced polarization homodyning, which enables shotnoise-limited detection at frequencies down to 10 kHz.

IV. RESULTS
In Fig. 1d, we present homodyne spectra recorded at the optical detuning ∆/(2π) = 7 GHz over a range of detection quadratures φ.In this measurement, dynamical backaction effects are small (ζ ≈ 0.01), and the probed spin behaves as a single oscillator subjected to position measurements.The data in Fig. 1d shows squeezing down to 7.5 dB, attained by the highlighted blue trace.From a global fit of the spectra at all quadratures, we infer the measurement rate Γ/(2π) = 13 kHz and the The trace with the largest squeezing is highlighted by the blue curve.The black curve is the theoretical prediction based on the global fit including all quadratures (see the SI).c) The spectra taken at the PL quadrature when the probe beam is Gaussian (blue curve) and tophat (orange curve).The gray curve is the shot noise.
The inset shows the beam intensity distributions over the 1 mm×1 mm channel cross section recorded without the cell.
quantum cooperativity C q = 11.The measurement rate can be verified directly from Fig. 1d via the width ∆Ω of the frequency band over which squeezing is present in any of the traces, which in the backaction-dominated regime is ∆Ω ∼ Γ.The envelope of the traces in Fig. 1d is described by the spectrum given by Eq. ( 2) minimized over the detection quadrature at each frequency.Neglecting the imaginary part of the response, the optimumquadrature spectrum is given by where The red curve plotted in Fig. 1d additionally accounts for 0.7 shot noise units of excess PL -quadrature noise from the thermal motion of fast-decaying spin modes (see Sec. V).This noise is the main limitation for the backaction-imprecision product in this measurement, which equals 1.5 × ( /2).
Due to the scaling Γ ∝ 1/∆ 2 , higher measurement rates are achievable with the probe laser tuned closer to the atomic transition.In Fig. 2a we present data obtained at the optical detuning of 3 GHz using 8.4 mW of probe power.In this measurement ζ = 0.054, in which case the dynamical backaction results in optical damping and hybridization of the oscillator modes, as well as optical squeezing in the XL -quadrature (see the green trace in Fig. 2a).Since the thermal decoherence of the oscillators is due to baths at a temperature close to zero, the optical damping improves the maximum magnitude of squeezing by about 0.5 dB.The minimum noise shown by the blue trace in Fig. 2a is 8.5 +0.1 −0.1 dB below the shot noise level.The overall detection efficiency of our setup is η = (91 ± 3) %, and the transmission loss at the exit window of the cell is 1.6 %, which means that the magnitude of the squeezing at the exit of the cell is 11.5 +2.5 −1.5 dB.The backaction-imprecision product in this measurement is 1.9×( /2), which is higher than in the measurement at 7 GHz detuning due to the higher excess PL -quadrature noise (two shot noise units).
The experimental spectra in Fig. 2a can be understood as arising from the coupled dynamics of two nearlydegenerate bright modes of the spin, which we refer to as modes a and b.To extract their effective parameters, we globally fit the set of spectra recorded over an extended range of quadrature angles (see the SI).We find the total measurement rate to be Γ/(2π) = 52 kHz, the individual quantum cooperativities to be C a q = 12 and C b q = 4, and the total cooperativity to be C q = 15.The lower envelope of the experimental traces is in agreement with the optimum-quadrature spectrum predicted by the single-oscillator model using the same Γ and C q .
The bright modes a and b emerge due to the coupling of the individual spin oscillators via the common reservoir of the probe optical modes with coupling rates proportional to ζ m and Γ m .To illustrate this effect, we set the laser detuning to 0.7 GHz, where the dynamical backation coefficient is larger, ζ = 0.18, and excite the oscillators with classical white noise applied via a magnetic field.The spectra of the P quadrature of the output light at different probe powers are shown in Fig. 2b.At the lowest power, the eight bare spin oscillators due to the transitions between adjacent m F levels are individually resolved.As the probe power is increased, the resonances first merge in two (the a and b modes) and then three.The macroscopic occupancies of different m F levels in the atomic ensemble remain the same at all powers, as we separately check, which means that the change in the output spectrum is only due to the coupled dynamics of the collective oscillators.
At the detuning of 0.7 GHz from the optical transition, the measurement rate of the spin motion can be as high as the oscillation frequency.While around the Larmor resonance, in a frequency band of approximately one hundred kHz, the coupling between individual spin oscillators is pronounced, at frequencies much lower than the resonance the spin behaves as a single oscillator, and the quantum measurement backaction manifests via broadband squeezing of light.In Fig. 3a, we present spectra recorded using 12.8 mW of optical probe power at two resonance frequencies, 1.09 MHz and 1.79 MHz, in which the bandwidth of low-frequency squeezing extends down to 30 kHz.The minimum noise levels of the homodyne signals (6.5 dB below the shot noise for the 1.09 MHz data) are consistent with the quantum cooperativity C q = 8.The measurement rate can be estimated from the signal-to-shot-noise ratio on the P quadrature in Fig. 3a using the formula which yields Γ/(2π) ≈ 2 MHz, a value higher than the resonance frequencies.To further corroborate the measurement rate, we perform a quadrature sweep with the resonance frequency set to 5 MHz and using 10.2 mW of probe power (Fig. 3b).From fitting this data, we find Γ/(2π) = 1.77MHz, which is consistent within ten percent with the previous estimate corrected for the difference in the probe powers.Theoretically, the optimumquadrature noise levels should saturate as the Fourier frequency approaches zero, to a value around 0.22 shotnoise units for the 1.09 MHz data in Fig. 3a, while experimental noise levels increase at low frequencies due to excess noise from the atomic ensemble.The backaction-imprecision product for the measurements in Fig. 3a is below 1.2 × ( /2) at frequencies higher than 100 kHz.This value is closer to saturating the Heisenberg uncertainty relation than the values in the slow-measurement experiments, because the fastdecaying modes are in the backaction-dominated regime, and do not contribute excess thermal noise.The limiting factors for the product in this case are the dynamical backaction and detection inefficiency.

V. FAST-DECAYING MODES
In addition to the collective oscillators described by the annihilation operators from Eq. ( 3), in which all atoms contribute equally, there are other modes of the spin in our system [38,39].The resonance frequencies of these modes coincide with Ω S , but their decay rates are limited by the rate of atoms flying through the probe field (γ 0,flight /(2π) ≈ 300 kHz) rather than collisions with the walls and other atoms (γ 0,coll /(2π) ≈ 200 Hz).The annihilation operators of these modes are b m = 1 ∆N m ∆g(t) 2 c N j=1 ∆g j (t) |m j m + 1| j , (7) where g j (t) are the coupling rates between the optical probe and the individual atoms (see the SI) and ∆g 2 c is the squared deviation of the coupling from the mean averaged over classical trajectories, assumed to be the same for all atoms.The measurement rate of the fastdecaying modes is ∝ ∆g 2 c , while the measurement rate of the slow-decaying modes is ∝ g 2 c .An enabling feature of our experiment is the high 3D uniformity of the optical probe field, achieved using a tophat beam configuration, which reduces ∆g 2 c and thus the readout of the fast-decaying modes.In Fig. 3c, we compare the spectra recorded at the PLquadrature using a tophat and a wide Gaussian probe beam with equal optical powers in the slow-measurement regime.The thermal noise contributed by the fastdecaying modes is reduced from 1 to 0.3 shot-noise units on resonance upon switching from the Gaussian to the tophat probe.The absolute non-uniformity of the coupling [40,41] for the tophat beam is estimated to be ∆g 2 c / g 2 c = 0.6 based on the camera imaging.

VI. OUTLOOK
Continuous measurements that combine high measurement rate, quantum cooperativity, and detection efficiency can be used for single-shot generation of spinsqueezed states and quantum state tomography [42].The entanglement link between the material spin and traveling light entailed by the squeezing enables quantumcoherent coupling of spins with other material systems [14,32].While the backaction-imprecision product in all our measurements is already within a factor of two from the Heisenberg bound, it can be further improved by optimizing the probe power for measurements of the PL -quadrature.Our measurements were optimized for quadratures intermediate between XL and PL (i.e. for "variational" readout [23]) which can yield superior results [43] in quantum sensing and control.
This work also establishes room-temperature atomic spin oscillators as a practical platform for engineering quantum light with high levels of squeezing, which is a basic resource for interferometric sensing and optical quantum information processing [20].The highest demonstrated squeezing, reaching 8.5 dB at the detection, is narrowband, but its frequency can be tuned by the magnetic field without degrading the level within the range of approximately 0.8 − 5 MHz in our experiments.and modes scrambled by the atomic motion.Individual atoms interact with the light field with the strengths g k (t) (where k = 1, .., N is the integer index that labels the atoms) that is proportional to the intensity of the light field at their instantaneous position.The interaction strengths randomly change in time as atoms move inside the cell.The motions of different atoms are assumed to have the same statistical properties and be uncorrelated between each other.The statistics of motion are characterized by decomposing the couplings into their mean value, ḡ, and deviations, ∆g k (t), and specifying the motional correlation function, R(τ ), where δ kl is the Kronecker symbol and • c denotes motional averaging (following the notation of Ref. [2], to separate from the quantum averaging • ).The normalization factor, ∆g(t) 2 c , is the mean squared deviation among the individual atom-light couplings.According to the ergodic hypothesis, the result of the averaging is the same regardless of whether it is done over the time or the realizations of the ensemble.The dispersive interaction between the light and the k−th atom in the ensemble is described by the Hamiltonian [3,4] where Ŝx,y,z are the Stokes parameters of the input light [3], Î is the intensity of the input light, and the parameters a 0,1,2 are functions of the level structure and the laser detuning from the optical transition [5].After linearization assuming a strong coherent y-polarized light probe with the mean amplitude ā, the Hamiltonian is expressed as where the Stark Hamiltonian Ĥ(k) The spin components of individual atoms ĵ(k) x,y,z can be expressed in terms of the jump operators σ(k) n,m between the ground state sublevels, where m, n = −F, ..., F is the projection of the angular momentum on the x axis (which coincides with the direction of the magnetic field), and F is the total angular momentum quantum number of the ground state level.In this notation, m,m is the Stark energy, and C m = F (F + 1) − m(m + 1) are Clebsch-Gordan coefficients.When transiting from Eq. (SI B.4) to Eq. (SI B.7) we neglected the terms involving secondorder coherences that only couple to Î and are negligibly small in our case.
The individual atomic spins are precessing in a homogeneous magnetic field directed along the x axis.Taking the zero of the energy scale to be the ground state energy of free atoms, the Hamiltonian of the precession is expressed as where E Zeem,m are the Zeeman energies of the magnetic sublevels that include contributions linear and quadratic in m.The total Hamiltonian of all atoms, can be expressed using collective operators: the total numbers of atoms in the magnetic sublevels, denoted by Nm , and two sets of coherences between neighboring m levels, denoted by Σm and Σ m .The operators are defined as where m = −F, ..., F − 1 for the Σ operators and m = −F, ..., F for the N operators.The expression for the Hamiltonian, neglecting a small contribution due to the inhomogeneity of the Stark shift, is where ζ m = 2(2m + 1)a 2 /a 1 , and E m = E Zeem,m + E Stark,m is the sum of the Zeeman and the Stark energies.In the limit of a large number of atoms in the ensemble, the two sets of Σm operators are independent and have constant commutators, where ∆N m = N m+1 − N m .The modes described by bm are those usually identified with the Larmor precession of the spin ensemble as a whole.They experience coupling to the probe light that is averaged over the atomic trajectories [6], and their coherence time is high, limited by the reorientation of individual spins due to the collisions with the walls and between each other, and by the spontaneous scattering of probe photons.The modes described by b m experience additional damping and decoherence due to the atoms flying in and out of the probe beam.We refer to them as the fast-decaying modes.Introducing the quadratures of the spin oscillators, which satisfy [ Xm , Pm ] = i and [ X m , P m ] = i, and using the fact that, in the Holstein-Primakoff approximation, the numbers of atoms in the m−th levels satisfy the total Hamiltonian in Eq. (SI B.11) is expressed as which is a Hamiltonian of 4F oscillators linearly coupled to a propagating field.The frequencies Ω m are determined by the energy splittings between different magnetic sublevels due to the Zeeman and Stark effects, and the measurement rates for the slow-and the fast-decaying modes are identified as The input-output relations for the quadratures of the light field are derived based on Eq. (SI B.18) as described in Ref. [7].They are given by Xout and the Heisenberg equations of motion for the slow-decaying modes are Eq. (SI B.23-SI B.24) show that the oscillators experience damping or antidamping by dynamical backaction with the rates γ DBA,m = 2ζ m Γ m , and are coupled between each other at the rates √ γ DBA,m γ DBA,n due to the interaction with the common optical bath.For practical calculations, intrinsic dissipation due to the atomic collisions and spontaneous scattering is added to Eq. (SI B.23-SI B.24) using the usual quantum Langevin approach [4].The temperatures of the effective thermal baths can be determined from the equilibrium numbers of excitation in the modes in the absence of probing, n th ≡ b † m bm = (N m /∆N m ), which are calculated directly from the definitions of bm under the assumption that the processes that determine the equilibrium populations N m affect all atoms independently.
The Heisenberg equations of motion describing the evolution of the modes from the fast-decaying family are identical to Eq. (SI B.23-SI B.24), except that they include additional terms due to the explicit time dependence of their operators.These terms are more convenient to present for the annihilation operators than for the quadratures, they are given by where −i[ b m , Ĥ] contributes the terms due to the coherent evolution and the coupling to the light field that are completely analogous to those present in Eq. (SI B.23-SI B.24).The added terms give rise to both extra dissipation and fluctuations.If the motional correlation function is exponential, ∆g k (t 1 ) ∆g k (t 2 ) ∝ e −γb|t1−t2|/2 , as it was suggested in [6], the stochastic evolution of ∆g k (t) can be modeled by the Ornstein-Uhlenbeck process, where In this case, the extra terms in the Heisenberg-Langevin equations for b can be re-expressed as where and n th = N m /∆N m is the thermal occupancy of the bath.While the atomic motion increases the decoherence rate, the thermal bath occupancies for the fast-and slow-decaying modes are the same.the mass sign on homodyne spectra is therefore the reflection of the spectra with respect to the Larmor frequency.We observe this in Fig. SI2d, where we invert the sign of the mass by changing the direction of the magnetic field.
where SN = 1/4 is the shot noise level, χ[ω] = −(1/2)/(∆Ω + iγ/2) is the RWA force susceptibility, ∆Ω = Ω − Ω S is the Fourier-detuning from the oscillator resonance, γ = γ 0 + 2ζΓ is the total oscillator linewidth, and the transduction factor A is (SI F.2) By minimizing Eq. (SI F.1) over the quadrature angle φ, we find the frequency-dependent maximum-squeezing angle where the total decoherence rate γ dec = γ th + γ QBA is the sum of the decoherence rates due to the intrinsic thermal noise, γ th and the quantum backaction, γ QBA which are defined as The shot-noise normalized signal spectrum at the optimum quadrature is where γ DBA = 2ζΓ is the contribution of the dynamical backaction to the total oscillator linewidth (the optical damping).The absolute minimum of the spectrum is found by further minimizing S φmin [Ω] over ∆Ω, which can be done analytically in the general case, but yields a cumbersome result.Instead of presenting this result, we restrict the attention to the case ζ 1, which is relevant to our experiments, and estimate the minimum noise level by evaluating S φmin [Ω] at ∆Ω min,ζ=0 = 1/2 γ(2γ dec + γ), the optimum Fourier detuning for ζ = 0.The result is When the thermal occupancy of the intrinsic bath is close to zero, and the quantum cooperativity is in the intermediate regime, such that γ dec has the same order of magnitude as γ 0 , there is an improvement in the minimum noise level from a small positive optical damping.Optical beams with tophat transverse profiles are commonly produced by passing a collimated Gaussian beam through an aspherical beam shaper, and focusing the beam after the shaper using a spherical lens.In this configuration, SI 10 the optimum tophat profile (giving the sharpest roll-off of the intensity distribution in the transverse direction) is realized before the focal point, and the beam is tightly focused.In our experiment, it is essential to create a beam in which the tophat profile coincides with the position of the beam waist, and has a relatively large transverse size, enabling a long Rayleigh length extending over the entire cell channel.
An intuition on how to produce a tophat beam that fulfills our criteria can be obtained by examining the setup shown in figure Fig. SI3a, which is a straightforward extension of the usual beam shaper application scheme with an addition of a negative lens f 2 .The optimum tophat transverse profile is realized at a distance one effective focal length (EFL) away from the first lens.The transverse width is proportional to the focal length f 1 .The beam is converging at the optimum point, because of the full fan angle of the tophat beam shaper (i.e. the divergence the shaper introduces in the beam).By placing an appropriate negative lens f 2 in the optimum point, the beam can be collimated, and its waist position made coincide with the optimum location of the transverse profile.The required focal length of the negative lens can be calculated given the size of the input Gaussian beam, w in , and the full fan angle of the beam shaper, φ FA , as f 2 = φFA/winf1 φFA/win−1/f1 .The setup in Fig. SI3a would be challenging to implement directly, because the waist position of the beam is located inside the cell, where placing a lens is hardly realistic.However, one can find an optical setup with an identical ray transfer matrix to the one in Fig. SI3a, but realized using a different physical arrangement of lenses.Such a setup is shown in Fig. SI3b.The transfer matrices for the two setups, M a and M b , are given by where the matrices for propagation in free space, S, and passing through a lens, L, respectively, are In our experiment, the setup in Fig. SI3b is implemented using lenses of pre-determined focal lengths F 1 and F 2 , while the separating distances L 1 , L 2 and L 3 are adjusted to meet the condition M a = M b .Additionally, the matrix M a is supplemented by an inversion in the transverse plane, which can be interpreted as passing the beam through an extra 4f optical system, which is done in order to have more flexibility in the choice of lenses and more control over the resulting distances.
) where Γ is the measurement rate proportional to the optical power.The measurement backaction force is FQBA = 2 √ Γ Xin L .The response of the spin to the measurement backaction in this situation is described by the Fourierdomain susceptibility χ[Ω] = Ω S /(Ω 2 S −Ω 2 −iΩγ 0 ), where Ω S is the resonance Larmor frequency and γ 0 is the intrinsic decay rate.The response induces correlations between Xout L and P out L that can be observed by detecting intermediate quadratures of light, Qφ L = sin(φ) Xout L +cos(φ) P out L .The two-sided spectra of those quadratures, detected by a homodyne with efficiency η, are given by S φ [Ω] = 1/4 + (ηΓ/2) Re (χ[Ω]) sin(2φ)

34 FIG. 2 .
FIG. 2. a) Homodyne signal PSDs at ∆/(2π) = 3 GHz and different detection angles φ indicated in the figure.The points are experimental data.The green and orange traces are obtained close to PL and XL, respectively, and the olive, blue and purple-at intermediate quadratures.The gray points show the shot-noise level.The black curves are theoretical predictions based on the global fit including the spectra at 15 quadratures (see the SI).The red curve is the optimum-quadrature squeezing spectrum predicted by the single-oscillator model.b) The spectra of classically driven motion of the collective spin.The eight peaks visible at low probe powers correspond to bare oscillator modes due to the transitions between adjacent mF levels.Their frequencies are determined by the linear and quadratic Zeeman energies, and magnitudes are determined by the macroscopic populations of the mF levels as shown in the inset.The spectra at high powers expose the hybridized oscillator modes.

Frequency
FIG.3.a-b) Homodyne signal PSDs at ∆/(2π) = 0.7 GHz.The gray curves show the experimental shot-noise levels, and the red curves are the theoretical optimum-quadrature squeezing spectra derived from Eq. (2).a) Spectra for |ΩS|/(2π) = 1.09MHz and 1.79 MHz.The orange and blue curves are measurements with the quadrature angle set to detect PL and a quadrature φ close to XL, respectively.LO: local oscillator, th: theoretical.b) Orange curves show homodyne spectra recorded at |ΩS|/(2π) = 5 MHz and at different quadratures φ.The trace with the largest squeezing is highlighted by the blue curve.The black curve is the theoretical prediction based on the global fit including all quadratures (see the SI).c) The spectra taken at the PL quadrature when the probe beam is Gaussian (blue curve) and tophat (orange curve).The gray curve is the shot noise.The inset shows the beam intensity distributions over the 1 mm×1 mm channel cross section recorded without the cell.

zÎ
describes the energy shifts due to the dynamic Stark effect, and XL and PL are the polarization quadratures of the light field normalized such that they satisfy the commutation relation [ XL (t 1 ), PL (t 2 )] = (i/2)δ(t 1 − t 2 ).(SI B.5) 14) where m, n = −F, ..., F − 1, and N m = N are the average macroscopic populations of the magnetic sublevels.By normalizing the Σ operators to satisfy the canonic commutation relations, we can introduce two sets of bosonic modes, bm and b m , that appear in the main text, bm = Σm / ∆N m , b m = Σ m / ∆N m , (SI B.15) FIG. SI2.a)-c) Power spectral densities (PSD) of homodyne signals recorded at different quadrature angles φ and laser detunings ∆.The points of different colors show the experimental spectra for different quadrature angles as labeled in the legends.The black curves show the results of global fits at each detuning performed as described in Sec.D. Gray points show the local oscillator shot noise.Panel a) displays only part of the 17 traces fitted in total.d) The effect of changing the oscillator mass, M , on the homodyne spectrum measured at a quadrature intermediate between XL and PL.The blue curve shows the spectrum recorded in a negative mass (M ) configuration, the orange curve shows the spectrum recorded in a positive mass configuration, and the gray curve shows the local oscillator (LO) shot noise.The sign of the mass was changed by inverting the direction of the magnetic field with respect to the x axis.The spectra were recorded using a 12 mW probe detuned from the optical transition by 3 GHz.

Appendix G :
FIG. SI3.Optical setups for the generation of collimated tophat beams.M a,b are ray transfer matrices.a) A simple setup.The dashed black line shows how the beam would propagate after passing the beam shaper and the lens f1, but without passing the negative lens f2.EFL: effective focal length.b) A realistic setup designed using the condition Ma = M b .Beam shaper: Gaussian-to-tophat beam-shaping lens.