## Introduction

Improving the understanding of magnetic interactions is of paramount importance due to the challenges associated with exotic magnetic phenomena, the potential connections to high-Tc superconductivity, and the widespread use of magnetic materials in current technology. A popular approach is to develop relatively simple, controllable “quantum simulators” of magnetic interactions using ultracold atoms loaded in optical lattices1. These simulators have been used to investigate classical2 and quantum3,4,5,6,7 magnetism, including the role played by the range of the spin-interaction8,9,10,11. Without externally applied optical lattices, spontaneous magnetization of spinor condensates12,13,14 has been observed. Magnetic coupling between atoms is obtained via artificial gauge fields15,16 or via dipole-dipole interaction of polar molecules17, highly magnetic atoms18, and Rydberg atoms5,9,10. Here we provide an alternative approach where spin-spin interactions in a cold atomic gas are mediated through light over length scales determined by diffractive dephasing. We demonstrate antiferromagnetic and ferrimagnetic phases. Transitions between magnetic phases are induced by external magnetic fields in close connection to the transverse (quantum) Ising model19,20,21,22,23,24,25,26.

## Results

### Diffractive coupling scheme and experimental setup

The matter density in ensembles of laser-cooled thermal atoms is typically too low to observe direct magnetic interactions between atoms in different Zeeman states. In our experiment, however, strong interactions and magnetic ordering are mediated through state-selective optical nonlinearities combined with diffractive light propagation in a feedback scheme as sketched in Fig. 1a27. A 100–200 μK cold atomic cloud is illuminated by a pump laser beam propagating along z and linearly-polarized along x. The transmitted beam is retro-reflected by a semi-transparent mirror (R > 95%) located at a distance d behind the cloud. The beam is detuned from the atomic transition by several linewidths such that single-pass absorption is moderate and the nonlinear effects described here are mainly dispersive in character. Under these conditions a spatial modulation of a state variable in the atomic system at a transverse wavenumber q will impart a phase modulation on the transmitted light. Diffractive dephasing in the feedback loop results in conversion of phase to amplitude modulations. This conversion is related to the Talbot effect28, a self-imaging effect for periodic light structures. The phase between the generated sidebands at q and the pump varies as exp(iq2z/(2k)) (k wavenumber of light) under the paraxial approximation. Hence for a wavenumber with exp(iq2(2d)/(2k)) = i the initial phase modulation is converted to an amplitude modulation for the reentrant field, which can provide positive feedback to the original fluctuation for a self-focusing situation in which the phase (or refractive index) increases with intensity. This provides the light-mediated coupling in our experiment. The spatial period for the ordered magnetization state is given by $${\mathrm{\Lambda }} = \sqrt {4\lambda d}$$27. The emerging spatial structure spontaneously breaks the translational and rotational symmetry in the x − y-plane transverse to the pump axis. This is similar to the symmetry breaking in multi-mode cavities29,30, but different from transversely pumped single-mode cavities31,32,33,34 in which the symmetry and orientation of the spatial structures is determined by the pump and cavity axis. For cold quasi-2-level atoms in non-cavity schemes, the spontaneous emergence of ordered structures was demonstrated previously in experiments using either opto-mechanical nonlinearities due to the dipole force35 or Sisyphus cooling-assisted bunching36,37 or inversion patterns due to the saturation of the atomic transition38.

In order to produce magnetic ordering, the magnetic substructure of the atomic states can be exploited. The experiment is performed on the F = 2 → F′ = 3 transition of the D2 line of 87Rb, but the simpler F = 1 → F′ = 2 transition (Fig. 1b) is sufficient to explain the observed features. Choosing the quantization axis along z, the pump beam axis, the interaction is described in terms of circularly polarized light components, σ+ and σ. When a low saturation parameter is employed, the optical properties of the gas are determined by the magnetization in the ground state. This contains magnetic dipole and quadrupole contributions which can be written in terms of the density matrix elements ρij. Here we focus on the magnetic spin w = ρ11 − ρ−1−1, which corresponds to a magnetic dipole in the z-direction. It is produced by pumping with an optical beam possessing a net spin, i.e. a non-zero circular polarization component. The σ+ (resp. σ) component induces population transfer toward the stretched state mF = +1 (resp. mF = −1) known as Zeeman pumping (Fig. 1b). In zero magnetic field, this process is described by

$$\dot w = - {\mathrm{\Gamma }}_ww + \frac{5}{{18}}D + {\mathrm{higher}}\,\, {\mathrm{order}}\,\, {\mathrm{multipoles}},$$
(1)

where D = P+ − P is the difference pump rate between the σ±-components (each being characterized by a pump rate $$P_ \pm \sim \left| {E_ \pm } \right|^2$$) and Γw = r + S/6 describes relaxation, r being an effective relaxation rate due to atomic motion and S = P+ + P the sum of the pump rates. The latter saturation term ensures that the spin orientation remains bounded.

As indicated, we concentrate on the interaction of the optical spin with the atomic dipole moment (tensor rank 1 of irreducible tensor components39). The F = 2 ground state used in the experiment and the F = 1 ground state used in the modeling allow for higher moments, in principle. A linearly polarized pump field will not induce an orientation but will induce alignment components (magnetic quadrupoles) X = ρ11 + ρ−1−1 − 2ρ00 and Δm = 2 coherences Φ = 2ρ1−1 = u + iv. Effects of these are under current investigation. For the full equations, we refer to Eqs. (8a)–(8l) of the Supplementary Note 1. It is important to stress that the magnetic fields induce coherent coupling within the dipole and quadrupole components but not between dipole and quadrupole components40, enabling us to treat the ordering of the dipole magnetization separately from the more complex atomic states to a large extent. The homogeneous quadrupole components are important on a quantitative level and responsible for symmetry breaking (see below) but they are not driving the magnetic ordering discussed here.

The action of the atomic orientation (spin) pattern on the optical spin structure, i.e. the circularly-polarized optical fields E±, is described by

$$\frac{\partial }{{\partial z}}E_ \pm = - i\frac{{k\chi _ \pm }}{2}\left[ {\left( {1 \pm \frac{3}{4}w} \right)E_ \pm + {\mathrm{higher}}\,\, {\mathrm{order}}\,\, {\mathrm{multipoles}}} \right].$$
(2)

Here, χ± = $$- b_0{\mathrm{/}}(kL) \times \left[ {2{\mathrm{\Gamma }}\left( {\delta \mp {\mathrm{\Omega }}_z} \right)} \right]$$/$$\left[ {{\mathrm{\Gamma }}^2 + 4\left( {\delta \mp {\mathrm{\Omega }}_z} \right)^2} \right]$$ is the linear susceptibility, where δ denotes the detuning, Γ the radiative decay rate of the transition, b0 the optical density at line center, L the medium length, and Ωz the Larmor frequency of the B-field in z-direction. In zero magnetic field, the linearly polarized pump field alone will not induce an orientation. As long as the spin system w(x, y) in the atomic cloud is disordered, the difference pump rate D of the fed back E±-fields is disordered, too. A localized (Fig. 2a) or periodic modulation (Fig. 2b, c lowest subpanel) will instead generate opposite modulations of phase shifts for the σ± components (see the ±w in Eq. (2), Fig. 2c central two subpanels) and, due to the diffractive phase-shifts in the feedback loop, a spatial separation of the σ± components arises (Fig. 2c uppermost subpanel). A self-organized optical spin pattern is then generated in the backward field reentering the cloud. The resulting non-zero difference pump rate, D, will drive the atomic spin ordering via Eq. (1). As the propagation time of the light field is much shorter than the time scales of the atomic dynamics, the ordering can be interpreted as to arise from an interaction of atoms in separate Zeeman states via light-mediated coupling. For F = 1/2 → F′ = 1/2 -transitions, a related instability is known in hot atomic vapors41,42,43,44,45,46, but these studies focused on the nonlinear optics aspects of the self-organization.

Polarization aspects were also investigated in the transverse instability occurring in counterpropagating beams in a nonlinear optical medium, although these studies focused on optical wave-mixing47,48,49,50,51,52,53. The counterpropagating beam instability is linked to the single-mirror feedback instability but requires a simultaneous treatment of diffraction and nonlinearity. If in a single-mirror setup the mirror position is in the medium, the threshold increases and the length scale does not decrease any more like $$\sqrt {\lambda d}$$ but is limited to slightly larger than $$\sqrt {\lambda L}$$, where L is the medium length, as a new significant diffractive length54. In our experiment, the mirror is typically placed just outside the cloud to reduce these effects, but close enough to the cloud to maximize the number of lattice sites within the pump beam by being close to the minimum lattice period. The resulting scale for the pattern period is about Λ ≈ 100 μm for the Nice experiment and Λ ≈ 50 μm for the Strathclyde experiment. The control of the position of the mirror is achieved via an imaging system (see ‘Experimental Details’ in the Methods Section) and hence can be both positive and negative.

### Ising-like interaction

Figure 2a illustrates that the spatial structure of the difference pump rate fed back supports an Ising-like interaction mechanism for the atomic spins on the lattice created by the instability which is much bigger than typical atomic separations. A perturbation of w ranging over half of the self-organized lattice period creates an optical spin structure that enhances the magnetization at the point of perturbation and at locations one period away (the nearest neighbors) but will also lead to negative, i.e. antiferromagnetic, coupling at half the period. This ferromagnetic, respectively antiferromagnetic, coupling increases with optical density and input intensity. For a harmonic perturbation w = δw cos(qx) a simple relation for the amplitude of the difference pump rate δD can be derived by a linear expansion of Eqs. (1) and (2),

$$\delta D = - 3P_0\phi _0\delta w,$$
(3)

where ϕ0 = b0δ/Γ/[1 + 4(δ/Γ)2] is the linear phase shift. For a localized perturbation like in Fig. 2a or general distributions there is no simple closed expression but the scalings hold, i.e. the interaction strength is given by the dispersive optical density (i.e. the linear phase shift) and the pump rate. The question whether it is possible to derive an effective Hamiltionian for the super-spins describing the dynamics of an open quantum system is open at this stage of investigations.

### Observation of antiferromagnetic ordering

For an observation of magnetic ordering, we typically set the laser detuning at a value of δ between −7Γ and −9Γ. The laser intensity used is as low as 1 mW/cm2, typically two orders of magnitude below non-magnetic instability thresholds35,38. Under these conditions, square lattices of large domains of spin-oriented atoms are observed in the near-field (NF) as can be seen in Figs. 1 and 3a,b. The two square lattices detected in the σ+ and σ channels are interlaced, as illustrated in Fig. 3c, where the NF intensity difference I(σ+) − I(σ) is plotted. This difference is an indicator of the orientation distribution, w(x, y). The plot in Fig. 3e corresponds to a cut of the experimental intensity difference image along the dashed line, and emphasizes the periodic, symmetrical arrangement in zones of opposite light helicity. This structure corresponds to an antiferromagnetic phase. Indeed, for Bz = 0, there is no Zeeman splitting and the system is fully symmetric for the σ+ and σ light components. Spontaneous symmetry breaking leads to an antiferromagnetic state with zero net magnetization and an arbitrary spatial phase. This analysis is confirmed by numerical simulations which solve the coupled equations for the light field and atomic variables in the presence of mirror feedback, Eqs. (8a)–(8d) of the Supplementary Note 1. The computed transverse spatial distribution of w is shown in Fig. 3d. The corresponding I(σ+) − I(σ) image (not shown) closely matches the experimental data.

The spin-oriented domains have similar size and contain a large number of atomic spins. A macroscopic orientation (super-spin) can be obtained by averaging over the atomic spins in each domain and by assigning positive or negative magnetization to the corresponding lattice points at the period of the w lattice. The macroscopic super-spins at each lattice point will then interact with their neighboring lattice spins via diffracted light (Fig. 2a), sustaining the antiferromagnetic ordering. Consequently this corresponds to a simulator of interacting semiclassical spins made of domains of spin-oriented cold atoms. We observe shot-to-shot fluctuations of both orientation and position of the structures between different realizations, as expected from the continuous symmetries that are spontaneously broken (translation and rotation in the transverse plane). Patterns can also include defects, deformations or irregularities of amplitudes.

### Observation of ferrimagnetic ordering

When a longitudinal magnetic field Bz is applied, a further symmetry breaking occurs, leading to a preferred sign of the orientation. However, for a small field Bz the antiferromagnetic square pattern survives, in accordance with the antiferromagnetic Ising model in a longitudinal field23. Increasing the Bz field to about 0.1 G we observe a transition from square (antiferromagnetic) to hexagonal (ferrimagnetic) lattices, as shown in Fig. 3f–h, j. The ferrimagnetic state exhibits striking differences between the two polarization channels, with hexagonally coordinated peaks in one component and honeycombs in the other (Fig. 3f, g). These are still interleaved with each other but the amplitudes of the components are very different as shown in Fig. 3h, j through the $$I_{\sigma ^ + } - I_{\sigma ^ - }$$ image. The 1D cut of this experimental image shows that the spins on the hexagonal lattice are mostly oriented along one direction, with only ≈10% of them in the opposite direction. This is a ferrimagnetic phase. The simulations show the corresponding spatial modulation of w in Fig. 3i. By flipping the sign of Bz, the roles of σ+ and σ are exchanged and the sign of the dominant magnetization inverted in response to the inversion symmetry breaking by the external field.

The transition from square to hexagonal structures can be understood using a symmetry argument55,56,57. In the absence of an inversion symmetry (e.g. when Bz ≠ 0) pattern selection is governed by quadratic interactions allowing three-wave mixing. This favors hexagonal patterns as the sum of two lattice wavevectors will yield a vector with the lattice wavenumber only for an angle of 120°. Other kinds of lattices (including squares) can emerge if inversion symmetry is respected (Bz = 0). Note also that an antiferromagnetic state cannot exist on a triangular (and thus hexagonal) lattice as this would lead to frustration2,58. Increasing the Bz-field it is observed that the clear hexagonal symmetry is destroyed and disordered spin patterns prevail. This transition is currently investigated in more detail. Symmetry breaking due to a longitudinal field occurs in standard antiferromagnetic Ising models but to an unstructured, paramagnetic state22. This paramagnetic phase is outside the capability of our current apparatus, but is expected for larger B-fields (several Gauss) such that Ωzδ. Transitions to a ferrimagnetic phase occur in models and materials consisting of two layers or two sublattices of magnetic sites59,60,61. This two layer structure resembles the situation here in which the hexagonally coordinated, dominant super-spins are surrounded by a weaker, opposite magnetization on the ridges of an interleaved honeycomb structure.

### Threshold behavior

Figure 4 shows an analysis of the diffracted power at the lattice wavevectors vs. pump. For zero magnetic field (red dots), it evidences a continuous, i.e. second order, transition at a threshold of 4.1 mW/cm2 to the antiferromagnetic state and a linear increase of diffracted power vs. pump intensity beyond threshold. As the diffracted power is proportional to the square of the modulation amplitude of w in linear expansion of Eq. (2), this indicates a square-root scaling of the magnetic order parameter vs. pump above threshold, as expected for the Ising model in mean field limit26 (staggered magnetization for the anti-ferromagnetic case). The transition to the ferrimagnetic hexagonal state at small longitudinal magnetic fields is abrupt, i.e. first order (Fig. 4, blue dots), which is related to the cooperative dynamics due to the breaking of the inversion symmetry as discussed before. For Bz = 0 and r = 2.29 × 10−4Γ2, R = 0.95 the theory, Eq. (9), indicates a threshold of 0.69 mW/cm2 neglecting contributions of all other multipoles (i.e. assuming a value of the homogeneous component of the alignment X0 = 0), lower than the experimentally one of 4.1 mW/cm2. For a more realistic component of X0 = 0.29 obtained from solving for the homogeneous solution of Eqs. (8a)–(8d) of the Supplementary Note 1 for these parameters, the threshold is 3.5 mW/cm2. The remaining differences can be easily explained by the inhomogeneous profile of the Gaussian input beam, transverse stray fields and corrections from the fact that the medium is not really diffractively thin as assumed in this treatment54.

### Dynamics in transverse fields and transverse Ising model

We now consider the antiferromagnetic ordered state when a purely transverse magnetic field is applied. The transverse field stimulates coherent coupling (tunneling) between spin states thus suppressing ordering as described by the transverse (also referred to as the quantum) Ising model24,25,26. This model shows a quantum phase transition between the ordered antiferromagnetic state and a paramagnetic state in which all spins are aligned to the transverse field, even at zero temperature. In our experiment the magnetic coupling induces a phenomenologically similar transition from the antiferromagnet to a state without structure. Figure 5a shows the diffracted power versus the transverse B-field for three different pump values. The diffracted power has an approximately parabolic dependence on the transverse B-field. The maximum diffracted power increases with pump power as does the critical transverse B-field, i.e. the B-field at which the ordered states vanishes, as expected (see Eqs. (11) and (12)). Normalizing the amplitude of the diffracted power to one and the magnetic field to the critical B-field, all the experimental curves fall on an universal parabola (Fig. 5b). As the diffracted power is proportional to the square of the orientation modulation amplitude, the parabola of Fig. 5b corresponds to the magnetization decaying as $$\sqrt {1 - \left( {B{\mathrm{/}}B_{{\mathrm{crit}}}} \right)^2}$$ in agreement with the transverse Ising model26.

Figure 5c compares the experimentally obtained threshold curve (Ith vs. By, black squares) with the prediction from Eq. (1) (blue line) incorporating the quantum coupling induced by the transverse field into the relaxation rate Γw (see Eq. (7)). The agreement is excellent, except at low magnetic fields. If one allows for a residual stray field of Bx = 35 mG (see Methods), the threshold at By ≈ 0 is also reproduced well.

It is important to note that the transverse magnetic field enters the equations as a coherent coupling as in the transverse Ising model24,25,26 as the operator describing the action of a transverse magnetic field does not commute with the z-component of the spin, the orientation w. This provides a coherent decay mechanism of the magnetization, often referred to as quantum tunneling24,25. As the incoherent decay rate is relatively small (r < 104 s−1), the decay dynamics can be easily dominated by coherent spin tunneling via stray and applied transverse fields (Fig. 5c) corresponding to the phase transition in the quantum critical regime at zero and low temperature in the transverse Ising model. Differently from the transverse Ising model, there is no magnetization without the instability (for Bz = 0), so if the spin-flips destroy the z-magnetization, the remaining state is magnetically disordered and not a quantum paramagnet aligned to the transverse field.

## Discussion

In conclusion, we demonstrated spontaneous magnetic ordering in a cloud of cold atoms due to light-mediated interactions. The behavior of the system in a longitudinal field is more complex than in the antiferromagnetic Ising model on a single layer as a ferrimagnetic state is obtained after a symmetry breaking transition. The decay of antiferromagnetic order due to quantum tunneling in a transverse magnetic field is also observed in agreement with the transverse Ising model. The role of internal spin temperature is played by the atomic kinetic temperature, but the threshold can be dominated by the quantum tunneling. Hence, this system constitutes a novel, unconventional open system for the study of magnetic ordering on large spatial scales and in a relatively simple cold atom setup. Future investigations consider fluctuations and correlations beyond the semiclassical treatment presented here, e.g. quenching dynamics4,62 and fluctuations at the classical-quantum boundary63,64. Of further interest are the influence of frustration in the transition between ferri- and antiferromagnetic order, the existence of magnons and the possibility of optically controllable localized magnetic nucleation domains65,66,67.

## Methods

### Experimental details

The results presented in this paper were obtained using two different experimental setups, one in Nice and the other in Strathclyde. Both experiments used a magneto-optical trap (MOT) to prepare a cold cloud (at temperature T ≈ 200 μK at Nice, T ≈ 120 μK at Strathclyde) of 87Rb atoms. The main difference between the two setups is the number of trapped atoms (1011 at Nice and 109 at Strathclyde). As a result, both the size L and the optical density (OD) b0 of the cloud are larger at Nice (L ≈ 14 mm, b0 ≈ 80, peak density about 5 × 1010 cm−3) than at Strathclyde (L ≈ 2.6 mm, b0 ≈ 27, peak density about 7 × 1010 cm−3).

The threshold for ordering is not given by the spin temperature that is ill-defined since the spins are well isolated from the environment in the vacuum chamber at the low densities employed. The main source of disorder to be overcome is the residual thermal motion of the atoms leading to a wash-out of the spin pattern and thus serving as a kind of effective spin temperature counteracting the magnetic ordering. Typical lattice periods are Λ  100 μm (Nice) and Λ  50 μm (Strathclyde) giving an estimation of the relaxation rate r of the spin structures due to atomic motion of r ≈ 2.8 × 103 s−1 and r ≈ 4.4 × 103 s−1, respectively (see Eqs. (19) and (20) of the Supplementary Note 1).

After the atomic sample is prepared in the MOT, the MOT is shut down by switching off both trapping lasers and magnetic field gradient. After a delay, the cloud is submitted to a pulsed retro-reflected laser beam using the configuration depicted in Fig. 1a. The duration of the pulse is typically 400 μs. The input laser beam, with a linear polarization, is detuned below the F = 2 → F′ = 3 transition of the D2 line of 87Rb, chosen as the transition is closed. The typical detuning is δ = −(7…9)Γ, where Γ = 2π × 6.06 MHz is the atomic linewidth. Since the optical pumping process tends to increase the population in the stretched states with the strongest dipole matrix element, the optical density increases due to the pumping process and the optical nonlinearity is self-focusing in spite of negative detuning. The beam diameters (FWHM) of the pump beams are 2.2 mm (Nice) and 0.8 mm (Strathclyde). Depending on the laser power, a superimposed weak repumping beam tuned to the F = 1 → F′ = 2 transition is added to prevent hyperfine pumping. Because the spin instability is highly magnetic field-dependent, the residual B has to be carefully controlled. This is achieved by a sufficiently long delay of 3 to 10 ms between MOT shutdown and the pattern-forming laser pulse and a set of three pairs of Helmholtz coils to control the bias fields. The residual magnetic field is compensated to best effort in all three dimensions. We cannot exclude uncontrolled magnetic fields on the order of some tens of mG.

The feedback mirror is located outside the vaccum chamber and imaged with a telescope closer to the atomic sample. Hence one can achieve also negative feedback distances54,68.

In Fig. 5b, the critical magnetic field used for scaling is expected to be proportional to the input intensity, Bcrit = ζIin, see Eq. (12). The scaling factor is obtained by first dividing the critical magnetic field values by the pump intensity and averaging the obtained numbers giving ζ = (8.7 ± 0.1) × 10−3 for the Bx-fields and ζ = (1.1 ± 0.1) × 10−2 for the By-fields.

### Pattern detection and analysis

The pattern detection relies on the imaging of the transverse intensity distribution of the weak beam transmitted by the semi-transparent feedback mirror. The small amount of light transmitted by the mirror is used to image the transverse intensity distribution of the beam either in near-field (NF, at a plane located at a distance 2d behind the cloud corresponding to the intensity distribution fed back to the atoms) or in far-field (FF, in the focal plane of a suitably positioned lens). This imaging can be performed simultaneously in two orthogonal polarization channels (either circular (σ+/σ) or linear). The NF images are employed to identify the nature of the magnetic phase. From Eq. (2) and the propagation phase shift of π/2, the difference between the transmitted intensities of the two σ−components is proportional to the magnetization w in linear order, justifying the visualization used in Fig. 3c, e, h, j. In the far field, the Fourier spectrum of the magnetic structure is available. The diffracted power used in Fig. 5a is obtained by integrating the intensity in FF images within the transverse wave number range of the structure. From Eq. (2), in linear order the field is proportional to the modulation amplitude of w at that wavenumber, whereas the recorded intensity is proportional to the square of the modulation amplitude of the magnetization. As the latter is proportional to the staggered order parameter of the antiferromagnetic state, the diffracted power is expected to indicate the square of the order parameter of the Ising model. The normalization constants for the diffracted power in Fig. 5b are obtained by fitting parabolas to the data in Fig. 5a.

### Theoretical model

Equations of motion for the atomic magnetization states in the electronic ground state are derived from the Liouville equation for the density matrix. The procedure is outlined in the Supplementary Note 1. The numerical simulations are performed in the representation outlined in the Supplementary Note 1, but it is useful to note that these equations can be rewritten in terms of irreducible tensor components of rank 1 describing orientation or magnetic dipoles and components of rank 2 describing alignment components or magnetic quadrupoles, respectively, and that the magnetic field provides only a coupling within the dipole, respectively quadrupole, components, but not between tensors of different rank. The coherent dynamics of the magnetic dipole vector m in the external magnetic field Ω is described by

$${\dot{\mathrm m}} = {\bf{\Omega }} \times {\bf{m}},$$
(4)

where the z-component of the magnetization represents the orientation introduced before, mz = w. As we concentrate on the dipole dynamics in this article and the mz component is the only one directly coupling to the optical field, we proceed by neglecting the contributions from the rank 2 components and solving for the mx, my components in steady state. This results in an effective equation for the orientation given by

$$\begin{array}{*{20}{l}} {\dot w} \hfill & = \hfill & { - \left( {r + \frac{1}{6}S} \right)\left[ {1 + \frac{{{\mathrm{\Omega }}_x^2 + {\mathrm{\Omega }}_y^2}}{{\left( {r + \frac{{17}}{{24}}S} \right)^2 + {\mathrm{\Omega }}_z^{\prime 2}}}\frac{{r + \frac{{17}}{{24}}S}}{{r + \frac{1}{6}S}}} \right] \times w} \hfill \\ {} \hfill & {} \hfill & { + \frac{5}{{18}}D,} \hfill \end{array}$$
(5)
$$= - {\mathrm{\Gamma }}_w\left[ {1 + \frac{{{\mathrm{\Omega }}_x^2 + {\mathrm{\Omega }}_y^2}}{{\left( {r + \frac{{17}}{{24}}S} \right)^2 + {\mathrm{\Omega }}_z^{\prime 2}}}\frac{{r + \frac{{17}}{{24}}S}}{{r + \frac{1}{6}S}}} \right]w + \frac{5}{{18}}D$$
(6)

which illustrates nicely the destruction of longitudinal magnetization by the transverse B-fields in addition to the decay rate Γw evaluated at Ωx = Ωy = 0. The dash in $${\mathrm{\Omega }}_z^\prime$$ illustrates that one might need to take into account light-shift contributions for a fully quantitative description, but the spatial average of light-shift is zero as long as Ωz = 0. For $${\mathrm{\Omega }}_z^\prime$$ = 0, Eq. (6) reduces then to

$$\dot w = - {\mathrm{\Gamma }}_w\left[ {1 + \frac{{{\mathrm{\Omega }}_x^2 + {\mathrm{\Omega }}_y^2}}{{\left( {r + \frac{{17}}{{24}}S} \right)\left( {r + \frac{1}{6}S} \right)}}} \right]w + \frac{5}{{18}}D,$$
(7)

which is used to analyze the destruction of magnetic order by the transverse field in Fig. 5.

### Symmetry breaking in a longitudinal magnetic field

For zero difference pump rate D, the homogeneous state of the z-magnetization is w = 0 from Eq. (6). A non-zero longitudinal field provides a symmetry breaking. However, the main effect does not arise directly via the dipole energy but via the chiral optical properties of the atom as light-mediated coupling is responsible for the z-magnetization. First, the difference pump rate D is different from zero even with linearly polarized input light, if the longitudinal magnetic field is nonzero, as the pump rates depend on detuning and hence on the Larmor frequency Ωz (nonlinear Faraday effect, see Eq. (3) of Supplementary Note 1). However, the effect is small as it is linear in Ωz/δ, D ≈ 4P0Ωz/δ (P0 input pump rate for one circular polarization at Ωz = 0), and the experiments operates at Bz ≈ 0.1 G and |δ| ≈ 7.5Γ giving |4Ωz/δ| ≈ 0.006. For the same reason, the symmetry breaking provided by the linear Faraday effect, the difference in linear optical phase shift between the two σ-components (Eq. (24) of Supplementary Note 1), is small. However, for sufficiently large Bz fields one can obtain a sizable homogeneous magnetization, which corresponds to the anticipated paramagnetic state mentioned in the article. The dominant symmetry breaking for small longitudinal magnetic fields is provided by the precession term between the coherences u and v in Eqs. (8a) and (8b) of the Supplementary Note 1. From this, a homogeneous component u0 created directly by the linearly polarized pump beam induces a homogeneous component v0, which in turn couples to the z-magnetization w via a term proportional v (see Eq. (8c) of the Supplementary Note 1). As the sign of v0 generated depends on the sign of Bz, this provides a symmetry breaking for w destabilizing the antiferromagnetic state. However, if the Bz-field is increased further, it leads to a destruction of the coherences by the precession around the Bz-field and thus to a reduction of symmetry breaking. This transition seems to be related to the transition between nicely ordered ferrimagnetic states and the disordered states mentioned. As at higher pump rate the coherence survives to larger Bz-fields, the range with nice, ordered hexagons changes being the main reason for a range of Bz mentioned in the caption of Fig. 3.

### Linear stability analysis

For the linear stability analysis of the homogeneous state we confine to perturbations in the orientation w as the numerical simulations suggest that the orientation is the driver for the magnetic ordering under consideration. Hence we solve Eq. (8) of Supplementary Note 1 for the homogeneous stationary state u0, v0, X0, w0 without a transverse magnetic field and set w = w0 + δw cos(qx) with a small perturbation δw at a wavenumber q. The resulting equation for the growth rate of δw in linear approximation for the wavenumber with the lowest threshold (i.e. the one of the emerging ordered state, $$q_c = \sqrt {\pi k{\mathrm{/}}(2d)}$$) is for Ωz = 0

$$\eta = - r - \frac{1}{6}S_0(1 + R) + R\left( {\frac{{X_0}}{6} - \frac{5}{{12}}} \right)S_0\phi _0,$$
(8)

where P0 is the input pump rate for one circular component of the linearly polarized light and S0 = 2P0 and ϕ0 = ϕ+z = 0) = ϕ+z = 0). The first term describes the destruction of ordering due to the residual atomic motion while the second term accounts for the fact that the orientation is bounded. The third term is the driving term. It is positive as ϕ0 < 0 for δ < 0. It is reduced by the development of a homogeneous alignment X0. Except for this X0 term, Eq. (8) has the same structure as the expression derived for a J = 1/2-system44,45 but with coefficients and signs adapted to the different term scheme. The threshold pump rate is obtained as

$$S_0 = \frac{r}{{ - \frac{1}{6}(1 + R) + \left( {\frac{{X_0}}{6} - \frac{5}{{12}}} \right)R\phi _0}},$$
(9)

giving a minimal linear phase shift to obtain ordering of

$$\left| {\phi _0} \right| = \frac{{1 + R}}{{6R\left( {\frac{5}{{12}} - \frac{{X_0}}{6}} \right)}}.$$
(10)

For r = 0 (zero effective temperature, i.e. no classical fluctuations are counteracting the ordering) the pump threshold is zero (without a transverse field), but a finite optical density (phase shift) is still needed to obtain magnetic ordering. Hence, ϕ0 is the cooperativity parameter characterizing the interaction strength. For X0 = 0, i.e. considering only the dipole components, Eq. (10) indicates the requirement of a minimum phase shift of 0.82.

Turning our attention to the consequences of the presence of the higher order magnetic multipoles, numerical simulations show that X0 varies roughly between 0.25 and 0.5. Hence, the corrections to the minimum required phase are quite small (0.91–1.03) and not very important except when the optical density is so low that the linear phase shift is close to the minimally required one. This is the case for the Strathclyde experiment which operates at $$\left| {\phi _0} \right| \approx 0.96$$. The consequences for the threshold are discussed in the final paragraph of Section ‘Threshold behavior’. For the higher density involved in the Nice experiment, $$\left| {\phi _0} \right| \approx 2.32$$ (b0 = 80, δ = −8.6Γ), the correction by X0 is not very important. For r = 4.4 × 103 s−1 and R = 0.95 one obtains Ith = 0.062 mW/cm2 for X0 = 0 and Ith = 0.076 mW/cm2 for X0 = 0.29.

As these differences are small, we have neglected higher order magnetic multipoles in the discussion of Fig. 5c and calculated the critical transverse field just from the tensor 1 components from Eq. (7),

$$\begin{array}{*{20}{l}} {{\mathrm{\Omega }}_{{\mathrm{c,trans}}}^{\mathrm{2}}} \hfill & = \hfill & { - \frac{5}{{12}}\left( {r + \frac{{17}}{{24}}S_0(1 + R)} \right)RS_0\phi _0} \hfill \\ {} \hfill & {} \hfill & { - \left( {r + \frac{{17}}{{24}}S_0(1 + R)} \right)\left( {r + \frac{1}{6}S_0(1 + R)} \right),} \hfill \end{array}$$
(11)

indicating a linear relationship between critical field and pump power for $$S_0 \gg r$$

$${\mathrm{\Omega }}_{{\mathrm{c,trans}}} = S_0\sqrt {\frac{{17}}{{24}}(1 + R)\left[ { - \frac{5}{{12}}R\phi _0 - \frac{1}{6}(1 + R)} \right]} .$$
(12)

The critical magnetic field also increases with increasing dispersive optical density ϕ0, i.e. increasing interaction strength.

### Data availability

Data used to plot the figures are available under https://doi.org/10.15129/386278bf-dd5f-4dd6-8808-2a2728beed93. Further relevant data are available on reasonable request from the authors.