Magneto-optical study of metamagnetic transitions in the antiferromagnetic phase of α-RuCl3

α-RuCl3 is a promising candidate material to realize the so far elusive quantum spin liquid ground state. However, at low temperatures, the coexistence of different exchange interactions couple the effective pseudospins into an antiferromagnetically zigzag (ZZ) ordered state. The low-field evolution of spin structure is still a matter of debate and the magnetic anisotropy within the honeycomb planes is an open and challenging question. Here, we investigate the evolution of the ZZ order parameter by second-order magneto-optical effects, the magnetic linear dichroism and magnetic linear birefringence. Our results clarify the presence and nature of metamagnetic transitions in the ZZ phase of α-RuCl3. The experimental observations show the presence of initial magnetic domain repopulation followed by a spin-flop transition for small in-plane applied magnetic fields (≈1.6 T) along specific crystallographic directions. In addition, using a magneto-optical approach, we detected the recently reported emergence of a field-induced intermediate phase before suppressing the ZZ order. The results disclose the details of various angle-dependent in-plane metamagnetic transitions quantifying the bond-anisotropic interactions present in α-RuCl3.

To date, several experimental techniques have been applied to map out the equilibrium phase diagram of α-RuCl 3 in the temperature and magnetic field plane 33,[36][37][38][39][40][41][42][43][44][45] . Signatures of fractionalized excitations have been detected by various spectroscopy techniques 11,15,16,46,47 , hinting towards a proximate spin-liquid behavior. Initially this has lead to a wide spreading in reported values for the possible interaction strengths and a controversial discussion on the effective spin Hamiltonian capturing the experimental observations 48 . Nowadays, the parameter space of the effective spin Hamiltonian and the size and sign of the present exchange interactions converge towards a unifying description of α-RuCl 3 physical properties. Especially the role of the symmetric off-diagonal exchange interaction Γ has been studied intensively and Sears et al. reported recently that its size is comparable to the anisotropic ferromagnetic Kitaev K exchange interaction 22 , indicating its key role in understanding α-RuCl 3 large anisotropic susceptibilities for magnetic fields applied within χ ∥ and perpendicular χ ⊥ to the honeycomb planes 32,39,49 . Despite this, also the orientation of a magnetic field applied within the honeycomb planes has been found to be crucial to resolve strongly angle-dependent low-energy excitations revealing fingerprints of a potential QSL state in α-RuCl 3 19,50,51 . Different experimental studies reported that α-RuCl 3 undergoes several phase transitions at low temperatures and small finite applied fields before entering the quantum-disordered phase 44,[52][53][54][55][56] , and the appearance of metamagnetic transitions has been predicted theoretically 31,57 . The emergence of an intermediate fieldinduced transition at around 6.2 T, depending on the orientation of the external in-plane magnetic field either along or perpendicular to the Ru-Ru bonds 44,[54][55][56] has been reported and points towards the necessity to include anisotropic inter-layer exchange interactions in the model Hamiltonian. In contrast, the low-field response for in-plane fields up to ≈2 T is still only partially understood, since the precise knowledge about the orientation of the order parameter is difficult to attain with thermodynamic probes. Therefore, its nature has been interpreted differently 15,33,39,52,58 , while there are first experimental signatures revealing the necessity to consider the present bond-anisotropy stemming from a small inequivalence in the Ru-Ru bond length in α-RuCl 3 59 . Nevertheless, the detailed magnetic-temperature (B, T) phase diagram of α-RuCl 3 is still under intense debate and an experimental probe that couples sensitively to the ZZ order parameter needed to be utilized. Magneto-optical (MO) spectroscopy is a well-established and powerful contactless technique to explore magnetic ordering phenomena and the emergence of topological magnetic structures on small scales with remarkable sensitivity [60][61][62][63][64][65][66][67][68][69][70][71] . Especially the quadratic MO effects are perfectly suited to study the antiferromagnetic ordering vector (L = M ↑ − M ↓ ≠ 0) and recently, direct optical probing of ZZ antiferromagnetic order via optical spectroscopy has been reported 72 . The main quadratic MO effects (even in M) in reflection are named magnetic linear dichroism (MLD) and magnetic linear birefringence (MLB) 62,73,74 , which are defined for the reflection of linearly polarized light under normal incidence and depend on the difference in the diagonal components of the dielectric tensor 67 . In this context, the origin of MLD and MLB can be understood in terms of different absorption (reflection) coefficients parallel and perpendicular to the magnetization M or Néel vector L. These effects manifest themselves in the polarization rotation θ (MLD) of a linearly polarized light upon reflection from the sample or give rise to an elliptical polarization η (MLB). They stem from the spin-orbit and anisotropic exchange interactions and can be related to spin-spin correlation functions 62,[75][76][77] . For symmetry reasons, it follows that the considered second-order MO effects are to lowest order quadratic in the antiferromagnetic order parameter such that the scaling (θ, η) ∝ L 2 holds 66 (Supplementary Information Notes 8,9 and 10).
Motivated by these intriguing questions, we performed a systematic MO spectroscopy study to track the evolution of the ZZ order parameter in α-RuCl 3 in thermodynamic equilibrium. The orientation-dependent MO response reveals the effect of the present magnetic anisotropy for magnetic fields applied perpendicular and parallel to the Ru-Ru bonds within the honeycomb layers. We show that the remarkable sensitivity of MO spectroscopy helps to clarify the emergence of two different intermediate field-induced and orientation-dependent metamagnetic transitions. Our results provide a detailed picture of the low-field behavior clarifying the influence of unidirectional bond-anisotropy within the honeycomb planes and we derive a value for the anisotropy field strength. (c) Fig. 1 Magneto-optical experiment. a Experimental setup (HW half-wave plate, P polarizer, C chopper, M beam-splitting mirror, L lens, S sample, PEM photo-elastic modulator, A analyzer, PD photo-diode). E i and E r correspond to the incident electric field polarization and the one upon reflection from the sample, respectively. The inset displays the effect of MO rotation Θ = θ + iη and the definition of the angles θ 0 , ϕ, θ. b Polarization dependence of the MO response as a function of the polarization orientation θ 0 of the incident light. Gray squares are data, solid line is a fit according to Equation (2). c Universal temperature dependence of the MO response for three different incident polarization orientations of θ 0 = 0 ∘ , 45 ∘ , 90 ∘ .

Magnetic linear dichroism
First, we explore the nature of the detected MO signal and its relation to the magnetic order. Here, we apply the Voigt geometry 67 , which is typically used to study antiferromagnets with spin alignments that are perpendicular to the light wave vector. Below, we derive the relation of the experimentally observed rotation of the polarization plane of the reflected linearly polarized light, but the same consideration applies to the change in the ellipticity. The rotation θ is related to both the amplitude of the order parameter and the relative orientation of the Néel vector to the electric field E of the incident linearly polarized light. In α-RuCl 3 the in-plane component of the Néel vector is oriented parallel to the ZZ chain direction. Figure 1a depicts the relative angles of the polarization of incident light and the Néel vector to the vertical polarization by θ 0 and ϕ, respectively. The MLD response is then given by 63,65 tanðθ MLD Þ ¼ More general and taking the presence of both linear and second-order MO effects into account, a total rotation of the polarization θ, in the limit of small rotations, can be expressed as The coefficients A Lin and A MLD are the amplitude of linear and quadratic MO effects, respectively. Since in the MLD geometry, the incident light is normal to the surface, the A Lin is mainly determined by the polar MO Kerr effect (PMOKE).
, where r ∥ and r ⊥ are the amplitude reflection coefficients of the light polarized parallel and perpendicular to the Néel vector, depends quadratically on the in-plane component of the antiferromagnetic order parameter L, i.e. A MLD $ L 266 (Supplementary Information Note 10). It is worth noting that the above relation indicates that the MLD signal, in contrast with the linear MO response, is a harmonic function of the incident polarization θ 0 , which becomes maximal for ϕ − θ 0 = 45 ∘ and it indicates an extreme sensitivity of the second-order MO response to the orientation of the incident linearly polarized light with respect to the spin-pointing direction. Recently it has been reported that the presence of antiferromagnetic ZZ chains can give rise to a polarization-dependent MO response, which is independent of the spin-pointing direction 72 . In α-RuCl 3 both, the ZZ chain direction and the spin-pointing direction are collinear in small magnetic fields, such that the spin-pointing and ZZ chain direction cause the MLD. Clearly, the MO response shown in Fig. 1b is polarizationdependent, which is manifested in a sinusoidal modulation as expected for MLD. At the same time, the MO response scales with A MLD / L 2 ðB; TÞ. Figure 1c shows the temperature dependence of the ZZ order parameter studied for three different polarization orientations of the incident light, θ 0 . Clearly, the magnitude of θ depends on θ 0 and for θ 0 = 45 ∘ becomes zero, while the qualitative temperature-dependent behavior remains, as expected, similar. In subsequent measurements, we chose the spolarized probe to obtain the maximum signal. Figure 2a shows the variations of θ MLD as a function of temperature. According to ref. 11 , the temperature dependence of the ZZ order parameter in zero magnetic field L(T, B = 0T) follows a power-law θ MLD / L 2 / ð1 À T=T N Þ 2β below the Néel temperature T N . The deduced Néel temperature T N = (7.19 ± 0.14) K indicates ABC-stacking in the samples under study as opposed to ABAB-stacking accompanied by stacking faults, which causes further transitions above T N 56 . Within the experimental uncertainty, the data can be fitted using a critical exponent β = 0.19 ± 0.07, which is close to the 2D Ising universality class 78 . Figure 2(b) displays the phase diagram in the (B, T) plane extracted from the MLD induced rotation at different fixed magnetic fields (isomagnetic). We find the evolution of the order parameter L(T, B) to scale proportional to ðH À H c Þ γ , where H c corresponds to the critical magnetic field above which the antiferromagnetic ZZ order is suppressed and the system enters the magnetically disordered state. The power-law fit exhibits a critical field value of H c = (7.48 ± 0.3) T and γ = 0.31 ± 0.07, which is again close to the theoretical value of 0.32 for the 2D Ising symmetry class 33 . These findings support that the MO response clearly displays the evolution of the order parameter. More details on the fitting of the critical behavior can be found in the SI.
Magnetic field orientation-dependent measurements Figure 2c displays the honeycomb structure, bond-anisotropy, and spin orientation in α-RuCl 3 . We point out the breaking of C 3 symmetry in α-RuCl 3 crystals originating from inequivalent Ru-Ru bond lengths, which leads to the existing monoclinic C2/m space group 34,35 . Phenomenologically, the inequivalence in the Ru-Ru bond lengths causes a change in the present interactions along the stretched bond indicated by J 0 ; K 0 and Γ 0 (cf. Fig. 2c). This causes the pseudospins, which are tilted by an angle of ≈32 ∘ out of the honeycomb plane 22 to have their in-plane projection being preferentially oriented perpendicular to the stretched bonds. This is a key point that needs to be taken into account to understand the anisotropic MO response in the following results. Nevertheless, the local C2/m symmetry has been found to be broken in multidomain samples 52 , due to a randomness in the monoclinic distortion of one Ru-Ru bond. Consequently, there can be three possible and symmetry-allowed ZZ domains in zero field cooled samples below T N , which are related by 120 ∘ rotations within the plane. However, applying a finite magnetic field along specific crystallographic orientations can change the spin-pointing direction within the honeycomb planes as will be discussed in the following and is schematically depicted in Fig. 2c via a spin-flop process.
Having established the iso-magnetic response of the order parameter L encoded in the MLD response θ MLD we turn now to the iso-thermal MLD response of α-RuCl 3 at a temperature of 3 K and magnetic field strengths up to ±7 T applied within the honeycomb planes along two different crystallographic directions to investigate the magnetic in-plane anisotropy. We studied two samples from the same batch which have been oriented along different crystallographic directions. Once, for sample (a) the inplane magnetic B ab field is applied perpendicular to a Ru-Ru bond, i.e., along one of the symmetry-equivalent {1, 1, 0} directions, while the field was directed parallel to the Ru-Ru bonds for sample (b), i.e., along one of the symmetry-equivalent {1, 0, 0} directions (Fig. 2c). The MO measurements were conducted with the magnetic field swept continuously from 0 to ±7 T to systematically track the dynamical MO response (Supplementary Information Notes 2 and 3). Figure 3a shows that the MO response θ MLD differs significantly for two distinct orientations of an in-plane magnetic field B ab , a first experimental evidence for the present in-plane magnetic anisotropy. The purely magnetic origin for this anisotropy is verified by the fact that the temperature-dependent evolution of θ MLD and η in zero magnetic field is similar for both samples ruling out possible temperature-related effects like strain or thermoelastic changes (Supplementary Information Note 2). The absolute field-induced change in the rotation defined as Δθ(B) = θ MLD (0T) − θ MLD (±7 T) for both field configurations at 3 K is in the order of 100 mdeg, which is large and underlines the microscopic impact of strong spin-orbit interactions (~100 meV in α-RuCl 3 79 ) on the second-order MO responses. The value at 7 T was set to zero to extract the field-induced changes in the rotation between 0 and 7 T at constant temperature (Supplementary Information Note 3). Despite the difference in the full hysteresis loops for the different field orientations, we divide the MO response shown in Fig. 3a into three field regimes to allow a simple description and comparison. Magnetic linear dichroism of α-RuCl 3 for different sample orientations. a Full magnetic field sweep scans of the magneto-optical response for B ab ∥{1, 1, 0} and B ab ∥{1, 0, 0} at 3 K. The hysteresis loops can be divided into three field regimes I, II, and III, which are discussed in the main text. b Temperature dependence of the MLD response as the magnetic field is aligned along the {1, 1, 0} direction. Labels are discussed in the main text. c Linear MO response extracted from the MLD response for the two different field orientations. The light gray area indicates field-regime I. d Zoom-in into field regime I and II for two subsequent field sweeps, which are divided into four steps as discussed in the main text. Inset shows the zero field time-dependent recovery of MO signal after the field sweep. e High-field regime III. The kink in the rotation for B ab ∥{1, 1, 0} highlighted by the dashed line is connected to the transition towards the new AFM phase. The shaded area indicates the emergence of the intermediate magnetic transition.
In each regime, the MO signal is comparable, while at the crossovers a vivid anisotropic response is present. The first transition at ±1.6 T is a large and steep change in the rotation θ MLD (absolute reduction of ≈50 mdeg) for the magnetic field applied perpendicular to the Ru-Ru bonds, whereas the rotation changes only slightly for the in-plane magnetic field applied parallel to the bonds. The marked difference for the two differently oriented samples in regimes I and II are displayed in Fig. 3d in more detail. This clear difference indicates the presence of a metamagnetic transition as will be discussed later. The second transition in the response occurs at around 6.2 T for the field applied perpendicular to the Ru-Ru bonds where an anomaly in θ MLD is observed by a kink. Reducing the magnetic field strength continuously from ±7 T back towards 0 T causes the emergence of significant hysteresis in the MLD response opening at a field strength of ≈6 T. The hysteresis loops close at a field strength of ≈0.7 T for both field orientations, but open again for small fields close to 0 T. Here, the integrated hysteresis weight is a factor of ≈1.5 larger for the field applied perpendicular to the bonds than along the bonds pointing again towards a difference in the inplane anisotropy energy. Figure 3b reports the iso-thermal MO responses as a function of the applied magnetic field. We found a similar behavior showing the three regimes I-III for three bath temperatures of 3, 4.5, and 6 K, corresponding to locations in the phase diagram deep insight the ZZ phase, in an intermediate range and close to the critical transition temperature towards the quantum paramagnetic phase. This finding indicates the purely magnetic nature of the MO field response.
In contrast, the odd MO response scales linearly in the applied magnetic field for both samples and shows no pronounced hysteresis for a whole field sweep (Fig. 3c). However, in regime I small spikes can be observed for both samples.
In order to understand the nature of these spikes, MOKE measurements with the sample rotated by 45 ∘ w.r.t. the incident light wave vector have been performed. We find a clear featureless linear scaling of the obtained rotation with the applied magnetic field ( Supplementary Information Note 6). Therefore, we assume that the small spikes are related to second-order MO effects. Since under the real experimental condition, the perfect symmetry cannot be achieved, minor differences in MLD obtained from negative and positive fields are unavoidable.
Low-field magneto-optical response Figure 3d compares the initial part of the hysteresis in regime I for both crystallographic orientations for subsequent field sweeps from 0 to 7 T. To emphasize similarities, we consider the normalized MO response. Here, we can divide the fielddependent alternation of MLD into the following four distinct steps.
(i) The first step shows the following characteristics. First, it displays an initial big, but a gradual and monotonic change in θ MLD , which starts immediately when applying a small inplane magnetic field. Second, it is seemingly independent on the field orientation in the honeycomb plane and terminates at a field strength of~0.7 T. Third, the absolute amplitude of this first step decreases for subsequent field sweeps, but displays a similar field dependence for both field orientations. Fourth, the MO response does not recover immediately when sweeping the field back to 0 T. However, there is a slow recovery of the MO response within tens of seconds at zero field to a saturating value (see inset 3d). The above mentioned distinct features of step 1, as we explain later, can reasonably be assigned to an initial field-driven gradual domain repopulation or ZZ domain switching. (ii) In step 2, no further change in the magneto-optical response can be observed for any of the subsequent field scans between 0.7 and 1.6 T with distinct in-plane field orientations. This indicates that in step 2 neither the ZZ chain orientation nor the spin-pointing direction and the amplitude of L are changed by the external magnetic field. This observation shows that at around 0.7-1.6 T the magnetic field changes between the three degenerate domains are terminated. (iii) A further increase in the applied field strength leads then to a sharp change (step 3) at ≈1.6 T for the in-plane field aligned perpendicular to the bond. In contrast, this abrupt change is not present for the field directed along the bond. This is a clear indication for a field-induced metamagnetic transition, which we will discuss later in terms of a spin-flop transition originating from the intra-layer bond-anisotropy of the frustrated honeycomb magnet. This anisotropic change in rotation at a field of ≈1.6 T is, in contrast to step 1, completely reproducible in subsequent field sweeps. Further, step 3 is clearly observed and remains sharp at all temperatures demonstrating the field evolution of the MO response is reproducible in the entire ZZ phase (cf. Fig. 3b). (iv) Further strengthen the applied field leads to the emergence of another steady state of the obtained signal from 2 to 3 T (step 4) pointing towards a homogeneously field-aligned ZZ ordering. Here, similar to step 2, neither the ZZ chain nor the spin-pointing direction change. Moreover, the different steps can be clearly observed in subsequent measurements at different temperatures as shown in Fig. 3b.
These different observations confirm the magnetic origin of the field response and provide first important key information to distinguish both the effects of fieldand anisotropy-related domain selection and an accompanied spin-flopped phase for inplane applied magnetic fields. In the first step, the in-plane field leads to an immediate population of ZZ domains with the easy ZZ chain axis closer to the field direction 44,52,77,80 . This interpretation is further supported by a recent study of the thermal and magneto-elastic properties of α-RuCl 3 for field applied in the ab-plane. It was found, that under an in-plane magnetic field α-RuCl 3 shows lattice contraction along the {1, 1, 0} direction 41 . Especially in low fields up to ≈0.5 T the magnetostriction coefficient changes continuously, such that the initial ZZ domain structure and distribution will be gradually changed by the external magnetic field. The fact that the MO hysteresis does not close while reducing the in-plane field back to 0 T points towards irreversible processes. This fits to the initial domain repopulation picture and hysteric magnetocaloric measurements below T N 41 .

High-field magneto-optical response
Turning to the high-field regime III, the kink which is observed only for one direction (Fig. 3e) indicates clear dependence on the orientation of B ab with respect to the pseudospin-bonds. This kink in the rotation is reproducible during different measurement cycles and likely related to the previously reported first-order transition into an intermediate differently ordered ZZ phase for inplane magnetic fields of ≈6−6.5 T aligned perpendicular to the Ru-Ru bond 44 . We observed this kink only at low temperatures since its intensity dramatically decreases by temperature (Fig. 6c) in agreement with the recent reports 44,81 . We point out that signatures of this metamagnetic transition seem much weaker compared to the spin-flop transition at a field of ≈1.6 T, which is related to the already suppressed amplitude of L close to the critical line and the fact that it might originate from a competition of anisotropic inter-layer exchange interactions as opposed to a change in the in-plane magnetic ordering 44 .

Low-field anomaly
In the following, we present the results of two individual experiments which further support the observed anisotropic MLD response at ≈1.6 T. Figure 4a shows the temperature dependence of the iso-magnetic MLB η. Since the birefringence originates from spin-spin correlations contributing to the magnetic energy, its derivative scales proportional to the magnetic part of the specific heat 62,82,83 . The obtained curves for [(dη/dT)/T] B are shown in the inset of Fig. 4a, which resolve changes in η more clearly and display a typical λ-shape as has been reported previously for the magnetic part of the specific heat at the transition for α-RuCl 3 20,84 . The red arrows indicate the peak values of these derivatives, which coincide with the vanishing MO rotation reported in Fig. 2b. As expected, the power-law fit ðH À H c Þ γ η gives similar values H c = (7.21 ± 0.09) T and γ η = 0.29 ± 0.04 as for the MLD response. Indications of the transition at 1.6 T are also visible in the temperature-dependence of η, where a small kink in the curves is indicated by a dashed line in Fig. 4b for all three temperatures.
Furthermore, we performed in-plane magnetometry measurements at 3 K (Fig. 5a) using a superconducting quantum interference device (SQUID, Quantum Design, MPMS). The obtained magnetization curves for both field orientations as in the MO experiments cross through the zero point, as expected for a compensated antiferromagnet, excluding any finite ferromagnetic contribution or background signal. Although at first glance, the magnetization curves M(H) do not exhibit any clear signs of a transition at 1.6 T, the standard deviation of the magnetization measurements illustrated in Fig. 5b give an indication (Supplementary Information Note 4). The spin-flop transition is of firstorder, i.e., close to the critical field value both the initial ZZoriented domains and some already spin-flopped domains coexist, such that the system is driven into a regime of large fluctuations stemming from the competition of anisotropy-stabilized ZZ and already field-driven spin-flopped domains. These fluctuations display the instability of the coexistence of energetically different ZZ phases near the critical spin-flop field strength, which leads to discontinuous jumps in the magnetization accompanied by irreversible behavior. The instability is clearly visible in the standard deviation of the magnetization measurements illustrated in Fig. 5b. This effect has been observed previously in susceptibility measurements at the phase boundary between the antiferromagnetic and spin-flopped phase in the hexagonal antiferromagnet NiO accompanied by an initial field-induced domain alignment 85 .

DISCUSSION
Based on these independent and consistent experimental observations, we elaborate on our main findings of the response to the magnetic field orientation along the two different crystallographic orientations and discuss the low-field response in detail. In zero field, the sample comprises three possible ZZ domains in which the spin direction differs by 120 ∘ with statistically distributed unequal but likely comparable populations. A small external applied field within the ab-plane will then favor ZZ domain(s) for which the Néel vector is most nearly perpendicular to B ab in order to maximize the susceptibility. Such a scenario has been discussed for a similar threefold degenerate domain structure in the uniaxial antiferromagnet NiO 85 . This is perfectly in line with the initial changes in the MO response indicative of the metastability of the domain population.
During the process of field-induced domain repopulation the volume of the three distinct ZZ chains changes, which immediately is reflected in changes of θ MLD 72 . However, if this domain repopulation would be the only mechanism causing the ZZ degeneracy to be lifted, a further increase of the magnetic field  strength should then only affect the amplitude of the order parameter L, while its orientation should be unchanged for small enough field strengths. It follows that no additional abrupt and anisotropic changes in θ MLD would be expected for just a gradual repopulation of the ZZ domains. In this regard, the key information provided by the MLD response is encoded in steps 2 and 3. In step 2 the system is approaching a steady-state in an interval of the applied field from ≈0.7 to 1.6 T, that is the initial and field-orientation independent domain reorientation is terminated. Hence, the clear difference at 1.6 T for the different orientations of the magnetic field gives experimental evidence that it has to be caused by a different mechanism than the already terminated domain repopulation. At 1.6 T an abrupt change for the field perpendicular to the bond appears, whereas no clear change is observable for the field along the bond. Hence, there is seemingly a competition between the external applied field and the anisotropic intra-layer interactions stabilizing the orientation of the Néel vector in some domains until the external field overcomes the anisotropy energy. This competition can be modeled based on a modified spin-flop theory and allows to derive a value for the anisotropy constant K a (Supplementary Information Note 5). We start with the free energy of the system at the ground state, which gives information about the spin-pointing direction and can be generally expressed as 77,86,87 Here μ 0 is the vacuum permeability, χ + and χ − correspond to the extrema of the in-plane oscillating susceptibility χ ∥ (ϕ) of α-RuCl 3 , where χ + and χ − occur for the magnetic field applied parallel or perpendicular to one of the Ru-Ru bond directions, respectively 59 . The values of χ + and χ − at a temperature of 2 K have been reported in ref. 59 . The angle of the pseudospins and the magnetic field is parametrized by α and Ψ is the angle between the applied field and the magnetic easy axis, which is perpendicular to one of the stretched Ru-Ru bonds. Equation (3) was numerically evaluated for every single ZZ domain and field orientation to find the angle α at fixed Ψ, which minimizes the energy for each field value H. For this, α was varied between 0 and π for each field value H. For the field oriented along the stretched bond, α = π/2 does not change for increasing field. For anisotropy-stabilized domains at a finite angle with the applied field, we find a continuous change in α(H), which converges towards α = π/2 for increasing field strength. Only for the field applied perpendicular to the stretched bond there is a discontinuity in α(H). In addition, we consider the initial field-induced changes in the ZZ domain volumes on a phenomenological level similar to reported in ref. 77 . More information regarding the modeling of the MO response can be found in Supplementary Information Note 5.
The model satisfactorily reproduces the field evolution of the MO response for both in-plane field directions (Fig. 6d). Further, the anisotropy constant K a can be estimated from the experimentally determined spin-flop field H sf = 1.6 T applied within the ab-plane The derived value for the effective anisotropy field K a /μ 0 is 0.0027 T. The interpretation of a spin-flop transition is supported by recently reported in-plane susceptibility measurements, where a two-fold oscillation has been observed for fields below 2 T pointing towards the intrinsic in-plane bond-anisotropy with a πperiodicity as opposed to a characteristic sixfold oscillation emerging for larger field strengths indicating the field-induced reorientation of the order parameter and hence spin-pointing direction in α-RuCl 3 44,59 . However, this produces only little changes in the magnetization (cf. Fig. 5a), although the orientation of L changes remarkably. This is a reasonable explanation why in previously reported magnetization data, which lack the sensitivity to small changes in L, a change in the low-field susceptibility has been overlooked, although some changes at around 1.5 T in dM/ dB have been found previously ( 39 , SI of ref. 43 ). Figure 6a, b summarizes the experimental results. Independent of the in-plane field orientation there is an initial change in the ZZ domain population in regime I. According to the direction of the applied field, some of the initial domains become more populated at the expense of the others, but the general evolution of the rotation θ MLD in small fields is similar for both field orientations. The experimental data show that this process terminates at a field strength of~0.7 T. For the field range of 0.7-1.6 T, i.e., step 2, there are already energetically optimal field-aligned domains, although some anisotropy-stabilized domains exist for the field applied along the {1, 1, 0} direction. Then at the transition into regime II at step 3, the external field wins the competition over the bond anisotropy for the magnetic field applied perpendicular to the Ru-Ru bonds, such that the pseudospins in all domains rotate perpendicular to the field directions pushing α-RuCl 3 into a single polarized spin-flopped state. A further increase in the field strength, i.e., moving closer to the critical line of the phase transition towards the quantum paramagnetic state, results in a natural decrease of the observed rotation, i.e., the magnitude of the order parameter. Before entering the quantum paramagnetic state above 7 T another anomaly is seen in the MO data. In contrast to the low-field transition, this anomaly at 6.2 T in dθ/dB is not immune to temperature changes and fades off for increasing temperatures from 2.8 to 3.5 K systematically for fields applied perpendicular to the bond (c.f. Fig. 6c). Its dependence on the inplane field angle is illustrated by the fact, that for B ab ∥{1, 0, 0} no signature of an intermediate field-induced transition in the vicinity of 6.2 T can be identified since dθ/dB is almost constant even at the lowest temperature of 2.8 K. The fact that this intermediate field-induced transition strictly depends on the orientation of B ab w.r.t. to the crystallographic axis conveys the emergence of an anisotropy-related novel magnetically ordered phase. This phase was named ZZ2 and its origin has been discussed in terms of inter-layer anisotropies very recently 44 . It was understood in terms of a field-induced inhomogeneous spin canting between different threefold and sixfold ZZ stackings for B ab ∥{1, 1, 0}, while the canting is homogeneous for B ab ∥{1, 0, 0} 44 . In addition, it has been discussed in terms of an inverse melting phenomenon 54 . It is worth to note that the MO response is sensitive to this transition and its first-order character fits to the observed large MO hysteresis effect. Nevertheless, a microscopic origin of this fieldinduced transition, nor an estimate for the underlying inter-layer exchange interactions can be given based on our observations, but calls for future MO measurements at elevated magnetic fields.
In summary, we have reported magneto-optical spectroscopy measurements on the Kitaev spin liquid candidate material α-RuCl 3 . Our study establishes magneto-optical spectroscopy as a versatile experimental tool to elucidate exotic phases of quantum materials. We observed two intermediate metamagnetic transitions at ≈1.6 T and 6.2 T for the magnetic field applied perpendicular to the Ru-Ru bond, while none of these transitions appears for the magnetic field aligned parallel to the Ru-Ru bonds. We clarified the nature of the low-field transition and discussed it in terms of a spin-flop transition, where the external field overcomes the anisotropy energy to align the Néel vector nearly orthogonal to the field direction. The effective anisotropy field has been determined to be 0.0027 T. Further, we confirmed the emergence of the previously observed high-field intermediate phase. Our results point out the importance of anisotropic intraand inter-layer bond anisotropies and the necessity to include those in future theoretical calculations. Besides that, we illuminated the importance of the in-plane field angle, which calls for future MO studies in the high-field proximate spin liquid phase of α-RuCl 3 . The spin-flop transition at a moderate field strength motivates further studies on α-RuCl 3 , as the reorientation of the Néel vector opens pathways to vary the magnetoresistance almost continuously. Here, measurements of the anisotropic magnetoresistance, which are also even in the magnetization as MLD and MLB 63 , should be considered for future experiments to access the precise control of the Néel vector 88 . This could in principle lead to future implementation of α-RuCl 3 in antiferromagnetic spintronics 89 .

Sample growth, characterization, and orientation
High-quality α-RuCl 3 crystals were prepared by vacuum sublimation 37 . The different samples of the same batch were characterized by SQUID magnetometry, showing a sharp transition at around T N ≈ 7 K in zero applied field corresponding to the ABC stacking order 35,56 . No additional magnetic transitions above T N are observed, which have been related to a different stacking order ABAB with a two-layer periodicity or strainintroduced stacking faults due to extensive handling or deformation of the crystals. This bulk technique can only provide the first indication of sample quality for an optics study. Cleaving or polishing introduces strain. In this regard, we refrained from any sample treatment and used an as-grown α-RuCl 3 sample. The temperature dependence of the equilibrium MO response shown in Fig. 1d shows a clear phase transition at T N ≈ 7 K, confirming good sample quality ( Supplementary Information Note 1).
The different α-RuCl 3 samples have been oriented via a standard X-ray Laue-diffractometer at room temperature.

Experimental procedures and measurement techniques
For the study of the MO response of α-RuCl 3 , the high-quality as-grown samples were placed in a helium-cooled cryostat (Oxford Spectromag) with temperatures down to 2.2 K inside the coils of a superconducting magnet with magnetic field strengths up to ±7 T. Figure 1a illustrates the experimental setup. The magnetic field was applied along different crystallographic directions within the crystallographic ab-plane, i.e. within the honeycomb layers. The polarization of incident light was rotated by a half waveplate and after initial tests, set to the purely linearly s-polarized setting, i.e., θ 0 = 0 for the maximum signal in zero field. The measurements of the second-order MO response were carried out in the so-called Voigt geometry 67 at near-normal incidence, such that the light wave propagation k was perpendicular to the honeycomb layer planes (k⊥ab) and magnetic field vector B ab (Fig. 1a). The MO measurements were performed using a continuous laser with a wavelength of 532 nm and the laser spot was focused to a spot size of 200 × 200 μm on the sample with the power set to 50 μW. Detection of the MO response Θ = θ + iη was done using a polarization modulation technique, in which the relative phase of two orthogonal linear polarizations was modulated that pass through a photoelastic modulator (PEM) 90 . The change in rotation θ and ellipticity η were probed simultaneously ( Supplementary Information Note 7).

DATA AVAILABILITY
The data supporting the findings of this study are available from the corresponding authors on reasonable request.