Strategy to extract Kitaev interaction using symmetry in honeycomb Mott insulators

Abstract

The Kitaev spin liquid, a ground state of the bond-dependent Kitaev model in a honeycomb lattice has been a center of attraction, since a microscopic theory to realize such an interaction in solid-state materials was discovered. A challenge in real materials though is the presence of the Heisenberg and another bond-dependent Gamma interactions detrimental to the Kitaev spin liquid, and there have been many debates on their relative strengths. Here we offer a strategy to extract the Kitaev interaction out of a full microscopic model by utilizing the symmetries of the Hamiltonian. Two tilted magnetic field directions related by a two-fold rotational symmetry generate distinct spin excitations originated from a specific combination of the Kitaev and Gamma interactions. Together with the in- and out-of-plane magnetic anisotropy, one can determine the Kitaev and Gamma interactions separately. Dynamic spin structure factors are presented to motivate future experiments. The proposed setups will advance the search for Kitaev materials.

Introduction

An electron’s orbital motion in an atom generates a magnetic field that influences its spin moment, known as spin-orbit coupling. When the coupling is strong in heavy atoms, the effective Hamiltonian is described by the spin-orbit-entangled pseudospin wave-function and the interactions among magnetic ions are highly anisotropic different from the standard Heisenberg interaction^1,2,3,4,5,6. A fascinating example is the Kitaev model with a bond-dependent interaction in a two-dimensional honeycomb lattice, whose ground state is a quantum spin liquid (QSL) with Majorana fermions and Z₂ vortex excitations⁷. There have been extensive studies on the model because in the Kitaev QSL non-Abelian excitations emerge under a magnetic field, and their braidings provide topological computation. Since a microscopic mechanism to generate such an interaction was uncovered⁸, intense efforts toward finding QSLs including a variety of candidate materials from spin S = 1/2^{9,10,11,12,13,14,15,16,17,18} to higher-spin S systems have been made^19,20,21,22. Despite such efforts, a confirmed Kitaev QSL is still missing.

One challenge in finding the Kitaev QSL in magnetic materials is the presence of other spin interactions which may generate magnetic orderings or other disordered phases^{23,24,25,26,27,28,29}. A generic nearest neighbor (n.n.) model in an ideal honeycomb was derived which revealed the isotropic Heisenberg interaction and another bond-dependent interaction named the Gamma (Γ)²⁵. Furthermore, there exist further neighbor interactions such as second and third n.n. Heisenberg interactions, which makes it difficult to single out the Kitaev interaction itself. There have been many debates on the relative strengths, especially between the dominant Kitaev and Gamma interactions in Kitaev candidate materials^13,18,28,30, and an experimental guide on how to extract the Kitaev interaction out of a full Hamiltonian is highly desirable.

In this work, we present a symmetry-based experimental strategy to determine the Kitaev interaction. Our proposal is based on the π-rotation around the a axis perpendicular to one of the bonds in the honeycomb plane, denoted by C_2a symmetry that is broken by a specific combination of the Kitaev and Γ interactions. This broken C_2a can be easily detected with the help of a magnetic field applied within the a−c plane where the c-axis is perpendicular to the honeycomb plane; spin excitations under the two field angles of θ and −θ, measured away from the honeycomb plane as shown in Fig. 1a, are distinct due to the combination of the Kitaev and Gamma interactions. The two field angles are related by the π-rotation around a axis, i.e., C_2a operation. Such differences are based on the symmetry and signal the relative strengths of these interactions. A magnetic ordering that further enhances the broken C_2a symmetry does not alter the asymmetry, but quantifying the interaction strengths requires the size of the magnetic ordering. For this reason, a polarized state in the high-field region would be ideal for our purpose.

Fig. 1: Crystal structure and direction of the magnetic field.

a Schematic of the honeycomb lattice of transition metal ions (light blue) in edge-sharing octahedra environment of anions (above the honeycomb plane: gray, below the plane: light gray). Octahedral xyz axes, abc axes, and the Kitaev bonds x (red), y (green), z (blue) are indicated. C_2a and C_2b symmetries (orange) are highlighted. The octahedra environment breaks C_2a, while C_2b symmetry is intact. b Direction of the external magnetic field $\overrightarrow{h}$ in abc axes where θ is measured from the a−b plane, and ϕ is from the a axis. The blue arrow $\overrightarrow{M}$ represents the magnetic moment direction with the angle θ_M. c δω_K(θ) in the a−c plane is the difference in the spin excitation energies ω between two field directions: ω(θ) (blue) and ω(−θ) (red). C_2b maps ω(θ) to ω(π + θ), so δω_K(π − θ) = −δω_K(θ).

To determine each of the interactions, one needs to use the conventional in- vs. out-of-plane anisotropy in spin excitations. We note that the Gamma interaction affects the conventional anisotropy, but the Kitaev does not when the field is large enough to compensate for the order by disorder effect³¹. Thus subtracting the Gamma contribution deduced from the conventional anisotropy allows us to estimate the Kitaev interaction from the measured spin excitations under the field angles of θ and −θ. Both the conventional anisotropy and the π-rotation-related spin excitations can be measured by angle-dependent ferromagnetic resonance (FMR) or inelastic neutron scattering (INS) techniques while sweeping the magnetic field directions in the a−c plane containing the C_2a rotation axis.

Below we present the microscopic model and main results based on the π-rotation symmetry around a-axis. To demonstrate our theory, we also show the FMR and dynamical spin structure factors (DSSF) obtained by exact diagonalization (ED). We analyze the different spin excitations under the two field angles at finite momenta using the linear spin-wave theory (LSWT), which further confirms our results based on the symmetry argument. Our results will guide a future search of Kitaev materials.

Result

Model

The generic spin-exchange Hamiltonian among magnetic sites with strong spin-orbit coupling for the ideal edge-sharing octahedra environment in the octahedral x−y−z axes shown in Fig. 1a contains the Kitaev (K), Gamma (Γ), and Heisenberg (J) interactions²⁵:

$${{{{{{{\mathcal{H}}}}}}}}=\mathop{\sum}\limits_{\langle ij\rangle \in \alpha \beta (\gamma )}\left[J{{{{{{{{\bf{S}}}}}}}}}_{i}\cdot {{{{{{{{\bf{S}}}}}}}}}_{j}+K{S}_{i}^{\gamma }{S}_{j}^{\gamma }+{{\Gamma }}\left({S}_{i}^{\alpha }{S}_{j}^{\beta }+{S}_{i}^{\beta }{S}_{j}^{\alpha }\right)\right],$$

(1)

where ${{{{{{{\bf{S}}}}}}}}=\frac{1}{2}\overrightarrow{\sigma }$ with ℏ ≡ 1 and $\overrightarrow{\sigma }$ is Pauli matrix, 〈ij〉 denotes the n.n. magnetic sites, and αβ(γ) denotes the γ bond taking the α and β spin components (α, β, γ ∈ {x, y, z}). The x-, y-, and z-bonds are shown in red, blue, and green colors, respectively in Fig. 1a. Further neighbor interactions and trigonal distortion allowed interactions, and their effects will be discussed later.

To analyze the symmetry of the Hamiltonian, we rewrite the model in the a−b−c axes^32,33,34:

$${{{{{{{\mathcal{H}}}}}}}}= \mathop{\sum}\limits_{\langle i,j\rangle }\left[{J}_{XY}\left({S}_{i}^{a}{S}_{j}^{a}+{S}_{i}^{b}{S}_{j}^{b}\right)+{J}_{Z}{S}_{i}^{c}{S}_{j}^{c}\right.\\ +{J}_{ab}\left[\cos {\phi }_{\gamma }\left({S}_{i}^{a}{S}_{j}^{a}-{S}_{i}^{b}{S}_{j}^{b}\right)-\sin {\phi }_{\gamma }\left({S}_{i}^{a}{S}_{j}^{b}+{S}_{i}^{b}{S}_{j}^{a}\right)\right]\\ -\left.\sqrt{2}{J}_{ac}\left[\cos {\phi }_{\gamma }\left({S}_{i}^{a}{S}_{j}^{c}+{S}_{i}^{c}{S}_{j}^{a}\right)+\sin {\phi }_{\gamma }\left({S}_{i}^{b}{S}_{j}^{c}+{S}_{i}^{c}{S}_{j}^{b}\right)\right]\right],$$

(2)

where ${\phi }_{\gamma }=0,\frac{2\pi }{3}$, and $\frac{4\pi }{3}$ for γ = z-, x-, and y-bond respectively, and the exchange interactions are given by

$${J}_{XY} = J+{J}_{ac},\,\,{J}_{Z}=J+{J}_{ab},\\ {J}_{ab} = \frac{1}{3}K+\frac{2}{3}{{\Gamma }},\,\,{J}_{ac}=\frac{1}{3}K-\frac{1}{3}{{\Gamma }}.$$

(3)

The Hamiltonian ${{{{{{{\mathcal{H}}}}}}}}$ is invariant under π-rotation around the b axis denoted by C_2b and $\frac{2\pi }{3}$-rotation around the c axis by C_3c in addition to the inversion and time-reversal symmetry.

Our proposed experimental design is based on the observation that the ${{{{{{{\mathcal{H}}}}}}}}$ is not invariant under π-rotation about the a axis C_2a due to the presence of only J_ac, i.e., if J_ac = 0, C_2a is also a symmetry of ${{{{{{{\mathcal{H}}}}}}}}$. Since the C_2a is broken by J_ac, if there is a way to detect the broken C_2a, that will signal the strength of J_ac. We note that the magnetic field sweeping from the c axis to a axis within the a−c-plane does the job. The fields with angles of θ (blue line) and −θ (red line) for $0 \, < \, \theta \, < \, \frac{\pi }{2}$ shown in Fig. 1b, c are related by C_2a rotation, and thus measuring the spin excitation difference between these two field directions will detect the strength of J_ac.

To prove our symmetry argument, we consider a full model with a magnetic field. Under a magnetic field, the total Hamiltonian including the Zeeman term is given by

$${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{tot}}}}}}}}}={{{{{{{\mathcal{H}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{B}={{{{{{{\mathcal{H}}}}}}}}-g\,{\mu }_{B}\mathop{\sum}\limits_{i}{\overrightarrow{S}}_{i}\cdot \overrightarrow{h},$$

(4)

where the external field $\overrightarrow{h}$ has the polar angle θ measured away from the a−b honeycomb plane and the azimuthal angle ϕ from the a-axis as shown in Fig. 1b. The magnetic anisotropy in the spin excitation energies is defined as ω_n(θ) = E_n(θ) − E₀(θ), where E_n and E₀ are the excited and ground-state energy respectively. This anisotropy is affected by all interactions other than the isotropic Heisenberg limit (J_XY = J_Z), making it difficult to quantify the effect of individual interactions. However, if we compare the two excitation anisotropies, ω_n(θ) and ω_n(−θ) for a given strength h and ϕ = 0 as shown in Fig. 1c, related by C_2a symmetry transformation, we can eliminate the effects of all other interactions except J_ac thanks to symmetries of the model. Since our theory relies on the symmetry of the Hamiltonian, the ground state should break the C_2a symmetry only explicitly from the J_ac term. The magnetic field also contributes to the C_2a breaking, but by comparing two angles of θ and −θ, the effect of J_ac is isolated.

We focus on the lowest energy excitation n = 1 which gives a dominant resonance at low temperatures, and drop the n in ω_n from now on for simplicity, even though our proposal works for all n. We define the excitation anisotropy between the magnetic field with angles of θ and −θ as δω_K(θ) ≡ ω(θ) − ω( −θ) for $0 \, < \, \theta \, < \, \frac{\pi }{2}$, and the conventional anisotropy between in- and out-of-plane fields as $\delta {\omega }_{A}\equiv \omega (\theta =0)-\omega (\theta =\frac{\pi }{2})$. Below we first show how δω_K arises from J_ac under the field in the a−c plane based on the symmetry.

Symmetry analysis

To understand the origin of a finite δω_K for ϕ = 0 under the magnetic field sweep, we first begin with a special case when $\phi =\frac{\pi }{2}$, i.e, when the external field is in the b−c plane. This is a special case where δω_K = 0 for the following reason.

The Zeeman terms due to the field with the angle θ and with −θ are related by a π rotation of the field about the $\hat{b}$ axis, denoted by

$${C}_{2b,\theta }:\,{{{{{{{{\mathcal{H}}}}}}}}}_{B}\propto (\cos \theta {S}_{i}^{b}+\sin \theta {S}_{i}^{c})\longrightarrow (\cos \theta {S}_{i}^{b}-\sin \theta {S}_{i}^{c}).$$

(5)

The same can be achieved by a π-rotation of the lattice,

$${C}_{2b}:\,({S}^{a},{S}^{b},{S}^{c})\to (-{S}^{a},{S}^{b},-{S}^{c})\,\,{{{{{{{\rm{and}}}}}}}}\,\,{\phi }_{x}\leftrightarrow {\phi }_{y},$$

(6)

which also indicates ${{{{{{{\mathcal{H}}}}}}}}$ is invariant under C_2b. While ${{{{{{{{\mathcal{H}}}}}}}}}_{B}$ breaks the C_2b symmetry of ${{{{{{{\mathcal{H}}}}}}}}$, the total Hamiltonian ${{{{{{{\mathcal{H}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{B}(\theta )$ and ${{{{{{{\mathcal{H}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{B}(-\theta )$ are related by C_2b and therefore, share the same eigenenergies, i.e., δω_K = 0. The difference due to the field is simply removed by a π rotation of the eigenstates about the $\hat{b}$ axis. The magnetic field sweeping from θ to −θ in the other planes equivalent to b−c plane by C_3c symmetry also gives δω_K = 0.

Now let us consider when the magnetic field sweeps in the a−c plane. Similarly, the magnetic field directions θ and −θ are related by

$${C}_{2a,\theta }:{{{{{{{{\mathcal{H}}}}}}}}}_{B}\propto (\cos \theta {S}_{i}^{a}+\sin \theta {S}_{i}^{c})\longrightarrow (\cos \theta {S}_{i}^{a}-\sin \theta {S}_{i}^{c}).$$

(7)

Considering a π rotation of the lattice about the $\hat{a}$ axis,

$${C}_{2a}:\,({S}^{a},{S}^{b},{S}^{c})\to ({S}^{a},-{S}^{b},-{S}^{c})\,\,{{{{{{{\rm{and}}}}}}}}\,\,{\phi }_{x}\leftrightarrow {\phi }_{y},$$

(8)

we find J_XY, J_Z, J_ab, terms are invariant under C_2a, while the J_ac terms transform as

$${C}_{2a}:\,{J}_{ac}\to -{J}_{ac}.$$

(9)

By the same argument, if J_ac = 0, ${{{{{{{\mathcal{H}}}}}}}}$ is invariant under C_2a, and the eigenenergies of the total Hamiltonian for θ and −θ are the same, i.e., δω_K = 0. If J_ac ≠ 0, the total Hamiltonian ${{{{{{{\mathcal{H}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{B}(\theta )$ and ${{{{{{{\mathcal{H}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{B}(-\theta )$ cannot be related by C_2a, and therefore, δω_K ≠ 0. We need to change the sign of J_ac for the C_2a relation to hold, i.e., the transformation of the external field angles of θ to −θ is equivalent to the change of J_ac to −J_ac. Thus, the lack of C_2a symmetry allows us to single out the J_ac interaction through δω_K.

Since J_ac contains a combination of the Kitaev and Γ interactions, we need other methods to subtract the Γ contribution. The in- and out-of-plane anisotropy, δω_A offers precisely the other information. We note that the in- and out-of-plane anisotropy δω_A is determined by J_Z − J_XY = Γ. Thus, for the ideal edge-sharing octahedral environment, we can first estimate Γ from the measured δω_A, and then extract the Kitaev strength by subtracting the Γ contribution from the measured δω_K(θ).

Below we show numerical results of spin excitations obtained by ED on a 24-site cluster which can be measured by angle-dependent FMR and INS techniques under magnetic field angles of θ and −θ with ϕ = 0.

Angle-dependent ferromagnetic resonance

FMR is a powerful probe to study ferromagnetic or spin correlated materials. FMR spectrometers record the radio-frequency (RF) electromagnetic wave that is absorbed by the sample of interest placed under an external magnetic field. To observe the resonance signal, the resonant frequency of the sample is changed to match that of the RF wave under a scan of the external magnetic field, so the excitation anisotropy δω(θ) leads to the anisotropy in the resonant magnetic field. FMR provides highly resolved spectra over a large energy range and has been used to investigate exchange couplings^35,36,37,38 and anisotropies^39,40 due to its dependence on the magnetic field angle. Here, for simplicity, we calculate the excitation energy probed by the RF field (details can be found in the Methods) with a set magnetic field strength for spin $\frac{1}{2}$ using ED on a C₃-symmetric 24-site cluster.

We set our units the magnetic field h = 1 and g = μ_B ≡ 1, leading to the excitation energy of a free spin, ω₀ = gμ_Bh = 1, so the excitation energies calculated are normalized by ω₀. A few sets of different interaction parameters (in units of ω₀) are investigated. Figure 2(a) shows the J = −1 and K = Γ = 0.5 case with no δω_K(θ) between −π/2 < θ < 0 (red line) and 0 < θ < π/2 (blue line), since J_ac = 0. The conventional anisotropy δω_A is finite, because the Γ interaction generates a strong anisotropy between the plane θ = 0 and the c-axis θ = π/2, i.e., J_XY ≠ J_Z due to a finite Γ contribution. The black line is for only J = −1 showing a uniform FMR independent of angles which serves as a reference. Figure 2b shows the J = −1, K = 1, and Γ = 0 case, which shows a finite δω_K(θ) between −π/2 < θ < 0 and 0 < θ < π/2 in the a−c plane. On the other hand, no δω_K(θ) by sweeping θ in the b−c plane (up and down triangles with green line) is observed, consistent with the symmetry analysis presented above. Note the conventional anisotropy δω_A in both a−c and b−c planes are not exactly zero, because the Kitaev interaction selects the magnetic moment along the cubic axes in the ferromagnetic state via order by disorder^31,41. This leads to a tiny anisotropy between the plane θ = 0 and the c-axis θ = π/2 when Γ = 0 and J_XY = J_Z. This anisotropy becomes weaker when the magnetic field increases, i.e, when the moment polarization overcomes the order by disorder effect. Supplementary Note 1 shows that the anisotropy is almost gone when the field is increased by three times with the same set of parameters, where the Heisenberg limit (black line) is added for reference. When Γ becomes finite favouring either the a−b plane or the c axis depending on the sign of the Γ, this conventional anisotropy is determined by the Γ interaction as shown in Fig. 2c, d, and the order by disorder effect becomes silent. Figure 2c shows the J = −0.5, Γ = 0.5, and K = 0 case. The Γ interaction alone can generate a finite δω_K due to the broken C_2a by J_ac. In addition, the Γ interaction generates a large δω_A, different from Fig. 2b. Figure 2d presents the J = −0.1, K = −1, and Γ = 0.5 case, which is close to a set of parameters proposed for J${}_{{{{{{{{\rm{eff}}}}}}}}}=\frac{1}{2}$ Kitaev candidate materials²⁸. Clearly, δω_K(θ) is significant due to a finite J_ac, and δω_A is also large due to a finite Γ. While a magnetic field of strength h = 1 is used to polarize the ground state where the finite-size effect is small as shown in Supplementary Note 2, our symmetry argument works for any finite field. However, we note that the finite-size effect of ED is minimal when the ground state is polarized.

Fig. 2: Angle-dependent spin excitations in ferromagnetic resonance (FMR) using exact diagonalizaiton on a C₃-symmetric 24-site cluster.

Various sets of parameters with Zeeman energy gμ_Bh = 1 are used. δω_A is the difference in the spin excitation energies ω between fields along a axis and c axis, and δω_K is the difference between ω(θ) (blue) and ω(−θ) (red), as highlighted by the arrows. J, K, and Γ are the Heisenberg, Kitaev and off-diagonal interactions respectively. a J = −1 and K = Γ = 0.5. b J = −1, K = 1, and Γ = 0. FMR in the b−c plane is shown in green: θ (up triangle) and −θ (down triangle). c J = −0.5, Γ = 0.5, and K = 0. d J = −0.1, K = −1, Γ = 0.5. See the FMR subsection for implication of the results.

Inelastic neutron scattering

Complementary to FMR, INS can measure excitations between different points in the reciprocal space based on the momentum transfer of the scattered neutrons. The magnon dispersions of the ordered states of magnetic materials measured via INS have been used to determine the spin-exchange Hamiltonian parameters^{10,17,42,43,44,45,46}. Figure 3a, b show the spin excitations at accessible wavevectors on a C₃-symmetric 24-site cluster with the same exchange parameters for Fig. 2d and with h = 1 and h = 8, respectively. The cluster and the accessible momenta are shown in c, d, respectively. We set the magnetic field angles θ = 30^∘ (blue) and θ = −30^∘ (red) in a−c plane. The square boxes denote the excitation energies obtained by the ED, and the color bars indicate the intensity of DSSF ∑_αS^αα(q, ω) (details can be found in the Methods). The structure factor is convolved with a Gaussian of finite width to emulate finite experimental resolution. We observe a clear difference between the two field directions, δω_K at every momentum points. In particular, δω_K is the largest at M₂-point, while it is tiny at the K₁-point. Note that M₁ and M₃ are related by the C_2b and inversion.

Fig. 3: Dynamic spin structure factor (DSSF) of the spin excitations at accessible wavevectors using exact diagonalizaiton (ED) on a C₃-symmetric 24-site cluster and linear spin-wave theory (LSWT).

The boxes and the dashed lines are DSSF obtained by ED and LSWT, respectively. The color bars represent the intensity of DSSF. The same parameters for Fig. 2d are used, i.e., (J, K, Γ) = (−0.1, −1, 0.5) in units of ω₀ = gμ_Bh = 1. The magnetic field angles in the a−c plane are 30^∘ (blue) and −30^∘ (red). b DSSF with the same parameters as a except a larger field gμ_Bh = 8, showing a better match between the ED and LSWT results; see the Inelastic Neutron Scattering subsection for further discussions. c C₃-symmetric 24-site cluster used for the ED. d Accessible momentum points labeled in the x axis of a and b.

To gain more insights of δω_K(θ) at finite momenta obtained by ED, we also perform LSWT calculations with the magnetization making an angle θ_M as indicated in Fig. 1b. θ_M is found via minimizing the classical ground state energy (details can be found in the Methods); the LSWT with the set of parameters used for Fig. 3a’s ED results leads to θ_M ~ 12.1^∘. The spin excitations within the LSWT are shown as dashed lines together with the ED results in Fig. 3a. The mismatch between LSWT and ED is visible at every momentum, which implies the significant effects of nonlinear terms⁴⁷.

However, when the field increases, the difference should decrease, since the magnetic polarization increases at a higher field. In Fig. 3b, we show both ED and LSWT with h = 8 and θ_M ~ 25.8^∘, where the two results match well as expected, and the nonlinear terms become less significant. In particular, the anisotropy δω_K at the K-point at the high-field limit given by the leading terms in 1/h, is simplified as

$$\delta {\omega }_{K}(\theta )= \frac{3}{8}\cos {\theta }_{M} \bigg(| 2\sqrt{2}{J}_{ac}\sin {\theta }_{M}-{J}_{ab}\cos {\theta }_{M}| -| 2\sqrt{2}{J}_{ac}\sin {\theta }_{M}+{J}_{ab}\cos {\theta }_{M}| \bigg)\,\\ +\frac{9\sqrt{2}{J}_{ac}{J}_{ab}\left(2\sin 2{\theta }_{M}+\sin 4{\theta }_{M}\right)}{128h\cos (\theta -{\theta }_{M})}+{{{{{{{\mathcal{O}}}}}}}}\left(\frac{1}{{h}^{2}}\right),$$

(10)

where θ_M(θ) → θ when h → ∞. This shows that both J_ac and J_ab should be finite for a finite δω_K at the K-point, which explains no splitting of δω_K at the K-point in Fig. 3b, as our choice of parameters gives J_ab = 0, i.e, Γ = −K/2. On the other hand, at the M₂-point, there is no simple expression, but the leading terms of δω_K(θ) in δθ_a/c around the a- and c axis (δθ_a = 0 − θ and δθ_c = θ − π/2) are given by

$$\delta {\omega }_{K}(\theta )\simeq \left\{\begin{array}{l}{J}_{ac}(\delta {\theta }_{a})A+{{{{{{{\mathcal{O}}}}}}}}(\delta {\theta }_{a}^{3})\\ {J}_{ac}(\delta {\theta }_{c})C+{{{{{{{\mathcal{O}}}}}}}}(\delta {\theta }_{c}^{3}),\end{array}\right.$$

(11)

where A and C are functions of other interactions given in Supplementary Note 3. Clearly, δω_K(θ) appears as odd powers of J_ac and δθ_a/c, consistent with the symmetry analysis presented above.

So far, we have focused on the ideal octahedra environment. However, trigonal distortion is often present, albeit small, which introduces extra exchange interactions. Below we discuss other contributions to δω_A complicating the isolation of K from J_ac and our resolution of such complication in order to estimate the Kitaev interaction out of a full Hamiltonian.

Effects of trigonal distortion and further neighbor interactions

In principle, there are other small but finite interactions; few examples in $\delta {{{{{{{{\mathcal{H}}}}}}}}}^{\prime}$ include

$$\delta {{{{{{{{\mathcal{H}}}}}}}}}^{\prime}= \mathop{\sum}\limits_{\langle ij\rangle \in \alpha \beta (\gamma )}\left[{{{\Gamma }}}^{\prime}({S}_{i}^{\alpha }{S}_{j}^{\gamma }+{S}_{i}^{\gamma }{S}_{j}^{\alpha }+{S}_{i}^{\beta }{S}_{j}^{\gamma }+{S}_{i}^{\gamma }{S}_{j}^{\beta })\right]\\ +{J}_{2}\mathop{\sum}\limits_{\langle \langle i,j\rangle \rangle }{{{{{{{{\bf{S}}}}}}}}}_{i}\cdot {{{{{{{{\bf{S}}}}}}}}}_{j}+{J}_{3}\mathop{\sum}\limits_{\langle \langle \langle i,j\rangle \rangle \rangle }{{{{{{{{\bf{S}}}}}}}}}_{i}\cdot {{{{{{{{\bf{S}}}}}}}}}_{j},$$

(12)

where ${{{\Gamma }}}^{\prime}$ is introduced when a trigonal distortion is present²⁶; J₂ and J₃ are the second and third n.n. Heisenberg interactions respectively. It is natural to expect that they are smaller than the n.n. Kitaev, Gamma, and Heisenberg interactions^18,27,28. Several types of interlayer exchange interactions are present, but they are even smaller than the terms considered in Eq. (12)¹⁸.

Let us investigate how they affect the above analysis done for the ideal n.n. Hamiltonian. First of all, the isotropic interactions such as further neighbor J₂, J₃, and the interlayer Heisenberg do not make any change to our proposal, since they do not contribute to δω_A nor δω_K. On the other hand, the ${{{\Gamma }}}^{\prime}$ modifies the exchange parameters as follows:

$${J}_{XY} = J+{J}_{ac}-{{{\Gamma }}}^{\prime},\,\,{J}_{Z}=J+{J}_{ab}+2{{{\Gamma }}}^{\prime},\\ {J}_{ab} =\frac{1}{3}K+\frac{2}{3}({{\Gamma }}-{{{\Gamma }}}^{\prime}),\,\,{J}_{ac}=\frac{1}{3}K-\frac{1}{3}({{\Gamma }}-{{{\Gamma }}}^{\prime}).$$

(13)

The conventional anisotropy δω_A is now due to ${{\Gamma }}+2{{{\Gamma }}}^{\prime}$ obtained from J_Z − J_XY. Thus to single out the Kitaev interaction, one has to find both Γ and ${{{\Gamma }}}^{\prime}$, as J_ac is a combination of K, Γ, and ${{\Gamma }}^{\prime}$. Once the trigonal distortion is present, the g-factor also becomes anisotropic, i.e., the in-plane g_a is different from the c axis g_c, which affects δω_A.

However, the g-factor anisotropy does not affect the δω_K, since the field angles of θ and −θ involve the same strength of in- and out-of-plane field components, i.e, ${{{{{{{\bf{h}}}}}}}}(\theta )={h}_{a}\hat{a}+{h}_{c}\hat{c}$ and ${{{{{{{\bf{h}}}}}}}}(-\theta )=-{h}_{a}\hat{a}+{h}_{c}\hat{c}$. Thus we wish to extract the information of K and ${{\Gamma }}-{{{\Gamma }}}^{\prime}$ from δω_K, as it is free from the g-factor anisotropy.

We note that δω_K at the K-point, Eq. (10) offers both J_ac and J_ab from the first term independent of the field and the next term proportional to 1/h_eff (${h}_{{{{{{{{\rm{eff}}}}}}}}}=h\sqrt{{g}_{a}^{2}{\cos }^{2}\theta +{g}_{c}^{2}{\sin }^{2}\theta }$). Once J_ac and J_ab are deduced, K and ${{\Gamma }}-{{\Gamma }}^{\prime}$ can be estimated from Eq. (13). The measurements of δω_K at the K-point with a large magnetic field then determine K and ${{\Gamma }}-{{{\Gamma }}}^{\prime}$ separately. Further neighbor Heisenberg interactions, J₂ and J₃ do not modify Eq. (10) in the high-field limit, so they do not affect our procedure.

Discussion

We propose an experimental setup to single out the Kitaev interaction for honeycomb Mott insulators with edge-sharing octahedra. In an ideal octahedra cage, the symmetry-allowed n.n. interactions contain the Kitaev, another bond-dependent Γ, and Heisenberg interactions. We prove that the magnetic anisotropy related by the π-rotation around the a-axis denoted by δω_K occurs only when a combination of K and Γ, i.e. K − Γ, is finite. This can be measured from the spin excitation energy differences under the magnetic field of angle sweeping from above to below the honeycomb plane using the FMR or INS techniques. Since the in- and out-of-plane magnetic anisotropy, δω_A is determined solely by Γ, one can estimate Γ strength first from δω_A and then extract the Kitaev interaction from δω_K.

While the trigonal distortion introduces an additional interaction, the Kitaev interaction is unique as it is the only interaction that contributes to δω_K without altering δω_A. Our theory is applicable to all Kitaev candidate materials including an emerging candidate RuCl₃. In particular, since the two dominant interactions are ferromagnetic Kitaev and positive Γ interactions in RuCl₃^3,5,18,27, leading to a large J_ac and a small J_ab, we predict that δω_K independent of the g-factor anisotropy is significant except at the K-point. Supplementary Note 4 shows the FMR and INS of a set of parameters with a small negative ${{{\Gamma }}}^{\prime}$ interaction to stabilize a zero-field zig-zag ground state as in RuCl₃^18,27,28. Another relevant perturbation in some materials is the effect of monoclinic structure which loses the C_3c symmetry of $R\bar{3}$, making the z-bond different from the x- and y-bonds. The current theory of finite δω_K due to a finite J_ac still works for C2/m structure. However, since the z-bond of ${J}_{ac}^{z}(={K}_{z}/3-{{{\Gamma }}}_{z}/3)$ is no longer the same as the x- and y-bonds of ${J}_{ac}^{x}(={J}_{ac}^{y})$ and C_2a symmetry relates between the x- and y-bonds, the anisotropy δω_K at different momenta, detecting both ${J}_{ac}^{x}={K}_{x}/3-{{{\Gamma }}}_{x}/3 \;{{{\mbox{and }}}}\; {J}_{ac}^{z}$, is required to determine different x- and z-bond strengths.

The symmetry-based theory presented here is also valid for higher-spin models with the Kitaev interaction such as S = 3/2 CrI₃ including a nonzero single-ion anisotropy^19,21,22,48 which generates a further anisotropy in δω_A but does not affect the δω_K. The next n.n. Dzyaloshinskii–Moriya interaction with the d-vector along the c-axis⁴⁹ is also invariant under the C_2a symmetry. Further studies for higher-spin models remain to be investigated to identify higher-spin Kitaev spin liquid. We would like to emphasize that the proposed setup is suitable for other experimental techniques such as low-energy terahertz optical and nuclear magnetic resonance spectroscopies that probe spin excitations in addition to the angle-dependent FMR and INS spectroscopy shown in this work as examples.

Methods

Exact Diagonalization Simulations

Numerical ED was used to compute spin excitations under a magnetic field. ED was performed on a 24-site honeycomb cluster with periodic boundary conditions, where the Lanczos method^50,51 was used to obtain the lowest-lying eigenvalues and eigenvectors of the Hamiltonian in Eq. (2). The 24-site honeycomb shape and accessible momentum points in the Brillouin zone are shown in Fig. 3c, d. The probability of the spin excitation of momentum q and energy ω is proportional to the DSSF⁵² given by

$${S}^{\alpha \beta }({{{{{{{\boldsymbol{q}}}}}}}},\omega ) =\frac{1}{N}\mathop{\sum }\limits_{i,j}^{N}{e}^{-i{{{{{{{\boldsymbol{q}}}}}}}}\cdot ({{{{{{{{\boldsymbol{R}}}}}}}}}_{i}-{{{{{{{{\boldsymbol{R}}}}}}}}}_{j})}\int\nolimits_{\infty }^{\infty }dt{e}^{i\omega t}\left\langle {S}_{i}^{\alpha }(t){S}_{j}^{\beta }(0)\right\rangle \\ =\frac{1}{N}\mathop{\sum }\limits_{i,j}^{N}{e}^{-i{{{{{{{\boldsymbol{q}}}}}}}}\cdot ({{{{{{{{\boldsymbol{R}}}}}}}}}_{i}-{{{{{{{{\boldsymbol{R}}}}}}}}}_{j})}\mathop{\sum}\limits_{\lambda ,{\lambda }^{\prime}}{p}_{\lambda }\langle \lambda | {S}_{i}^{\alpha }| {\lambda }^{\prime}\rangle \langle {\lambda }^{\prime}| {S}_{j}^{\beta }| \lambda \rangle \delta (\hslash \omega +{E}_{\lambda }-{E}_{{\lambda }^{\prime}})\\ =\mathop{\sum}\limits_{\lambda ,{\lambda }^{\prime}}{p}_{\lambda }\langle \lambda | {S}_{-{{{{{{{\boldsymbol{q}}}}}}}}}^{\alpha }| {\lambda }^{\prime}\rangle \langle {\lambda }^{\prime}| {S}_{{{{{{{{\boldsymbol{q}}}}}}}}}^{\beta }| \lambda \rangle \delta (\hslash \omega +{E}_{\lambda }-{E}_{{\lambda }^{\prime}}),$$

(14)

where the Lehmann representation is used; $\left|\lambda \right\rangle$ and $\left|{\lambda }^{\prime}\right\rangle$ are the eigenstates with the thermal population factor p_λ, and S^α,β are the spin operators. In the low temperatures, we take $\left|\lambda \right\rangle$ to be the ground state $\left|0\right\rangle$ and we are interested in the lowest energy excitation to $\left|1\right\rangle$ with a nonzero probability. For optical spectroscopies such as FMR, α = β = direction of the RF electromagnetic field and q = 0, so $\left|0\right\rangle$ and $\left|1\right\rangle$ belong to the same momentum sector. The structure factor simplifies to

$${S}^{\alpha \alpha }(\omega )=\frac{1}{N}{\left|\langle 1| \mathop{\sum}\limits_{i}{S}_{i}^{\alpha }| 0\rangle \right|}^{2}\delta (\hslash \omega +{E}_{0}-{E}_{1}).$$

For INS, the finite q must match the difference in the momenta of $\left|0\right\rangle$ and $\left|1\right\rangle$. For simplicity, we calculate the DSSF for α = β,

$$\mathop{\sum}\limits_{\alpha }{S}^{\alpha \alpha }({{{{{{{\bf{q}}}}}}}},\omega )=\mathop{\sum}\limits_{\alpha }{\left|\langle 1| {S}_{{{{{{{{\boldsymbol{q}}}}}}}}}^{\alpha }| 0\rangle \right|}^{2}\delta (\hslash \omega +{E}_{0}-{E}_{1}).$$

Linear spin-wave theory

The Hamiltonian in Eq. (2) is bosonized by the standard Holstein-Primakoff transformation⁵³ expanded to linear order in the spin S:

$${S}_{j}^{+} ={S}_{j}^{a}+i{S}_{j}^{b}=\sqrt{2S}\left({a}_{j}-\frac{{a}_{j}^{{{{\dagger}}} }{a}_{j}{a}_{j}}{4S}+{{{{{{{\mathcal{O}}}}}}}}\left(\frac{1}{{S}^{2}}\right)\right)\simeq \sqrt{2S}{a}_{j}\\ {S}_{j}^{-} ={S}_{j}^{a}-i{S}_{j}^{b}=\sqrt{2S}\left({a}_{j}^{{{{\dagger}}} }-\frac{{a}_{j}^{{{{\dagger}}} }{a}_{j}^{{{{\dagger}}} }{a}_{j}}{4S}+{{{{{{{\mathcal{O}}}}}}}}\left(\frac{1}{{S}^{2}}\right)\right)\simeq \sqrt{2S}{a}_{j}^{{{{\dagger}}} }\\ {S}_{j}^{c} ={S}^{c}-{a}_{j}^{{{{\dagger}}} }{a}_{j},$$

(15)

where the quantization axis is parallel to the c-axis. The Fourier transforms are ${a}_{j}=\frac{1}{\sqrt{N}}{\sum }_{{{{{{{{\boldsymbol{k}}}}}}}}}{e}^{i{{{{{{{\boldsymbol{k}}}}}}}}\cdot {{{{{{{{\boldsymbol{r}}}}}}}}}_{j}}{a}_{{{{{{{{\boldsymbol{k}}}}}}}}}$ for sublattice A and ${b}_{j}=\frac{1}{\sqrt{N}}{\sum }_{{{{{{{{\boldsymbol{k}}}}}}}}}{e}^{i{{{{{{{\boldsymbol{k}}}}}}}}\cdot ({{{{{{{{\boldsymbol{r}}}}}}}}}_{j}+\delta )}{b}_{{{{{{{{\boldsymbol{k}}}}}}}}}$ for sublattice B, where δ is the vector pointing to nearest neighbors. The resulting quadratic Hamiltonian has the form ${{{{{{{\mathcal{H}}}}}}}}={\sum }_{{{{{{{{\boldsymbol{k}}}}}}}}}{{{{{{{{\rm{X}}}}}}}}}^{{{{\dagger}}} }H({{{{{{{\boldsymbol{k}}}}}}}}){{{{{{{\rm{X}}}}}}}}$, where ${{{{{{{{\rm{X}}}}}}}}}^{{{{\dagger}}} }=(\begin{array}{cccc}{a}_{{{{{{{{\boldsymbol{k}}}}}}}}}^{{{{\dagger}}} },&{b}_{{{{{{{{\boldsymbol{k}}}}}}}}}^{{{{\dagger}}} },&{a}_{-{{{{{{{\boldsymbol{k}}}}}}}}},&{b}_{-{{{{{{{\boldsymbol{k}}}}}}}}}\end{array})$. Diagonalizing this BdG Hamiltonian following standard methods⁵⁴ gives two spin-wave excitation branches.

For a general field, the Hamiltonian in Eq. (2) is first written in new axes ${{{{{{{{\boldsymbol{a}}}}}}}}}^{\prime}{{{{{{{{\boldsymbol{b}}}}}}}}}^{\prime}{{{{{{{{\boldsymbol{c}}}}}}}}}^{\prime}$. ${{{{{{{{\boldsymbol{a}}}}}}}}}^{\prime}=(\sin {\theta }_{M}\cos {\phi }_{M},\sin {\theta }_{M}\sin {\phi }_{M},-\cos {\theta }_{M})$, ${{{{{{{{\boldsymbol{b}}}}}}}}}^{\prime}=(-\sin {\phi }_{M},\cos {\phi }_{M},0)$ and ${{{{{{{{\boldsymbol{c}}}}}}}}}^{\prime}=(\cos {\theta }_{M}\cos {\phi }_{M},\cos {\theta }_{M}\sin {\phi }_{M},\sin {\theta }_{M})$. ${{{{{{{{\boldsymbol{c}}}}}}}}}^{\prime}$ is parallel to the magnetization S(θ_M, ϕ_M), which is not in the same direction as the magnetic field, unless the field is very large to fully polarize the moment. The magnetization angles (θ_M, ϕ_M) are obtained by minimizing the classical ground state energy, and LSWT is applied to the ground state⁴⁷. Arbitrary ${{{{{{{{\boldsymbol{a}}}}}}}}}^{\prime}$ and ${{{{{{{{\boldsymbol{b}}}}}}}}}^{\prime}$ axes obtained by rotation around ${{{{{{{{\boldsymbol{c}}}}}}}}}^{\prime}$ are valid and do not affect the result.