Introduction

The search for quantum spin liquids (QSLs) has been at the forefront of condensed matter physics for many decades because they represent novel quantum phases of matter beyond the Landau paradigm of symmetry breaking—instead they are characterized by fractionalized excitations and non-local quantum entanglement1,2,3,4. A paradigmatic example of a two-dimensional (2D) QSL is the seminal Kitaev honeycomb model (KHM)5, which was initially derived to illustrate the basic ideas of topological quantum computation6. Remarkably, the model has an exact solution which shows that its excitations are free Majorana fermions with a Dirac dispersion and gapped conserved plaquette fluxes which couple to the Majoranas via a static Z2 gauge field. In the context of frustrated magnetism research, the KHM provided a first rigorous example how a QSL with fractionalized excitations and emergent gauge fields can emerge in a concrete microscopic 2D spin model.

In the last years, the KHM has transformed from a theoretical toy model to one of experimental relevance because a flurry of spin-orbit-coupled 4d and 5d transition metal compounds7,8,9,10,11,12 has been proposed as candidates for realizing its bond-anisotropic Ising interactions. Remarkably, experiments have also observed signatures of the proximate Kitaev spin liquid (KSL) in several materials with effective spin 1/2 moments, such as α-RuCl313,14,15,16,17,18 and (Na1−xLix)2IrO319,20, despite the residual zigzag ordered state which appears at low temperature21,22 because of additional interactions, e.g., an off-diagonal symmetric Γ exchange13. However, the exchange frustration of the KHM is not restricted to spin 1/2 and recently some promising realizations of higher-spin Kitaev materials have been proposed based upon 3d orbitals23,24,25, in which the QSL-disrupting non-Kitaev exchanges might be reduced26,27,28. In particular, a microscopic derivation of the S=3/2 KHM model with an extra single ion anisotropy (SIA) has been established for the quasi 2D systems CrI3 and CrGeTe323,24.

After the original proposal of the S=1/2 KHM5, much effort has been devoted to investigating the KHM models for S > 1/2, which have not found an exact solution29,30,31,32,33,34,35,36. Nevertheless, Baskaran et al.30 showed early on that a generic spin-S KHM still has conserved Z2 fluxes for each elementary hexagon and suggested via a semiclassical analysis that the ground state of KHMs for all values of S exhibits a homogeneous flux configuration in which all values of Z2 fluxes are + 130. Subsequently, it has been proposed that the ground states of the S > 3/2 KHMs are Z2 QSLs described by an effective toric code on a honeycomb superlattice, but the employed semi-classical analysis breaks down precisely at S=3/234. The S=1 KHM is amenable to numerical investigations and studies using exact diagonalization33 and density matrix renormalization group (DMRG)35 point to a gapless QSL ground state, whereas a tensor network approach proposes a gapped QSL for the isotropic model36. Overall, the S=3/2 KHM seems to be the least understood of all Kitaev models — the conserved plaquette fluxes alone do not help to gain an analytical understanding and a high density of low energy excitations lead to strong finite-size effects for numerical investigations.

Here, we report new exact properties of the S=3/2 KHM and provide a systematic understanding of its ground-state phase diagram and excitations. We introduce an SO(6) Majorana representation for the spin-3/2’s which permits an exact mapping of the spin model to one of fermions coupled to a static Z2 gauge field. The latter determines the conserved plaquette flux just like in the original S=1/2 KHM. Within a given gauge field configuration, the Hamiltonian still contains quartic and even sextic fermion interaction terms but we construct a parton mean-field (MF) theory which turns out to be even quantitatively reliable. Our theory is furthermore corroborated by the addition of an extra [001] SIA to the S=3/2 KHM, which still preserves the conservation of the Z2 fluxes and allows us to map out the phase diagram consisting of two gapless Dirac and two gapped QSLs. In the limit of large [001] SIA, we find an exact solution of an effective S=1/2 KSL, which is continuously connected to one of the two gapless phases of the pure S=3/2 KHM. We also investigate the model using DMRG37,38 and find the numerical results to be in quantitative agreement with the predictions from our parton MF theory. In the presence of a weak [111] SIA, we can derive an effective Hamiltonian within the zero-flux sector and argue that a chiral KSL is established.

Results

Model Hamiltonian

The Hamiltonian of the S=3/2 KHM reads

$${{{{{{{\mathcal{H}}}}}}}}=\mathop{\sum}\limits_{{\langle ij\rangle }_{a}}{J}_{a}{S}_{i}^{a}{S}_{j}^{a},$$
(1)

where \({S}_{i}^{a}\) (a = x, y, z) are three components of an S=3/2 spin at site i and 〈ij〉a denotes the nearest neighbor (NN) bonds of a-type S=3/2 Ising interactions (see Fig. 1). There exist commuting plaquette operators Wp for each hexagon p (see Fig. 1) as \({W}_{p}\equiv -{e}^{i\pi ({S}_{1}^{x}+{S}_{2}^{y}+{S}_{3}^{z}+{S}_{4}^{x}+{S}_{5}^{y}+{S}_{6}^{z})}\)30. By noticing that \([{W}_{p},{{{{{{{\mathcal{H}}}}}}}}]=0\), the total Hilbert space of Hamiltonian (1) can be divided into orthogonal sectors characterized by flux configurations {wp = ± 1}, where wp is the eigenvalue of Wp.

Fig. 1: Graphic representation of the S=3/2 KHM (3) and the SO(6) Majorana representation.
figure 1

Down (up) triangles stand for S=3/2 spins on A (B) sublattice. Each spin is represented by SO(6) Majoranas, e.g., three gauge Majoranas ηx,y,z (circles) and three itinerant Majoranas θx,y,z (dots). The blue, green, and red bold lines denote the static Z2 gauge fields ux, uy, and uz for x-, y-, and z-bond Ising interactions, respectively. The plaquette operator \({W}_{p}\equiv -{e}^{i\pi ({S}_{1}^{x}+{S}_{2}^{y}+{S}_{3}^{z}+{S}_{4}^{x}+{S}_{5}^{y}+{S}_{6}^{z})}\) can be expressed as the product of \({u}_{ij}^{a}\) around hexagon p. The gray dashed line stands for the mirror symmetry Mz with Jx = Jy.

Full size image

SO(6) Majorana representation

We introduce three gauge Majorana fermions \({\eta }_{i}^{a}\) and three itinerant Majorana fermions \({\theta }_{i}^{a}\)(a = x, y, z), to obtain the SO(6) Majorana representation for spin-3/2’s39,40,41,42,43,44,45,46,47,48,49,50,51,52: \({S}_{i}^{a}=\frac{i}{4}{\epsilon }_{abc}{\eta }_{i}^{b}{\eta }_{i}^{c}-\frac{i}{2}{\eta }_{i}^{a}{\tilde{\theta }}_{i}^{a},\) where \({\tilde{\theta }}_{i}^{x(y)}={\theta }_{i}^{z}-(+)\sqrt{3}{\theta }_{i}^{x}\), \({\tilde{\theta }}_{i}^{z}=-2{\theta }_{i}^{z}\), and ϵabc is the Levi-Civita tensor (summation over repeated indices throughout). This parton representation doubly enlarges the Hilbert space and the physical Hilbert space of spin-3/2’s can be restored by imposing the local constraint \({D}_{i}=i{\eta }_{i}^{x}{\eta }_{i}^{y}{\eta }_{i}^{z}{\theta }_{i}^{x}{\theta }_{i}^{y}{\theta }_{i}^{z}=1\). One can then obtain \(i{\eta }_{i}^{b}{\eta }_{i}^{c}={\epsilon }_{abc}{\eta }_{i}^{a}{\theta }_{i}^{x}{\theta }_{i}^{y}{\theta }_{i}^{z}\) and rewrite the spin operators as

$${S}_{i}^{a}=\frac{i}{2}{\eta }_{i}^{a}\left({\theta }_{i}^{xyz}-{\tilde{\theta }}_{i}^{a}\right),\,(a=x,y,z),$$
(2)

where \({\theta }_{i}^{xyz}=-i{\theta }_{i}^{x}{\theta }_{i}^{y}{\theta }_{i}^{z}\).

Eq. (1) then becomes an effective Hamiltonian for Majoranas

$$H=-\frac{i}{4}\mathop{\sum}\limits_{{\langle ij\rangle }_{a}}{J}_{a}{u}_{ij}^{a}\left({\theta }_{i}^{xyz}-{\tilde{\theta }}_{i}^{a}\right)\left({\theta }_{j}^{xyz}-{\tilde{\theta }}_{j}^{a}\right),$$
(3)

where \({u}_{ij}^{a}=i{\eta }_{i}^{a}{\eta }_{j}^{a}\). One can now verify that \([{u}_{ij}^{a},{u}_{kl}^{b}]=0\) for all different bonds and \([{u}_{ij}^{a},H]=0\). Therefore, \({u}_{ij}^{a}\) with eigenvalues ± 1 is a static Z2 gauge field! Similar to the S=1/2 KHM, the plaquette operator Wp is exactly mapped to a product of \({u}_{ij}^{a}\) around hexagon p, e.g., \({W}_{p}={u}_{12}^{z}{u}_{32}^{x}{u}_{34}^{y}{u}_{54}^{z}{u}_{56}^{x}{u}_{16}^{y}\) (see Fig. 1). In a fixed Z2 gauge field configuration the Hamiltonian only depends on the itinerant Majoranas \({\theta }_{i}^{a}\).

The microscopic derivation of the S=3/2 KHM introduced in Ref. 23,24,25 suggests that it is usually accompanied by an extra [111] SIA term of the form \({\left({S}_{i}^{c}\right)}^{2}\) with \({S}_{i}^{c}=\frac{1}{\sqrt{3}}\left({S}_{i}^{x}+{S}_{i}^{y}+{S}_{i}^{z}\right)\). In addition to the [111] SIA, we also consider a simplified [001] SIA term and focus on the Hamiltonian

$${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{D}}}}}}}}}={{{{{{{\mathcal{H}}}}}}}}+\mathop{\sum}\limits_{i}\left[{D}_{z}{({S}_{i}^{z})}^{2}+{D}_{c}{\left({S}_{i}^{c}\right)}^{2}\right].$$
(4)

Note that \([{W}_{p},\,{\sum }_{i}{\left({S}_{i}^{z}\right)}^{2}]=0\) but the [111] SIA breaks the conservation of fluxes, namely, \([{W}_{p},\,{\sum }_{i}{\left({S}_{i}^{c}\right)}^{2}]\,\ne\, 0\). Therefore we always treat Dc as a small perturbation to ensure that the system is close to the Kitaev limit.

The [001] SIA limit

First, we focus on the Dc = 0 limit with conserved fluxes. Since the local S=3/2 states of \(\left|{S}_{i}^{z}=\pm\! \frac{3}{2}\right\rangle\) \(\left(\left|{S}_{i}^{z}=\pm \!\frac{1}{2}\right\rangle \right)\) will be energetically favored when Jz (Dz) dominates, we expect that the competition of Jz and Dz leads to a rich phase diagram. Moreover, below we show that for Dz → ∞, we can recover an effective S=1/2 KSL.

By using \({\left({S}_{i}^{z}\right)}^{2}=-i{\theta }_{i}^{x}{\theta }_{i}^{y}\) (a constant of 5/4 has been omitted), the effective Majorana Hamiltonian can be divided into two parts. One is a quadratic Hamiltonian \({H}^{(2)}(\{u\})\equiv \frac{-i}{4}{\sum }_{{\langle ij\rangle }_{a}}{J}_{a}{u}_{ij}^{a}{\tilde{\theta }}_{i}^{a}{\tilde{\theta }}_{j}^{a}-i{D}_{z}{\sum }_{i}{\theta }_{i}^{x}{\theta }_{i}^{y}\), and the other is an interacting Hamiltonian consisting of quartic and sextic terms, which we treat within a MF analysis. In our decoupling scheme, the MF Hamiltonian reads

$${H}_{{{{{{{{\rm{MF}}}}}}}}}(\{u\})= \; {H}^{(2)}(\{u\})+\mathop{\sum}\limits_{{\langle ij\rangle }_{a}}\frac{i{J}_{a}{u}_{ij}^{a}}{4}\left\{\frac{{\epsilon }_{opq}{\epsilon }_{rst}}{4}\langle {\theta }_{i}^{o}{\theta }_{i}^{p}{\theta }_{j}^{r}{\theta }_{j}^{s}\rangle {\theta }_{i}^{q}{\theta }_{j}^{t}\right.\\ +\frac{{\epsilon }_{lmn}}{2}\left({\theta }_{i}^{m}{\theta }_{i}^{n}\langle {\theta }_{i}^{l}{\theta }_{j}^{x}{\theta }_{j}^{y}{\theta }_{j}^{z}\rangle -{\theta }_{j}^{m}{\theta }_{j}^{n}\langle {\theta }_{j}^{l}{\theta }_{i}^{x}{\theta }_{i}^{y}{\theta }_{i}^{z}\rangle \right)\\ \left.+\frac{{\epsilon }_{uvw}}{2}\left[\left({Q}_{i}^{uv}{\theta }_{i}^{w}{\tilde{\theta }}_{j}^{a}-{{{\Delta }}}_{ij}^{w\tilde{a}}{\theta }_{i}^{u}{\theta }_{i}^{v}+i{Q}_{i}^{uv}{{{\Delta }}}_{ij}^{w\tilde{a}}\right)-(i\leftrightarrow j)\right]\right\}$$
(5)

with the on-site parameters \({Q}_{i}^{ab}\equiv -\langle i{\theta }_{i}^{a}{\theta }_{i}^{b}\rangle\) (a ≠ b) and bond parameters \({{{\Delta }}}_{ij}^{ab}\equiv \langle i{\theta }_{i}^{a}{\theta }_{j}^{b}\rangle\)\(\left({{{\Delta }}}_{ij}^{a\tilde{b}}\equiv \langle i{\theta }_{i}^{a}{\tilde{\theta }}_{j}^{b}\rangle \right)\). In accordance with Wick’s theorem, the average of quartic terms can be decoupled as \(\langle {\theta }_{i}^{o}{\theta }_{i}^{p}{\theta }_{j}^{r}{\theta }_{j}^{s}\rangle =-{Q}_{i}^{op}{Q}_{j}^{rs}+{{{\Delta }}}_{ij}^{or}{{{\Delta }}}_{ij}^{ps}-{{{\Delta }}}_{ij}^{os}{{{\Delta }}}_{ij}^{pr}\) and \(\langle {\theta }_{i}^{l}{\theta }_{j}^{x}{\theta }_{j}^{y}{\theta }_{j}^{z}\rangle ={{{\Delta }}}_{ij}^{lx}{Q}_{j}^{yz}+{{{\Delta }}}_{ij}^{ly}{Q}_{j}^{zx}+{{{\Delta }}}_{ij}^{lz}{Q}_{j}^{xy}\). Eventually, HMF is parameterized by MF parameters Q and Δ, which we determine self-consistently (see Supplementary Note 1). In the presence of Z2 flux conservation, we can restrict to antiferromagnetic couplings of Ja > 0 since in the parton level a sign change of Ja → − Ja can be resolved by \({u}_{ij}^{a}\to -{u}_{ij}^{a}\). For simplicity, we focus on the case of Jx = Jy = 1 with a mirror symmetry Mz across the z-bonds (see Fig. 1) which leads to a vanishing of Qyz,zx = 0. The only nonzero on-site parameter is \({Q}^{xy}\equiv -\langle i{\theta }^{x}{\theta }^{y}\rangle =\langle {\left({S}^{z}\right)}^{2}\rangle\) which is a time-reversal-invariant spin quadrupolar component characterizing the MF ground states of HMF({u}) (see Supplementary Note 2).

Following the proposal of Ref. 30 that for generic spin-S KHMs the ground-state always exhibits a zero-flux configuration with {wp} = 1, we will mainly focus on the zero-flux sector and fix {u} = 1 for studying the Majorana excitations of HMF. Note that the quadratic Hamiltonian H(2)({u}) alone always energetically favors a zero-flux state.

The Hamiltonian HMF({u} = 1) displays four phases which are characterized by their distinct Majorana excitations and the value of the quadrupolar parameter Qxy, as shown in Fig. 2a. (i) At the isotropic point of Jz = 1 and Dz = 0, the ground state is a Dirac QSL with Qxy = 0. The Majorana band structure of HMF({u} = 1) at the isotropic point is almost the same as that of the quadratic Hamiltonian H(2)({u} = 1), except that the exact flat bands populated by θy in Fig. 3a acquire a very narrow dispersion as shown in Fig. 3b. Consequently, in this phase HMF({u} = 1) has a Dirac point at the K-point, around which there exist two gapless Majorana bands crossing linearly at zero energy but one of whose velocity is close to zero. (ii) In the A0 phase, the ground state is a Dirac QSL coexisting with a spin quadrupolar parameter Qxy < 0. Because of a nonzero Qxy < 0, only one branch of gapless spin excitations remains around the Dirac point (see Fig. 3c). (iii) In the Az phase, the effect of Dz remains, but a relatively large anisotropy of Jz gaps out all Majorana excitations and the ground state is a gapped QSL coexisting with a spin quadrupolar parameter Qxy < 0. (iv) In the B phase, the dominant Jz leads to a positive Qxy > 0 and all Majorana excitations are gapped. Fig. 3d shows a typical Majorana band structure for the B phase. We conclude that the transition between the B phase and the A0 (Az) phase is of first-order since Qxy shows a discontinuous bump at the phase boundary in Fig. 2b, c.

Fig. 2: The ground-state phase diagram and the values of parameter Qxy.
figure 2

a The MF phase diagram of Hamiltonian (5) in zero-flux sector on a 36 × 36 torus. The A0 phase is a Dirac QSL with spin quadrupolar parameter Qxy < 0. In the Az (B) phase, the Majorana excitations are gapped with Qxy < 0 (Qxy > 0). At the isotropic point, the ground state is a QSL with Qxy = 0. The bold blue line at Dz = ∞ with Jz < 8 (Jz > 8) represents the effective gapless (gapped) S=1/2 KSL. The gapless phases in S=3/2 and S=1/2 KHMs can continuously connect to each other through the A0 phase without energy gap opening. The orange stars (green triangles) represent the ground states obtained by DMRG on a 3 × 4 torus with zero-flux (disordered-flux) configurations with bond dimension χ = 4000. The values of Qxy as a function of Jz with Dz = 0 (b) and Dz with Jz = 7 (c). The computations of DMRG and parton MF theory are performed on a 3 × 4 torus.

Full size image
Fig. 3: Band structure of itinerant Majorana fermions.
figure 3

a The Majorana band structure of the quadratic Hamiltonian H(2) at the isotropic point Jz = 1 and Dz = 0. The Majorana band structures of the MF Hamiltonian HMF with (b) Jz = 1 and Dz = 0, (c) Jz = 1.2 and Dz = 0.4, and (d) Jz = 1.4 and Dz = 0. Here Jx = Jy = 1, and a zero-flux configuration of {u} = 1 is used. The inset in (b) shows the zoom-in around the K point.

Full size image

Effective S=1/2 KSL

A key advantage of using the [001] SIA is that the Hamiltonian can be solved exactly in the limit of Dz → ∞, because the high energy local states of \(\left|{S}_{i}^{z}=\pm \frac{3}{2}\right\rangle\) are removed and the ground-state subspace of Hamiltonian (4) is spanned by the local states of \(\left|{S}_{i}^{z}=\pm 1/2\right\rangle\) only. Within the framework of the SO(6) Majorana representation, the itinerant Majoranas \({\theta }_{i}^{x}\) and \({\theta }_{i}^{y}\) are paired up subjected to the constraint \(i{\theta }_{i}^{x}{\theta }_{i}^{y}=1\) (Qxy = − 1). Then the three S=3/2 operators in Eq. (2) projected onto the subspace of \(i{\theta }_{i}^{x}{\theta }_{i}^{y}=1\) become

$${S}_{i}^{z}\simeq \frac{i}{2}{\eta }_{i}^{z}{\theta }_{i}^{z},\quad \quad {S}_{i}^{x(y)}\simeq i{\eta }_{i}^{x(y)}{\theta }_{i}^{z}.$$
(6)

Obviously, Eq. (6) is equivalent to Kitaev’s original four-Majorana representation5. It is remarkable that an effective S=1/2 KHM with a renormalized coupling constant Jz → Jz/4 emerges in our S=3/2 Hamiltonian (3) for Dz → ∞ with an effective gapless (gapped) S=1/2 KSL for Jz < (>)8. This connection gives additional support to the assumption of a zero-flux ground state and our MF treatment, which is known to exactly capture the phase diagram and nature of excitations of the S=1/2 KHM53,54. Indeed, we find that the gapless KSL of the emergent S=1/2 KHM is continuously connected to the gapless A0 phase of the pure S=3/2 KHM in Fig. 2a.

Numerical results

Since Lieb’s theorem55 may not be applicable for the interacting Hamiltonian (3), we examine the zero-flux ground-state configuration, which is a pivotal assumption in our parton theory. We find that the ground states in our DMRG simulations always exhibit a zero-flux configuration in the A0 and Az phases. In contrast, for the B phase, DMRG does not converge to a unique ground-state flux configuration but instead leads to a disordered-flux ground state in which the measured flux for each plaquette is neither 1 nor −1. The data points for different DMRG-obtained ground-state flux configurations are shown in Fig. 2a.

The disordered-flux ground states obtained by DMRG indicate an extremely small flux gap above the zero-flux state in the B phase. This flux gap can also be evaluated within our MF theory in the B phase. To connect to our DMRG simulations, we have performed the MF calculations on the same 3 × 4 torus. We find that the energy of a pair of neighboring fluxes is E2fluxes ≃ 10−6 for Jz = 1.2 and Dz = 0, which is as small as the corresponding DMRG truncation error ϵ ≃ 10−6. The flux gap E2fluxes rapidly decreases as Jz increases, which explains why DMRG fails to capture the conserved Z2 flux in the B phase.

Next, we investigate the local spin quadrupolar parameter \({Q}_{i}^{xy}\) using DMRG and find that the \({Q}_{i}^{xy}\) are spatially uniform (on the torus and in the bulk of cylinders) as assumed in our parton description. A remarkable observation is that, if we ignore small discrepancies near the phase boundaries, the values of the DMRG-obtained Qxy are even in quantitative agreement with those given by the parton MF theory within the zero-flux sector, as shown in Fig. 2b, c. For the purpose of comparison, we have chosen the same lattice geometry, e.g., a 3 × 4 torus, to calculate the MF and DMRG values of Qxy. The difference between the MF Qxy for a 3 × 4 torus and its extrapolation to an infinite lattice is of the magnitude of 10−2 (except near the phase boundaries), thus finite-size effects for Qxy are small. This remarkable quantitative agreement demonstrates that our SO(6) Majorana MF theory provides a compelling scenario for describing the S=3/2 KHM. For example, we can now understand why the S=3/2 KHM at the isotropic point is so challenging for numerical methods, e.g., the extreme system-size dependence encountered in our DMRG simulations, because the almost flat Majorana bands, see Fig. 3b, lead to a large pile up of close to zero energy states.

The [111] SIA limit

Next, we study the experimentally more relevant case, i.e., Jx = Jy = Jz = J, Dz = 0, and Dc ≪ ∣J∣. A finite [111] SIA breaks the flux conservation, leading to a dynamical gauge field. In analogy to the procedure of treating a magnetic field in the S=1/2 KHM5, we circumvent this problem by deriving an effective three-body quadrupolar interaction within the zero-flux sector. This can be further motivated by noticing that \({\left({S}_{i}^{c}\right)}^{2}=-\frac{i}{3}\left({\eta }_{i}^{x}+{\eta }_{i}^{y}+{\eta }_{i}^{z}\right){\theta }_{i}^{y}\) plays a similar role as the [111] magnetic field in the S = 1/2 KHM5. The effective term H(3) described by three-body quadrupolar interactions is represented by

$${H}^{(3)}=\kappa \mathop{\sum}\limits_{{\left\langle ij\right\rangle }_{a}{\left\langle jk\right\rangle }_{b}}{u}_{ij}^{a}{u}_{jk}^{b}\left(i{\theta }_{i}^{y}{\theta }_{j}^{z}\right)\left(i{\theta }_{j}^{x}{\theta }_{k}^{y}\right),$$
(7)

where \(\kappa \sim {D}_{c}^{3}/{{{\Delta }}}_{{{{{{{{\rm{flux}}}}}}}}}^{2}\) (Δflux ≈ 0.093J) and 〈ij〉α and 〈ij〉β are two NN bonds connected by site j (see Fig. 4a). Eq. (7) clearly commutes with the Z2 gauge fields but is quartic in the itinerant Majoranas. Its most general decoupling reads

$${H}_{{{{{{{{\rm{MF}}}}}}}}}^{(3)}(\{u\}=1)= \; \kappa \mathop{\sum}\limits_{{\left\langle ij\right\rangle }_{a}{\left\langle jk\right\rangle }_{b}}i\left({{{\Delta }}}_{ij}^{yz}{\theta }_{j}^{x}{\theta }_{k}^{y}+{{{\Delta }}}_{kj}^{yx}{\theta }_{j}^{z}{\theta }_{i}^{y}\right)\\ +\kappa \mathop{\sum}\limits_{{\left\langle ij\right\rangle }_{a}{\left\langle jk\right\rangle }_{b}}i\left(-{Q}_{j}^{zx}{\theta }_{i}^{y}{\theta }_{k}^{y}+{\xi }_{ik}^{yy}{\theta }_{j}^{x}{\theta }_{j}^{y}\right),$$
(8)

where \({\xi }_{ik}^{yy}=-\langle i{\theta }_{i}^{y}{\theta }_{k}^{y}\rangle\) is the hopping parameter on the 2nd NN bond. A nonzero Qzx breaks time-reversal symmetry (TRS) so that \({H}_{{{{{{{{\rm{MF}}}}}}}}}^{(3)}\) naturally describes a chiral KSL analogous to the S = 1/2 case5. A key difference is that here TRS is spontaneously broken since Eqs. (4) and (7) are even under time-reversal.

Fig. 4: Chiral KSL induced by [111] SIA.
figure 4

a Qzx as a function of κ. Inset: A sketch for the three-body quadrupolar interactions in Eq. (7). The blue, green, and red bonds stand for the x-, y-, and z-type S=3/2 Ising interactions, respectively. The summation in Eq. (7) takes place over the triangle with vertexes i, j, and k (and symmetry-equivalent ones). b The band structure and corresponding Chern numbers for the parton MF theory in Eq. (8) with κ = 0.01.

Full size image

\({H}_{{{{{{{{\rm{MF}}}}}}}}}^{(3)}\) favors the S = 3/2 chiral KSL, which undergoes a first-order phase transition on the parton MF level. For small κ = 0.01, the self-consistent solution converges to parameters Qzx ≈ 0.28 and ∣ξyy∣ ≈ 0.115. Fig. 4a presents the evolution of Qzx as a function of positive κ—solutions for negative κ are obtained by changing the signs of Qzx, ξyy, and Δyx(z).

Next, we study properties of this chiral KSL. The Majorana hybridization induced by a nonzero Qzx narrows all dispersions and separates the six bands from each other, see Fig. 4b. We evaluate the topological characteristics of these bands in terms of the Chern number, which is

$${C}^{\prime}{{{{{{{\rm{s}}}}}}}}=\left(1,0,-1,1,0,-1\right)$$

for the lowest to the highest band. Notice that in contrast to the S = 1/2 chiral KSL, the sum of Chern numbers over the negative energy bands is zero. Therefore, no quantized thermal Hall conductivity is expected in the low temperature limit, but an activated signal emerges for increasing temperatures.

Discussion

In summary, we have studied the ground-state phase diagram and excitations of the S=3/2 KHM with additional SIA terms. We employed a parton theory based on the SO(6) Majorana representation of spin=3/2’s, which is supported by DMRG simulations. We have shown that the conserved flux for each honeycomb plaquette can be represented exactly via a static Z2 gauge field similar to the well-known S=1/2 KHM, which is key for identifying the correct MF decoupling of the parton description. For a [001] SIA, DMRG calculations are shown to agree with our self-consistent MF theory qualitatively and even quantitatively. We uncover a rich phase diagram characterized by distinct Majorana excitations and different phases with spin quadrupolar parameter \({Q}^{xy}=\langle {\left({S}^{z}\right)}^{2}\rangle\): (i) a gapless Dirac QSL with Qxy = 0 and an additional almost flat Majorana band close to zero energy at the isotropic point (Jz = 1, Dz = 0); (ii) a gapless Dirac QSL with Qxy < 0 in the A0 phase; (iii) a gapped QSL with Qxy < 0 in the Az phase; and (iv) a gapped QSL with Qxy > 0 in the B phase. For a dominating [001] SIA, the low energy sector of the S=3/2s reduces to effective S=1/2s which allows us to continuously connect the gapless A0 phase of the pure S=3/2 KHM to that of the well-known Dirac QSL of the S=1/2 KHM. In the B phase, we found that DMRG fails to capture the conservation of Z2 fluxes because of an extremely small flux gap above the zero-flux ground state, which is again accounted for in our parton MF theory. In the presence of a small [111] SIA, we establish an effective model in the zero-flux sector with three-body quadrupolar interactions. Our parton MF study indicates an emergent chiral KSL spontaneously breaking TRS.

We argue that our SO(6) Majorana parton theory efficiently describes the different QSLs of the S=3/2 KHM, which also provides a compelling scenario for explaining the difficulties encountered in the numerical studies. Hence, it will provide an good starting point for studying the robustness of the QSL regimes with respect to additional terms in the Hamiltonian, for example different exchange interactions and the SIA relevant for microscopic realizations of the S=3/2 KHM24. The connection to the S=1/2 KHM indicates that in particular the ferromagnetic QSLs will be very fragile and, in general, the formation of conventional magnetic order will be further facilitated because of flux-fermion bound state formation involving the almost flat Majorana bands. In that context, large-scale numerical studies for S=3/2 KHM with non-Kitaev interactions like [111] SIA and Heisenberg terms are still highly demanded, and the quality of our parton MF states can be further improved by efficient tensor network representation with Gutzwiller projection56,57,58 or by including different flux sectors in the variational ansatz59.

In the future, it will be worthwhile to study the effect of applying a magnetic field and the ensuing QSL phases of the S=3/2 KHM. Similarly, it would be desirable to generalize our Majorana parton construction to higher-spin systems whose dimension of local Hilbert space is 2n (with n integer), i.e., the S=7/2 KHM could possibly have a similar exact static Z2 gauge field permitting an efficient description via an eight Majorana representation for spin-7/2’s.

Methods

In order to examine the reliability of our parton MF theory in the case of a [001] SIA, we employ state-of-the-art DMRG method37,38 to study the ground state of Hamiltonian (4). DMRG is a very powerful numerical approach for studying 1D strongly-correlated systems. To perform DMRG calculations on a 2D honeycomb lattice of L1 × L2 unit cells, we consider a cylindrical geometry for which the periodic boundary condition (PBC) is imposed along the shorter direction (e.g., the circumference L1), while the longer (e.g., the length L2) is left open. Moreover, we also adopt small tori with PBCs along both directions to strictly preserve the A/B sublattice and translational symmetries. The DMRG simulations are performed on a 3 × 4 torus as well as a 4 × 8 cylinder (L1 = 4). The bond dimension of DMRG is kept as large as χ = 4000, resulting in a typical truncation error of ϵ ≃ 10−6 (ϵ ≃ 10−4 close to the isotropic point Jz = 1 and Dz = Dc = 0). In general, we encounter significant finite-size effects in our numerical studies in contrast to checks on the S=1/2 and S=1 KHMs.