## Abstract

A family of spin–orbit coupled honeycomb Mott insulators offers a playground to search for quantum spin liquids (QSLs) via bond-dependent interactions. In candidate materials, a symmetric off-diagonal Γ term, close cousin of Kitaev interaction, has emerged as another source of frustration that is essential for complete understanding of these systems. However, the ground state of honeycomb Γ model remains elusive, with a suggested zigzag magnetic order. Here we attempt to resolve the puzzle by perturbing the Γ region with a staggered Heisenberg interaction which favours the zigzag ordering. Despite such favour, we find a wide disordered region inclusive of the Γ limit in the phase diagram. Further, this phase exhibits a vanishing energy gap, a collapse of excitation spectrum, and a logarithmic entanglement entropy scaling on long cylinders, indicating a gapless QSL. Other quantities such as plaquette-plaquette correlation are also discussed.

### Similar content being viewed by others

## Introduction

The ongoing search for exotic magnetic states in highly frustrated antiferromagnets^{1,2,3,4,5,6} has been extended to a new class of correlated materials with a two-dimensional honeycomb structure^{7,8,9,10} and its three-dimensional variants^{11}. It is suggested that bond-dependent interactions could be realized in the spin–orbit coupled Mott insulators with the aforementioned lattice geometry^{12}. In particular, the Kitaev honeycomb model exhibits a novel Kitaev quantum spin liquid (QSL), which hosts fractionalized Majorana fermions and flux excitations^{13}. Realization of Kitaev interaction in real materials was first proposed in iridates^{14,15,16}, and then turned toward *α*-RuCl_{3} in which Ru^{3+} ions are arranged in a honeycomb lattice and carry effective spin-1/2 particles^{7,9}. Although *α*-RuCl_{3} displays long-range zigzag magnetic order at low temperature^{17,18,19,20,21,22}, it is argued to be proximate to the Kitaev QSL owing to the broad continuum of magnetic excitations identified in Raman scattering^{23,24} and inelastic neutron scattering^{9,25,26,27}.

In spite of massive research efforts, it has been challenging to determine exchange parameters of the proposed spin Hamiltonian for *α*-RuCl_{3} (see refs. ^{28,29} and references therein). However, there is a broad consensus on a sizable off-diagonal Γ interaction^{30,31}, which is antiferromagnetic (AFM) and is potentially comparable to the celebrated Kitaev interaction^{26,32,33}. Crucially, it is shown that the Γ interaction could help enhance the mass gap of Majorana fermions^{34} and is responsible for the strongly anisotropic responses to the magnetic field observed in *α*-RuCl_{3} provided that the Landé *g*-factor anisotropy is modest^{28,30,35}. In contrast to the Kitaev model^{13}, analytical solution of the honeycomb Γ model has not been found yet^{36}. Previous classical studies have demonstrated that its ground state is a classical spin liquid^{37} followed by a flux-ordered spin liquid, which is stabilized in a finite temperature window^{38}. Given the infinite classical ground-state degeneracy^{37}, determining the precise quantum nature of Γ model is nontrivial, and existing numerical works have already led to conflicting results. Parallel works by exact diagonalization^{39} and density-matrix renormalization group (DMRG) study of a cylinder with a width of three unit cells^{40} both claim that the ground state is a nonmagnetic phase. A variational Monte Carlo simulation, on the other hand, suggests that it is a zigzag order^{41}. Furthermore, a recent study proposes that it is a nematic paramagnet that spontaneously breaks the lattice rotational symmetry^{42}.

In this work, we study a model which consists of the Γ term and of a staggered Heisenberg (\(\tilde{J}\)) interaction along the bonds, dubbed the bond-modulated\(\tilde{J}\)–Γ model (see Eq. 1). Depending on the sign of \(\tilde{J}\), it could either favor the zigzag order (\(\tilde{J}\,>\,0\)) or stripy order (\(\tilde{J}\,<\,0\)). If the ground state of Γ model is a zigzag ordered phase, then the zigzag order protruding from the pure Γ limit should compete with and survive up to a finite ferromagnetic \(\tilde{J}\) interaction. Otherwise, there will be an intermediate phase sandwiched between the two magnetically ordered states. Thus, this model works as a virtuous arena to clarify the debates by unfolding the competing states, although it is not a description of any particular material. By employing the DMRG method on both finite cylinders with circumferences of up to 10 sites and *C*_{3}-symmetric hexagonal clusters^{43,44}, we identify a disordered state in between. This phase manifests characters of a gapless QSL including a dense excitation spectrum, logarithmic entanglement entropy scaling, and short-range plaquette–plaquette correlation. The pure Γ limit belongs to this QSL and is separated from the zigzag order by a first-order transition.

## Results

### Model

The Hamiltonian of the bond-modulated \(\tilde{J}\)–Γ model reads

where \({S}_{{\rm{i}}}^{\gamma }\) (*γ* = *x*, *y*, *z*) is the *γ*-component of a spin-1/2 operator at site *i*, and *α* and *β* are the two other bonds on a honeycomb lattice. *η*_{γ} = 1 for the bond 〈*i**j*〉_{γ} along the horizontal direction and equals to −1 otherwise (see Fig. 1). \(\tilde{J}\) and Γ are parameterized using *ϑ* ∈ [0, *π*] so as to \(\tilde{J}=\cos \vartheta\) and \({{\Gamma }}=\sin \vartheta \,(\ge 0)\).

In what follows, we carry out a hierarchical study of Eq. (1) to provide multi-faceted evidences of the gapless QSL nature of Γ magnet. We start by mapping out the classical phase diagram via the parallel tempering Monte Carlo simulation^{45,46}, and conclude that the ground state of Γ model sits exactly at the classical transition point on the verge of the zigzag phase. This makes sense because the zigzag ordering belongs to the macroscopic ground-state manifold of the classical Γ model^{37}. Next, we show the energy reduction and sublattice magnetization within the linear spin-wave analysis (for a review, see ref. ^{47}). Afterwards, we present a quantum phase diagram obtained by large-scale DMRG calculations on various distinct cluster geometries.

### Classical phase diagram

Figure 2 shows the classical phase diagram of Eq. (1) obtained by Monte Carlo simulations^{45,46}, coincide exactly with the subsequent results of energy optimization method (see "Methods” section). Due to the bond-modulated *η*_{γ}-term, the conventional zigzag and stripy orderings perpendicular to the **Z** bonds are induced when *ϑ*/*π* is 0 or 1, respectively. By introducing AFM Γ interaction, the ground state becomes more competitive, triggering the possibility of other magnetic orderings in the moderate interaction regime. The ground-state energy *e*_{g} = *E*_{g}/(*N**S*^{2}) (*E*_{g} is the total energy) is shown in Fig. 2a, while selected spin configurations of the corresponding phases are depicted in Fig. 2b–d. In the phase diagram, the leftmost is the zigzag order with \({e}_{{\rm{g}}}^{{\rm{zz}}}=-(2{{\Gamma }}+3\tilde{J})/2\) and its magnetic moment direction is \({\bf{n}}\ [11\bar{1}]\). The rightmost is occupied by the stripy order with \({e}_{{\rm{g}}}^{{\rm{st}}}=-({{\Gamma }}-3\tilde{J})/2\). Its spins are perpendicular to \({\bf{n}}\,[11\bar{1}]\), but could vary freely in the plane spanned by \(\widetilde{{\bf{a}}}\,[112]\) and \(\widetilde{{\bf{b}}}\,[1\bar{1}0]\), showing an emergent continuous symmetry. Further, an extensive intermediate region appears in between. It is dominated by a so-called mixed phase in which the AFM order and two twining zigzag orders are degenerate with energy \({e}_{{\rm{g}}}^{{\rm{mixed}}}=-(2{{\Gamma }}-\tilde{J})/2\). Here, twining zigzag orders refer to the other two zigzag orders whose spin orientations are different from the one shown in Fig. 2b (for spin configurations, see Supplementary Note 1). The zigzag–mixed transition takes place exactly at \({\vartheta }_{{\rm{t,l}}}^{{\rm{cl}}}/\pi =0.5\), reflecting the classical spin liquid of the Γ model^{37}. There is no direct transition between the mixed phase and the stripy phase expected to occur at \({\vartheta }_{{\rm{t,r}}}^{{\rm{cl}}}/\pi\) = \(1-\frac{1}{\pi }{\rm{atan}}\ 2\) ≈ 0.6476. Instead, a noncollinear phase (see Fig. 2d) with *e*_{g} = \(-\sqrt{{\tilde{J}}^{2}+{{{\Gamma }}}^{2}/16}\) − \({{\Gamma }}/\sqrt{2}\) appears in a narrow window of *ϑ*/*π* that is less than 0.02, see inset of Fig. 2a.

### Spin-wave theory

To understand the role played by quantum fluctuations and for the sake of comparison with the DMRG results later, we have performed the linear spin-wave calculation^{47} based on the quadratic Hamiltonian \(\bar{{\mathcal{H}}}={E}_{{\rm{g}}}[{S}^{2}\to S(S+1)]+\frac{S}{2}{\sum }_{{\boldsymbol{q}}}{\psi }_{{\boldsymbol{q}}}^{\dagger }{{\mathcal{M}}}_{{\boldsymbol{q}}}{\psi }_{{\boldsymbol{q}}}\), where \({\psi }_{{\boldsymbol{q}}}^{\dagger }=\left(\right.{a}_{{\boldsymbol{q}}}^{\dagger },{b}_{{\boldsymbol{q}}}^{\dagger },\cdots \ ,{a}_{-{\boldsymbol{q}}},{b}_{-{\boldsymbol{q}}},\cdots \left)\right.\) is the Nambu spinor, and \({{\mathcal{M}}}_{{\boldsymbol{q}}}\) is a 2 × 2 block matrix (see “Methods” section and Supplementary Note 2). There are four spin-wave dispersion branches *ω*_{qυ} (*υ* = 1–4) for the four-sublattice (*n*_{s} = 4) zigzag and stripy orderings. In the zigzag order, there exists a magnon gap Δ at **M** point in the Brillouin zone (see inset of Fig. 3a). When approaching Γ limit, \(\tilde{J}/{{\Gamma }}\ll 1\), the lowest magnon branch is softened and the gap vanishes as \({{\Delta }}/{{\Gamma }}\simeq \frac{\sqrt{30}}{3}\sqrt{\tilde{J}/{{\Gamma }}}\). Therefore, the zigzag order could only survive for AFM \(\tilde{J}\) (i.e., *ϑ*/*π* < 0.50), beyond which the magnon branch becomes imaginary and should be terminated by a transition.

The magnon spectra for the representative stripy order with *ϑ*/*π* = 0.75 are shown in Fig. 3a where the wave vector ** q** is parameterized in units of (

*h*,

*k*) as

**= \(\left(\right.\frac{2\pi }{{a}_{1}}h,\frac{2\pi }{{a}_{2}}k\left)\right.\)**

*q*^{15}. The spectra are symmetric with the middle of the

**Γ**–

**M**line, so the

**Γ**and

**M**points are equivalent. Due to the emergent continuous symmetry of the classical stripy order, the magnon spectra are gapless. In the presence of quantum fluctuations, however, the degeneracy is lifted via order-by-disorder mechanism

^{48}, selecting two of them that are either parallel or antiparallel to \(\widetilde{{\bf{b}}}\) axis (see inset of Fig. 3b). To illustrate it, we firstly define the quantum energy correction \({{\Delta }}E(\phi )=S{e}_{{\rm{g}}}^{{\rm{st}}}+\frac{S}{2{n}_{{\rm{s}}}}{\sum }_{\upsilon }\int \frac{{d}^{2}{\bf{q}}}{{(2\pi )}^{2}}{\omega }_{{\boldsymbol{q}}\upsilon }(\phi )\), where

*ϕ*is the angle in the \(\widetilde{{\bf{a}}}\)-\(\widetilde{{\bf{b}}}\) plane

^{49}. For

*ϑ*/

*π*= 0.75, which is deep in the stripy order, we show Δ

*E*(

*ϕ*) vs.

*ϕ*in the inset of Fig. 3b. The energy correction has its minima at

*ϕ*=

*π*/2 or 3

*π*/2, corresponding to the two mostly favored configurations at the quantum level. The energy barrier

*δ*

*E*, defined as the energy difference between Δ

*E*(0) and Δ

*E*(

*π*/2), is approximately 0.0175. The main panel of Fig. 3b shows energy barrier at different

*ϑ*/

*π*in the stripy order. When

*ϑ*/

*π*= 1.00 the energy barrier is zero, consistent with the gapless Goldstone modes thereof. Beyond that, the energy barrier is finite, indicating that the stripy order should also be twofold degenerate in the quantum case.

Due to the magnon instabilities of zigzag and stripy orderings, they could only exist in their classically allowed regions. The corresponding phase transitions could be illuminated by their sublattice magnetization. It is observed that magnetization of the zigzag order is almost saturated when *ϑ*/*π* < 0.4. As *ϑ*/*π* approaches 0.5, it undergoes a considerable suppression and the lowest branch of the magnetization nearly vanishes at \({\vartheta }_{{\rm{t,l}}}^{{\rm{cl}}}/\pi =0.50\) (see Supplementary Fig. 5). The stripy order is more stable and its magnetization only has a small reduction at \({\vartheta }_{{\rm{t,r}}}^{{\rm{cl}}}/\pi \approx 0.6476\). However, spin-wave energy in the mixed phase, say AFM order, is overwhelmingly higher than its neighbors. It thus implies that the genuine phase in the intermediate region should be different from its classical counterpart, imposing restrictions on the applicability of the spin-wave analysis.

### Intervening magnetically disordered state

As discussed, the spin-wave calculation fails in the intermediate regime, hence the quantum study is necessary. We have performed the standard DMRG computation on three XC clusters of 12 × 6 (*n* = 6), 16 × 8 (*n* = 8), and 20 × 10 (*n* = 10) and compute the ground-state energy *e*_{g} = *E*_{g}/*N* which is shown in Fig. 4. The energy curves in the middle are very flat, while they have two sharp downwarping when away from the middle region, leading to two well-marked kinks that are signals of first-order transitions. These discontinuous phase transitions could also be advocated by the entanglement entropy, which has a trend to jump as the system size is increased (see Supplementary Figs. 9 and 11). Therefore, the DMRG result supports an intermediate region that impedes a direct transition between the zigzag and stripy phases^{50,51}. For comparison, we also depict the spin-wave energy of the zigzag order (green belt) and stripy order (blue belt) in Fig. 4. It is clearly found that there is a further energy reduction of the zigzag order beyond the linear approximation, in accord with the dramatic suppression of magnetic order parameter which will be clarified later. Strikingly, as shown in the inset of Fig. 4, the energy *e*_{g} of Γ model (*ϑ*/*π* = 0.5) exhibits a nonmonotonic scaling behavior^{52}, indicative of a possible periodicity as revealed in the Kitaev model^{13}. At each fixed circumference *n*, the energy is linearly decreasing with length *L*_{x} of the cylinder. By varying the circumference *n* from 4 to 10, the extrapolated energy has an oscillation in a window of −0.357 < *e*_{g} < −0.352. Therefore, we estimate that the energy of Γ model is *e*_{g} = −0.354(3) in the thermodynamic limit.

In order to unveil the nature of the intermediate region and to pin down the precise phase boundaries, we resort to the magnetic order parameter, which is defined as \({M}_{{\rm{N}}}({\bf{Q}})=\sqrt{{{\mathbb{S}}}_{{\rm{N}}}({\bf{Q}})/N}\) where \({{\mathbb{S}}}_{{\rm{N}}}({\bf{Q}})\) is the magnetic structure factor with **Q** being the ordering wavevector (see “Methods” section for definition). The zigzag phase has a peak at **M** point in the Brillouin zone, while the stripy phase possesses both peaks at **M** and \({\bf{M}}^{\prime}\) points. Crucially, magnetic structure factor of the intermediate region is diffuse with a soft peak. Figure 5a displays order parameters *M*_{N}(**Q**) of the zigzag and stripy phases on four distinct XC clusters with a circumference ranging from 4 to 10. Akin to their spin-wave results, magnetization of the zigzag and stripy phases exhibits maxima at *ϑ*/*π* ≈ 0.25 and 0.75, respectively. This implies that these magnetic orderings are most stable, when Γ term is approximately of equal strength to the Heisenberg interaction. Away from these points, quantum fluctuations are enhanced so that the magnetic ordering in the intermediate region is dramatically suppressed, followed by an algebraically decay with the circumference *n* (see Fig. 5c, d). After a careful inspection of the finite-size effect, we conclude that the magnetization will disappear eventually as the best fitting gives *M* → 0.0 for Γ model (*ϑ*/*π* = 0.5).

The entire quantum phase diagram of Eq. (1) is presented in Fig. 5b. In addition to the conventional zigzag and stripy orderings, there is a disordered phase, which is later interpreted as a QSL, is stabilized in a large region between *ϑ*_{t,l} and *ϑ*_{t,r} with *ϑ*_{t,l}/*π* ≃ 0.50 and *ϑ*_{t,r}/*π* = 0.66(1). It should be noticed that even though the bond-modulated Heisenberg interaction has a strong tendency to favor the zigzag ordering, the ground state of Γ model remains disordered albeit the transition point *ϑ*_{t,l} is so close to *π*/2. A more elaborative study on hexagonal clusters of *N* = 24 and 32 suggests that *ϑ*_{t,l}/*π* = 0.498(1) (see Supplementary Fig. 12). For other less sensitive perturbations such as the third-nearest-neighbor Heisenberg (*J*_{3}) interaction, the zigzag order could only exist for *J*_{3}/Γ > 0.075 (see Supplementary Fig. 20). While for another off-diagonal \({{\Gamma }}^{\prime}\) interaction, the zigzag order is generated at \({{\Gamma }}^{\prime} /{{\Gamma }}<-0.015\)^{53}. These, in turn, confirm the robust nonmagnetic character of Γ model notwithstanding the aggressive zigzag ordering.

### Gapless excitation and entropy scaling

Next, we investigate the nature of the disordered phase by calculating the excitation gap and entanglement entropy. For this purpose, we target the first sixteen energy states on a 24-site hexagonal cluster by the DMRG method (see “Methods” section), and the first fifteen low-lying excitation gaps, Δ_{υ} = *E*_{υ} − *E*_{g} (*υ* = 1−15), is shown in Fig. 6a. It can be seen that Δ_{1} is vanishing small while Δ_{2} survives in the zigzag/stripy phases, indicative of the doubly degenerate ground states predicted by the semi-classical analysis. In the intermediate region, the ground state is unique and the density of state in the low-energy spectrum is higher than its neighbors. Such a collapse of excitation gaps could be interpreted as a sign of gapless spectrum^{54,55}. To check the behavior of the lowest excitation gap as *N* is varied, we focus on four selected points at *ϑ*/*π* = 0.40, 0.50, 0.60, and 0.80, under hexagonal cluster of *N* = 18, 24, and 32. As can be seen from Table 1, the lowest excitation gap Δ_{2} of the zigzag phase (*ϑ*/*π* = 0.40) and stripy phase (*ϑ*/*π* = 0.80) is considerably large and slightly grows with the increasing of system size. For the intermediate phase, however, the lowest excitation gap Δ_{1} at *ϑ*/*π* = 0.50 and 0.60 declines quickly when *N* changes from 18 to 32, indicating that the excitation gap tends to close eventually.

We then turn to XC cylinders which enable us to calculate the excitation gaps on large system sizes. Above all, we calculate the first fifteen excitation gaps on a XC cluster of 8 × 4 and we find that there is unlikely a big ground-state degeneracy in the intermediate region since the gap increases gradually without abrupt change (see Supplementary Fig. 8). Therefore, we only present the two lowest excitation gaps on three larger XC clusters up to 200 sites (see Fig. 6b). The gaps in the middle are rather small, indicative of a gapless region. We also take a closer look of the gaps at *ϑ*/*π* = 0.50, where different YC clusters are also adopted. For either XC or YC cluster, Δ_{1,2} show a decreasing trend with the increasing of circumference *n*. In spite of the oscillation in value, they appear to vanish within a reasonable round-off, showing the gaplessness of the excitation spectrum in Γ model (see Supplementary Fig. 13). As a final consistency check for the gapless nature, we calculate the excitation gap on cylinder geometry of 2 × *L*_{x} × *L*_{y} (for geometry, see inset of Fig. 7b) with *N* = 2*L*_{x}*L*_{y} sites in total. Although the three-leg cylinder (*L*_{y} = 3) is gapped, excitation gap on cylinder of *L*_{y} = 4 decreases quickly with *L*_{x} and is expected to disappear as *L*_{x} → *∞* (see Supplementary Fig. 14). Such a strong size-dependent behavior of the excitation gap is typical of gapless systems.

The entanglement entropy has appeared as a versatile tool in diagnosing quantum critical systems described by conformal field theory. In this regard, the von Neumann entanglement entropy is introduced and it is defined as \({\mathcal{S}}(l)=-{\rm{Tr}}(\rho {\mathrm{ln}}\,\rho )\) where *ρ* is the reduced density matrix of a subregion with length *l*^{56}. Figure 7a shows the representative behavior of \({\mathcal{S}}(l)\) in the Γ model on a 2 × 16 × 4 cylinder, which contains eight sites along each column. When *l* is a multiply of 8, it corresponds to a neat edge-cutting where the two halves have smooth margins. The entanglement entropy is minimized and forms a lower branch as marked by solid symbols. Otherwise the subsystems will be more entangled, gaining extra entropy over the lower bound. Therefore, we shall fix *l* = *N*/2 to extract the central charge upon a series of finite cylinders. For such a critical system, it is recognized that the entanglement entropy scaling takes the form of \({{\mathcal{S}}}_{{\rm{vN}}}\equiv {\mathcal{S}}(N/2)=\frac{c}{6}{\mathrm{ln}}\,\left(\right.\frac{2{L}_{x}}{\pi }\left)\right.+c^{\prime}\) where *c* is the central charge and \(c^{\prime}\) is a non-universal constant^{56}. In Fig. 7b, we shows the logarithmic fitting of \({{\mathcal{S}}}_{{\rm{vN}}}\) for cylinders of length *L*_{x} = 8, 16, 24, and 32. It is found that \({{\mathcal{S}}}_{{\rm{vN}}}\) obeys the formula well with the fitting constants \((c,c^{\prime} )\approx (2.92,1.09)\), showing that the central charge is close to 3. An alternative fitting of the lowest branch of the entropy on each cylinder also demonstrates that *c* ≃ 3 (see Supplementary Fig. 16). By contrast, for the three-leg cylinder the entropy is extremely insensitive to the length (see Fig. 7b), revealing a central charge of 0. The fact that the central charge depends highly on the width (*L*_{y}) of cylinders may imply the existence of spinon Fermi surface (SFS). In this scenario, the pockets of SFS might be detected by different cuts in the Brillouin zone, and thus the central charge could vary for different *L*_{y}. We note in passing that the central charge argument has also been used to explore the possible SFS in the field-induced gapless QSL in the Kitaev model^{57,58,59}. Another possibility is a Dirac QSL^{6,60} with three Dirac Fermions around **M** points, which is potentially consistent with the central charge 3 on four-leg cylinders. Information of the central charge on wider cylinders should be helpful to distinguish between the two scenarios. Regardless of different QSL natures, the vanishing magnetization and excitation gap in the Γ model, together with the distinct central charges on cylinders of *L*_{y} = 3 (*c* = 0) and *L*_{y} = 4 (*c* = 3), manifest that its ground state is likely a gapless QSL.

### Flux-like density and plaquette correlation

So far, we have confirmed that there is no magnetic ordering in the honeycomb Γ model, yet little is known about the lattice symmetry breaking. Very recently, there is a proposal of plaquette ordering stemming from a broken translational symmetry in the *classical* Γ model^{38}. It is thus of interest to examine whether there is a plaquette ordering in the quantum situation. To this end, we study the hexagonal plaquette operator \({\hat{W}}_{{\rm{p}}}\) and its correlation. Actually, \({\hat{W}}_{{\rm{p}}}\) also has its own merit as it can capture the associated phase transitions^{33}. The six-body plaquette operator is known as^{13}

which is the product of spin operators on out-going bonds around a plaquette (see Fig. 1).

Figure 8 a shows the flux-like density \(\langle {\overline{W}}_{{\rm{p}}}\rangle ={\sum }_{{\rm{p}}}\langle {\hat{W}}_{{\rm{p}}}\rangle /{N}_{{\rm{p}}}\) where *N*_{p} = *N*/2 is the number of hexagonal plaquette on clusters of *N* = 24 and 32. Starting from *ϑ*/*π* = 0.0, \(\langle {\overline{W}}_{{\rm{p}}}\rangle\) is zero, followed by a continuous decrease before arriving at the transition point, *ϑ*/*π* ≃ 0.50. Afterwards, it begins to increase and then surpasses the critical line to enter into the stripy phase where \(\langle {\overline{W}}_{{\rm{p}}}\rangle\, >\,0\). Recalling the quantum phase diagram shown in Fig. 5b, our result corroborates that the flux-like density could signal phase transitions. For Γ model we have \(\langle {\overline{W}}_{{\rm{p}}}\rangle =-0.25(2)\), which is about a quarter of that in the Kitaev model^{13}. Based on the plaquette–plaquette correlation \(\langle {\hat{W}}_{{\rm{p}}}{\hat{W}}_{{\rm{q}}}\rangle\), we then introduce the plaquette order parameter \({{\mathcal{P}}}_{{{\rm{N}}}_{{\rm{p}}}}({\bf{Q}})\) via the plaquette structure factor \({{\mathcal{W}}}_{{{\rm{N}}}_{{\rm{p}}}}({\bf{Q}})\)^{38} (see “Methods” for definition). In Fig. 8b, we show that the QSL phase has a vigorous peak at **Γ** point in the reciprocal space but a weaker intensity at **K** point, signifying a perceptible plaquette correlation. Nevertheless, The fact that the strength of \({{\mathcal{P}}}_{{{\rm{N}}}_{{\rm{p}}}}({\bf{K}})\) goes down rapidly suggests that there is unlikely a plaquette ordering in the region, further corroborating a QSL without a broken symmetry.

## Discussion

Ever since the seminal proposal of the Kitaev interaction in heavy 4*d*/5*d* transition metal oxides^{12}, which triggers the thriving research direction of Kitaev materials, tremendous efforts have been devoted to realizing the Kitaev QSL in real materials, yet hampered by the ineluctable non-Kitaev terms such as the off-diagonal Γ interaction. Whereas the honeycomb Γ model has drawn enormous attention, its quantum nature is still under debate. To this end, we introduce a bond-modulated Heisenberg interaction to check its tendency towards probable magnetic orderings. For the magnetic phase diagram of the proposed bond-modulated \(\tilde{J}\)–Γ model, we find an intermediate region which is intervened between the zigzag and stripy phases. Though exhibiting magnetic order at the classical level, quantum fluctuations suppress such ordering since it acquires a large energy according to the spin-wave result. In the quantum case, it turns out to be disordered and is separated from its two neighbors by first-order transitions. By taking massive numerical efforts on the Γ model, we are able to confirm the following three subtle physical issues. (i) The low-energy spectrum is rather dense on a 24-site hexagonal cluster, and the lowest excitation gap goes down gradually with the expansion of cluster size. The empirical extrapolation on large cylinders up to 200 sites gives a vanishing energy gap, in line with the logarithmic behaviors of entanglement entropy. (ii) The zigzag magnetic ordering vanishes eventually, consistent with the suppression of magnetization of the zigzag order by spin-wave analysis. (iii) In the plaquette structure factor, there is a perceptible short-range plaquette correlation because of a subleading peak at **K** point. These findings strongly corroborate the ground state of Γ model is a gapless QSL rather than a zigzag order, despite the latter being close in energy.

We would like to mention that, due to the gapless nature and for the lack of continuous spin symmetry, it is exceedingly challenging to capture the fractionalized excitation in the proposed QSL. The flux insertion method, which pumps fractional particles from one edge to the other, is usually a promising way to elucidate the topological characters of the ground state. It is performed by adiabatically twisting boundary conditions of the Hamiltonian so that the *U*(1) symmetry is required. The discrete symmetries of the Γ model thus inherently hinder this trick. Actually, topological degeneracy is in general not well-defined for a gapless QSL, as different gauge sectors are closely connected due to gapless excitations. Nonetheless, the Kitaev QSL is special because flux \({\hat{W}}_{{\rm{p}}}\) is a conserved quantity, allowing for the identification of different flux sectors by the vison insertion^{55}. Also of note is that a recent study suggests the existence of a nematicity due to the lattice rotational symmetry breaking^{42}. We would like to stress that the symmetry of Γ model itself is discrete and the asymmetrical boundary condition could cause instability on the landscape of bond energy, making it hard to determine the nematicity in the thermodynamic limit. However, the pending lattice nematicity does not alter our proposal of the QSL, because it could be accompanied by a broken lattice symmetry as reported in other theoretical models^{61,62}. Despite such challenges, our work emphasizes on the inspiring and intractable quantum nature of the Γ model. Notably, the dominating Γ region could be realized in *α*-RuCl_{3} under compression where the magnitude of Kitaev interaction is small^{53}. In short, our results provide a significant guidance to further theoretical and experimental studies on honeycomb magnets.

## Methods

### Density matrix renormalization group

In order to check for finite-size effects, we have performed large-scale DMRG calculations^{43,44} on three kinds of cluster geometries. Firstly, the frequently used geometry is a *L*_{x} × *L*_{y} XC*n* cluster under cylindrical boundary condition (see Fig. 1). Here, X indicates the orientation of the cylinder, while *n* is the circumference of the cylinder. We consider even circumferences *n* (=*L*_{y}/*a*_{0}) ranging from 4 to 10 lattice spacing *a*_{0}, and use fixed ratio *L*_{x}/*L*_{y} = 2 unless stated explicitly otherwise. *N* = *L*_{x}*L*_{y} is the total number of spins. Secondly, we consider the honeycomb cylinder of 2 × *L*_{x} × *L*_{y} where *L*_{x} (*L*_{y}) is the number of unit cell along \({{\bf{e}}}_{1}=(\sqrt{3},0)\) (\({{\bf{e}}}_{2}=(1/2,\sqrt{3}/2)\)) direction (see the inset of Fig. 7). Due to the limitation of modern computational capability, we focus primarily on four-leg cylinders (*L*_{y} = 4). The maximal value of *L*_{x} is 32, and the total number of spins *N* = 2*L*_{x}*L*_{y}. Lastly, we also use the *C*_{3} symmetric hexagonal clusters with *N* = 24 and 32 sites under full periodic boundary conditions. In all cases, we keep up to *m* = 3000–5000 states and perform about 12 sweeps in the calculation, so as to ensure the truncation error is smaller than 10^{−6}. When targeting the first few low-lying energy levels, we diagonalize a subspace of a sparse Hermitian matrix iteratively by Davidson algorithm, with the precision of each eigenvalue maintained at a desired standard. In addition, all the targeted states are used with an equal weight to construct the reduced density matrix.

The magnetic order parameter is defined by \({M}_{{\rm{N}}}({\bf{Q}})=\sqrt{{{\mathbb{S}}}_{{\rm{N}}}({\bf{Q}})/N}\) where \({{\mathbb{S}}}_{{\rm{N}}}({\bf{Q}})={\sum }_{\alpha \beta }{\delta }_{\alpha \beta }{{\mathbb{S}}}_{{\rm{N}}}^{\alpha \beta }({\bf{Q}})\) is the total static magnetic structure factor, with

Here, *R*_{i} is the position of spin and **Q** is the ordering wavevector. We also calculate the plaquette–plaquette correlation \(\langle {\hat{W}}_{{\rm{p}}}{\hat{W}}_{{\rm{q}}}\rangle\), where \({\hat{W}}_{{\rm{p}}}\) is the hexagon plaquette operator (see Eq. 2). Likewise, we define the static plaquette structure factor

where *R*_{p} is the central position of each plaquette, and *N*_{p} = *N*/2 is the number of plaquette. To eliminate the strong finite-size effect due to the identity \(\left\langle \right.{({\hat{W}}_{{\rm{p}}})}^{2}\rangle\) = 1, we introduce the plaquette order parameter (see Supplementary Note 5)

### Simulation and energy optimization

We use the parallel tempering Monte Carlo simulation with the heat-bath algorithm to prevent the possible metastable state at low temperatures^{45,46}. The simulation is carried out in a temperature range with a hundred of replicas. The heat-bath algorithm is performed at given temperature, followed by a so-called thermal replicas, where configuration swaps between different temperatures are allowed with a probability according to a detailed balance condition. The simulations are performed on three XC clusters of 16 × 16, 24 × 24, and 32 × 32, under toroidal boundary condition.

The resulting energy-optimized spin configuration is then served as the benchmark for the analytical calculation. The classical spin can be written as

where *θ*_{i} ∈ [0, *π*) and *ϕ*_{i} ∈ [0, 2*π*). Taking the zigzag (zz) order and stripy (st) order for instance, their classical energy are

and

where the explicit form of the auxiliary function is

Mathematically, the maximum of Eq. (9) is 2 with \((\theta ,\phi )=(\pi -{\rm{atan}}(\sqrt{2}),\pi /4)\) or \((\theta ,\phi )=({\rm{atan}}(\sqrt{2}),5\pi /4)\). This means that the classical energy of the zigzag phase is \({e}_{{\rm{g}}}^{{\rm{zz}}}=-(2{{\Gamma }}+3\tilde{J})/2\) with the classical magnetic direction \({\bf{n}}=[11\bar{1}]\). The mimimum of Eq. (9) is −1, and the energy of the stripy phase is \({e}_{{\rm{g}}}^{{\rm{st}}}=-({{\Gamma }}-3\tilde{J})/2\). Its moment direction is free to vary in a plane that is perpendicular to **n**, indicating an emergent continuous symmetry in the classical stripy phase. The analytical energy and spin configurations of the other magnetic phases are shown in Supplementary Note 1.

### Linear spin-wave theory

We summarize the derivation of the spin wave spectra of the zigzag phase, which is one of the degenerate ground states of the classical Γ model. In the framework of linear spin-wave theory, the local spin operator \({{\bf{S}}}_{{\rm{i}}}=\left({S}_{{\rm{i}}}^{{\rm{x}}},{S}_{{\rm{i}}}^{{\rm{y}}},{S}_{{\rm{i}}}^{{\rm{z}}}\right)\) is represented by bosonic creation and annihilation operators *a*_{i} and \({a}_{{\rm{i}}}^{\dagger }\). By virtue of the Holstein-Primakoff transformation, we have

Here, \({\tilde{S}}_{{\rm{i}}}^{{\rm{n}}}\equiv ({\bf{S}}\cdot {\bf{n}})\) is the spin component along the classical ordered moment **n** and \({\tilde{S}}_{{\rm{i}}}^{\pm }\equiv ({{\bf{S}}}_{{\rm{i}}}\cdot {\bf{e}})\pm \imath [{{\bf{S}}}_{{\rm{i}}}\cdot ({\bf{n}}\times {\bf{e}})]\) are the ladder operators consisting of the orthogonal spin components, with **e** being an (arbitrary) unit vector perpendicular to **n** and satisfying the right-hand rule^{47}. The spin operator is thus

where *τ* is introduced for classical spin which is either parallel (*τ* = +1) or antiparallel (*τ* = −1) to **n**. The *γ*-component of the spin \({S}_{\tau ,{\rm{i}}}^{\gamma }={{\bf{S}}}_{\tau ,{\rm{i}}}\cdot {{\bf{e}}}_{\gamma }\).

For the zigzag phase, we choose the following orthogonal axis, **e** = [112], \({\bf{n}}\times {\bf{e}}=[1\bar{1}0]\), and \({\bf{n}}=[11\bar{1}]\). Because of the four-sublattice (*n*_{s} = 4) nature, the magnetic unit cell is taken as the rectangle of the area *a*_{1} × *a*_{2} with *a*_{1} = 3*a*_{0} and \({a}_{2}=\sqrt{3}{a}_{0}\), see Fig. 1. Within the magnetic unit cell, the wave vector ** q** could be parameterized in units of (

*h*,

*k*) as

**= \(\left(\right.\frac{2\pi }{{a}_{1}}h,\frac{2\pi }{{a}_{2}}k\left)\right.\)**

*q*^{15}. In this notation,

**M**and \({\bf{M}}^{\prime}\) points in the Brillouin zone could be rewritten as (1, 0) and (0, 1), respectively. By introducing four flavors of Holstein–Primakoff bosons and using the Fourier transformation, we arrive at the following spin-wave Hamiltonian

where \({\hat{{\bf{x}}}}_{{\boldsymbol{q}}}^{\dagger }=\left(\right.{a}_{{\boldsymbol{q}}}^{\dagger },{b}_{{\boldsymbol{q}}}^{\dagger },{c}_{{\boldsymbol{q}}}^{\dagger },{d}_{{\boldsymbol{q}}}^{\dagger },{a}_{{\boldsymbol{-q}}}^{},{b}_{{\boldsymbol{-q}}}^{},{c}_{{\boldsymbol{-q}}}^{},{d}_{{\boldsymbol{-q}}}^{}\left)\right.\) is a vector of length 2*n*_{s} and \({\hat{{\bf{H}}}}_{{\boldsymbol{q}}}\) is a 2*n*_{s} × 2*n*_{s} matrix of the form

with

and

The parameters in Eqs. (14) and (15) are given by

where *ϱ* = *e*^{ıπh/3}. Since \({B}_{-{\boldsymbol{q}}}={B}_{{\boldsymbol{q}}}^{* }\), \({C}_{-{\boldsymbol{q}}}={C}_{{\boldsymbol{q}}}^{* }\), \({E}_{-{\boldsymbol{q}}}={E}_{{\boldsymbol{q}}}^{* }\), and \({D}_{-{\boldsymbol{q}},\tau }={D}_{{\boldsymbol{q,-\tau }}}^{* }\), we hence deduce that \({\hat{{{\Delta }}}}_{{\boldsymbol{q}}}^{\dagger }={\hat{{{\Delta }}}}_{{\boldsymbol{q}}}\) and \({\hat{{{\Lambda }}}}_{-{\boldsymbol{q}}}^{T}={\hat{{{\Lambda }}}}_{{\boldsymbol{q}}}^{\dagger }\).

The quadratic Hamiltonian Eq. (12) can be diagonalized via a bosonic Bogoliubov transformation *T*(** q**). To preserve the canonical commutation rules of the bosons, it should satisfy the orthogonality relations

*T*Σ

*T*

^{†}=

*T*

^{†}Σ

*T*= Σ where Σ = diag(1, −1). The eigenvalues of \({{\Sigma }}{\hat{{\bf{H}}}}_{{\boldsymbol{q}}}\) give the magnon spectrum \({{\Omega }}({\boldsymbol{q}})={\rm{diag}}({\omega }_{{\boldsymbol{q,1}}},{\omega }_{{\boldsymbol{q,2}}},\cdots \ ,{\omega }_{{\boldsymbol{q}},{{\rm{n}}}_{{\rm{s}}}})\). The spin-wave dispersions of other magnetic orderings (including the mixed phase and the noncollinear phase) are shown in Supplementary Note 2.

## Data availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

## References

Balents, L. Spin liquids in frustrated magnets.

*Nature***464**, 199–208 (2010).Han, T.-H. et al. Fractionalized excitations in the spin-liquid state of a kagome-lattice antiferromagnet.

*Nature***492**, 406–410 (2012).Li, Y. et al. Rare-earth triangular lattice spin liquid: a single-crystal study of YbMgGaO

_{4}.*Phys. Rev. Lett.***115**, 167203 (2015).Liao, H. J. et al. Gapless spin-liquid ground state in the

*S*= 1/2 kagome antiferromagnet.*Phys. Rev. Lett.***118**, 137202 (2017).Wang, L. & Sandvik, A. W. Critical level crossings and gapless spin liquid in the square-lattice spin-1/2

*J*_{1}-*J*_{2}Heisenberg antiferromagnet.*Phys. Rev. Lett.***121**, 107202 (2018).Hu, S., Zhu, W., Eggert, S. & He, Y.-C. Dirac spin liquid on the spin-1/2 triangular Heisenberg antiferromagnet.

*Phys. Rev. Lett.***123**, 207203 (2019).Plumb, K. W. et al.

*α*-RuCl_{3}: a spin-orbit assisted Mott insulator on a honeycomb lattice.*Phys. Rev. B***90**, 041112(R) (2014).Kim, H.-S., Shankar, V. V., Catuneanu, A. & Kee, H.-Y. Kitaev magnetism in honeycomb RuCl

_{3}with intermediate spin-orbit coupling.*Phys. Rev. B***91**, 241110(R) (2015).Banerjee, A. et al. Proximate Kitaev quantum spin liquid behaviour in a honeycomb magnet.

*Nat. Mater.***15**, 733–740 (2016).Li, Y. D., Yang, X., Zhou, Y. & Chen, G. Non-Kitaev spin liquids in Kitaev materials.

*Phys. Rev. B***99**, 205119 (2019).Rau, J. G., Lee, E. K.-H. & Kee., H.-Y. Spin-orbit physics giving rise to novel phases in correlated systems: Iridates and related materials.

*Annu. Rev. Condens. Matter Phys.***7**, 195–221 (2016).Jackeli, G. & Khaliullin, G. Mott insulators in the strong spin-orbit coupling limit: from Heisenberg to a quantum compass and Kitaev models.

*Phys. Rev. Lett.***102**, 017205 (2009).Kitaev, A. Anyons in an exactly solved model and beyond.

*Ann. Phys.***321**, 2–111 (2006).Ye, F. et al. Direct evidence of a zigzag spin-chain structure in the honeycomb lattice: a neutron and x-ray diffraction investigation of single-crystal Na

_{2}IrO_{3}.*Phys. Rev. B***85**, 180403(R) (2012).Choi, S. K. et al. Spin waves and revised crystal structure of honeycomb iridate Na

_{2}IrO_{3}.*Phys. Rev. Lett.***108**, 127204 (2012).Chun, S. H. et al. Direct evidence for dominant bond-directional interactions in a honeycomb lattice iridate Na

_{2}IrO_{3}.*Nat. Phys.***11**, 462–466 (2015).Sears, J. A. et al. Magnetic order in

*α*-RuCl_{3}: a honeycomb-lattice quantum magnet with strong spin-orbit coupling.*Phys. Rev. B***91**, 144420 (2015).Leahy, I. A. et al. Anomalous thermal conductivity and magnetic torque response in the honeycomb magnet

*α*-RuCl_{3}.*Phys. Rev. Lett.***118**, 187203 (2017).Sears, J. A. et al. Phase diagram of

*α*-RuCl_{3}in an in-plane magnetic field.*Phys. Rev. B***95**, 180411(R) (2017).Baek, S.-H. et al. Evidence for a field-induced quantum spin liquid in

*α*-RuCl_{3}.*Phys. Rev. Lett.***119**, 037201 (2017).Wolter, A. U. B. et al. Field-induced quantum criticality in the Kitaev system

*α*-RuCl_{3}.*Phys. Rev. B***96**, 041405(R) (2017).Zheng, J. et al. Gapless spin excitations in the field-induced quantum spin liquid phase of

*α*-RuCl_{3}.*Phys. Rev. Lett.***119**, 227208 (2017).Sandilands, L. J. et al. Scattering continuum and possible fractionalized excitations in

*α*-RuCl_{3}.*Phys. Rev. Lett.***114**, 147201 (2015).Li, G. et al. Raman evidence for dimerization and Mott collapse in

*α*-RuCl_{3}under pressures.*Phys. Rev. Mater.***3**, 023601 (2019).Do, S.-H. et al. Majorana fermions in the Kitaev quantum spin system

*α*-RuCl_{3}.*Nat. Phys.***13**, 1079–1084 (2017).Ran, K. et al. Spin-wave excitations evidencing the Kitaev interaction in single crystalline

*α*-RuCl_{3}.*Phys. Rev. Lett.***118**, 107203 (2017).Winter, S. M. et al. Breakdown of magnons in a strongly spin-orbital coupled magnet.

*Nat. Commun.***8**, 1152 (2018).Janssen, L., Andrade, E. C. & Vojta, M. Magnetization processes of zigzag states on the honeycomb lattice: Identifying spin models for

*α*-RuCl_{3}and Na_{2}IrO_{3}.*Phys. Rev. B***96**, 064430 (2017).Laurell, P. & Okamoto, S. Dynamical and thermal magnetic properties of the Kitaev spin liquid candidate

*α*-RuCl_{3}.*npj Quantum Mater.***5**, 2 (2020).Rau, J. G., Lee, E. K.-H. & Kee, H.-Y. Generic spin model for the honeycomb iridates beyond the Kitaev limit.

*Phys. Rev. Lett.***112**, 077204 (2014).Yadav, R. et al. Kitaev exchange and field-induced quantum spin-liquid states in honeycomb

*α*-RuCl_{3}.*Sci. Rep.***6**, 37925 (2016).Wang, W., Dong, Z.-Y., Yu, S.-L. & Li, J.-X. Theoretical investigation of magnetic dynamics in

*α*-RuCl_{3}.*Phys. Rev. B***96**, 115103 (2017).Gordon, J. S., Catuneanu, A., Sßorensen, E. S. & Kee, H.-Y. Theory of the field-revealed Kitaev spin liquid.

*Nat. Commun.***10**, 2470 (2019).Takikawa, D. & Fujimoto, S. Impact of off-diagonal exchange interactions on the Kitaev spin-liquid state of

*α*-RuCl_{3}.*Phys. Rev. B***99**, 224409 (2019).Lampen-Kelley, P. et al. Anisotropic susceptibilities in the honeycomb Kitaev system

*α*-RuCl_{3}.*Phys. Rev. B***98**, 100403(R) (2018).Samarakoon, A. M. et al. Classical and quantum spin dynamics of the honeycomb Γ model.

*Phys. Rev. B***98**, 045121 (2018).Rousochatzakis, I. & Perkins, N. B. Classical spin liquid instability driven by off-diagonal exchange in strong spin-orbit magnets.

*Phys. Rev. Lett.***118**, 147204 (2017).Saha, P., Fan, Z., Zhang, D. & Chern, G.-W. Hidden plaquette order in a classical spin liquid stabilized by strong off-diagonal exchange.

*Phys. Rev. Lett.***122**, 257204 (2019).Catuneanu, A., Yamaji, Y., Wachtel, G., Kim, Y.-B. & Kee, H.-Y. Path to stable quantum spin liquids in spin-orbit coupled correlated materials.

*npj Quantum Mater.***3**, 23 (2018).Gohlke, M., Wachtel, G., Yamaji, Y., Pollmann, F. & Kim, Y. B. Quantum spin liquid signatures in Kitaev-like frustrated magnets.

*Phys. Rev. B***97**, 075126 (2018).Wang, J., Normand, B. & Liu, Z.-X. One proximate Kitaev spin liquid in the

*K*-*J*-Γ Model on the honeycomb lattice.*Phys. Rev. Lett.***123**, 197201 (2019).Gohlke, M., Chern, L. E., Kee, H.-Y. & Kim, Y. B. Emergence of nematic paramagnet via quantum order-by-disorder and pseudo-Goldstone modes in Kitaev magnets.

*Phys. Rev. Res.***2**, 043023 (2020).White, S. R. Density matrix formulation for quantum renormalization groups.

*Phys. Rev. Lett.***69**, 2863 (1992).Stoudenmire, E. M. & White, S. R. Studying two dimensional systems with the density matrix renormalization group.

*Annu. Rev. Condens. Matter Phys.***3**, 111–128 (2012).Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. & Teller, E. Equation of state calculations by fast computing machines.

*J. Chem. Phys.***21**, 1087–1092 (1953).Hukushima, K. & Nemoto, K. Exchange Monte Carlo method and application to spin glass simulations.

*J. Phys. Soc. Jpn***65**, 1604–1608 (1996).Janssen, L. & Vojta, M. Heisenberg-Kitaev physics in magnetic fields.

*J. Phys.***31**, 423002 (2019).Henley, C. L. Ordering due to disorder in a frustrated vector antiferromagnet.

*Phys. Rev. Lett.***62**, 2056–2059 (1989).Zhu, Z., Maksimov, P. A., White, S. R. & Chernyshev, A. L. Disorder-induced mimicry of a spin liquid in YbMgGaO

_{4}.*Phys. Rev. Lett.***119**, 157201 (2017).Huang, Y.-Z. et al. Quantum phase transition, universality, and scaling behaviors in the spin-1/2 Heisenberg model with ferromagnetic and antiferromagnetic competing interactions on a honeycomb lattice.

*Phys. Rev. E***93**, 062110 (2016).Huang, Y.-Z. & Su, G. Quantum Monte Carlo study of the spin-1/2 honeycomb Heisenberg model with mixed antiferromagnetic and ferromagnetic interactions in external magnetic fields.

*Phys. Rev. E***95**, 052147 (2017).Luo, Q., Zhao, J., Wang, X. & Kee, H.-Y. Unveiling the phase diagram of a bond-alternating spin-\(\frac{1}{2}\)

*K*-Γ chain.*Phys. Rev. B***103**, 144423 (2021).Luo, Q., Stavropoulos, P. P. & Kee, H.-Y. Spontaneous chiral-spin ordering in spin-orbit coupled Honeycomb magnets. Preprint at https://arxiv.org/abs/2010.11233 (2020).

Zhu, Z., Kimchi, I., Sheng, D. N. & Fu, L. Robust non-Abelian spin liquid and a possible intermediate phase in the antiferromagnetic Kitaev model with magnetic field.

*Phys. Rev. B***97**, 241110(R) (2018).Hickey, C. & Trebst, S. Emergence of a field-driven

*U*(1) spin liquid in the Kitaev honeycomb model.*Nat. Commun.***10**, 530 (2019).Eisert, J., Cramer, M. & Plenio, M. B. Colloquium: Area laws for the entanglement entropy.

*Rev. Mod. Phys.***82**, 277–306 (2010).Jiang, H. -C., Wang, C. -Y., Huang, B. & Lu, Y.-M. Field induced quantum spin liquid with spinon Fermi surfaces in the Kitaev model. Preprint at https://arxiv.org/abs/1809.08247 (2018).

Patel, N. D. & Trivedi, N. Magnetic field-induced intermediate quantum spin liquid with a spinon Fermi surface.

*Proc. Natl Acad. Sci. USA***116**, 12199–12203 (2019).Zou, L. & He, Y.-C. Field-induced neutral Fermi surface and QCD

_{3}-Chern-Simons quantum criticalities in Kitaev materials.*Phys. Rev. Res.***2**, 013072 (2020).Iqbal, Y., Becca, F., Sorella, S. & Poilblanc, D. Gapless spin-liquid phase in the kagome spin-\(\frac{1}{2}\) Heisenberg antiferromagnet.

*Phys. Rev. B***87**, 060405(R) (2013).Gong, S.-S., Zhu, W., Sheng, D. N. & Yang, K. Possible nematic spin liquid in spin-1 antiferromagnetic system on the square lattice: implications for the nematic paramagnetic state of FeSe.

*Phys. Rev. B***95**, 205132 (2017).Hu, W.-J. et al. Nematic spin liquid phase in a frustrated spin-1 system on the square lattice.

*Phys. Rev. B***100**, 165142 (2019).

## Acknowledgements

We thank M. Gohlke, J.S. Gordon, Y.-Z. Huang, T. Li, H.-J. Liao, K. Liu, Z.-X. Liu, B. Xi, W. Yu, and especially S. Hu for many inspiring and helpful discussions. X.W. was supported by the National Program on Key Research Project (Grant No. 2016YFA0300501) and by the National Natural Science Foundation of China (Grant No. 11974244). J.Z. was supported by the the National Natural Science Foundation of China (Grant No. 11874188). H.-Y.K. was supported by the NSERC Discovery Grant No. 06089-2016 and acknowledged funding from the Canada Research Chairs Program. X.W. also acknowledged additional support from a Shanghai talent program. The computations were mostly performed on the Tianhe-2JK at the Beijing Computational Science Research Center (CSRC).

## Author information

### Authors and Affiliations

### Contributions

J.Z. and X.W. initiated and supervised the project. Q.L. performed the spin-wave analysis and numerical calculations. H.-Y.K. guided the analysis of entanglement entropy scaling and plaquette correlation. J.Z., H.-Y.K. and X.W. checked the calculations. All authors together discussed the numerical details and drafted the article.

### Corresponding authors

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Luo, Q., Zhao, J., Kee, HY. *et al.* Gapless quantum spin liquid in a honeycomb Γ magnet.
*npj Quantum Mater.* **6**, 57 (2021). https://doi.org/10.1038/s41535-021-00356-z

Received:

Accepted:

Published:

DOI: https://doi.org/10.1038/s41535-021-00356-z