Introduction

The role of strong interaction between electrons in the emergence of topological phases of matter is currently a topic of intensive research. The archetypal example of a topological phase with strong electron interaction is the quantum spin liquid,1 in which the elementary excitations are charge-neutral fractionalized particles. While a lot of progress has been made on the theoretical understanding of the quantum spin liquid phase, its direct experimental confirmation has remained elusive despite various studies on a number of candidate materials.2,3,4,5,6 Significant progress, however, has recently been made due to the availabilty of a new class of correlated materials, where strong spin-orbit coupling leads to various bond-dependent spin interactions,7,8,9 thus resulting in magnetic frustation. These materials are Mott insulators with 4d and 5d transition metal elements, which include iridates and ruthenates10–14 and come in two-dimensional (2D) or three-dimensional (3D) honeycomb variants. They have been particularly exciting because they intrinsically have a strong Kitaev interaction and therefore could potentially realize the Kitaev spin liquid (KSL) phase—an example of a \({\Bbb Z}_2\) quantum spin liquid where the electron’s spin \(- \frac{1}{2}\) fractionalizes into two degrees of freedom: itinerant Majorana fermions and \({\Bbb Z}_2\) fluxes.

While the Kitaev interaction (K) in these materials is large, it competes with symmetry allowed nearest-neighbor (n.n.) symmetric off-diagonal (Γ) and Heisenberg (J) spin interactions.15 For example, in α-RuCl3 (RuCl3), an actively studied KSL candidate, comprehensive ab initio computations and recent dynamical studies16,17 suggest that ferromagnetic K and antiferromagnetic Γ interactions are dominant and comparable in magnitude, while J is negligible.17,19,20 The balance of these and additional small further neighbor interactions causes RuCl3 and other KSL candidates to magnetically order at low temperature; however, it is still unclear whether or not the often-large Γ interaction prefers magnetic ordering. Meanwhile, the community has attempted to revive the possibility of a KSL in RuCl3 by applying a small magnetic field, with the effect of entering a potential spin liquid phase with no magnetization.20

Since a weak magnetic field takes RuCl3 out of the ordered phase, it lends credence to the idea that the zig-zag phase is stabilized by small interactions at comparable energy scale to the magnetic field, such as a 3rd n.n. Heisenberg J317,19,20 term or terms coming from slight trigonal distortion.21 This calls into question the role of the Γ interaction. Interestingly, a recent analysis of the Γ model revealed a macroscopically degenerate classical ground state.22

In this work we will thus investigate if a model with K and Γ hosts an extended quantum spin liquid phase. A previous exact diagonalization (ED) study on a 24-site honeycomb cluster hints that the ferromagnetic KSL is unstable after perturbing with a small Γ, but the resulting phase is not orderered.15 On the other hand, it is known that the KSL is stable upon introducing bond anisotropy, which is present in real materials as depicted in Fig. 1a. We indeed find that such anisotropy can extend the KSL phase between the −K and Γ limits, as shown in Fig. 1b.

Fig. 1
figure 1

a Shorter z-bond leading to stronger \(K^z\) and weaker \(K^{x,y}\) interactions, parameterized by \(a_K\) in Eq. 1. b \(a_K\) phase diagram of Eq. 1. A line of first order phase transitions (dashed black line), terminating below \(a_K = 0.06\) (black dot), separates the −K and Γ limits. Anisotropy allows the KSL to be adiabatically connected to the Γ-phase (arrow). Red, green, and blue triangles indicate first-order phase transitions when \(a_K = 0.0,\,0.02\), and 0.04 respectively, as seen in N = 18 and 24-site exact diagonalization calculations. A subsequent iDMRG calculation38 has further corroborated the phase transition labeled by the red triangle

Full size image

We consider the following nearest-neighbor (n.n.) model on a 2D honeycomb lattice:

$$H = \mathop {\sum}\limits_{\gamma \in x,y,z} H^\gamma ,$$
(1)

where

$$H^z = \mathop {\sum}\limits_{\langle ij\rangle \in z - bond} [K_zS_i^zS_j^z + \Gamma _z(S_i^xS_j^y + S_i^yS_j^x)]$$
(2)

and Hx,y are defined similarly with corresponding Kx, y and Γx, y. Each Hγ represents the n.n spin interactions along one of the three bond directions, γ = x, y, z. The model is parameterized by K z  = −(1 + 2a K ) cos ϕ, Kx, y = −(1−a K ) cos ϕ, Γx, y, z = sin ϕ, with a K characterizing bond anisotropy. When ϕ = 0, π (i.e., Γ γ  = 0), this model reduces to the exactly solvable Kitaev model with the KSL ground state.

We have studied this model using a combination of three different, corroborating, numerical methods: ED on a 24-site honeycomb cluster, the method of thermal pure quantum states,22,24,25,26,27 and infinite time-evolution block decimation (iTEBD). We have first reproduced the earlier work in the isotropic a K  = 0 limit, showing a strong first-order transition between −K and Γ limits (see Supplementary Materials (SM)). We present the following results using our numerical techniques:

  1. 1.

    When a K  ≥ 0.06, we find that the −K γ (0 ≤ ϕ ≤ π/2) and Γ γ (ϕ/π = 0.5) limits are adiabatically connected as shown in Fig. 1b. Thus we find evidence for an extended quantum spin liquid phase in the presence of anisotropy, a K .

  2. 2.

    An intervening magnetically ordered phase separates the spin liquid phase near the pure Γ γ limit and the antiferromagnetic KSL at ϕ/π = 1.

  3. 3.

    The specific heat C(T) and entropy S(T) at finite temperatures suggest a smooth crossover from the ferromagnetic Kitaev limit to the pure Γ γ limit, consistent with our ED results.

  4. 4.

    Zig-zag spin correlations become dominant upon perturbing the quantum spin liquid phase in 0 < ϕ < π/2 by J3, indicating the enhancement of zig-zag order by J3.

Results

Extended spin liquid state in global phase diagram

The ground state energy per site E0/N of Eq. 1 was computed for ϕ/π ∈ (0, 1), and for different anisotropy parameters by ED on a 24-site cluster using periodic boundary conditions (see SM). Discontinuities in \(\frac{1}{N}\frac{{\partial E_{\mathrm{0}}}}{{\partial {\it{\phi }}}}\) were used to identify possible phase transitions. Remarkably, when ϕ/π ∈ (0, 0.5) and 0 ≤ a K  < 0.06, there is a line of first order phase transitions that terminate at a K  = 0.06. Above a K  = 0.06, the first derivative of the energy presents no sharp features suggesting that the ground state of the Γ-limit (ϕ/π = 0.5) is adiabatically connected to the ferromagnetic KSL (ϕ/π = 0) as depicted in Fig. 1b for a K  = 0.1.

In the antiferromagnetic region of phase space, there are two large discontinuities in \(\frac{1}{N}\frac{{\partial E_0}}{{\partial {\it{\phi }}}}\) that encompass a large region of phase space separating the Γ-limit and the exactly solvable antiferromagnetic Kitaev limit at ϕ/π = 1. These peaks coincide with kinks in E0/N (solid yellow) shown in Fig. 2a. Two smaller discontinuities can also be seen near ϕ/π = 0.75, however these are not present when a K  = 0, while the larger jumps near ϕ/π = 0.5 and 1 appear consistently for different a K . The small discontinuities can thus be considered spurious and a consequence of the finite cluster size. Similar finite size effects were also found for ϕ/π ∈ (0, 0.5) when a K  = 0, as discussed in the SM.

Fig. 2
figure 2

a Top: E0/N (yellow), \(\frac{1}{N}\frac{{\partial E_0}}{{\partial \phi }}\) (purple), and \(- \frac{1}{N}\frac{{\partial ^2E_0}}{{\partial \phi ^2}}\) (light purple) for anisotropy parameter \(a_K = 0.1\). Bottom: S q for anisotropy parameter \(a_K = 0.1\) at Γ (black), M (blue), Y (cyan), K (red), and Γ (green) in the reciprocal lattice (inset). b Representation of S q , averaged over domains in a real material, when \(a_K = 0.1\) for various ϕ in the phase diagram

Full size image

Magnetic order and perturbations

The ground state wavefunction of Eq. 1 computed by ED is used to evaluate real-space spin–spin correlation functions \(\langle S_i \cdot S_j\rangle,\) where i and j are site indices on the honeycomb lattice. By Fourier transform, we obtain the static structure factor (SSF) given by \(S_q = \frac{1}{N}\mathop {\sum}\nolimits_{i,j} e^{i(r_i - r_j) \cdot q}\langle S_i \cdot S_j\rangle,\) where q is a vector in the reciprocal lattice. The SSF at various points in the Brillouin zone (BZ) is plotted over the phase space in the bottom panel of Fig. 2a.

The discontinuities in the SSF can be directly matched with those in \(\frac{1}{N}\frac{{\mathrm{\partial} E_{\mathrm{0}}}}{{\mathrm{\partial} {\it{\phi }}}}\). Visualizations of the SSF over the BZ for representative ϕ in the phase diagram are presented in Fig. 2b. The SSF in Fig. 2b is obtained by computing the average of \(\langle S_i \cdot S_j\rangle\) over all n.n. bonds, 2nd n.n., etc. This calculation reflects the presence of different domains in the crystal, in which either of x, y, z bond interactions can be stronger and thus, over the whole crystal, these domains result in an isotropic S q despite the inherent bond anistropy in Eq. 1. The SSF varies adiabatically when a K  = 0.1 for ϕ ∈ (0, π/2) and the spin correlations at the Γ-points and M-points are comparable in intensity when \(\Gamma \simeq K_\gamma\), leading to a “star”-shaped structure in the SSF as seen in Fig. 2b (e.g., ϕ/π = 0.2). The extended phase separating ϕ/π = 0.5 and 1 is characterized by dominating spin correlations at the K-points and Γ′-points in the reciprocal lattice (ϕ/π = 0.75 in Fig. 2b). Contained within this phase is the exactly solvable point with hidden SU(2) symmetry at ϕ/π = 0.75, which features K-point correlations28 consistent with the results presented here. Thus the extended spin liquid phase for ferromagnetic K is separated from the antiferromagnetic KSL at ϕ/π = 1 by a magnetically ordered phase.

These results can be connected to real materials, particularly RuCl3 in which a zig-zag magnetic ordering has been observed.28,30,31 Previous studies have shown that in addition to the n.n. ferromagnetic Kitaev and antiferromagnetic Γ interactions, a 3rd n.n. antiferromagnetic Heisenberg interaction \(J_3\mathop {\sum}\nolimits_{\langle \langle \langle i,j\rangle \rangle \rangle } S_i \cdot S_j\) is non-vanishing and plays a role in determining the magnetic ordering in RuCl3.18,19 Figure 3 shows that the effect of perturbing Eq. 1 by J3 is to enhance (suppress) the M-point (Γ-point) spin correlations, consistent with a zig-zag magnetically ordered state observed in experiments. This result indicates that by tuning J3 in the real material, an alternate path to achieve a spin liquid phase may be realized.

Fig. 3
figure 3

SSF at the Γ-points (black) and M-points (blue) in the BZ for small J3. Solid and dashed curves correspond to ϕ/π = 0.5 and 0.2, respectively. The inset shows the dramatic change in the relative intensity at the Γ-points and M-points for small J3 > 3

Full size image

Specific heat and thermal entropy

Previous study on the finite temperature properties of the Kitaev model has shown that KSLs feature two peaks in the heat capacity C(T) and a \(\frac{1}{2} -\) plateau in the entropy S(T), which is attributed to the thermal fractionalization of spin degrees of freedom.32 Here we go beyond the Kitaev limit and investigate the heat capacity and entropy at finite temperature in the presence of Γ, which is expected to compete with K γ in RuCl3, using the method of thermal pure quantum states (see SM).

The dependence of C(T) on ϕ when a K  = 0.1 is plotted in the top panel of Fig. 4. The expected two peak structure in C(T) is observed when ϕ/π = 0, and is seen to be maintained continuously as ϕ/π approaches 0.5 so that the Γ-limit shows a qualitatively similar behavior in C(T) to the KSL. Evidence for a phase transition can be seen when ϕ ≳ 0.7 on account of the abrupt change in C(T), resembling that of the heat capacity in trivially ordered phases.33 This finding is consistent with our ED results and the 120-order at ϕ/π = 0.75 seen in refs. 15 28. The dependence of S(T) on ϕ is plotted in the bottom panel of Fig. 4 with a clear \(\frac{1}{2} -\) plateau observed when ϕ/π = 0, consistent with the expected KSL behavior. In addition, a plateau of about 1/5 the total entropy is observed when ϕ/π = 0.5. Another plateau is observed in the magnetically ordered phase around ϕ/π = 0.75; however, this feature can be attributed to finite-size effects as follows. The (N + 1)-fold ground state degeneracy at ϕ/π = 0.75 due to the hidden SU(2) symmetry28 is only slightly broken away from this point, inducing a plateau in S(T) with height given by \({\mathrm{ln}}\,(N + 1)/N\,{\mathrm{ln}}\,2 \simeq 0.1935 \sim 1/5\) when N = 24. By contrast, the height of the plateau around ϕ/π = 0.5 is independent of N. (The height of the plateau in the temperature dependence of entropy is also examined by using a 32 site cluster).

Fig. 4
figure 4

Results of the method of thermal pure quantum states on a 24-site cluster with anisotropy parameter \(a_K = 0.1\). Plotted in solid curves are the temperature dependence of heat capacity C(T) (top) and entropy S(T) (bottom) for various ϕ in the phase diagram. The shaded regions represent the estimated errors on the results. The temperature Γ is expressed in units, where \(\sqrt {(K_z{\mathrm{/}}(1 + 2a_K))^2 + \Gamma _z^2}\) = \(\sqrt {(K_{x,y}{\mathrm{/}}(1 - a_K))^2 + \Gamma _{x,y}^2}\) = 1

Full size image

The physical origin of the two peak structure in C(T) and the plateau in S(T) can be traced to the energy scales of the thermal fluctuations of the underlying quasiparticles in the spin liquid.32 In the KSL at zero temperature, the low-lying quasiparticle excitations are characterized by itinerant Majorana fermions which disperse in a background of zero flux.34,35 It has been shown that as temperature increases, the flux degrees of freedom begin to fluctuate and lead to the low temperature peak seen in C(T), resulting in the plateau seen in S(T). Furthermore, the high temperature peak in C(T) is attributed to the development of short range spin correlations.32 Our results show that the two-peak structure in C(T) is qualitatively maintained and further suggests that no phase transition has taken place.

Similarities on the infinite tree

We further studied Eq. 1 on an infinite Cayley tree with z = 3 connectivity, using the iTEBD36 (see SM). Classically, the ground state in the Γ-limit on the infinite tree is macroscopically degenerate because a different state with the same energy can be constructed by flipping the sign of one spin component on an infinite string of neighboring spins. The Γ-limit on the 2D honeycomb and 3D hyper-honeycomb21 lattices also feature similar classical degeneracy. The similarity at the classical level of the Γ-limit on the infinite tree to the 2D and 3D lattices prompts us to study the quantum model on the infinite tree for further insight.

Figure 5 shows results of the eight-site iTEBD calculation with bond dimension χ = 10, and anisotropy a K  = 0.1. In this calculation, we have also introduced an anisotropy to Γ γ such that Γ x  = Γ y  = (1−aΓ)sin ϕ and Γ z  = (1 + 2aΓ)sin ϕ in order to apply the iTEBD method (see SM). No transition is found when ϕ/π ∈ (0, 0.5) and the obtained state is a highly entangled paramagnet, with \(S_E\sim 0.8\) for strong (z) bonds, while for weak (x, y) bonds, \(S_E\sim 0.4\). Deep in the gapped phase of the Kitaev model, with large anisotropy a K , one finds \(S_E\sim {\mathrm{log}}2\sim 0.693\) for the strong bonds and much smaller values of S E for the weak bonds. Both, however, increase as the anisotropy is reduced—perhaps due to a finite contributions from the Majorana fermions.37 An increase is expected in the entanglement entropy as one approaches a phase transition, however no such peaks are seen for 0 < ϕ < π/2. Similarly, there are no sharp features in the ground state energy E0 as a function of ϕ, which indicates that this phase is adiabatically connected to the KSL at ϕ = 0.

Fig. 5
figure 5

iTEBD results for small anisotropy \(a_K = a_\Gamma = 0.1\), and bond dimension χ = 10. a Néel order parameter MNéel, and ground state energy per site, E0/N. b Entanglement entropy S E associated with splitting the system along the different bonds in the eight-site unit cell. In most of the phase diagram all the strong z bonds are identical, and so are all the weak, x and y bonds. In the transition region near ϕ = 0.6π, the symmetry between like bonds is broken, perhaps indicating a first order transition. At ϕ/π = 0.75, the entanglement entropy vanishes since the system is in a product state. Inset: schematic of Cayley tree with z = 3 connectivity

Full size image

There is an apparent first order transition around ϕ = 0.6π into a Néel state with spins ordered in the (111) direction, accompanied by a dramatic lowering of S E on both strong and weak bonds into this region. This Néel state becomes a simple product state when ϕ/π = 0.75, as seen by the vanishing of S E . A final transition into a paramagnetic state is seen before the antiferromagnetic Kitaev limit.

Discussion

The highlight of our numerical results is that, in the presence of bond anisotropy a K , there exists an extended quantum spin liquid region which is adiabatically connected to the ferromagnetic KSL. The model we have studied is motivated by experiments on RuCl3 and earlier ab initio computations.17,19,20 In a recent inelastic neutron scattering experiment on RuCl3, it is found that the continuum of finite energy excitations exists both below and above the magnetic transition temperature despite that the low temperature ground state is the zig-zag long-range ordered state.38 The inelastic neutron scattering data for the continuum show the star-shape intensity that extends from the zone center towards the M-points of the BZ.

Recall that the SSF in our ED study shows enhanced (decreased) short-range spin correlations at the M-point (zone center) of the BZ as one moves from the ferromagnetic Kitaev limit to the pure Γ γ limit. When the strength of the ferromagnetic Kitaev interaction and the Γ γ interaction become comparable, both of the short-range spin correlations at the M and the zone center would show significant intensity, which leads to the star-shape structure in momentum space. This behavior may be favorably compared to the finite-energy short-range spin correlations seen in RuCl3. Given that the ab initio computations suggest comparable magnitudes of the ferromagnetic Kitaev and Γ γ interactions in RuCl3,18 it is conceivable that RuCl3 may be very close to the quantum spin liquid phase found in our model and, as shown in our work, the introduction of small J3 would favor the zig-zag magnetically ordered phase observed in RuCl3.

Finally, more analytical understanding of the connection between the pure Kitaev limit and the quantum spin liquid phases identified in our numerical work would be extremely valuable for future applications on real materials. Note that a possible incommensurate magnetic ordering cannot be ruled out due to finite cluster size. However, based on the results of a recent iDMRG study,39 no evidence of incommensurate ordering allowed by their momentum cuts was found. It is also interesting to note that quantum fluctuations do not lift the infinite ground state degeneracy of the classical model for positive Γ, while they may lead to incommensurate ordering for negative Γ.40 Thus it is likely that the positive Γ regime studied here possesses a spin liquid ground state, and the precise nature of the spin liquid is an excellent topic for future study.

Methods

Our results were obtained using the combination of the three independent numerical techniques listed below.

Exact diagonalization

Exact diagonalization was performed on a 24-site cluster with periodic boundary conditions. This cluster allows all the symmetries present in the infinite honeycomb lattice and has been used reliably in previous related classical and quantum studies.15,40 The Hamiltonian given by Eq. 1 in the main text does not have the U(1) symmetry associated with Sz conservation, making it impossible to block diagonalize into magnetization sectors. Therefore, the translational symmetry of the 24-site cluster was used to block diagonalize into different momentum sectors to gain more information about its energy spectrum. The lowest energies and corresponding wavefunctions of each block were then numerically obtained using the Lanczos method. Further details and calculations can be found in the SM.

Thermal pure quantum states

We used the method of thermal pure quantum states26,27 in our specific heat and thermal entropy calculations. A part of the TPQ results were checked by a program package, HΦ.41 Details about the construction of thermal pure quantum states and the subsequent calculation of specific heat and entropy can be found in the SM.

Infinite time-evolving block decimation algorithm

The Hamiltonian given by Eq. 1 was studied on an infinite Cayley tree with z = 3 connectivity using the iTEBD.36 Details about the method and the construction of the ground state can be found in the SM.

Data availability

All relevant data is available from the corresponding author.