Observation of plaquette fluctuations in the spin-1/2 honeycomb lattice

Quantum spin liquids are materials that feature quantum entangled spin correlations and avoid magnetic long-range order at T = 0 K. Particularly interesting are two-dimensional honeycomb spin lattices where a plethora of exotic quantum spin liquids have been predicted. Here, we experimentally study an effective S = 1/2 Heisenberg honeycomb lattice with competing nearest and next-nearest-neighbour interactions. We demonstrate that YbBr3 avoids order down to at least T = 100 mK and features a dynamic spin–spin correlation function with broad continuum scattering typical of quantum spin liquids near a quantum critical point. The continuum in the spin spectrum is consistent with plaquette type fluctuations predicted by theory. Our study is the experimental demonstration that strong quantum fluctuations can exist on the honeycomb lattice even in the absence of Kitaev-type interactions, and opens a new perspective on quantum spin liquids.


INTRODUCTION
Magnetism arises because of the quantum mechanical nature of the electron spin, yet for the understanding of many materials, particularly those used in today's applications, a classical approach is sufficient. Materials with strong quantum fluctuations are rare, but attract significant research attention since they hold enormous potential for future technologies 1 that make use of the long-range entanglement for quantum communication 2,3 . Fault-tolerant quantum computers are proposed to operate with anyon quasi-particles 2 which exist in a class of quantum spin liquids 4,5 .
Quantum spin liquids (QSL) are caused by quantum fluctuations which reduce the size of the ordered magnetic moment of static magnetic structures and can affect the dynamics of the spin excitations. This happens in the S = 1/2 frustrated antiferromagnetic square lattice, with competing nearest and next-nearestneighbour interactions, J 1 , J 2 , where the zone boundary spinwaves develop a dispersion due to the presence of quantum dimer-type fluctuations between nearest neighbours 6 . These fluctuations are similar to the resonant valence bond fluctuations predicted in the frustrated triangular lattice 7 , which are believed to be relevant for high-temperature superconductors 8 . Frustration can be induced by competing interactions and depending on their relative strength, incommensurate magnetic phases, valence bond solids with periodic ordering of local quantum states, or QSLs with different symmetry are theoretically predicted 4,[9][10][11][12][13][14][15] . In particular, it is expected that frustration enforces a quantum phase transition at which fractionalization of magnons into deconfined spinons occurs 16 .
It has been a challenge to identify and understand appropriate model systems to study QSLs. In general, lowering the dimension will increase quantum fluctuations. In one-dimension QSLs have been identified in antiferromagnetic (AF) spin chains. Case in point are KCuF 3 17 and Cu(C 6 D 5 COO) 2 ⋅ 3D 2 O 18 . In two-and threedimensions, quantum fluctuations can be enhanced by frustration, and there are several routes to achieve this: The inherent geometrical frustration of kagome 19 , triangular 20 , spinel 21 and pyrochlore 22 lattices may prohibit long-range ordering at low temperatures. Another promising candidate is the honeycomb lattice which has received relatively little attention until Kitaev's work 4 when it was realized that bond-dependent anisotropic interactions can stabilize a new form of QSL whose properties are known exactly. Representatives materials are α-RuCl 3 23 , Li 2 IrO 3 24 and H 3 LiIr 2 O 6 25 which show signatures of spin correlations due to quantum entanglement.
Quantum fluctuations are enhanced in the honeycomb lattice compared to the square lattice since the number of neighbours of each spin is lower, thus placing it closer to the quantum limit. When next-nearest-neighbour frustrating exchange interactions are sufficiently large compared to the nearest-neighbour exchange, theories predict a quantum phase transition from a Néel ground state into a quantum entangled state. However, there is no consensus on the nature of this ground state: Theories predict either a QSL 11,12 or a plaquette valence bond crystal (pVBC) [13][14][15] with different magnetic excitations which include spinons 11 , rotons 15 or plaquette fluctuations 26 .
Here, we study the magnetic properties of the trihalide twodimensional compound YbBr 3 that forms a realization of the undistorted S = 1/2 honeycomb lattice with frustrated interactions. Short-range magnetic correlations between the Yb moments develop below T ≈ 3 K, but the correlation length is only of the order of the size of an elementary honeycomb plaquette at T = 100 mK, consistent with a QSL ground state. Despite this short correlation length, inelastic neutron measurements reveal well-defined dispersive low energy magnetic excitations close to the Brillouin zone centre. At high energies and at the zone boundary, we observe a continuum of excitations that we interpret as quantum fluctuations on an elementary hexagonal plaquette.

RESULTS
Crystal structure and susceptibility YbBr 3 crystallizes with the BiI 3 layer structure in the rhombohedral space group R3 (148), where the Yb ions form perfect twodimensional (2D) honeycomb lattices perpendicular to the c-axis, as shown in Fig. 1. The atomic positions are given in Table 1 in Supplementary Note 1. The temperature dependence of the magnetic susceptibility has a broad maximum around T = 3 K, but as shown below, there is no evidence for long-range magnetic order down to at least T = 100 mK. In the low-temperature regime below 10 K we observe χ a ≈ 1.3χ c which reflects a small easy-plane anisotropy.
The rare-earth ion Yb 3+ features a J = 7/2 ground-state multiplet that is split by the crystal-electric field (CEF), giving rise to a total of four Kramers doublets with the three excited CEF levels being observable via neutron scattering. The first excited level is observed at~15 meV (see Supplementary Fig. 1) and the groundstate doublet is an effective S = 1/2 state. From an analysis of the measured susceptibility and the inelastic neutron data, we obtain the CEF parameters listed in Supplementary Note 2. They result in ground state expectation values of 〈J ⊥ 〉 = 1.2 and 〈J ∥ 〉 = 0.8 where the subscript indicates spin orientations measured relative to the c-axis.
Magnetic ground state Figure 2a shows the neutron diffraction pattern of the energy integrated magnetic scattering of Yb 3 that was determined as the difference between diffraction patterns taken at T = 100 mK and T = 10 K in order to eliminate the contributions of nuclear scattering. No magnetic Bragg peaks are visible in the diffraction pattern, demonstrating that YbBr 3 avoids magnetic order down to at least this temperature.
Diffuse magnetic scattering is centred at (1, 0, 0) and equivalent wave-vectors, which implies that the short-range correlations are described by a propagation vector Q 0 = (0, 0, 0). Figure 2b shows the diffuse scattering as obtained from the 2D spin-wave theory described below, which reproduces both position and intensity of the observed diffuse scattering. Figure 2c shows a cut along the Q = (q, 0, 0) direction which reveals diffuse scattering with Lorentzian line shape that reflects short-range magnetic order 27 . From a fit to the neutron intensity I ∝ κ 2 /(q 2 + κ 2 ), we determine an in-plane correlation length between the Yb moments of ξ = 1/κ ≈ 10 Å at T = 100 mK, comparable to the fourth nearest-neighbour distance of 10.66 Å which is~1.25 times the diameter of an Yb 6 -hexagon plaquette.

Magnetic excitations
We measured well-defined magnetic excitations at T = 250 mK along three cuts in the hexagonal plane. Within experimental resolution, we observed a single excitation branch and no spin gap at the zone centre. As shown in the constant-energy-scans in Fig. 2d and in Fig. 3, the magnetic excitations are sharp close to the Brillouin zone centre. One of the key results of this study is the observation of a broadening of the spectrum when the dispersion approaches the zone boundary, as shown in Fig. 3. In fact, the inelastic neutron spectrum close to the zone boundary exhibits a continuum which extends to over twice the energy of the welldefined magnetic excitation. While low-lying excitations are sharp, these broad excitations are only observed at higher energies.
While it may appear surprising that we observe well-defined excitations even in the presence of a correlation length of merely 10 Å, this agrees with the predictions of Schwinger-Boson 28 and modified spin-wave 29 theories which show that spin waves can propagate in low-dimensional systems with short-range Néel order. The well-defined excitations in YbBr 3 can be described by an effective S = 1/2 Hamiltonian including nearest and nextnearest-neighbour Heisenberg exchange coupling, and dipolar interactions between the CEF ground-state doublets, where J α;β ði; jÞ ¼ g 2 α δ αβ Jði; jÞ þ g α g β D α;β ði; jÞ, with α, β = x, y, z cartesian coordinates of the hexagonal cell, and S α i is the αcomponent of a spin-1/2 operator at site i. Here, J(i, j) are the exchange coupling constants between distinct sites i and j, while D(i, j) denotes the dipolar interactions. For the calculation of the spin-wave dispersion, we use the random-phase approximation (RPA) around the Néel state with spins in the hexagonal plane and S = 1/2 (see Supplementary Note 3). Our measurements allow the determination of the exchange couplings, while the dipolar coupling is fixed by the magnetic moment. As shown in Fig. 3, we find good agreement between measured and calculated spin-wave dispersions. The nearest-and next-nearest-neighbour exchange interactions J 1 , J 2 are obtained from a least-square fit to the data. We obtained g 2 J 1 = −0.69(8) meV and g 2 J 2 = −0.09 (2) meV that correspond to a calculated ground state with Q 0 = (0, 0, 0) (see Supplementary Note 4). We note that our spin-wave theory does not describe all aspects of our experimental results: It predicts an optical branch for values of the easy-plane anisotropy that corresponds to the measured susceptibility (see Fig. 1), while we do not find experimental evidence for such a second branch. Also, it does not explain the existence of an excitation continuum as we shall discuss next.   Fig. 3, the magnetic excitation spectrum also features weaker broad scattering at energies where the optical branch is expected. This is particularly evident near the M-points at (0.5, 0.5, 0) and (0.5, −1, 0), where the excitations extend to 0.8-1 meV and are reminiscent of scattering observed in other low-dimensional antiferromagnets 30,31 . In most materials, spin-waves are long-lived excitations that are resolution-limited as a function of energy. When the spin waves are damped or interact with other spin-waves they have a finite life-time and the line-shape of the dynamical structure factor S(Q, ω) broadens 32 . We have simulated the line-shape of S(Q, ω) derived from our model and convoluted it with the resolution of the spectrometer obtained from the Takin software 33 (see 'Methods'). While the spin-wave model adequately explains the dispersion and intensity distribution close to the Brillouin zone centres, it does not reproduce the inelastic neutron line-shape close to the maximum of the dispersion of the spin-wave branch as shown in Fig. 4.

DISCUSSION
Although YbBr 3 only exhibits short-range magnetic order, the dispersion of the sharp magnetic excitations can be well described by a spin-1/2 Heisenberg Hamiltonian with easy-plane anisotropy and dipolar interactions. For a honeycomb lattice classical theories predict instability of the Néel state for J 2 /J 1 ≈ 0.1 10,29 and quantum fluctuation in linear spin-wave theory destroy long-range Néel order. We note that other theoretical approaches find that quantum fluctuations may stabilize the Néel phase up to somewhat higher ratios of J 2 /J 1 . These approaches include Schwinger-Boson approach 11 , variational wave functions 15,34 and exact diagonalization 13 which all yield a critical ratio J 2 /J 1 ≈ 0.2. Since we find only magnetic short-range correlations between the Yb moments, we conclude that YbBr 3 must be in close proximity of such a quantum phase transition.
In YbBr 3 the Yb-ion has a large magnetic moment of the order of 2μ B and therefore the dipolar interactions cannot be neglected. At the classical level, one can show that they favour antiferromagnetic Néel order with the spins along the c-axis 35 enabled by a spin gap at the zone centre of~200 μeV. This spin gap caused by the dipolar interaction is reduced by the CEF easy-plane anisotropy which contributes to a destabilization of the Néel state at finite temperature (see Fig. 2. in Supplementary Note 3). At g crit ≈ g zz /g xx ≡ 0.985 the spin gap closes and quantum fluctuations will be enhanced. Below that value, the spins rotate into the basal plane. Linear spin-wave theory predicts that easy-plane anisotropy entails a lifting of the degeneracy of the two spin-wave branches at the zone centre, and the splitting increases with increasing anisotropy. A large anisotropy in YbBr 3 would then become measurable since the branch separation becomes large enough to be resolved. A computation of S(Q, ω) at g crit is shown in Fig. 3 and describes the observed dispersion and intensities of the sharp excitations very well.
Experimentally, we have observed neither a splitting of spin waves nor a spin gap within the available energy resolution. This suggests that the absence of long-range order in YbBr 3 at T = 100 mK is caused by the competition between easy-plane anisotropy which favours spins in the plane, and dipolar interactions that favour Ising order along the c-axis. These opposing trends will enhance quantum fluctuations which places YbBr 3 close to the quantum critical point towards a QSL of the spin-1/2 Heisenberg Hamiltonian on the honeycomb lattice.
Our experiment provides clear evidence for the presence of a continuum of excitations at high energies in YbBr 3 . We observe that the intensity of the continuum is stronger at the M' points along (h, −1, 0) and (h, h, 0) directions whereas it is weak along (0, k, 0) and at the Γ and Γ 0 points, in contrast to calculations of the two-magnon cross-section for the Heisenberg Hamiltonian (see Supplementary Fig. 3). We found, as shown in Fig. 5, that this modulation of the neutron intensity associated with the continuum can be reproduced by a RPA calculation for a hexamer plaquette with the exchange parameters obtained from the spinwave calculations (see 'Methods'). This picture of local excitations in YbBr 3 is supported by an analogous calculation of the magnetic susceptibility which shows a broad maximum at T ≃ 4 K (see Fig. 4 in Supplementary Note 5). Similar excitations associated with small spin clusters were also observed in the spinel lattice 36 . Our neutron measurements are also in agreement with recent Monte-Carlo calculations of the dynamical structure factor for the frustrated honeycomb lattice 15 that show a deconfined twospinon continuum 37 with enhanced intensity at the zone boundary due to proximity of a quantum critical point.
In summary, we have shown that the magnetic ground state of YbBr 3 exhibits only short-range order well below the maximum in the static susceptibility. Analysis of the dispersion of the magnetic excitations reveals competition between the nearest-neighbour and next-nearest-neighbour exchange interactions, but we did not observe the mode softening at the K-point which has been predicted 15 for large J 2 . This could be related to the large experimental uncertainty in the value of J 2 . Also, it is known that a large value of J 2 is not necessary for the existence of fractional excitations in 2D systems 38 . However, an unfrustrated Heisenberg model (J 2 = 0) would for our value of J 1 result in a long correlation length at T = 100 mK that corresponds to a narrow resolutionlimited peak which is at variance with our observations 39 . We observed a continuum of excitations with the spectrum of excitations extending to approximately twice the energy of the position of the maximum in S(Q, ω). The neutron inelastic intensity due to the continuum follows the modulation expected for the fluctuations of a honeycomb spin plaquette. Our results demonstrate that YbBr 3 is a two-dimensional S = 1/2 system on the honeycomb lattice with spin-liquid properties without Kitaev-type interactions. The observation of the continuum associated with localized plaquette excitations supports the view of a deconfined quantum critical point 40 in the frustrated honeycomb lattice, in agreement with results from coupled-cluster methods, density matrix renormalization group calculations and Monte-Carlo simulations 14,15,41 . Our measurements set a quantitative benchmark for future theoretical work.

Crystal growth and sample preparation
An YbBr 3 single crystal of cylindrical shape (15-mm diameter, 18-mm height) was grown from the melt in a sealed silica ampoule by the Bridgman method, as previously described for ErBr 3 42 . YbBr 3 was prepared from Yb 2 O 3 (6N, Metall Rare Earth Ltd.) by the NH 4 Br method 43 and sublimed for purification. All handling of the hygroscopic material was done under dry and O-free conditions in glove boxes or closed containers.

Magnetic susceptibility
The magnetic susceptibility was determined with a MPMS SQUID system (Quantum Design).

Neutron scattering experiments
The neutron experiments were performed at the Swiss Spallation Neutron Source (SINQ) utilizing different instruments. On all instruments, filters were used to reduce contamination of the beam by higher-order neutron wavelengths.
The crystal structure of YbBr 3 was refined using diffraction data collected with the high-resolution powder diffractometer HRPT at the wavelength of λ = 1.494 Å at room temperature. The crystal structure and lattice parameters were refined with Fullprof.
The magnetic ground state was investigated with the multi-counter diffractometer DMC at the wavelength λ = 2.4576 Å which integrates fluctuations up to a maximum of~13.5 meV. The measured neutron intensity is proportional to the equal time spin-spin correlation function.
The crystal-field splitting of the Yb 3+ ions was determined on the thermal three-axis spectrometer EIGER operated in the constant finalenergy mode with k f = 2.662 Å −1 at T = 1.5 K and |Q| = 1.5 Å −1 . With that configuration the energy resolution is 0.8 meV.
The dispersion of the magnon excitations is bound by hω(q) < 1 meV in YbBr 3 which required the use of cold neutrons that provide an improved energy resolution. Therefore the measurements of the spin waves were performed with the TASP three-axis spectrometer using k f = 1.3 Å −1 which resulted in an energy resolution of 80 μeV. To maximize the intensity, the measurements were performed without collimators in the beam and the analyser was horizontally focusing.