Introduction

The nature of QSLs has been well established in one-dimensional (1D) spin systems. However, it still remains unclear how QSLs emerge in dimensions greater than one. The celebrated resonating-valence-bond theory on a 2D triangular lattice1,2 puts forward the possibility that geometrical frustration plays an important role in stabilizing QSLs. In fact, a few candidate materials hosting QSLs have now been reported in materials with 2D triangular lattices3,4,5,6,7,8. Nevertheless, according to subsequent theoretical studies9,10, the effect of geometrical frustration in the triangular lattice is insufficient to stabilize QSLs, leading to a number of proposed mechanisms that may stabilize the QSL states found in the candidate materials11. One of the most promising approaches is to utilize a coupling of spins with charges and orbitals; the former has been discussed near a Mott-insulator-to-metal transition where the charge degrees of freedom begin to delocalize12,13,14,15,16,17,18 and the latter has been considered in the framework of a spin–orbital coupling19. Such strategies, however, have been limited to the utilization of internal degrees of freedom of electrons.

The hydrogen-bonded organic Mott insulator κ-H3(Cat-EDT-TTF)2 (hereafter abbreviated as H-Cat)5,20,21,22 may serve as a candidate for a different class of QSLs, where H2Cat-EDT-TTF is catechol-fused ethylenedithiotetrathiafulvalene (see Fig. 1a–c). H-Cat forms a 2D spin-1/2 Heisenberg triangular lattice of Cat-EDT-TTF dimers20 (Fig. 1e). Despite the antiferromagnetic interaction energy J/k B of ~80 K, no magnetic order has been observed down to 50 mK; this indicates the realization of a QSL state5. A distinct feature of H-Cat is that the 2D π-electron layers are connected by hydrogen bonds20,21 (Fig. 1a, c), which is in marked contrast to other 2D organic QSL materials such as κ-(BEDT-TTF)2Cu2(CN)3 (ref. 3) and EtMe3Sb[Pd(dmit)2]2 (ref. 4), where the 2D spin systems are separated by non-magnetic insulating layers. This structural feature of H-Cat is highlighted by deuteration of the hydrogen bonds21; specifically, in the deuterated analog of H-Cat, κ-D3(Cat-EDT-TTF)2 (denoted as D-Cat), deuterium localization occurs at T c = 185 K, accompanied by charge disproportionation within the Cat-EDT-TTF layers, resulting in a non-magnetic ground state (Fig. 1d, f). This demonstrates that the hydrogen bonds in this system strongly couple with the charge and spin degrees of freedom of the π-electrons in the Cat-EDT-TTF dimers.

Fig. 1
figure 1

Crystal structures of H-Cat and D-Cat. a Crystal structure of H-Cat viewed along the b axis. b Molecular structure of H2Cat-EDT-TTF. c Hydrogen-bonded molecular unit H3(Cat-EDT-TTF)2 (supramolecule) in H-Cat. The supramolecules are formed by two [H(Cat-EDT-TTF)]0.5+ molecules connected by the hydrogen bond. The supramolecules are stacked along the (a + c) direction, as shown in a. For clarity, a part of the stacking columns is colored red and purple in a. In the b –c plane, two face-to-face [H(Cat-EDT-TTF)]0.5+ molecules form a strongly dimerized unit (gray ellipsoid), which generates the 2D π-electron layers connected by the O–H–O hydrogen bonds (light blue circle). d Supramolecular unit in D3(Cat-EDT-TTF)2. Note that δ in c, d denotes the displacement of the hydrogen and deuterium atoms from the center of the O–H–O and  O–D–O bonds, respectively. e Spin and charge structures of the 2D π-electron layer in H-Cat. The π-dimers form a slightly anisotropic triangular lattice with S = 1/2 spins (magenta arrows). Dotted and solid lines show the inter-dimer hopping integrals, t and t′, respectively. f Spin and charge structures of a 2D π-electron layer in D-Cat below the phase transition temperature T c of 185 K. Charge disproportionation associated with deuterium localization leads to a non-magnetic ground state below T c . The blue- and orange-colored dimers indicate the charge-rich (+0.94) and charge-poor (+0.06) sites, respectively21. g, h Schematics of the potential energy curves of the hydrogen atoms in H-Cat (g) and the deuterium atoms in D-Cat (h). In H-Cat, the potential energy curve is suggested to change from a double-well structure (dashed line) to a very shallow and anharmonic single-well structure (red solid line) owing to the many-body effect arising from the network of hydrogen bonds and π-electrons22,23. In sharp contrast, the energy curve in D-Cat retains a double-well structure above T c = 185 K (dashed line), leading to the deuterium localization at T c (blue solid line)

In contrast to D-Cat, the hydrogen atoms in H-Cat do not localize down to low temperatures21. This is inconsistent with the fact23 that the potential energy curve of the hydrogen bonds calculated for an isolated supramolecule has a double minimum potential with a large energy barrier of ~800 K (see Fig. 1c, g), which should localize the hydrogen atoms in H-Cat at low temperatures. Recent theoretical calculations22,23 have pointed out that the potential energy curve has a single-well structure and its bottom becomes very shallow and anharmonic (see Fig. 1g) owing to a many-body effect arising from the network of hydrogen bonds and π-electrons. In this case, the zero-point motion of the hydrogen atoms (termed “proton fluctuations”) can be strongly enhanced by the anharmonic potential curve. In contrast to D-Cat, the enhanced proton fluctuations may delocalize the hydrogen atoms down to absolute zero, providing an opportunity for realizing a QSL state through strong coupling between the hydrogen bonds and the π-electrons. However, it has not been established whether such strong quantum proton fluctuations are indeed present in H-Cat, and if so, how the quantum fluctuations affect the QSL state.

Here we show, by using a combination of dielectric permittivity and thermal conductivity measurements, that the quantum proton fluctuations presented in H-Cat provide a quantum-disordered state of magnetic and electric dipoles through the coupling between π-electrons and hydrogen atoms. These methods are particularly suitable because the dielectric permittivity is sensitive to local electric-dipole moments arising from the hydrogen-bond dynamics24, whereas the thermal conductivity is a powerful probe to detect itinerant low-lying energy excitations associated with the nature of QSL states25,26.

Results

Dielectric permittivity measurements

Figure 2 shows the temperature dependence of the dielectric constant \(\epsilon _{\mathrm{r}}\)(T) for H-Cat and D-Cat. In H-Cat, \(\epsilon _{\mathrm{r}}\)(T) steeply increases with decreasing temperature and then saturates below ~2 K. In sharp contrast, \(\epsilon _{\mathrm{r}}\)(T) of D-Cat is temperature-independent owing to deuterium localization (Fig. 2a). The temperature dependence of \(\epsilon _{\mathrm{r}}\) for H-Cat is a typical dielectric behavior observed in quantum paraelectric (QPE) materials such as SrTiO3 (ref. 27), in which long-range electric order is suppressed by strong quantum fluctuations. In the QPE state, \(\epsilon _{\mathrm{r}}\)(T) is described by the so-called Barrett formula28:

$$\epsilon _{\mathrm{r}}(T) = A + \frac{C}{{\frac{{T_1}}{2}{\mathrm{coth}}\left( {\frac{{T_1}}{{2T}}} \right) - T_0}}.$$
(1)

Here, A is a constant offset, C =  2/k B is the Curie constant (where n is the density of dipoles, μ is the local dipole moment, and k B is the Boltzmann constant), T 0 is the Curie–Weiss (CW) temperature in the classical limit (that is, a temperature at which (anti)ferroelectric order occurs in the absence of strong quantum fluctuations) and T 1 is the characteristic crossover temperature from the classical CW regime to the QPE regime. As shown in the solid line in Fig. 2b, \(\epsilon _{\mathrm{r}}\)(T) of H-Cat is well fitted by the Barrett formula with T 0 = −6.4 K and T 1 = 7.7 K; this confirms that strong quantum fluctuations suppress long-range electric order. The relative strength of quantum fluctuations among different QPE materials can be evaluated by the ratio of T 1 to T 0. The value of T 1/T 0 in H-Cat is 1.2, which is smaller than that of the typical QPE material SrTiO3 (T 1/T 0 = 2.3, see ref. 27). This is consistent with the experimental fact that the QPE behavior of SrTiO3 is more significant than that of H-Cat. The obtained negative value of T 0 immediately indicates the presence of an antiferroelectric (AFE) interaction in H-Cat, which is consistent with the AFE configuration resulting from deuterium localization in D-Cat (see the inset of Fig. 2b). Therefore, the observed quantum paraelectricity in H-Cat clearly shows that strong quantum fluctuations that suppress the hydrogen-bond order as observed in D-Cat arise from the potential energy curve of H-Cat, consequently leading to the persistence of enhanced proton fluctuations down to low temperatures. The presence of strong quantum fluctuations is consistent with the recent theoretical calculations that highlight the importance of strong many-body effects imposed by the proton–π-electron network on the potential energy curve of H-Cat22,23.

Fig. 2
figure 2

Dielectric permittivity of H-Cat and D-Cat. a Temperature dependence of the dielectric constant \(\epsilon _{\mathrm{r}}\)(T) = \(\epsilon\)/\(\epsilon _{\mathrm{0}}\) in H-Cat (red, left axis) and D-Cat (green, right axis), where \(\epsilon\) is the dielectric permittivity and \(\epsilon _{\mathrm{0}}\) is the vacuum permittivity. The inset illustrates our measurement configuration. The dielectric permittivity was measured by applying an a.c. electric field (E a.c.) of 1 kV/cm with a frequency of 1 MHz along the a* axis. The sample, d.c. bias and frequency dependence of the dielectric behavior is discussed in Supplementary Note 1 (see also Supplementary Fig. 1). b The same data for H-Cat plotted on a logarithmic temperature scale. The solid line is a fit to the Barrett formula (see the main text), demonstrating the emergence of a QPE state in H-Cat. The inset shows the electric dipole moments induced within the deuterium bonds and the Cat-EDT-TTF dimers in D-Cat. Because of deuterium localization, the adjacent deuterium bonds possess the local electric dipoles oriented in antiparallel directions (green arrows) below T c . Concomitantly, charge disproportionation is induced within the Cat-EDT-TTF dimers (Fig. 1d), giving rise to the local electric dipoles (blue arrows). Thus, the antiferroelectric interactions are induced by the hydrogen/deuterium-bond dynamics

Thermal conductivity measurements

Using thermal conductivity measurements, we next examine how the proton dynamics in the QPE state affects the nature of the QSL state in H-Cat. Figure 3 shows the temperature dependence of the thermal conductivity of H-Cat (κ H) and D-Cat (κ D). The heat in H-Cat is carried by the phonons \(( {\kappa _{{\mathrm{ph}}}^{\mathrm{H}}} )\) and the spin excitations \(( {\kappa _{{\mathrm{sp}}}^{\mathrm{H}}} )\), whereas in non-magnetic D-Cat, it is transported only by phonons \(( {\kappa _{{\mathrm{ph}}}^{\mathrm{D}}} )\). Assuming that H-Cat and D-Cat share almost identical phonon thermal conductivity \(( {\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\sim \kappa _{{\mathrm{ph}}}^{\mathrm{D}}} )\), the relation \(\kappa ^{\mathrm{H}} = \kappa _{{\mathrm{ph}}}^{\mathrm{H}} + \kappa _{{\mathrm{sp}}}^{\mathrm{H}} \ge \kappa _{{\mathrm{ph}}}^{\mathrm{D}} = \kappa ^{\mathrm{D}}\) holds. Unexpectedly, however, we find that κ H < κ D above 2 K (see Fig. 3a), indicating that \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\) is much more suppressed than \(\kappa _{{\mathrm{ph}}}^{\mathrm{D}}\).

Fig. 3
figure 3

Thermal conductivity of H-Cat and D-Cat. a Temperature dependence of κ/T of H-Cat (filled red symbols) and D-Cat (open green symbols) below 10 K in the zero-field. b Low-temperature thermal conductivity for H-Cat and D-Cat below 0.8 K in the zero-field. The extrapolated data (dashed lines) assume that κ phT 2 (for details, see Supplementary Note 3). A clear residual κ/T in the zero-temperature limit is resolved for H-Cat. The peak structure of κ H/T around 0.3 K is considered to arise from both \(\kappa _{{\mathrm{sp}}}^{\mathrm{H}}\) and \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\) (See Supplementary Note 6)

To investigate the origin of this suppression, we employ the Callaway model29, which describes the heat transport of acoustic phonons. Above 2 K, κ H is reproduced by the model including a single resonance scattering mode with a resonance energy of ħω 0/k B ~ 5–10 K in addition to standard scattering processes (Supplementary Fig. 2; Supplementary Table 1; Supplementary Note 2). This energy scale is close to the proton fluctuations (T 1 = 7.7 K), indicating that resonance scattering arises between the acoustic phonons and the optical mode from the hydrogen bonds. Here, it should be noted that we can rule out the possibility of a spin–phonon scattering for the suppression of \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\), because the spin–orbit coupling of H-Cat is very weak, as confirmed by the small field dependence of κ H (Fig. 4a). Thus, it appears that the thermal fluctuations of the hydrogen bonds strongly suppress \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\) above 2 K.

Fig. 4
figure 4

Paraelectric and spin liquid phases of H-Cat. a Temperature dependences of κ(H) for H-Cat at 0 and 10 T. b A combination of the temperature dependences of the dielectric constant \(\epsilon _{\mathrm{r}}\) (blue, left axis), the thermal conductivity divided by temperature κ/T (red, left axis) and the magnetic susceptibility χ (green, right axis) for H-Cat. The values of χ are taken from ref. 5. The dashed line is an eye guide. The shaded region represents the QPE and QSL phases. c Magnetic field dependence of the thermal conductivity κ(H) normalized by the zero field value [κ(H) − κ(0)]/κ(0) for H-Cat (filled symbols) and EtMe3Sb[Pd(dmit)2]2 (open circles)26

Below 2 K, κ H rapidly increases and eventually exceeds κ D (Fig. 3a). This rapid increase of κ H may come from increases in \(\kappa _{{\mathrm{sp}}}^{\mathrm{H}}\) as well as \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\). We now investigate the behavior of κ H at lower temperatures where \(\kappa _{{\mathrm{sp}}}^{\mathrm{H}}\) becomes dominant over \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\); this provides essential information on the low-lying excitation spectrum characterizing the QSL state25,26. As shown in the dotted lines in Fig. 3b, both \(\kappa _{{\mathrm{ph}}}^{\mathrm{H}}\) and \(\kappa _{{\mathrm{ph}}}^{\mathrm{D}}\) exhibit a T 2-dependence rather than the conventional T 3-dependence. This originates from the influence of high-quality crystals with specular surfaces (Supplementary Fig. 3; Supplementary Note 3). The zero-temperature extrapolation of κ H/T shows a finite residual (Fig. 3b; Supplementary Fig. 4), thereby demonstrating a gapless spin excitation with high mobility (the mean free path of the gapless spin excitations l sp is estimated to be ~120 nm; See Supplementary Note 4). This result is consistent with recent magnetic torque measurements5 of H-Cat. Here, we stress that we can exclude the possibility that the itinerant low-energy excitations are due to either phonons or electric dipoles (see Supplementary Note 5).

Discussion

A key question raised here is how the gapless QSL state is stabilized in H-Cat. In organic QSL candidates with a triangular lattice charge fluctuations near a Mott transition12,13,14,15,16,17,18 have been pointed out to play an important role for stabilizing the QSLs. However, H-Cat is located deeper inside the Mott-insulating phase compared to the other organic QSL candidates, κ-(BEDT-TTF)2Cu2(CN)3 (ref. 3) and EtMe3Sb[Pd(dmit)2]2 (refs. 4,30). The distance from a Mott transition is inferred from the ratio of the on-site Coulomb repulsion U to the transfer integral t, which is given by \(U{\mathrm{/}}t\sim t{\mathrm{/}}J\). Whereas the transfer integrals are comparable among the three materials, J for H-Cat is ~1/3 compared to that for the other two (see Supplementary Note 7); this means that U/t is significantly large in H-Cat. Indeed, H-Cat sustains an insulating behavior even at 1.6 GPa (ref. 20), whereas the other two compounds become metallic at 0.4–0.6 GPa (ref. 31); this result also supports that H-Cat is far from the Mott transition. Therefore, the QSL in H-Cat should be stabilized using a different mechanism.

Figure 4b shows the temperature dependence of the dielectric constant \(\epsilon _{\mathrm{r}}\), the thermal conductivity divided by temperature κ/T and the magnetic susceptibility χ (ref. 5) for H-Cat. Below 2 K, the thermal conductivity increases upon entering the QPE state, where \(\epsilon _{\mathrm{r}}\) saturates. The characteristic temperature coincides with the temperature at which the susceptibility becomes constant; this occurs when the spin correlation develops in the QSL state5. The coincidence of the QPE and QSL states is surprising and strongly suggests that the development of the quantum proton fluctuations triggers the emergence of the QSL. We now theoretically analyze the effects of proton dynamics on the QSL state. In H-Cat, the charge degrees of freedom of the hydrogen bonds and the π-electrons are strongly coupled because of the charge neutrality within the H3(Cat-EDT-TTF)2 supramolecule (see Fig. 1c, d). A minimal and realistic model that describes the coupling between π-electrons and hydrogen bonds is the extended Hubbard model coupled with the proton degree of freedom; this model captures the essence of this system, namely that the proton and charge degrees of freedom are strongly entangled in H-Cat (Supplementary Fig. 5; Supplementary Note 8). According to the present model, charge fluctuations inside a dimer and/or between two neighboring dimers are affected by proton fluctuations for the following reasons: the magnetic exchange processes are governed by the virtual electron hopping (mainly the second term in Supplementary Eq. (7)), which is of the order of 100 meV in the present system20,21,22. In contrast, the time scale of the proton fluctuations T 1 = 7.7 K (~1 meV) is two orders of magnitude slower than that of the electron hopping. Such low-energy proton fluctuations modulate the amplitude of the electron transfers and the energy levels of the molecular orbitals. These effects may induce a dynamical modulation of J as well as a reduction of the on-site Coulomb repulsion U due to the bi-polaron effect32, both of which appear to destabilize the magnetic long-range order, that is, induce a QSL state.

Finally, we discuss the magnetic field dependence of the thermal conductivity in the QSL state of H-Cat. As shown in Fig. 4c, the field dependence of the thermal conductivity at low temperatures in H-Cat is negligibly small (a slight increase of 1–2% against the magnetic field of 10 T). In contrast, in κ-(BEDT-TTF)2Cu2(CN)3 (ref. 25) and EtMe3Sb[Pd(dmit)2]2 (ref.26), the magnetic field dependence of the thermal conductivity shows a gap-like behavior, which has been discussed in terms of an inhomogeneous QSL3,25,33,34,35 and a dichotomy of gapless and gapped excitations26,30, respectively. Therefore, the observed negligibly small field dependence in H-Cat indicates the absence of gapped excitations with magnetic field, which may suggest a more globally homogeneous QSL with gapless excitations. One of the possible explanations is a gapless spinon Fermi surface over the whole k-space36 (for details, see Supplementary Note 9). Recent torque measurements5 of H-Cat have shown that spin excitations behave as Pauli-paramagnetic-like low-energy excitations where the Fermi temperature T F is estimated to be 350 K. In such a case, in the regime where \(T_{\mathrm{F}} \gg g\mu _{\mathrm{B}}H\) (here, μ B is the Bohr magneton and g is the g-factor of Cat-EDT-TTF dimer with spin-1/2), the total number of spin excitations in the applied magnetic field becomes constant in a 2D system, and the velocity of the spin excitations v sp is assumed to be almost field-independent. As a result, κ sp becomes essentially field-independent.

The homogeneous gapless QSL insensitive to magnetic fields may originate from the structure of the present system; the 2D π-electron layers of H-Cat are connected by hydrogen bonds, whereas in the other organic QSL candidates they are separated by anion layers that may induce randomness in the π-electron system37. This structural difference enables a different mechanism to stabilize the QSL state in H-Cat compared to previous organic QSL candidates. The QSL realized in H-Cat can be induced by quantum proton fluctuations rather than charge fluctuations near a Mott transition. Thus, our findings suggest that a quantum-disordered state of magnetic and electric dipoles emerges in H-Cat from cooperation between the electron and proton degrees of freedom. Utilizing such a strong coupling between multiple degrees of freedom will advance our explorations of quantum phenomena such as orbital–spin liquids19,38 and electric–dipole liquids17,39.

Methods

Sample preparation

Single crystals of κ-H3(Cat-EDT-TTF)2 and κ-D3(Cat-EDT-TTF)2 were prepared by the electrochemical oxidation method, as described in refs20,21. A typical sample size for both compounds is ~0.03 × 0.12 × 1.0 mm3. In H-Cat, the anisotropy parameter t′/t at 50 K is estimated to be 1.48 by the extended Hückel method5 or 1.25 by the first principles DFT calculations22 (see Fig. 1e). In contrast, the value of t′/t for D-Cat is estimated to be 1.36 at 270 K by the extended Hückel method21 (see Fig. 1f), which is close to t′/t = 1.37 at 298 K for H-Cat obtained by the same method40.

Dielectric measurements

The dielectric permittivity measurements were carried out down to 0.4 K in a 3He cryostat using an LCR meter (Agilent 4980A) operated at 100 Hz–1 MHz along the a* direction, which is perpendicular to the b-c plane. The experiment was limited to the a*-axis direction by the plate-like shape of the sample. The applied a.c. voltage was 2 V. We confirmed the voltage-independent response of \(\epsilon _{\mathrm{r}}\) up to 2 V below 20 K. The dielectric permittivity was measured by sweeping both temperature and frequency. The electrical contacts were made using carbon paste. The open/short correction was performed before connecting the sample to the measurement system.

Thermal conductivity measurements

The thermal-transport measurements were performed by a standard steady-state heat-flow technique in the temperature range from 0.1 to 10 K using a dilution refrigerator. The heat current was applied along the c axis. The magnetic field was applied perpendicularly to the b-c plane up to 10 T. Two RuO2 thermometers precisely calibrated in the magnetic field and one heater were attached on the sample through gold wires.

Data availability

The data that support the findings of this study are available on request from the corresponding authors (M.S. or K.H.).