## Introduction

Organic solids offer plenty of playgrounds of conspicuous phenomena including superconductivity competing with charge and spin orders, and semi-metallic excitations with Dirac dispersions1,2. However, the severe competitions of the diverse phases and mechanisms of the phenomena found in complex crystal structures of the organic solids remain challenges of condensed matter physics.

Despite a large number of atoms in the unit cell of such organic compounds, however, the band structure near the Fermi level is mostly simple consisting of only LUMO (lowest unoccupied molecular orbital) and HOMO (highest occupied molecular orbital) around the Fermi level. Large inter-molecular distance leads to narrow bandwidths while poor screening of Coulomb interaction due to the sparse bands near the Fermi level leads to strongly correlated electron systems with various types of Mott insulators.

Among them, the QSL phase was proposed as novel states of matter in two families of correlation driven Mott insulating compounds, namely, κ-(ET)2X (ET= bis(ethylenedithio) tetrathiafulvalene) with the anion X=Cu2(CN)33 and $$\beta ^{\prime}$$-X[Pd(dmit)2]2 (dmit=1,3-dithiole-2-thione-4,5-dithiolate) with the cation X=EtMe3Sb4, where apparent spontaneous symmetry breaking including the antiferromagnetic (AF) order is unusually absent even at several orders of magnitude lower temperatures than the spin-exchange interaction. Although the geometrical frustration arising from the triangular lattice structure of the dimerized Pd(dmit)2 or ET molecule hosting a spin-1/2 electronic spin looks important for the absence of the AF order, its nature and mechanism of the emergence endorsed by a quantitative estimate are missing. In addition to the QSL, if one replaces X with other ions, it shows a variety of phases including the AF order, charge order and valence bond solid1,2,5. Because of severe competitions of these phases, ab initio approaches without adjustable parameters are desired particularly for the enigmatic QSL to reach realistic understanding.

We focus on a typical candidate of QSL, X[Pd(dmit)2]2. In fact, it shows4,6 smaller broadening of the nuclear magnetic resonance (NMR) spectra than κ-(ET)2Cu2(CN)31,7 implying smaller effects of extrinsic spatial inhomogeneity. At low temperatures of the QSL material, EtMe3Sb[Pd(dmit)2]2, the specific heat8 and the thermal conductivity κ9 are reported to be proportional to temperature T, although controversies exist for the latter10,11,12. The magnetic susceptibility stays at a nonzero constant13. These are roughly similar to the conventional Fermi-liquid metal but of course the electrons are frozen in the Mott insulator and the spin degrees of freedom must be responsible for them. In addition, the relaxation rate of NMR, 1/T1 seems to be scaled by T2 in the range 0.05K ≤ T < 1 K6 in contrast to the conventional Fermi-liquid behavior T. It is then desired to gain insights into the experimental consistency from ab initio calculations.

Here, we thoroughly study purely ab initio electronic Hamiltonians without any adjustable parameters for five dmit compounds using the experimental structure without optimizing lattice structures except for positions of hydrogen atoms. This ab initio study allows us to correctly reproduce the overall experimental phase diagram at low temperatures of $$\beta ^{\prime}$$-X[Pd(dmit)2]2 with X=Me4P, Me4As, Me4Sb, Et2Me2As exhibiting the AF order and with X=EtMe3Sb exhibiting QSL phases. We do not consider $$\beta ^{\prime}$$-Et2Me2Sb[Pd(dmit)2]2, because the observed charge order with the simultaneous spontaneous lattice distortion in this compound14,15 requires structural optimization, which is out of the scope of this paper. Then we conclude that the QSL phase has a character of primarily 1D spin liquid but combined with 2D nature, which accounts for the experimental properties consistently. From the study, we attempt to pin down the nature of QSL observed in the experiments, which is one of the central issues in condensed matter physics.

## Results

### Ab initio framework

Before going into our results, we summarize our framework of the calculation for the sake of readers to understand definitions of quantities, which we will analyze. We use ab initio low-energy effective Hamiltonian consisting of the half-filled HOMO band crossing the Fermi level derived for $$\beta ^{\prime}$$-X[Pd(dmit)2]216,17,18. For the procedure and the obtained parameters for the Hamiltonians, see Methods. It is built on the weakly coupled two-dimensional layers and the Pd(dmit)2 dimer forms an anisotropic triangular lattice within a layer. We assign the $$\tilde{a}$$, $$\tilde{b}$$ and $$\tilde{c}$$ bonds, for which the Hamiltonian presented below (Eq. (1)) contains the transfer parameters ta, tb and tc, respectively, as is illustrated in the middle panel of Fig. 1. The bonds are chosen so that ta, tb and tc are ordered from the stronger to weaker amplitudes. In the actual calculation, we take a deformed lattice structure illustrated in the right panel of Fig. 1 just for computational simplicity. We sometimes use x and y axes for the plot of the 2D Brillouin zone and note that x and y correspond to $$\tilde{a}$$ and $$\tilde{c}$$ directions, respectively. (Do not confuse the crystallographic a, b and c axes with the present assignment of $$\tilde{a}$$, $$\tilde{b}$$ and $$\tilde{c}$$ axes).

Because of very weak interlayer hopping (the largest interlayer hopping is less than 0.8 meV), the 2D plane is sufficient to capture physics of our interest about the ground state here. Then our Hamiltonian is built on the two-dimensional plane of the dmit salts. We take into account the effect of interlayer interaction through the dimensional downfolding method established in organic solids and iron-based superconductors16 (see Methods). The maximally localized Wannier function of the HOMO orbital, whose band crosses the Fermi level, is constructed for the molecular orbital19,20. The lattice structure in the simulation is depicted later in Fig. 6, which is, despite the deformation of the shape, able to sufficiently describe the ab initio Hamiltonian as the network of the Pd(dmit)2 dimers, where a Wannier orbital is extended over a dimer.

The resultant ab initio single-band effective Hamiltonian derived in refs. 17,18 has the form

$$\begin{array}{l}{{{\mathcal{H}}}}=-\mathop{\sum}\limits_{\sigma }\mathop{\sum}\limits_{i,j}{t}_{ij}{c}_{i\sigma }^{{\dagger} }{c}_{j\sigma }+\mathop{\sum}\limits_{i}U{n}_{i\uparrow }{n}_{i\downarrow }\\ +\mathop{\sum}\limits_{i < j}{V}_{ij}({n}_{i}-1)({n}_{j}-1),\end{array}$$
(1)

where i, j represent the dimer indices, and $${c}_{i\sigma }^{{\dagger} }$$ (ciσ) is the creation (annihilation) operator of electrons with spin σ (= or ) at the ith Wannier orbital, and the number operator is ni = ∑σniσ with $${n}_{i\sigma }={c}_{i\sigma }^{{\dagger} }{c}_{i\sigma }$$. Here, tij is the hopping parameters depending on the relative coordinate vector rirj, where ri is the position vector of the center of the ith Wannier orbital. In the present study, for tij, we retain the nearest neighbor pair of ri and rj in each direction of $$\tilde{a}$$, $$\tilde{b}$$ and $$\tilde{c}$$ as is illustrated in Fig. 1, while Vij is retained up to the third neighbor (see Fig. 6). Here, U and Vij are the screened on-site and off-site Coulomb interactions, respectively, as is illustrated in Fig. 6. Note that the Hamiltonian parameters of both transfer and interaction terms contain neither adjustable parameters nor fitting and are determined solely by using the experimental lattice structure at low temperatures and by following the established procedure of the maximally localized Wannier functions. The spatial ranges of the transfer and the interaction, which we take are sufficient to reach the convergence for the present study and the values at longer distance are small. We also ignore the direct exchange interactions because they are at most 3 meV and we expect it does not alter our conclusions. See Methods for the parameter values of Hamiltonian (1) used in the present study.

We apply a variational Monte Carlo (VMC) method21,22,23 to the ab initio Hamiltonians to reach highly accurate ground states. For details of the numerical method, see Methods.

### Phase diagram

Figure 2 is one of our central results of this paper, showing the agreement between the experimental (bottom plane) and the calculated (vertical plane) material dependences of the low-temperature phases. In the calculated results, the collinear AF and the QSL states are the two lowest energy states among severely competing various candidates all through the five compounds studied. Therefore, we plot the calculated energy difference between these two in the vertical plane. We find that the experimental QSL is reproduced in the calculated ground state for X=EtMe3Sb. Aside from a severe competition for X=Et2Me2As, which is indeed suggested by the experimental coexistence of AF and QSL, Fig. 2 shows that our ab initio results successfully reproduce the AF ground state observed in the experiments of X=Me4P, Me4As, and Me4Sb, as shown in the bottom plane. The AF state has the Bragg peak at (π, 0) in the notation of Fig. 1. The AF stability was recently studied by the quasi-1D approach based on the random phase approximation24.

Note that the abscissa of Fig. 2 shows only (tctb)/ta dependence among the ab initio parameters, while the plots of the five materials are the results of computation by using the full ab initio parameters. The overall monotonic dependence of Δe shows that (tctb)/ta is indeed a principally important parameter for the evaluation of the phase diagram. The experimental Néel temperature in the bottom plane is also consistently ordered with (tctb)/ta and further supports the relevance and accuracy of the ab initio effective Hamiltonian, because it is natural to expect that the Néel temperature is linked to the relative stability of the ground state. Within the parameter control of (tctb)/ta, the transition between the AF and QSL phases is a clear first-order transition, represented by the level crossing. The importance of the full anisotropy of the transfer in the inequilateral triangular lattice to stabilize the QSL phase was pointed out by the projected BCS study of the Hubbard-type model by using the ab initio hopping25.

### Nature of quantum spin liquid and antiferromagnetic state

Figure 3 shows the spin structure factor S(q) (the spin correlation in the Fourier space) for X=Me4P as a representative case of the AF ordered compounds and for the QSL compound, X=EtMe3Sb (see Methods for details of the calculation method and definition of physical quantities). For the AF state of X=Me4P, a strong sharp peak at q = (π, 0) indeed indicates the conventional collinear 2D AF order. The 2D ordering pattern is illustrated in Fig. 2. Other compounds, X=Me4As, Me4Sb, and Et2Me2As also show the same type of the order. In the QSL state for EtMe3Sb, S(q) also has a peak at the same momentum q = (π, 0), but with a substantially reduced height. Interestingly, a ridge line along qx = π emerges in S(q) of the QSL state, implying anisotropy between the x and y directions (namely between the chain ($$\tilde{a}$$) and interchain ($$\tilde{b}$$) directions). Such anisotropy is in sharp contrast with the isotropic order in the AF phase. Furthermore, the feature of the anisotropy is largely different from the previous numerical studies for the anisotropic triangular Hubbard or Heisenberg model26. Figure 3(c) shows the size dependence of the peak value of S(q) divided by the system size Ns = L2. Because the AF long-range order is signaled by a nonzero S(q)/Ns in the thermodynamic limit 1/L → 0, AF order exists for X=Me4P and it is absent for X=EtMe3Sb.

The spin-spin correlations in real space C(r) for the QSL state of X=EtMe3Sb shown in Fig. 3d with the log-log plot in the x (namely, ta) direction indicates a power law C(r) rp with p = 1.88 ± 0.01 suggesting the algebraic QSL with the gapless excitation. On the other hand, Fig. 3(e) shows a semilog plot of the correlations in the y direction, for which the exponential form $$C({{{\boldsymbol{r}}}})\propto \exp [-| {{{\boldsymbol{r}}}}| /{\xi }_{\perp }]$$ with the correlation length ξ ~ 1.33 ± 0.04 is appropriate and thus the excitation is gapped in the interchain (y) direction at least within the system size studied here. Of course, the same exponential decay of the correlation is observed in the tb (namely $$\tilde{b}$$) and tc (namely, $$\tilde{c}$$) directions. The anisotropic correlation develops the ridge in the qy dependence of S(q).

Finally, the spin Drude weight Ds calculated in the QSL state for X=EtMe3Sb is shown in Fig. 4. The size dependence of Ds for the QSL shown in Fig. 4(b) assures that Ds is nonzero and large in the thermodynamic limit, consistently with the power-law decay of the spin correlation in the x direction. In contrast, the spin Drude weight for the interchain y component is vanishing.

All the results support that the QSL with one-dimensional and gapless excitation is realized for X=EtMe3Sb, which is our second central result of the paper.

## Discussion

The QSL of $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]2 is characterized by the gapless spin excitation and associated algebraic power-law decay of the spin correlation in the direction of the largest transfer ta (namely, along the chain direction). The power p ~ 1.9 for $$C({{{\boldsymbol{r}}}})\propto \exp [i{{{\boldsymbol{Q}}}}\cdot {{{\boldsymbol{r}}}}]{{{{\boldsymbol{r}}}}}^{-p}$$ is similar to the value obtained for the QSL in the square-lattice J1J2 Heisenberg model with the nearest (J1) and the next-nearest-neighbor (J2) exchange interactions, where p ~ 1.4–1.727. However, in contrast to isotropic 2D spin correlation in the J1J2 model, the correlation decays exponentially in the interchain direction with the correlation length ξ ~ 1.3 lattice constant. The anisotropy makes the peak height of S(q) non-divergent in the thermodynamic limit in contrast with the 2D J1J2 model.

One might regard the present QSL as essentially the same as the 1D spin liquid26,28,29 smoothly connected to an effective 1D Heisenberg or Hubbard model. However, the revealed properties are not so simple. First, the nonzero correlation length in the interchain direction makes a prominent peak at (π, 0) in S(q) commonly to the case of the 2D J1J2 Heisenberg model. In the decoupled arrays of 1D Heisenberg chains, one would expect an equal-height ridge along the line (π, 0)-(π, π) with logarithmically divergent height where p = 1 and without qy dependence. In the present QSL, the ridge exists but the height is not divergent even at (π, 0) because p > 1. For the moment, we leave the issue about the possible presence or absence of the transition between the pure isolated chain and the present case for the future study. Substantial reduction of S(q) toward (π, π) from (π, 0), namely partially 2D-like correlation implies a small but nonzero dispersion of the excitation in the interchain direction, which may make an energetic hierarchy.

For the 2D J1J2 model, it was suggested that the gapless excitation is well accounted for by the fractionalization of a spin into two spin-1/2 spinons, where the spinon has the Dirac-like linear dispersion at (±π/2, ±π/2)27,30. Let us discuss the possibility of the fractionalization for the present QSL of $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]2. To gain insight into the possible existence of spinon and to elucidate its dispersion in the present QSL, we have analyzed the structure of our wavefunction for the QSL ground state of $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]2 as is described in Methods. The result shows that the elementary excitation is consistently described by the spinon born out as a consequence of the fractionalization of a spin, where the spinon has Dirac-type gapless dispersion around (±π/2, 0) and (±π/2, π) as is schematically illustrated in Fig. 5. Because the measurable spin is constructed from the two-spinon excitation, the gapless points for the triplet excitations appear at around (0, 0), (±π, 0), (0, ±π), and (±π, ±π). This structure has an essential similarity to the 2D J1J2 Heisenberg model27, implying the 2D nature of the QSL.

However, the exponential decay of the spin correlation and vanishing spin Drude weight in the interchain direction suggest a very anisotropic and small or incoherent dispersion away from the nodal point in the interchain direction in contrast to the isotropic conic dispersion for the 2D J1J2 model, generating the QSL of roughly 1D-like nature.

By considering the one-dimensional anisotropy, it is insightful to compare the present QSL with that of the 1D antiferromagnetic Heisenberg model defined by

$$H=J\mathop{\sum}\limits_{i}[{S}_{i}^{x}{S}_{i+1}^{x}+{S}_{i}^{y}{S}_{i+1}^{y}+{S}_{i}^{z}{S}_{i+1}^{z}]$$
(2)

with J > 0 for the spin-half operator $${{{{\boldsymbol{S}}}}}_{i}=({S}_{i}^{x},{S}_{i}^{y},{S}_{i}^{z})$$ at site i. The specific heat of the 1D Heisenberg model is given by the T-linear coefficient $$\gamma =2{k}_{{{{\rm{B}}}}}^{2}{N}_{{{{\rm{A}}}}}/3J$$ above 1K31, where kB is Boltzmann constant and NA is Avogadro constant. If we employ the strong coupling expansion for the ab initio Hamiltonian, the superexchange interaction Ja for the two neighboring spins in the $$\tilde{a}$$ (namely, ta) direction is estimated as $${J}_{a}=4{t}_{a}^{2}/(U-{V}_{1}) \sim 0.031$$ eV for $$\beta ^{\prime}$$−EtMe3Sb[Pd(dmit)2]2, by using the ab initio parameters listed in Table 1 of Methods. Then γ is estimated as γ ~ 15 mJ K−2 mole−1, comparable with the experimental value for $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]2, $${\gamma }^{\exp } \sim 15\!-\!20$$ mJ K−2 mole−18,12.

The uniform magnetic susceptibility of the 1D Heisenberg model is $${\chi }_{{{{\rm{1D}}}}}={g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}/({J}_{a}{\pi }^{2})$$ with the g-factor and the Bohr magneton μB32,33. By using Ja ~ 0.031eV, and g ~ 2, we obtain $$\chi /{\mu }_{{{{\rm{B}}}}}^{2} \sim 13$$ eV−1, while the experimental value is $$\chi /{\mu }_{{{{\rm{B}}}}}^{2} \sim 12\!-\!24$$ eV−1 per the dimer4,13, which is consistent with each other by considering the experimental uncertainty.

Frequency dependent thermal conductivity κ of the 1D Heisenberg model is estimated to be κ1D(ω) = κsδ(ω) with κs/T = π3J/(6a2), where a is the lattice constant34. If the delta function is replaced by the Lorentzian $${\omega }_{{{{\rm{W}}}}}/\pi ({\omega }^{2}+{\omega }_{{{{\rm{W}}}}}^{2})$$ to take into account the lifetime τ = 2π/ωW, we obtain $${\kappa }_{{{{\rm{s}}}}}^{{{{\rm{1D}}}}}(\omega =0)/T={\pi }^{2}{J}_{a}/(6a{\hslash }^{2}{\omega }_{{{{\rm{W}}}}})$$. By considering a ~ 10−9m and Ja ~ 0.031eV, $${\kappa }_{{{{\rm{s}}}}}^{{{{\rm{1D}}}}}(\omega =0)/T$$ is estimated as 7.7 × 1022/ωW[Jms]−1. Then if the experimental value reported in Ref. 9 is employed, and the one-to-one correspondence between the 1D Heisenberg and the experimental value is assumed, $${\kappa }_{\exp }^{{{{\rm{dmit}}}}}/T \sim 0.2/{k}_{{{{\rm{B}}}}}^{2}$$[Jms]−1 corresponds to the carrier relaxation time τ = 2π/ωW ~ 9.0 × 10−12s. A simple expectation of the spin velocity vs = Jaa/ results in the mean free path s = vsτ = 0.42 μm, which is a value comparable to the reported estimate ~1 μm9. More precisely, the experimental value interpreted by the 1D Heisenberg model suggests ωW/J ~ 0.015 implying small damping in the order of 10−2 for the propagation induced plausibly by the spinon dynamics driven by the energy scale of J. Therefore, although controversies exist as is mentioned above, if a large thermal conductivity experimentally reported is intrinsic, it can be essentially accounted for by the interpretation of 1D-like QSL. The obtained s is substantially longer than that of the inorganic compound Cs2CuCl435, where a 1D QSL was claimed. The present result implies that the 1D QSL found here has potentially much longer τ. Then all of the above thermodynamic and transport properties can be interpreted by the essentially 1D-like QSL found here.

The spin-lattice relaxation time T1 reported as scaled by 1/T1T2 below 1K6, implying the point-like gapless triplet excitation looks contradicting the T-linear specific heat and nonzero magnetic susceptibility compatible with the constant density of states. However, a consistent picture may emerge, if a spinon with a highly anisotropic nodal and gapless dispersion exists at ~(±π/2, 0) and ~(±π/2, π), where the 2D nature could be detectable only at low temperatures below 1K and by measuring the gapless momenta with an appropriate form factor. The power of the dispersion around the node in the chain direction is an intriguing issue but is beyond the scope of the present paper.

It turned out that the Dirac-type gapless excitation for the 2D QSL can be connected to the vanishing spinon dispersion along the Fermi line for T > 1 K, which behaves as if it is essentially the 1D QSL. $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]2 is located in this parameter space close to the 1D QSL. Furthermore, the controversy and sensitivity of the thermal conductivity10,11,12 are well understood from the one-dimensional nature: The exponential decay of the spin correlation and the vanishing Drude weight imply the vanishing thermal conductivity in the interchain direction. Then even small angle misalignment of the single crystal either in samples or in measurements causes serious reduction of κ and serious sample or measurement dependence. Effects of randomness may also seriously disturb the ideal behavior of the 1D-like spin liquid11. Observed very sensitive dependence might be able to be interpreted from the 1D nature of the QSL, which has also been pointed out in ref. 36 (See also Methods).

In summary, we have studied the family of dmit organic salts by using the ab initio Hamiltonians of 5 compounds. The obtained low-temperature phases are consistent with the experimental reports as the AF state for X=Me4P, Me4As, Me4Sb, Et2Me2As and the QSL for EtMe3Sb. The relative stabilities of the four AF compounds increasing with decreasing tctb are correctly reproduced in the ab initio calculation in agreement with the experimental trend. Thanks to this firm correspondence, the nature of the QSL in the real material is identified as the 1D-like spin liquid: The spin correlation decays exponentially and the spin Drude weight vanishes in the interchain direction. Though a controversy exists in the experimental reports of the thermal conductivity, the specific heat, the susceptibility and potentially the thermal conductivity show overall consistency with the experimentally observed values essentially in terms of the 1D QSL. However, the signature of the 2D properties is found in a prominent peak of the spin structure factor at (π, 0) and in the structure of the ground-state wavefunction itself. In addition, the exponent of the algebraic correlation is different from the known 1D QSL found in the Heisenberg or Hubbard model. The temperature dependence of NMR $${T}_{1}^{-1}$$ at T < 1 K (T2) supports vanishing density of states of spin excitation around zero energy and implies the existence of highly anisotropic but point-like nodes consistently with the present results. The present QSL offers a unified picture that bridges the 1D and 2D QSL.

It is desired to calculate the dynamical spin structure factor in the future to more directly clarify the spin dynamics with nodal excitation suggested by the structure of the wavefunction. An intriguing issue is the relation to recent studies on the charge dynamics above 2K for $$\beta ^{\prime}$$-EtMe3Sb[Pd(dmit)2]237. It might be associated with essentially the 1D-like spin liquid, where the dynamical singlet formation may couple to the dimerization fluctuation of the lattice. The spin transport can be measured by attaching ferromagnetic metal leads and by estimating the difference of the spin conduction between the cases of parallel and anti-parallel magnetization for the anode and cathode. The ac response may also help to estimate the spin Drude weight studied in the present study without the sensitivity to the misalignment or randomness. These are left for future studies to stringently verify the relevance of the present finding in the collaboration of experiment and theory.

## Methods

### Effective Hamiltonian parameters

We have used the parameters of the effective Hamiltonian (1) derived for the low-temperature experimental crystal structure below 10 K and construct Hamiltonians on a 2D plane. Here the parameters are listed in Table 1 for the self-contained description. The interaction is screened by the interlayer screening, which must be taken into account when one solves a single layer Hamiltonian. It is established that this dimensional downfolding effect is well represented by a constant reduction of all the interactions by the amount ΔDDF from the values for the three dimensional Hamiltonian, if the subtracted value exceeds zero, and by truncation to zero if the subtracted value becomes negative16. For dmit compounds, it was estimated as ΔDDF = 0.18 eV. The interaction beyond the 3rd neighbor site (namely beyond V4 and its geometrically equivalent sites) becomes small after the dimensional downfolding and we ignore them. We take into account up to the first neighbor transfer in each $$\tilde{a},\tilde{b}$$ and $$\tilde{c}$$ direction. The exchange interaction is also ignored because it is not expected to play any visible role in the present study. In this work, we analyze the above Hamiltonian in the form of the lattice structure shown in Fig. 6 at half filling in the canonical ensemble, with Ns = L × L sites. We consider systems under the antiperiodic-periodic boundary condition. In all the cases of the compounds studied, the ground state is the Mott insulator.

### Variational Monte Carlo method

In our simulations, we used a variational Monte Carlo method21,22,23. Our variational wavefunction takes the following form:

$$\left|\psi \right\rangle ={{{\mathcal{C}}}}\left|\phi \right\rangle$$
(3)

with

$${{{\mathcal{C}}}}={{{{\mathcal{P}}}}}^{{{{\rm{G}}}}}{{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{c}}}}}}{{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{s}}}}}}{{{{\mathcal{M}}}}}^{{{{\rm{s}}}}}{{{{\mathcal{M}}}}}^{{{{\rm{c}}}}}.$$
(4)

Here, $${{{{\mathcal{P}}}}}^{{{{\rm{G}}}}}=\exp \left({\sum }_{i}{\alpha }_{i}^{{{{\rm{G}}}}}{n}_{i\uparrow }{n}_{i\downarrow }\right)$$, $${{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{c}}}}}}=\exp ({\sum }_{i\,{ < }\,j}{\alpha }_{ij}^{{{{{\rm{J}}}}}_{{{{\rm{c}}}}}}{n}_{i}{n}_{j})$$ and $${{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{s}}}}}}=\exp ({\sum }_{i,j}{\alpha }_{ij}^{{{{{\rm{J}}}}}_{{{{\rm{s}}}}}}{S}_{i}^{z}{S}_{j}^{z})$$ are the Gutzwiller factor38, the long-range Jastrow correlation factors39,40, and the long-range spin Jastrow correlation factor41, respectively. Here, $${S}_{i}^{z}={n}_{i\uparrow }-{n}_{i\downarrow }$$. In practice, we impose the translational symmetry on the Gutzwiller and Jastrow factors.

To enhance the accuracy and to make the variance extrapolation explained in the next subsection easy, we also used two types of restricted Boltzmann machine (RBM) correlators23,42,43$${{{{\mathcal{M}}}}}^{{{{\rm{s}}}}}$$ and $${{{{\mathcal{M}}}}}^{{{{\rm{c}}}}}$$, which are defined in the following equations:

$${{{{\mathcal{M}}}}}^{t}={e}^{a\mathop{\sum}\nolimits_{i,\sigma }{p}_{i,\sigma }^{t}}\mathop{\prod }\limits_{\alpha }^{{N}_{\alpha }^{t}}\mathop{\prod }\limits_{r}^{N}2\cosh \left({b}_{\alpha }+\mathop{\sum}\limits_{i,\sigma }{W}_{i+r,\sigma ,\alpha }{p}_{i,\sigma }^{t}\right),$$
(5)
$${p}_{i,\sigma }^{s}={S}_{i}^{z}=\left\{\begin{array}{ll}1\quad &\,{{\mbox{for}}}\,{S}_{i}^{z}\left|x\right\rangle =\left|x\right\rangle {{\mbox{}}}\\ 0\quad &\,{{\mbox{for}}}\,{S}_{i}^{z}\left|x\right\rangle =0{{\mbox{}}}\\ -1\quad &\,{{\mbox{for}}}\,{S}_{i}^{z}\left|x\right\rangle =-\left|x\right\rangle {{\mbox{}}}\end{array}\right.,$$
(6)
$${p}_{i,\sigma }^{c}=2{({n}_{i}-1)}^{2}-1=\left\{\begin{array}{ll}1\quad &\,{{\mbox{for}}}\,{n}_{i}\left|x\right\rangle \ne \left|x\right\rangle {{\mbox{}}}\\ -1\quad &\,{{\mbox{for}}}\,{S}_{i}^{z}\left|x\right\rangle \ne 0{{\mbox{}}}\end{array}\right..$$
(7)

Here, t in Eq. (5) represents a type of the RBM correlators, i.e. t = c or s. $${N}_{\alpha }^{t}$$ denotes the ratio of the number of the hidden units in the hidden layer to the number of the physical sites Ns. $$\left|x\right\rangle$$ is a real space configuration of electrons in the sector where the total Sz is zero. a, b, and W are complex variational parameters. For the measurements of the physical quantities defined in Methods, we use $${N}_{\alpha }^{c}=4$$ and $${N}_{\alpha }^{c}={N}_{\alpha }^{s}=2$$ for the nonmagnetic states and the antiferromagnetic state, respectively.

$$\left|\phi \right\rangle$$ in Eq. (3) is a fermionic wavefunction. As $$\left|\phi \right\rangle$$, the generalized pairing wavefunction is employed, which is defined by

$$\left|{\phi }^{{{{\rm{pair}}}}}\right\rangle ={\left(\mathop{\sum}\limits_{i\sigma ,j\sigma ^{\prime} }{f}_{i\sigma ,j\sigma ^{\prime} }{c}_{i\sigma }^{{\dagger} }{c}_{j\sigma ^{\prime} }^{{\dagger} }\right)}^{{N}_{{{{\rm{e}}}}}/2}\left|0\right\rangle ,$$
(8)

where $${f}_{i\sigma ,j\sigma ^{\prime} }$$ are variational parameters and Ne is the total number of electrons. The spin indices σ and $$\sigma ^{\prime}$$ can be arbitrary with singlet and triplet combinations. This can accommodate the Hartree-Fock-Bogoliubov type wavefunction with antiferromagnetic (AF), charge and superconducting orders21,44, and flexibly describes these states and further paramagnetic metals as well. In this study, we extend it by introducing the dependence on the local density of $$\left|x\right\rangle$$. Our extended pairing wavefunction is defined as follows:

$$\left|{\phi }^{{{{\rm{ext}}}}}\right\rangle =\mathop{\sum}\limits_{x}{{{\rm{Pf}}}}M(x)\left|x\right\rangle ,$$
(9)
$${M}_{nm}(x)={f}_{{{{{\boldsymbol{r}}}}}_{n}\sigma ,{{{{\boldsymbol{r}}}}}_{m}\sigma ^{\prime} }^{{{{\rm{ext}}}}}(x)-{f}_{{{{{\boldsymbol{r}}}}}_{m}\sigma ^{\prime} ,{{{{\boldsymbol{r}}}}}_{n}\sigma }^{{{{\rm{ext}}}}}(x),$$
(10)
$${f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{ext}}}}}(x)=\left\{\begin{array}{ll}{f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{ss}}}}}\quad &\,{{\mbox{for}}}\,{n}_{i,-\sigma }\left|x\right\rangle =0\,{{\mbox{,}}}\,{n}_{j,-\sigma ^{\prime} }\left|x\right\rangle =0{{\mbox{}}}\\ {f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{sd}}}}}\quad &\,{{\mbox{for}}}\,{n}_{i,-\sigma }\left|x\right\rangle =0\,{{\mbox{,}}}\,{n}_{j,-\sigma ^{\prime} }\left|x\right\rangle =\left|x\right\rangle {{\mbox{}}}\\ {f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{ds}}}}}\quad &\,{{\mbox{for}}}\,{n}_{i,-\sigma }\left|x\right\rangle =\left|x\right\rangle \,{{\mbox{,}}}\,{n}_{j,-\sigma ^{\prime} }\left|x\right\rangle =0{{\mbox{}}}\\ {f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{dd}}}}}\quad &\,{{\mbox{for}}}\,{n}_{i,-\sigma }\left|x\right\rangle =\left|x\right\rangle \,{{\mbox{,}}}\,{n}_{j,-\sigma ^{\prime} }\left|x\right\rangle =\left|x\right\rangle {{\mbox{}}}\end{array}\right.,$$
(11)

where n and m are electron’s indices in the sample $$\left|x\right\rangle$$. PfM is the Pfaffian of a skew-symmetric matrix M. This extension improves accuracy of the fermionic part of the trial wavefunction especially for nonmagnetic states. We treated $${f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{ss}}}}}$$, $${f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{sd}}}}}$$, $${f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{ds}}}}}$$, and $${f}_{i\sigma ,j\sigma ^{\prime} }^{{{{\rm{dd}}}}}$$ as complex variational parameters.

In order to reduce the computational cost by saving the number of independent variational parameters, we assume that $${f}_{ij}^{{{{\rm{ext}}}}}$$ have a sublattice structure such that $${f}_{ij}^{{{{\rm{ext}}}}}$$ depends on the relative vector ri − rj and a sublattice index of the site j which is denoted as η(j). Thus, we can rewrite it as $${f}_{\eta (j)}^{{{{\rm{ext}}}}}({{{{\boldsymbol{r}}}}}_{i}-{{{{\boldsymbol{r}}}}}_{j})$$. In the present study on the nonmagnetic gapless states, we assumed a fully translational invariance (1 × 1 sublattice structure) and do not optimize triplet pairings (represented by the case$$\sigma =\sigma ^{\prime}$$). For studies on the AF states, we extended the sublattice structure of $${f}_{ij}^{{{{\rm{ext}}}}}$$ to 2 × 2. We did not use $${{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{s}}}}}}$$ and $${{{{\mathcal{M}}}}}^{{{{\rm{s}}}}}$$ for the optimization of the nonmagnetic states. All the variational parameters are simultaneously optimized by using the stochastic reconfiguration method45.

### Variance extrapolation

The AF and quantum spin liquid (QSL) states as well as the valence bond state are severely competing. To determine the lowest energy state among them, we performed extrapolations of energies to the limit of the zero energy variance45,46,47. For this purpose, we combined with the restricted Boltzmann machine algorithm23,42 together with the 1st Lanczos step48. To obtain the ground-state energy in the zero-variance limit, we perform the linear regression by using $${N}_{\alpha }^{c}=2$$ and 4 for nonmagnetic states. For the collinear antiferromagnetic state, we use $${N}_{\alpha }^{c}={N}_{\alpha }^{s}=0$$ and 2. Examples of the extrapolations in the present studies are found in Fig. 7, where the variance dependence of the ground-state energy for the ab initio Hamiltonian for X=EtMe3Sb is plotted. The limit of the zero variance is our estimate of the ground-state energy. We find severe competition between the AF and the QSL. The energies of the correlated metal and the dimer state with a finite spin gap are much higher than those of the AF and the QSL. We note that the error bars in Fig. 2 of the main text are obtained based on the propagation of errors as $$\varepsilon =\sqrt{{\varepsilon }_{{{{\rm{AF}}}}}^{2}+{\varepsilon }_{{{{\rm{QSL}}}}}^{2}}$$, where εAF and εQSL are the error arising from the linear regression for the AF and QSL states, respectively.

### Physical quantities

To elucidate the nature of the ground state, we analyze the following quantities: The primarily important quantity is the spin correlation

$$C({{{\boldsymbol{r}}}})=\frac{1}{{N}_{s}}\mathop{\sum}\limits_{{{{\boldsymbol{r}}}}^{\prime} }\langle {{{{\boldsymbol{S}}}}}_{{{{\boldsymbol{r}}}}^{\prime} }\cdot {{{{\boldsymbol{S}}}}}_{{{{\boldsymbol{r}}}}+{{{\boldsymbol{r}}}}^{\prime} }\rangle$$
(12)

between the site r and $${{{\boldsymbol{r}}}}+{{{\boldsymbol{r}}}}^{\prime}$$ and its Fourier transform called the spin structure factor

$$S({{{\boldsymbol{q}}}})=\frac{1}{N_s}\mathop{\sum}\limits_{{{{\boldsymbol{r}}}},{{{\boldsymbol{r}}}}^{\prime} }\langle {{{{\boldsymbol{S}}}}}_{{{{\boldsymbol{r}}}}}\cdot {{{{\boldsymbol{S}}}}}_{{{{\boldsymbol{r}}}}^{\prime} }\rangle {e}^{i{{{\boldsymbol{q}}}}\cdot ({{{\boldsymbol{r}}}}-{{{\boldsymbol{r}}}}^{\prime} )},$$
(13)

where Sr is the spin operator at the site r.

The spin Drude weight in u direction (u = x, y) is defined by the second derivative of the energy with respect to the vector potential49

$${D}_{{{{\rm{s}}}}}^{u}=\frac{1}{4\pi }\frac{{\partial }^{2}E}{\partial {A}_{u}^{2}},$$
(14)

where A is inserted to the transfer by replacing as

$${t}_{ij\sigma }\to {t}_{ij\sigma }\exp [i\sigma \pi {{{\boldsymbol{A}}}}\cdot ({{{{\boldsymbol{r}}}}}_{i}-{{{{\boldsymbol{r}}}}}_{j})/L].$$
(15)

The spin Drude weight is associated with the spin conductivity σ(ω) as

$${D}_{{{{\rm{s}}}}}=\int\nolimits_{0}^{{{\Lambda }}}\sigma (\omega )$$
(16)

with the cutoff Λ to represent the weight around ω = 0. If the peak around ω = 0 is given by the delta function, it is reduced to

$$\sigma (\omega )={D}_{{{{\rm{s}}}}}\delta (\omega ).$$
(17)

We compute the statistical error of a physical quantity O arising from the Monte Carlo sampling estimated as

$${\sigma }_{{{{\rm{err}}}}}(O)=\frac{1}{\sqrt{{N}_{{{{\rm{bin}}}}}}}\sqrt{\frac{{\sum }_{i}{[{\left\langle O\right\rangle }_{i}-\frac{1}{{N}_{{{{\rm{bin}}}}}}\left({\sum }_{i}{\left\langle O\right\rangle }_{i}\right)]}^{2}}{{N}_{{{{\rm{bin}}}}}-1}},$$
(18)

where $${\left\langle O\right\rangle }_{i}$$ is the expectation value of O at the ith bin in the Monte Carlo sampling. In general, we use about 106–107 Monte Carlo samples in each bin. Nbin is the total number of bins and we typically set Nbin = 5–10.

### Gapless structure of wavefunction

The method to analyze the structure of Fourier transform of the singlet pairing amplitude, fij = fi,j, denoted by f(k) is discussed here. Although the correlation factors $${{{\mathcal{C}}}}$$ in Eq. (4) and the density-dependent pairing amplitude in Eq. (9) largely modify the wavefunction character, the nodal structure is expected to be governed by fij in Eq. (8) for $$\sigma \,\ne\, \sigma ^{\prime}$$. The Fourier transform f(k) can be interpreted as the solution of the BCS mean-field Hamiltonian

$${{{{\mathcal{H}}}}}_{{{{\rm{BCS}}}}}=-\mathop{\sum}\limits_{{{{\boldsymbol{k}}}},\sigma }\tilde{\epsilon }({{{\boldsymbol{k}}}}){c}_{{{{\boldsymbol{k}}}}\sigma }^{{\dagger} }{c}_{{{{\boldsymbol{k}}}}\sigma }+\mathop{\sum}\limits_{k}\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})[{c}_{{{{\boldsymbol{k}}}}\sigma }^{{\dagger} }{c}_{-{{{\boldsymbol{k}}}}-\sigma }^{{\dagger} }+H.c]$$
(19)

as50

$$f({{{\boldsymbol{k}}}})=\frac{\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})}{\tilde{\epsilon }({{{\boldsymbol{k}}}})+\sqrt{\tilde{\epsilon }{({{{\boldsymbol{k}}}})}^{2}+\tilde{{{\Delta }}}{({{{\boldsymbol{k}}}})}^{2}}}.$$
(20)

To obtain the fitted f(k) defined in Eq. (20), we use the following form of $$\tilde{\epsilon }({{{\boldsymbol{k}}}})$$ and $$\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})$$:

$$\tilde{\epsilon }({{{\boldsymbol{k}}}})=-2\left(\cos {k}_{x}+\tilde{{t}_{b}}\cos ({k}_{x}+{k}_{y})+\tilde{{t}_{c}}\cos {k}_{y}\right)-\tilde{\mu },$$
(21)
$$\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})=2\left(\tilde{{{{\Delta }}}_{a}}\cos {k}_{x}+\tilde{{{{\Delta }}}_{b}}\cos ({k}_{x}+{k}_{y})+\tilde{{{{\Delta }}}_{c}}\cos {k}_{y}\right).$$
(22)

We also introduce the uniform scale factor C of f(k) because the original pairing amplitudes have the ambiguity for its norm. The parameters $$\tilde{{t}_{b}},\tilde{{t}_{c}},\tilde{\mu },\tilde{{{{\Delta }}}_{a}},\tilde{{{{\Delta }}}_{b}}$$, $$\tilde{{{{\Delta }}}_{c}}$$ and C are simultaneously optimized by using the differential evolution method51 implemented in SciPy52, which is one of the global optimization methods. We confirmed that the optimized results are similar even when we use other optimization methods such as the simplicial homology global optimization53. We ignore the pairing amplitudes smaller than 10−5 during the optimization. The fitting by $$\tilde{\epsilon }({{{\boldsymbol{k}}}})$$ and $$\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})$$ reproduces the original data Fig. 8b quite well and is not distinguishable in the color plot. Note that the original data are obtained from the optimized $${{{{\mathcal{P}}}}}^{{{{\rm{G}}}}}{{{{\mathcal{P}}}}}^{{{{{\rm{J}}}}}_{{{{\rm{c}}}}}}\left|{\phi }^{{{{\rm{pair}}}}}\right\rangle$$ for $$\sigma \,\ne\, \sigma ^{\prime}$$ with the real variational parameters $${\alpha }_{i}^{{{{\rm{G}}}}},{\alpha }_{ij}^{{{{{\rm{J}}}}}_{{{{\rm{c}}}}}}$$ and fij.

Figure 8b shows the fitted $$\tilde{\epsilon }({{{\boldsymbol{k}}}})$$ and $$\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})$$ by using Eq. (20). Since the excitation of the Hamiltonian is represented by $$\sqrt{\tilde{\epsilon }{({{{\boldsymbol{k}}}})}^{2}+\tilde{{{\Delta }}}{({{{\boldsymbol{k}}}})}^{2}}$$, the gapless point appears at the cross points of $$\tilde{\epsilon }({{{\boldsymbol{k}}}})=0$$ and $$\tilde{{{\Delta }}}({{{\boldsymbol{k}}}})=0$$, as are shown as red circles. Though change in quantitative slopes of dispersion and broadening may take place, these nodal points may not be seriously altered by the correlation factors such as the Gutzwiller factor $${{{{\mathcal{P}}}}}^{{{{\rm{G}}}}}$$30. The Fourier transform of fij denoted by f(k) can be associated with the excitation spectra50 through the fitting to the ground state of a mean-field BCS Hamiltonian with the d-wave type superconducting order. Since the quasiparticle excitation of the BCS Hamiltonian corresponds to the spin-1/2 spinon, the excitation of the QSL is inferred to be characterized by the spin-1/2 Dirac-type spinon around the nodes of the d-wave superconducting state at around (±π/2, 0) and (±π/2, π). The spinon is an excitation resulted from the fractionalization of the spin and is confined. Measurable ordinary triplet excitations are given by a combination of two spinons generating the gapless points for the triplet excitations at around (0, 0), (±π, 0), (0, ±π), and (±π, ±π).

### Strong coupling picture

The effective exchange coupling in the strong coupling expansion in terms of either ta/(UV1), tb/(UV2) or tc/(UV3) can be easily derived using $${J}_{a}=4{t}_{a}^{2}/(U-{V}_{1})$$ etc., which is summarized in Table 2. For X=EtMe3Sb, Ja = 30.5 meV, Jb = 19.6 meV, and Jc = 14.4 meV (we ignore the contribution from the direct exchange because its small and more or less isotropic values less than 3 meV18, which have only minor effect): The one-dimensionality is in fact as a consequence of not Jb/Ja (or Jc/Ja) but small (JcJb)/Ja. Nonetheless, this factor is 0.17, which is substantially larger than the value by Kenny et al., 0.05336. In fact, with our present ab initio parameters, their way of estimate of the phase diagram using the above Ja, Jb, Jc and their estimate of the ring exchange interaction $${K}_{4}=80{t}_{b}^{2}{t}_{c}^{2}/{(U-({V}_{1}+{V}_{2}+{V}_{3})/3)}^{3}$$ giving K4/Ja ~ 0.11 indicate that the EtMe3Sb[Pd(dmit)2]2 is located near the border of the antiferromagnetic phase but in the QSL side, consistently with our full quantum calculation. The estimates of the ground states for the other compounds based on their criterion are also consistent with our results. However, their recent estimate of the instability to the antiferromagnetic order contradicts our results24. It could be related to the neglect of the ring exchange interaction and partly because of the limitation of the random phase approximation based on the quasi-one-dimensionality employed by them24. In any case, it is clear that the present analysis is quantitatively the most comprehensive and accurate analysis because all orders of expansion even beyond the ring exchange from the viewpoint of the strong coupling expansion are included and the present original itinerant-electron treatment is the most strict first principles analysis.