Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Quantum spin liquid and cluster Mott insulator phases in the Mo3O8 magnets

## Abstract

We unveil the microscopic origin of largely debated magnetism in the Mo3O8 quantum systems. Upon considering an extended Hubbard model at 1/6 filling on the anisotropic kagomé lattice formed by the Mo atoms, we argue that its ground state is determined by the competition between kinetic energy and intersite Coulomb interactions, which is controlled by the trimerisation of the kagomé lattice into the Mo3O13 clusters, and the sign of hopping parameters, specifying the electron localisation at such clusters. Based on first-principles calculations, we show that the strong interaction limit reveals a plaquette charge order with unpaired spins at the resonating hexagons that can be realised in LiZn2Mo3O8, and whose origin is solely related to the opposite signs of intracluster and intercluster hoppings, in contrast to all previous scenarios. On the other hand, both Li2InMo3O8 and Li2ScMo3O8 are demonstrated to fall into the weak interaction limit where the electrons are well localised at the Mo3O13 clusters. While the former is found to exhibit long-range antiferromagnetic order, the latter is more likely to reveal short-range order with quantum spin liquid-like excitations. Our results not only reproduce most of the experimentally observed features of the Mo3O8 systems, but will also help to describe various properties in other quantum cluster magnets.

## Introduction

Geometrically frustrated quantum systems lie at the core of research activity revolving around a putative quantum spin liquid (QSL) state that displays long-range quantum entanglement, charge fractionalisation and emergent gauge structures1,2,3,4. Of particular importance are spin models on the triangular and kagomé lattices featuring various types of QSL5,6,7,8,9, whose material realisation has been an ongoing endeavour in condensed matter physics with only a few reasonable candidates proposed so far10,11,12,13,14,15.

During the past few years, the Mo3O8 cluster magnets have attracted a great deal of both experimental and theoretical attention as a candidate to host QSL. In these compounds, the Mo atoms arranged in anisotropic kagomé layers are trimerised, and the [Mo3O13]15− clusters form a triangular lattice, as shown in Fig. 1a16,17. Six out of seven valence electrons in the Mo3O13 cluster are responsible for a strong intracluster metal–metal bonding, and the seventh electron remains unpaired occupying a totally symmetric molecular a1 state.

LiZn2Mo3O8 was first reported to exhibit a QSL behaviour18,19,20. The magnetic susceptibility of LiZn2Mo3O8 has been experimentally shown to follow a Curie–Weiss law with low- and high-temperature regimes transitioning at TC ~ 96 K, whose Curie constants are related as CL ≈ CH/3 and where the disappearance of 2/3 of the paramagnetic spins was attributed to valence bond condensation on the triangular lattice of the Mo3O13 clusters. In a first attempt to explain these unusual features, the authors of ref. 21 suggested the formation of an emergent honeycomb lattice due to opposite rotations of the Mo3O13 clusters effectively decoupling the central cluster with an orphan paramagnetic spin. Another scenario was outlined in ref. 22, where a plaquette charge order (PCO) existing in a Mott insulator on the anisotropic kagomé lattice at 1/6 filling was conjectured to host a U(1) QSL state with the spinon Fermi surface that is reconstructed at low temperatures filling 2/3 of the spinon states.

However, the adequacy of the proposed mechanisms was questioned with recently synthesised Li2InMo3O8 and Li2ScMo3O8, both featuring magnetic moments well localised at the Mo3O13 clusters. While the former was identified with a Neél 120 magnetic order at TN = 12 K, for the latter no magnetic ordering has been observed down to 0.5 K23,24. Instead, muon spin rotation and inelastic neutron scattering measurements suggested that Li2ScMo3O8 shows a short-range magnetic order below 4 K with QSL-like excitations. Despite having similar crystal structures, these Mo3O8 systems manifest essentially unalike magnetic properties, whose enigmatic origin remains an unsolved problem.

Abstract electronic and spin models abounding with fascinating theories have been indispensable in advancing our understanding of QSLs. However, due to the complexity of the problem, first-principles calculations, which were proven to be an extremely valuable theoretical tool in a broad range of subfields in materials science, stay in the shadow of many hypothetical proposals that are often disconnected from real systems, seriously hampering the progress. In this work, upon revising a single-orbital extended Hubbard model on the anisotropic kagomé lattice at 1/6 filling, we uncover two different regimes of the plaquette charge ordered and cluster Mott insulator states governed by the interplay of kinetic energy and intersite Coulomb interactions, that were overlooked in all previous studies. By means of first-principles calculations, we will determine the material-specific model parameters for the family of the Mo3O8 quantum magnets and demonstrate that these states can indeed be realised in LiZn2Mo3O8, Li2ScMo3O8 and Li2InMo3O8. In particular, we will show that the opposite sign of hopping parameters is peculiar to the trimerised kagomé layers of the Mo3O8 systems, while their strength specifies the degree of electron localisation at the Mo3O13 clusters.

## Results

The model of interest shown in Fig. 1b is the extended Hubbard model on the kagomé lattice at 1/6 filling:

$$\begin{array}{lll}{\mathcal{H}}\,=\,\mathop{\sum}\limits_{\langle mm^{\prime} \rangle \in {\mathcal{T}}, \sigma }t\left(\right.{c}_{m}^{\dagger \sigma }{c}_{m^{\prime} }^{\ \sigma }+{\rm{H.c.}}\left)\right.+V{n}_{m}{n}_{m^{\prime} }+U\mathop{\sum}\limits_{m}{n}_{m}^{\uparrow }{n}_{m}^{\downarrow }\\ \,+\,\mathop{\sum}\limits_{\langle mm^{\prime} \rangle \in {\mathcal{T}}^{\prime}, \sigma ^{\prime} }t^{\prime} \left(\right.{c}_{m}^{\dagger \sigma }{c}_{m^{\prime} }^{\ \sigma }+{\rm{H.c.}}\left)\right.+V^{\prime} {n}_{m}{n}_{m^{\prime} },\end{array}$$
(1)

where $${c}_{m}^{\dagger \sigma }$$ ($${c}_{m}^{\sigma }$$) creates (annihilates) an electron with spin σ at site m, $${n}_{m}^{\sigma }={c}_{m}^{\dagger \sigma }{c}_{m}^{\sigma }$$ is the density operator ($${n}_{m}={n}_{m}^{\uparrow }+{n}_{m}^{\downarrow }$$), t and $$t^{\prime}$$ stand for intracluster and intercluster hopping parameters defined on the $${\mathcal{T}}$$ and $${\mathcal{T}}^{\prime}$$ triangles, respectively, and the interaction terms include the on-site U, intracluster V and intercluster $$V^{\prime}$$ Coulomb repulsions (see Supplementary Note 1). Taking a shorter bond length in $${\mathcal{T}}$$, the $${\mathcal{T}}$$ triangles correspond to the Mo3O13 clusters accommodating one unpaired electron and forming a triangular lattice.

### First-principles calculations

The calculated band structures of LiZn2Mo3O8, Li2ScMo3O8 and Li2InMo3O8 are shown in Fig. 2a, indicating the a2 and e2 states below the Fermi level, which are responsible for the Mo–Mo bonding in the Mo3O13 cluster, and the nonbonding a1 and e1 states well separated from the rest of the spectrum. According to our results, the splitting Δ of the a1 and e1 levels varies significantly within the systems, being relatively small in LiZn2Mo3O8 and gradually increasing in Li2ScMo3O8 and Li2InMo3O8. Given that one electron occupies these states, the value of Δ suggests the degree of electron localisation at the molecular a1 level, which is, in turn, responsible for the formation of localised magnetic moments. As will be shown below, this fact is one of the main ingredients to understand distinct magnetic behaviour in these materials.

The extended Hubbard model, Eq. (1), for the systems in question was constructed in the basis of Wannier functions corresponding to the a1 and e1 bands, as shown in Fig. 2b. The full set of model parameters is given in Table 1 (see Supplementary Note 4). Due to the trimerisation of the kagomé lattice, $$V^{\prime} < V \ll U$$ and $$| t^{\prime} |\ <\,| t|$$ hold for each system. But importantly t and $$t^{\prime}$$ always have opposite signs, and t < 0 while $$t^{\prime}\, >\,0$$. This can be related to the fact that in the Mo3O13 cluster the direct dd (always negative) hopping dominates due to shorter Mo–Mo bonds. Because this term decays rapidly with the metal–metal distance (~1/r5, ref. 25), the hopping process via common oxygens having the opposite sign starts to prevail between the clusters, and $$t^{\prime}$$ turns out to be positive. In a more general way, the sign alternation of hopping parameters can be regarded to occur due to the formation of the Mo3O13 clusters, thus enforcing the localisation of a single unpaired electron at the molecular a1 level.

The competition between intracluster and intercluster interactions along with the electron filling and geometrical frustration brings about different regimes of electron localisation. Taking into account the calculated values of $$t/V^{\prime}$$ and $$t^{\prime} /V$$, we will address two limits of Eq. (1).

### Plaquette charge order

Let us consider $$| t| \ll V^{\prime}$$ and $$| t| ^{\prime} \ll V$$. Due to 1/6 filling, the Hubbard U cannot localise electrons at the lattice sites, and as a result they move without encountering any double occupancy. Since U is not operative, it is the intersite V and $$V^{\prime}$$ that are responsible for electron localisation leading to a highly degenerate charge ordered state, where each corner-sharing triangle hosts exactly one electron. This degeneracy is further lifted by hopping parameters that induce collective tunnelling processes, when the electrons hop either clockwise or counter-clockwise along the $${\mathcal{T}}$$ and $${\mathcal{T}}^{\prime}$$ bonds stabilising a charge pattern with three electrons at the hexagons, as shown in Fig. 3a, b. To lowest order in $$t/V^{\prime}$$ and $$t^{\prime} /V$$, it corresponds to the quantum dimer model for two plaquette states $$\left|{\mathbb{A}}\right\rangle ={c}_{5}^{\dagger \sigma }{c}_{3}^{\dagger \sigma ^{\prime} }{c}_{1}^{\dagger \sigma ^{\prime\prime} }\left|0\right\rangle$$ and $$\left|{\mathbb{B}}\right\rangle ={c}_{6}^{\dagger \sigma }{c}_{4}^{\dagger \sigma ^{\prime} }{c}_{2}^{\dagger \sigma ^{\prime\prime} }\left|0\right\rangle$$, $${{\mathcal{H}}_{\hexagon}}={\sum_{\hexagon}}$$ $${\sum }_{\sigma \sigma ^{\prime} \sigma ^{\prime\prime} }({g}_{1}+{g}_{2})\left(\right.\left|{\mathbb{A}}\right\rangle \left\langle {\mathbb{B}}\right|+\left|{\mathbb{B}}\right\rangle \left\langle {\mathbb{A}}\right|\left)\right.$$ with $${g}_{1}=6t{^{\prime} }^{3}/{V}^{2}$$ and $${g}_{2}=6{t}^{3}/V{^{\prime} }^{2}$$, where the sum runs over all hexagons26,27. When mapped onto the dual hexagonal lattice, the ground state of $${\mathcal{H}}_{\hexagon}$$ for spinless electrons is described by the PCO (Fig. 3c) with an emergent triangular lattice of resonating hexagons28,29. Calculations within exact diagonalization for finite geometries show that the PCO is the ground state in the case of spinful electrons (see Supplementary Note 2), that will be regarded as the strong interaction limit of Eq. (1).

One can further include antiferromagnetic spin fluctuations between next-nearest neighbours in each hexagon, $${{\mathcal{H}}}_{{S}}={J}_{{\rm{nn}}}{\sum }_{\langle \langle ij\rangle \rangle }{n}_{i}{n}_{j}\left(\right.{{\bf{S}}}_{i}\cdot {{\bf{S}}}_{j}-\frac{1}{4}\left)\right.$$, where $${J}_{{\rm{nn}}}=4{t}_{{\rm{nn}}}^{2}/U,$$ and tnn is the corresponding hopping (Fig. 3a). Assuming that the PCO effectively decouples hexagons, $${\mathcal{H}}_{D}={\mathcal{H}}_{\hexagon}+{\mathcal{H}}_{S}$$ for a single hexagon can be solved exactly yielding four fourfold degenerate states (see Supplementary Note 2). When g1 and g2 have opposite signs (g1 > 0 and g2 < 0), regardless of the value of Jnn the ground state of $${{\mathcal{H}}}_{{D}}$$ is represented by:

$$\begin{array}{lll}\left|{\psi }_{1}\right\rangle =\frac{1}{2}\left(\right.{\left|\uparrow \uparrow \downarrow \right\rangle }_{{\mathbb{A}}}-{\left|\downarrow \uparrow \uparrow \right\rangle }_{{\mathbb{A}}}\\- \frac{{g}_{1}-{g}_{2}}{\tilde{g}}{\left|\uparrow \uparrow \downarrow \right\rangle }_{{\mathbb{B}}}-\frac{{g}_{2}}{\tilde{g}}{\left|\uparrow \downarrow \uparrow \right\rangle }_{{\mathbb{B}}}+\frac{{g}_{1}}{\tilde{g}}{\left|\downarrow \uparrow \uparrow \right\rangle }_{{\mathbb{B}}}\left)\right.,\\ \left|{\psi }_{2}\right\rangle =\frac{1}{2}\left(\right.{\left|\uparrow \downarrow \uparrow \right\rangle }_{{\mathbb{A}}}-{\left|\downarrow \uparrow \uparrow \right\rangle }_{{\mathbb{A}}}\\+ \frac{{g}_{2}}{\tilde{g}}{\left|\uparrow \uparrow \downarrow \right\rangle }_{{\mathbb{B}}}-\frac{{g}_{1}}{\tilde{g}}{\left|\uparrow \downarrow \uparrow \right\rangle }_{{\mathbb{B}}}+\frac{{g}_{1}-{g}_{2}}{\tilde{g}}{\left|\downarrow \uparrow \uparrow \right\rangle }_{{\mathbb{B}}}\left)\right.,\end{array}$$
(2)

with $$\tilde{g}=\sqrt{{g}_{1}^{2}-{g}_{1}{g}_{2}+{g}_{2}^{2}}$$, $$\left|{\psi }_{3}\right\rangle$$ and $$\left|{\psi }_{4}\right\rangle$$ can be obtained by applying time-reversal symmetry to the above states. As seen from Eq. (2) and schematically shown in Fig. 4a, the resulting ground state of the resonating hexagon is given as a superposition of the plaquette states $$\left|{\mathbb{A}}\right\rangle$$ and $$\left|{\mathbb{B}}\right\rangle$$, each having valence bonds resonating in the hexagon and leaving one spin unpaired, whose location is smeared out in the hexagon owing to the resonating nature of valence bonds. Such an unusual entanglement with dangling spins originates solely from the asymmetry of tunnelling processes that, in turn, facilitates partial singlet pairing between the resonating electrons, while the unpaired spins behave paramagnetically in a thermodynamic limit.

From our first-principles calculations, a small splitting Δ of the a1 and e1 levels is found in LiZn2Mo3O8 preventing the electrons from being localised at the molecular states, and the calculated $$t/V^{\prime}$$ and $$t^{\prime} /V$$ suggest the PCO with unpaired spins at the resonating hexagons as a possible ground state for LiZn2Mo3O8. Given g1 = 13.5 meV, g2 = −40.1 meV and Jnn = 1.4 meV, the calculated TC ~ 92.0 K between two paramagnetic regimes is in excellent agreement with neutron powder diffraction data18 (see Supplementary Fig. 5).

Originally, the PCO state on the kagomé lattice in the strong interaction limit was studied for spinless fermions at 1/3 filling27,30. For spinful electrons, ferromagnetism has been proposed in ref. 31 as one of the ground states at 1/6 and 5/6 fillings. This scenario turns out to be justified only when both t and $$t^{\prime}$$ have the same sign, so that the Hamiltonian of the quantum dimer model $${\mathcal{H}}_{\hexagon}$$ can be chosen to contain exclusively positive entries owing to the bipartite lattice, and the Perron–Frobenius theorem can be used for a finite geometry. When the hopping parameters have opposite signs, the Perron–Frobenius theorem is no longer applicable, as is also confirmed by our calculations for a single cluster and finite geometries with periodic boundary conditions, where antisymmetric configurations with minimal total spin are found to be more favourable (see Supplementary Fig. 3). Several studies have also considered the same model for spinful electrons at 1/3 filling with hopping parameters of the same sign32,33, where the PCO was predicted in the limit of strong intersite Coulomb interactions. However, no possible scenario for the two paramagnetic regimes has been proposed.

The PCO state was reconsidered in ref. 22, where the disappearance of 2/3 of the spins was attributed to a conjectural U(1) QSL that reconstructs the spinon Fermi surface at low temperatures partially filling the spinon states, inert to external magnetic field. A striking difference with the present study is that the hopping parameters in ref. 22 were assumed to have the same sign. In this case, one can find that the ground state for g1 < 0 and g2 < 0 on a single hexagon is given by a symmetric configuration of the plaquette states, $$(\left|{\mathbb{A}}\right\rangle +\left|{\mathbb{B}}\right\rangle )/\sqrt{2}$$ (see Supplementary Note 2), and the PCO itself does not feature any pairing of spins with unpaired electrons. In fact, the emergence of the U(1) QSL suggested in ref. 22 was introduced phenomenologically by “artificially” modifying the ring tunnelling processes to mimic the PCO-driven breaking of translational symmetry. In contrast, in the present case with g1 < 0 and g2 > 0 the PCO state is represented by antisymmetric configurations of $$\left|{\mathbb{A}}\right\rangle$$ and $$\left|{\mathbb{B}}\right\rangle$$, where each plaquette is, in turn, given as an antisymmetric configuration of spins forming valence bonds with one dangling spin in a hexagon.

Finally, the unusual pairing given by Eq. (2) can also be realised when g1 and g2 have the same sign (g1 < 0 and g2 < 0) and the antiferromagnetic coupling between next-nearest neighbours is large, $${J}_{{\rm{nn}}}\, > \, \frac{2}{3}(-{g}_{1}-{g}_{2}-\tilde{g})$$, as was suggested in refs. 22,34. Although the PCO can be destroyed by strong spin fluctuations32, the calculated thermodynamic properties for a single hexagon shown in Fig. 4b clearly demonstrate that two paramagnetic regimes possess a much higher TC when g1 and g2 have opposite signs. Moreover, the negligibly small Jnn obtained for LiZn2Mo3O8 shows that the partial spin pairing in the resonating hexagons is driven solely by g1 > 0 and g2 < 0. Thus, the PCO state presented in our study arises when t and $$t^{\prime}$$ have opposite signs, and the partial pairing of spins in resonating hexagons comes about in a straightforward way from the asymmetry of tunnelling processes.

Signatures of the partial spin pairing driven by the PCO state can be directly traced from the spin–spin correlation functions (see Supplementary Fig. 4 for the density–density and plaquette–plaquette correlation functions). As visualised in Fig. 5a, the correlation function for the PCO state with g1 < 0 and g2 < 0 is essentially positive, indicating no spin pairing, and the calculated structure factor has maxima at the Γ and $$\bar{{K}}$$ points of the extended Brillouin zone. On the contrary, the correlation function for the PCO state with g1 < 0 and g2 > 0 shown in Fig. 5b changes the sign between next-nearest neighbours, and the structure factor reveals maxima at the K points of the original Brillouin zone. This data can further be used in the analysis of inelastic neutron scattering experiments to verify our findings.

### Cluster Hubbard model

As t increases, the electrons start moving freely within the $${\mathcal{T}}$$ triangle, and the number of electrons at the adjacent $${\mathcal{T}}^{\prime}$$ triangles fluctuates. When $$| t| \sim V^{\prime}$$, the perturbation theory considered above breaks down, and the electrons minimise their energy by forming bound “molecular” states. As a result, the original model in Eq. (1) can be reformulated as a three-orbital extended Hubbard model on the triangular lattice formed by the $${\mathcal{T}}$$ triangles:

$${{\mathcal{H}}}_{{\rm{CF}}}=\frac{{{\Delta }}}{3}\sum _{mm^{\prime} \in {\mathcal{T}},\sigma }{c}_{im}^{\dagger \sigma }{\left(\begin{array}{lll}0&1&1\\ 1&0&1\\ 1&1&0\\ \end{array}\right)}_{mm^{\prime} }\,\,{c}_{im^{\prime} }^{\sigma }$$
(3)

with Δ = 3t. As follows, $${{\mathcal{H}}}_{{\rm{CF}}}$$ has the form of crystal field that splits the electronic states at the $${\mathcal{T}}$$ triangle into the single a1 and double degenerate e1 states with energy levels $$\frac{2{{\Delta }}}{3}$$ and $$-\frac{{{\Delta }}}{3}$$, respectively: $$\left|{a}_{1}\right\rangle =\frac{1}{\sqrt{3}}\left(\right.\left|1\right\rangle +\left|2\right\rangle +\left|3\right\rangle \left)\right.$$, $$|{e}_{1}^{(1)}\rangle =\frac{1}{\sqrt{3}}(w|1\rangle +\bar{w}|2\rangle +|3\rangle ).$$ and $$\left|{e}_{1}^{(2)}\right\rangle =\frac{1}{\sqrt{3}}\left(\right.\bar{w}\left|1\right\rangle +w\left|2\right\rangle +\left|3\right\rangle \left)\right.$$ with w = e2πi/3. Importantly, the a1 state is occupied when Δ < 0 (t < 0). Despite the weak interaction limit, the electrons are localised at the $${\mathcal{T}}$$ triangles due to the large splitting of molecular levels. We refer to this state as a cluster Mott insulator as opposed to the PCO phase, where the electron delocalisation, in turn, is entirely driven by intersite Coulomb interactions.

When both t < 0 and $$t^{\prime}\, >\,0$$ are large, the electrons localised at the $${\mathcal{T}}$$ triangles can develop long-range magnetic order. In this limit, the on-site $$\widetilde{U}=\frac{U+2V}{3}$$ comes back into play and forbids any double occupancy at the $${\mathcal{T}}$$ triangles, and the corresponding spin model $${{\mathcal{H}}}_{\bigtriangleup }={\sum }_{\langle ij\rangle }{J}_{\bigtriangleup }{{\bf{S}}}_{i}\cdot {{\bf{S}}}_{j}$$ on the triangular lattice can be derived to second order in $$t^{\prime} /U$$ and $$t^{\prime} /{{\Delta }}$$ (see Supplementary Note 3):

$${J}_{\bigtriangleup }=-\frac{8t{^{\prime} }^{2}}{3(2V+3| {{\Delta }}| -2V^{\prime} )}+\frac{4t{^{\prime} }^{2}}{3(U+2V-2V^{\prime} )}+\frac{8t{^{\prime} }^{2}}{3(U+2V+3| {{\Delta }}| -2V^{\prime} )},$$
(4)

which can be both ferro- and antiferromagnetic, that can explain why some of the recently found Mo3O8 systems are ferromagnetic insulators35. Stability of the magnetic order is directly related to the strength of t and $$t^{\prime}$$ in the sense that it can be suppressed by thermal or quantum fluctuations when t or $$t^{\prime}$$ are not strong enough to avoid electron number fluctuations at the $${\mathcal{T}}^{\prime}$$ triangles.

As shown above, Li2ScMo3O8 and Li2InMo3O8 have large values of Δ, and the ratio $$t/V^{\prime}$$ favours electron localisation at the Mo3O13 clusters stabilising a cluster Mott insulator phase. Indeed, having the largest $$t/V^{\prime}$$ and $$t^{\prime} /V$$, Li2InMo3O8 reveals an antiferromagnetic order with $$J_{\bigtriangleup}$$ = 9.5 meV (109.8 K) in good agreement with the experimental value of 112 K23. On the other hand, $$J_{\bigtriangleup}$$ = 4.0 meV (46.7 K) in Li2ScMo3O8, being consistent with the experimental value of 67 K, is close to the instability region where $$J_{\bigtriangleup}$$ is small, as clearly seen in Fig. 6. Consequently, although the electrons tend to localise at the Mo3O13 clusters, Li2ScMo3O8 is more likely to fall into the intermediate regime, where any long-range magnetic order is suppressed by quantum fluctuations down to low temperatures. Since the number of electrons at the $${\mathcal{T}}^{\prime}$$ triangles is allowed to fluctuate when $$t/V^{\prime}$$ and $$t^{\prime} /V$$ are not strong, we conclude that the magnetic order in Li2ScMo3O8 is short range with possible QSL-like excitations.

## Discussion

Having considered an extended Hubbard model on the anisotropic kagomé lattice at 1/6 filling as the low-energy model for the Mo3O8 cluster magnets, we showed that it features two different limits: a PCO of resonating hexagons with valence bond condensation and orphan spins, as realised in quantum paramagnet LiZn2Mo3O8, and a cluster Mott insulator with the electrons localised at the kagomé triangles, as revealed in Li2InMo3O8 and Li2ScMo3O8 showing a Néel-type antiferromagnetic order and QSL behaviour, respectively. Based on first-principles calculations, we demonstrated that their manifestation can be attributed to the trimerisation of the kagomé lattice controlling the competition between kinetic energy and intersite Coulomb interactions and specifying the character of electron localisation at the Mo3O13 clusters, that can help to unravel a largely speculated origin of magnetism in these systems.

We believe that the opposite sign of t and $$t^{\prime}$$ is a peculiar aspect of the trimerised kagomé lattice inherent to the Mo3O8 quantum magnets that reflects the bonding character of the Mo3O13 clusters and plays an important role in their different magnetic behaviour. According to the general Jahn–Teller theorem, a lattice trimerisation should lift the ground-state degeneracy so that a single electron resides at the a1 orbital of the $${\mathcal{T}}$$ triangle forming a one-dimensional representation of the point group, that only occurs when t < 0 and $$t^{\prime}\, >\,0$$.

The key point of our study is that the magnetism of the Mo3O8 systems can be described within a single model, where both strong localisation and itineracy appear as the opposing limits of competing interactions. Our calculations provide a good agreement with experimental data for both Li2InMo3O8 and Li2ScMo3O823,24 and can explain the difference in their magnetic properties. The results obtained for LiZn2Mo3O8 also agree with inelastic neutron scattering data for powder samples19,20, reporting a gapless spectrum of magnetic excitations and short-range spin correlations, which are inherent to the PCO state, as well as the absence of detectable sharp excitations that suggests a disordered valence bond solid or a resonating valence bond state. Available momentum dependencies of the inelastic magnetic scattering intensity reveal scatterings concentrated at small momenta q < 1.0 Å−1 with the maximum at q ~ 0.41 Å−1 at 1.7 K19, which is within the range of the first Brillouin zone (the length of the in-plane reciprocal lattice vector is ~1.25 Å−1 for the experimental structure of LiZn2Mo3O836) and is also consistent with our calculations of the static spin structure factor indicating the maximum weight at the K point of the first Brillouin zone, as shown in Fig. 5c (~0.72 Å−1 for the experimental structure of LiZn2Mo3O8). That being said, the possible origin of paramagnetism in LiZn2Mo3O8 can still be speculated.

Our first-principles calculations would possibly rule out some previous scenarios proposed within the same model for decoupling 2/3 of the spins in LiZn2Mo3O8 at low temperatures, such as a U(1) QSL state with the spinon Fermi surface22 and strong coupling between next-nearest neighbours34,37. Cooperative rotations of the Mo3O13 clusters resulting in an emergent hexagonal lattice21 assume electrons to be well localised at the clusters and are at variance with our results in the strong interaction limit. The presence of mobile Li ions at ~50 − 100 K can also locally contribute to the electron localisation18,19,20. However, the peaks from the Zn/Li sites observed in Li NMR measurements have different maxima below the paramagnetic phase transition, and the Li NMR data from ref. 20 show a minimal spin concentration from impurities, indicating that the origin of magnetic response in LiZn2Mo3O8 arises from the Mo layers. Nevertheless, the dynamical and disorder effects cannot be unequivocally excluded from the consideration and should be carefully addressed in further studies. Importantly, the low-energy models derived in this study suggest that the delocalised nature of electrons in LiZn2Mo3O8 plays a very intricate role in defining magnetic properties advancing short-range spin dynamics over long-range ordering38. We believe that further experimental studies on single crystals are important to verify the origin of unusual magnetic behaviour in LiZn2Mo3O8, as was previously done to identify fractional spinon excitations of the quantum antiferromagnetic chains39 and continuous spin excitations of the triangular-lattice QSL40.

Finally, it is known that spin-$$\frac{1}{2}$$ systems with an odd number of electrons can reveal both long-range order and short-range correlations with topological excitations41, and various scenarios with unusual magnetic properties at low temperature are expected to be realised in other cluster magnets. For example, Na3A2(MoO4)2Mo3O8 (A = In, Sc) was experimentally shown to feature a paramagnetic behaviour similar to LiZn2Mo3O842, or Nb3Cl8 with the same number of electrons and similar crystal structure was found to be completely nonmagnetic below 90 K, in contrast to the Mo3O8 systems43,44. The analysis presented in this work can be generally applied to these and other trimerised quantum systems, such as Li2In1−xScxMo3O837 and ScZnMo3O845, and further experimental and theoretical studies of these materials hold promise for discovering novel quantum effects.

## Methods

### First-principles calculations

Electronic structure calculations were performed within local density approximation46 and projected augmented waves47, as implemented in the Vienna ab initio simulation package (VASP)48. Band structure calculations including spin-orbit coupling were carried out by using norm-conserving pseudopotentials as implemented in the Quantum ESPRESSO package (QE)49. The plane wave cutoff was set to 600 and 1360 eV for VASP and QE, respectively, the Brillouin zone was sampled by a 10 × 10 × 5 Monkhorst–Pack k-point mesh50, and the convergence criteria for the total energy calculations was 10−9 eV. High-symmetry k-points used for the band structure calculations are $${K}=(\frac{1}{3},\frac{1}{3},0)$$, $${M}=(\frac{1}{2},0,0)$$, $${A}=(0,0,\frac{1}{2})$$, $${H}=(\frac{1}{3},\frac{1}{3},\frac{1}{2})$$, $${L}=(\frac{1}{2},0,\frac{1}{2})$$.

To calculate model parameters of Eq. (1), we employed the maximally localised Wannier functions as implemented in the wannier90 package51,52. The parameters of Coulomb interactions were calculated within constrained random-phase approximation in the basis of Wannier functions53,54.

The crystal structure parameters of Li2InMo3O8, Li2ScMo3O8 and LiZn2Mo3O8 adopted from refs. 23,36 are given in Supplementary Tables 1 and 2. For LiZn2Mo3O8, the disorder of the Li and Zn atoms was fixed, and their arrangement was checked to give a small effect on the final set of model parameters. It is worth noting that there is a small splitting of the a1 and e1 states corresponding to the inequivalent Mo layers due to the Li/Zn disorder. Both octahedral sites having a higher Li occupancy20 and tetrahedral sites can contribute to the local splitting of molecular levels leading to a small change of the model parameters.

The crystal structures were visualised with VESTA55.

### Exact diagonalisation

Exact diagonalisation of the quantum dimer model $${{\mathcal{H}}}_{{D}}$$ was performed on finite clusters containing N = 27 and 36 sites with periodic boundary conditions. The spin–spin correlation function is computed as:

$${C}^{{\rm{s}}}({{\bf{r}}}_{0},{{\bf{r}}}_{k})=\langle {{\bf{S}}}_{0}\cdot {{\bf{S}}}_{k}\rangle -\langle {{\bf{S}}}_{0}\rangle \cdot \langle {{\bf{S}}}_{k}\rangle ,$$
(5)

where 〈...〉 is the thermal average. The corresponding structure factor is calculated on the extended Brillouin zone (double of the original one):

$${S}^{{\rm{s}}}({\bf{q}})=\frac{1}{N}\sum _{k}{e}^{-i{\bf{q}}\cdot ({{\bf{r}}}_{k}-{{\bf{r}}}_{0})}{C}^{{\rm{s}}}({{\bf{r}}}_{0},{{\bf{r}}}_{k}).$$
(6)

Specific heat and spin susceptibility for $${{\mathcal{H}}}_{{D}}$$ on a single hexagon are calculated as:

$$\begin{array}{lll}{C}_{V}&=&{\beta }^{2}\left(\right.\langle {E}^{2}\rangle -{\langle E\rangle }^{2}\left)\right.,\\ \chi &=&\beta \left(\right.\langle {m}_{z}^{2}\rangle -{\langle {m}_{z}\rangle }^{2}\left)\right.,\end{array}$$
(7)

respectively, where β is inverse temperature, and mz is the total magnetic moment of the plaquette state.

## Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

## References

1. 1.

Moessner, R. Magnets with strong geometric frustration. Can. J. Phys. 79, 1283–1294 (2001).

2. 2.

Lee, P. A. An end to the drought of quantum spin liquids. Science 321, 1306–1307 (2008).

3. 3.

Balents, L. Spin liquids in frustrated magnets. Nature 464, 199–208 (2010).

4. 4.

Savary, L. & Balents, L. Quantum spin liquids: a review. Rep. Prog. Phys. 80, 016502 (2017).

5. 5.

Misguich, G., Lhuillier, C., Bernu, B. & Waldtmann, C. Spin-liquid phase of the multiple-spin exchange Hamiltonian on the triangular lattice. Phys. Rev. B 60, 1064–1074 (1999).

6. 6.

Motrunich, O. I. Variational study of triangular lattice spin-1/2 model with ring exchanges and spin liquid state in κ − (ET)2 Cu2 (CN)3. Phys. Rev. B 72, 045105 (2005).

7. 7.

Bauer, B. et al. Chiral spin liquid and emergent anyons in a kagome lattice Mott insulator. Nat. Commun. 5, 5137 (2014).

8. 8.

Gong, S.-S., Zhu, W. & Sheng, D. N. Emergent chiral spin liquid: fractional quantum hall effect in a kagome Heisenberg model. Sci. Rep. 4, 6317 (2014).

9. 9.

Zhou, Y., Kanoda, K. & Ng, T.-K. Quantum spin liquid states. Rev. Mod. Phys. 89, 025003 (2017).

10. 10.

Hiroi, Z. et al. Spin-1/2 kagomé-like lattice in volborthite Cu3 V2 O7 (OH)2 2H2 0. J. Phys. Soc. Jpn. 70, 3377–3384 (2001).

11. 11.

Helton, J. S. et al. Spin dynamics of the spin-1/2 kagome lattice antiferromagnet ZnCu3 (OH)6 Cl2. Phys. Rev. Lett. 98, 107204 (2007).

12. 12.

Olariu, A. et al. 17O NMR study of the intrinsic magnetic susceptibility and spin dynamics of the quantum kagome antiferromagnet ZnCu3 (OH)6 Cl2. Phys. Rev. Lett. 100, 087202 (2008).

13. 13.

Shimizu, Y. et al. Spin liquid state in an organic Mott insulator with a triangular lattice. Phys. Rev. Lett. 91, 107001 (2003).

14. 14.

Itou, T. et al. 13C NMR study of the spin-liquid state in the triangular quantum antiferromagnet EtMe3 Sb[Pd(dmit)2]2. J. Phys. Conf. Ser. 145, 012039 (2009).

15. 15.

Itou, T. et al. Quantum spin liquid in the spin-1/2 triangular antiferromagnet EtMe3 Sb[Pd(dmit)2]2. Phys. Rev. B 77, 104413 (2008).

16. 16.

McCarroll, W. H. Structural relationships in ARMo3O8 metal atom cluster oxides. Inorg. Chem. 16, 3351–3353 (1977).

17. 17.

Cotton, F. A. Metal atom clusters in oxide systems. Inorg. Chem. 3, 1217–1220 (1964).

18. 18.

Sheckelton, J. P., Neilson, J. R., Soltan, D. G. & McQueen, T. M. Possible valence-bond condensation in the frustrated cluster magnet LiZn2 Mo3O8. Nat. Mater. 11, 493–496 (2012).

19. 19.

Mourigal, M. et al. Molecular quantum magnetism in LiZn2 Mo3O8. Phys. Rev. Lett. 112, 027202 (2014).

20. 20.

Sheckelton, J. P. et al. Local magnetism and spin correlations in the geometrically frustrated cluster magnet LiZn2 Mo3 O8. Phys. Rev. B 89, 064407 (2014).

21. 21.

Flint, R. & Lee, P. A. Emergent honeycomb lattice in LiZn2 Mo3 O8. Phys. Rev. Lett. 111, 217201 (2013).

22. 22.

Chen, G., Kee, H.-Y. & Kim, Y.-B. Cluster Mott insulators and two Curie-Weiss regimes on an anisotropic kagome lattice. Phys. Rev. B 93, 245134 (2016).

23. 23.

Yu, H. et al. Spin-liquid behavior in the spin-frustrated Mo3 cluster magnet Li2 ScMo3O8 in contrast to magnetic ordering in isomorphic Li2 InMo3 O8. Phys. Rev. B 92, 014409 (2015).

24. 24.

Iida, K. et al. Quantum magnetisms in uniform triangular lattices Li2 AMo3 O8 (A = In, Sc). Sci. Rep. 9, 1826 (2019).

25. 25.

Harrison, W. A. Elementary Electronic Structure (World Scientific, 1999).

26. 26.

Runge, E. & Fulde, P. Charge degrees of freedom in frustrated lattice structures. Phys. Rev. B 70, 245113 (2004).

27. 27.

Nishimoto, S., Nakamura, M., O’Brien, A. & Fulde, P. Metal-insulator transition of fermions on a kagome Lattice at 1/3 filling. Phys. Rev. Lett. 104, 196401 (2010).

28. 28.

Moessner, R., Sondhi, S. L. & Chandra, P. Phase diagram of the hexagonal lattice quantum dimer model. Phys. Rev. B 64, 144416 (2001).

29. 29.

Schlittler, T. M., Mosseri, R. & Barthel, T. Phase diagram of the hexagonal lattice quantum dimer model: order parameters, ground-state energy, and gaps. Phys. Rev. B 96, 195142 (2017).

30. 30.

O’Brien, A., Pollmann, F. & Fulde, P. Strongly correlated fermions on a kagome lattice. Phys. Rev. B 81, 235115 (2010).

31. 31.

Pollmann, F., Fulde, P. & Shtengel, K. Kinetic ferromagnetism on a kagome lattice. Phys. Rev. Lett. 100, 136404 (2008).

32. 32.

Pollmann, F., Roychowdhury, K., Hotta, C. & Penc, K. Interplay of charge and spin fluctuations of strongly interacting electrons on the kagome lattice. Phys. Rev. B 90, 035118 (2014).

33. 33.

Ferhat, K. & Ralko, A. Phase diagram of the 1/3-filled extended Hubbard model on the kagome lattice. Phys. Rev. B 89, 155141 (2014).

34. 34.

Chen, G. & Lee, P. A. Emergent orbitals in the cluster Mott insulator on a breathing kagome lattice. Phys. Rev. B 97, 035124 (2018).

35. 35.

Sinclair, R. P. The synthesis and characterization of new triangular lattice compounds with exotic magnetic ground states (PhD diss., University of Tennessee, 2018). https://trace.tennessee.edu/utk_graddiss/4953.

36. 36.

Sandvik, K. E. et al. Controlling the stoichiometry of the triangular lattice antiferromagnet Li1+x Zn2−y Mo3 O8. J. Solid State Che. 271, 216–221 (2019).

37. 37.

Akbari-Sharbaf, A. et al. Tunable quantum spin liquidity in the 1/6th-filled breathing kagome lattice. Phys. Rev. Lett. 120, 227201 (2018).

38. 38.

Moessner, R. & Sondhi, S. L. Resonating valence bond liquid physics on the triangular lattice. Prog. Theor. Phys. Suppl. 145, 37–42 (2002).

39. 39.

Mourigal, M. et al. Fractional spinon excitations in the quantum Heisenberg antiferromagnetic chain. Nat. Phys. 9, 435–441 (2013).

40. 40.

Paddison, J. A. M. et al. Continuous excitations of the triangular-lattice quantum spin liquid YbMgGaO4. Nat. Phys. 13, 117–122 (2017).

41. 41.

Hastings, M. B. Lieb-Schultz-Mattis in higher dimensions. Phys. Rev. B 69, 104431 (2004).

42. 42.

Haraguchi, Y., Michioka, C., Ueda, H. & Yoshimura, K. Highly spin-frustrated magnetism in the topochemically prepared triangular lattice cluster magnets Na3 A2 (MoO4)2 Mo3 O8 (A = In, Sc). Chem. Eur. J. 27, 15879–15883 (2017).

43. 43.

Yu, H. et al. Magnetic-nonmagnetic phase transition with interlayer charge disproportionation of Nb3 trimers in the cluster compound Nb3 Cl8. Inorg. Chem. 56, 3483–3488 (2017).

44. 44.

Sheckelton, J. P. et al. Rearrangement of van der Waals stacking and formation of a singlet state at T = 90 K in a cluster magnet. Inorg. Chem. Front. 4, 481–490 (2017).

45. 45.

Torardi, C. C. & McCarley, R. E. Synthesis, crystal structures, and properties of lithium zinc molybdenum oxide (LiZn2 Mo3 O8), zinc molybdenum oxide (Zn3 Mo3 O8), and scandium zinc molybdenum oxide (ScZnMo3 O8), reduced derivatives containing the Mo3 O13 cluster unit. Inorg. Chem. 24, 476–481 (1985).

46. 46.

Kohn, W. & Sham, L.-J. Self-consistent equations including exchange and correlation effects. Phys. Rev. A 140, A1133–1138 (1965).

47. 47.

Blochl, P. E. Projector augmented-wave method. Phys. Rev. B 50, 17953 (1994).

48. 48.

Kresse, G. & Hafner, J. Ab initio molecular dynamics for liquid metals. Phys. Rev. B 47, 558–561 (1993).

49. 49.

Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. Condens. Matter 21, 395502 (2009).

50. 50.

Monkhorst, H. J. & Pack, J. D. Special points for Brillouin-zone integrations. Phys. Rev. B 13, 5188–5192 (1976).

51. 51.

Mostofi, A. A. et al. An updated version of wannier90: a tool for obtaining maximally-localised Wannier functions. Comput. Phys. Commun. 185, 2309–2310 (2014).

52. 52.

Marzari, N. et al. Maximally localized Wannier functions: theory and applications. Rev. Mod. Phys. 84, 1419–1475 (2012).

53. 53.

Springer, M. & Aryasetiawan, F. Frequency-dependent screened interaction in Ni within the random-phase approximation. Phys. Rev. B 57, 4364–4368 (1998).

54. 54.

Aryasetiawan, F. et al. Frequency-dependent local interactions and low-energy effective models from electronic structure calculations. Phys. Rev. B 70, 195104 (2004).

55. 55.

Momma, K. & Izumi, F. VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data. J. Appl. Crystallogr. 44, 1272–1276 (2011).

## Acknowledgements

The authors thank Wei Ren, Yuya Haraguchi, V. Yu. Irkhin, Yu. N. Skryabin and D.I. Khomskii for various discussions. I.V.S. and S.V.S. were supported by projects RFBR 20-32-70019, programs AAAA-A18-118020190095-4 (Quantum) and contract No. 02.A03.21.0006.

## Author information

Authors

### Contributions

S.A.N. conceived the study. All model and first-principles calculations were performed by S.A.N. All authors participated in analysing the data and discussions. S.A.N. wrote the paper.

### Corresponding author

Correspondence to S. A. Nikolaev.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Nikolaev, S.A., Solovyev, I.V. & Streltsov, S.V. Quantum spin liquid and cluster Mott insulator phases in the Mo3O8 magnets. npj Quantum Mater. 6, 25 (2021). https://doi.org/10.1038/s41535-021-00316-7