## Abstract

We unveil the microscopic origin of largely debated magnetism in the Mo_{3}O_{8} quantum systems. Upon considering an extended Hubbard model at 1/6 filling on the anisotropic kagomé lattice formed by the Mo atoms, we argue that its ground state is determined by the competition between kinetic energy and intersite Coulomb interactions, which is controlled by the trimerisation of the kagomé lattice into the Mo_{3}O_{13} clusters, and the sign of hopping parameters, specifying the electron localisation at such clusters. Based on first-principles calculations, we show that the strong interaction limit reveals a plaquette charge order with unpaired spins at the resonating hexagons that can be realised in LiZn_{2}Mo_{3}O_{8}, and whose origin is solely related to the opposite signs of intracluster and intercluster hoppings, in contrast to all previous scenarios. On the other hand, both Li_{2}InMo_{3}O_{8} and Li_{2}ScMo_{3}O_{8} are demonstrated to fall into the weak interaction limit where the electrons are well localised at the Mo_{3}O_{13} clusters. While the former is found to exhibit long-range antiferromagnetic order, the latter is more likely to reveal short-range order with quantum spin liquid-like excitations. Our results not only reproduce most of the experimentally observed features of the Mo_{3}O_{8} systems, but will also help to describe various properties in other quantum cluster magnets.

## Introduction

Geometrically frustrated quantum systems lie at the core of research activity revolving around a putative quantum spin liquid (QSL) state that displays long-range quantum entanglement, charge fractionalisation and emergent gauge structures^{1,2,3,4}. Of particular importance are spin models on the triangular and kagomé lattices featuring various types of QSL^{5,6,7,8,9}, whose material realisation has been an ongoing endeavour in condensed matter physics with only a few reasonable candidates proposed so far^{10,11,12,13,14,15}.

During the past few years, the Mo_{3}O_{8} cluster magnets have attracted a great deal of both experimental and theoretical attention as a candidate to host QSL. In these compounds, the Mo atoms arranged in anisotropic kagomé layers are trimerised, and the [Mo_{3}O_{13}]^{15−} clusters form a triangular lattice, as shown in Fig. 1a^{16,17}. Six out of seven valence electrons in the Mo_{3}O_{13} cluster are responsible for a strong intracluster metal–metal bonding, and the seventh electron remains unpaired occupying a totally symmetric molecular *a*_{1} state.

LiZn_{2}Mo_{3}O_{8} was first reported to exhibit a QSL behaviour^{18,19,20}. The magnetic susceptibility of LiZn_{2}Mo_{3}O_{8} has been experimentally shown to follow a Curie–Weiss law with low- and high-temperature regimes transitioning at *T*_{C} ~ 96 K, whose Curie constants are related as *C*_{L} ≈ *C*_{H}/3 and where the disappearance of 2/3 of the paramagnetic spins was attributed to valence bond condensation on the triangular lattice of the Mo_{3}O_{13} clusters. In a first attempt to explain these unusual features, the authors of ref. ^{21} suggested the formation of an emergent honeycomb lattice due to opposite rotations of the Mo_{3}O_{13} clusters effectively decoupling the central cluster with an orphan paramagnetic spin. Another scenario was outlined in ref. ^{22}, where a plaquette charge order (PCO) existing in a Mott insulator on the anisotropic kagomé lattice at 1/6 filling was conjectured to host a *U*(1) QSL state with the spinon Fermi surface that is reconstructed at low temperatures filling 2/3 of the spinon states.

However, the adequacy of the proposed mechanisms was questioned with recently synthesised Li_{2}InMo_{3}O_{8} and Li_{2}ScMo_{3}O_{8}, both featuring magnetic moments well localised at the Mo_{3}O_{13} clusters. While the former was identified with a Neél 120^{∘} magnetic order at *T*_{N} = 12 K, for the latter no magnetic ordering has been observed down to 0.5 K^{23,24}. Instead, muon spin rotation and inelastic neutron scattering measurements suggested that Li_{2}ScMo_{3}O_{8} shows a short-range magnetic order below 4 K with QSL-like excitations. Despite having similar crystal structures, these Mo_{3}O_{8} systems manifest essentially unalike magnetic properties, whose enigmatic origin remains an unsolved problem.

Abstract electronic and spin models abounding with fascinating theories have been indispensable in advancing our understanding of QSLs. However, due to the complexity of the problem, first-principles calculations, which were proven to be an extremely valuable theoretical tool in a broad range of subfields in materials science, stay in the shadow of many hypothetical proposals that are often disconnected from real systems, seriously hampering the progress. In this work, upon revising a single-orbital extended Hubbard model on the anisotropic kagomé lattice at 1/6 filling, we uncover two different regimes of the plaquette charge ordered and cluster Mott insulator states governed by the interplay of kinetic energy and intersite Coulomb interactions, that were overlooked in all previous studies. By means of first-principles calculations, we will determine the material-specific model parameters for the family of the Mo_{3}O_{8} quantum magnets and demonstrate that these states can indeed be realised in LiZn_{2}Mo_{3}O_{8}, Li_{2}ScMo_{3}O_{8} and Li_{2}InMo_{3}O_{8}. In particular, we will show that the opposite sign of hopping parameters is peculiar to the trimerised kagomé layers of the Mo_{3}O_{8} systems, while their strength specifies the degree of electron localisation at the Mo_{3}O_{13} clusters.

## Results

The model of interest shown in Fig. 1b is the extended Hubbard model on the kagomé lattice at 1/6 filling:

where \({c}_{m}^{\dagger \sigma }\) (\({c}_{m}^{\sigma }\)) creates (annihilates) an electron with spin *σ* at site *m*, \({n}_{m}^{\sigma }={c}_{m}^{\dagger \sigma }{c}_{m}^{\sigma }\) is the density operator (\({n}_{m}={n}_{m}^{\uparrow }+{n}_{m}^{\downarrow }\)), *t* and \(t^{\prime}\) stand for intracluster and intercluster hopping parameters defined on the \({\mathcal{T}}\) and \({\mathcal{T}}^{\prime}\) triangles, respectively, and the interaction terms include the on-site *U*, intracluster *V* and intercluster \(V^{\prime}\) Coulomb repulsions (see Supplementary Note 1). Taking a shorter bond length in \({\mathcal{T}}\), the \({\mathcal{T}}\) triangles correspond to the Mo_{3}O_{13} clusters accommodating one unpaired electron and forming a triangular lattice.

### First-principles calculations

The calculated band structures of LiZn_{2}Mo_{3}O_{8}, Li_{2}ScMo_{3}O_{8} and Li_{2}InMo_{3}O_{8} are shown in Fig. 2a, indicating the *a*_{2} and *e*_{2} states below the Fermi level, which are responsible for the Mo–Mo bonding in the Mo_{3}O_{13} cluster, and the nonbonding *a*_{1} and *e*_{1} states well separated from the rest of the spectrum. According to our results, the splitting Δ of the *a*_{1} and *e*_{1} levels varies significantly within the systems, being relatively small in LiZn_{2}Mo_{3}O_{8} and gradually increasing in Li_{2}ScMo_{3}O_{8} and Li_{2}InMo_{3}O_{8}. Given that one electron occupies these states, the value of Δ suggests the degree of electron localisation at the molecular *a*_{1} level, which is, in turn, responsible for the formation of localised magnetic moments. As will be shown below, this fact is one of the main ingredients to understand distinct magnetic behaviour in these materials.

The extended Hubbard model, Eq. (1), for the systems in question was constructed in the basis of Wannier functions corresponding to the *a*_{1} and *e*_{1} bands, as shown in Fig. 2b. The full set of model parameters is given in Table 1 (see Supplementary Note 4). Due to the trimerisation of the kagomé lattice, \(V^{\prime} < V \ll U\) and \(| t^{\prime} |\ <\,| t|\) hold for each system. But importantly *t* and \(t^{\prime}\) always have opposite signs, and *t* < 0 while \(t^{\prime}\, >\,0\). This can be related to the fact that in the Mo_{3}O_{13} cluster the direct *d*–*d* (always negative) hopping dominates due to shorter Mo–Mo bonds. Because this term decays rapidly with the metal–metal distance (~1/*r*^{5}, ref. ^{25}), the hopping process via common oxygens having the opposite sign starts to prevail between the clusters, and \(t^{\prime}\) turns out to be positive. In a more general way, the sign alternation of hopping parameters can be regarded to occur due to the formation of the Mo_{3}O_{13} clusters, thus enforcing the localisation of a single unpaired electron at the molecular *a*_{1} level.

The competition between intracluster and intercluster interactions along with the electron filling and geometrical frustration brings about different regimes of electron localisation. Taking into account the calculated values of \(t/V^{\prime}\) and \(t^{\prime} /V\), we will address two limits of Eq. (1).

### Plaquette charge order

Let us consider \(| t| \ll V^{\prime}\) and \(| t| ^{\prime} \ll V\). Due to 1/6 filling, the Hubbard *U* cannot localise electrons at the lattice sites, and as a result they move without encountering any double occupancy. Since *U* is not operative, it is the intersite *V* and \(V^{\prime}\) that are responsible for electron localisation leading to a highly degenerate charge ordered state, where each corner-sharing triangle hosts exactly one electron. This degeneracy is further lifted by hopping parameters that induce collective tunnelling processes, when the electrons hop either clockwise or counter-clockwise along the \({\mathcal{T}}\) and \({\mathcal{T}}^{\prime}\) bonds stabilising a charge pattern with three electrons at the hexagons, as shown in Fig. 3a, b. To lowest order in \(t/V^{\prime}\) and \(t^{\prime} /V\), it corresponds to the quantum dimer model for two plaquette states \(\left|{\mathbb{A}}\right\rangle ={c}_{5}^{\dagger \sigma }{c}_{3}^{\dagger \sigma ^{\prime} }{c}_{1}^{\dagger \sigma ^{\prime\prime} }\left|0\right\rangle\) and \(\left|{\mathbb{B}}\right\rangle ={c}_{6}^{\dagger \sigma }{c}_{4}^{\dagger \sigma ^{\prime} }{c}_{2}^{\dagger \sigma ^{\prime\prime} }\left|0\right\rangle\), \({{\mathcal{H}}_{\hexagon}}={\sum_{\hexagon}}\) \({\sum }_{\sigma \sigma ^{\prime} \sigma ^{\prime\prime} }({g}_{1}+{g}_{2})\left(\right.\left|{\mathbb{A}}\right\rangle \left\langle {\mathbb{B}}\right|+\left|{\mathbb{B}}\right\rangle \left\langle {\mathbb{A}}\right|\left)\right.\) with \({g}_{1}=6t{^{\prime} }^{3}/{V}^{2}\) and \({g}_{2}=6{t}^{3}/V{^{\prime} }^{2}\), where the sum runs over all hexagons^{26,27}. When mapped onto the dual hexagonal lattice, the ground state of \({\mathcal{H}}_{\hexagon}\) for spinless electrons is described by the PCO (Fig. 3c) with an emergent triangular lattice of resonating hexagons^{28,29}. Calculations within exact diagonalization for finite geometries show that the PCO is the ground state in the case of spinful electrons (see Supplementary Note 2), that will be regarded as the strong interaction limit of Eq. (1).

One can further include antiferromagnetic spin fluctuations between next-nearest neighbours in each hexagon, \({{\mathcal{H}}}_{{S}}={J}_{{\rm{nn}}}{\sum }_{\langle \langle ij\rangle \rangle }{n}_{i}{n}_{j}\left(\right.{{\bf{S}}}_{i}\cdot {{\bf{S}}}_{j}-\frac{1}{4}\left)\right.\), where \({J}_{{\rm{nn}}}=4{t}_{{\rm{nn}}}^{2}/U,\) and *t*_{nn} is the corresponding hopping (Fig. 3a). Assuming that the PCO effectively decouples hexagons, \({\mathcal{H}}_{D}={\mathcal{H}}_{\hexagon}+{\mathcal{H}}_{S}\) for a single hexagon can be solved exactly yielding four fourfold degenerate states (see Supplementary Note 2). When *g*_{1} and *g*_{2} have opposite signs (*g*_{1} > 0 and *g*_{2} < 0), regardless of the value of *J*_{nn} the ground state of \({{\mathcal{H}}}_{{D}}\) is represented by:

with \(\tilde{g}=\sqrt{{g}_{1}^{2}-{g}_{1}{g}_{2}+{g}_{2}^{2}}\), \(\left|{\psi }_{3}\right\rangle\) and \(\left|{\psi }_{4}\right\rangle\) can be obtained by applying time-reversal symmetry to the above states. As seen from Eq. (2) and schematically shown in Fig. 4a, the resulting ground state of the resonating hexagon is given as a superposition of the plaquette states \(\left|{\mathbb{A}}\right\rangle\) and \(\left|{\mathbb{B}}\right\rangle\), each having valence bonds resonating in the hexagon and leaving one spin unpaired, whose location is smeared out in the hexagon owing to the resonating nature of valence bonds. Such an unusual entanglement with dangling spins originates solely from the asymmetry of tunnelling processes that, in turn, facilitates partial singlet pairing between the resonating electrons, while the unpaired spins behave paramagnetically in a thermodynamic limit.

From our first-principles calculations, a small splitting Δ of the *a*_{1} and *e*_{1} levels is found in LiZn_{2}Mo_{3}O_{8} preventing the electrons from being localised at the molecular states, and the calculated \(t/V^{\prime}\) and \(t^{\prime} /V\) suggest the PCO with unpaired spins at the resonating hexagons as a possible ground state for LiZn_{2}Mo_{3}O_{8}. Given *g*_{1} = 13.5 meV, *g*_{2} = −40.1 meV and *J*_{nn} = 1.4 meV, the calculated *T*_{C} ~ 92.0 K between two paramagnetic regimes is in excellent agreement with neutron powder diffraction data^{18} (see Supplementary Fig. 5).

Originally, the PCO state on the kagomé lattice in the strong interaction limit was studied for spinless fermions at 1/3 filling^{27,30}. For spinful electrons, ferromagnetism has been proposed in ref. ^{31} as one of the ground states at 1/6 and 5/6 fillings. This scenario turns out to be justified only when both *t* and \(t^{\prime}\) have the same sign, so that the Hamiltonian of the quantum dimer model \({\mathcal{H}}_{\hexagon}\) can be chosen to contain exclusively positive entries owing to the bipartite lattice, and the Perron–Frobenius theorem can be used for a finite geometry. When the hopping parameters have opposite signs, the Perron–Frobenius theorem is no longer applicable, as is also confirmed by our calculations for a single cluster and finite geometries with periodic boundary conditions, where antisymmetric configurations with minimal total spin are found to be more favourable (see Supplementary Fig. 3). Several studies have also considered the same model for spinful electrons at 1/3 filling with hopping parameters of the same sign^{32,33}, where the PCO was predicted in the limit of strong intersite Coulomb interactions. However, no possible scenario for the two paramagnetic regimes has been proposed.

The PCO state was reconsidered in ref. ^{22}, where the disappearance of 2/3 of the spins was attributed to a conjectural *U*(1) QSL that reconstructs the spinon Fermi surface at low temperatures partially filling the spinon states, inert to external magnetic field. A striking difference with the present study is that the hopping parameters in ref. ^{22} were assumed to have the same sign. In this case, one can find that the ground state for *g*_{1} < 0 and *g*_{2} < 0 on a single hexagon is given by a symmetric configuration of the plaquette states, \((\left|{\mathbb{A}}\right\rangle +\left|{\mathbb{B}}\right\rangle )/\sqrt{2}\) (see Supplementary Note 2), and the PCO itself does not feature any pairing of spins with unpaired electrons. In fact, the emergence of the *U*(1) QSL suggested in ref. ^{22} was introduced phenomenologically by “artificially” modifying the ring tunnelling processes to mimic the PCO-driven breaking of translational symmetry. In contrast, in the present case with *g*_{1} < 0 and *g*_{2} > 0 the PCO state is represented by antisymmetric configurations of \(\left|{\mathbb{A}}\right\rangle\) and \(\left|{\mathbb{B}}\right\rangle\), where each plaquette is, in turn, given as an antisymmetric configuration of spins forming valence bonds with one dangling spin in a hexagon.

Finally, the unusual pairing given by Eq. (2) can also be realised when *g*_{1} and *g*_{2} have the same sign (*g*_{1} < 0 and *g*_{2} < 0) and the antiferromagnetic coupling between next-nearest neighbours is large, \({J}_{{\rm{nn}}}\, > \, \frac{2}{3}(-{g}_{1}-{g}_{2}-\tilde{g})\), as was suggested in refs. ^{22,34}. Although the PCO can be destroyed by strong spin fluctuations^{32}, the calculated thermodynamic properties for a single hexagon shown in Fig. 4b clearly demonstrate that two paramagnetic regimes possess a much higher *T*_{C} when *g*_{1} and *g*_{2} have opposite signs. Moreover, the negligibly small *J*_{nn} obtained for LiZn_{2}Mo_{3}O_{8} shows that the partial spin pairing in the resonating hexagons is driven solely by *g*_{1} > 0 and *g*_{2} < 0. Thus, the PCO state presented in our study arises when *t* and \(t^{\prime}\) have opposite signs, and the partial pairing of spins in resonating hexagons comes about in a straightforward way from the asymmetry of tunnelling processes.

Signatures of the partial spin pairing driven by the PCO state can be directly traced from the spin–spin correlation functions (see Supplementary Fig. 4 for the density–density and plaquette–plaquette correlation functions). As visualised in Fig. 5a, the correlation function for the PCO state with *g*_{1} < 0 and *g*_{2} < 0 is essentially positive, indicating no spin pairing, and the calculated structure factor has maxima at the Γ and \(\bar{{K}}\) points of the extended Brillouin zone. On the contrary, the correlation function for the PCO state with *g*_{1} < 0 and *g*_{2} > 0 shown in Fig. 5b changes the sign between next-nearest neighbours, and the structure factor reveals maxima at the *K* points of the original Brillouin zone. This data can further be used in the analysis of inelastic neutron scattering experiments to verify our findings.

### Cluster Hubbard model

As ∣*t*∣ increases, the electrons start moving freely within the \({\mathcal{T}}\) triangle, and the number of electrons at the adjacent \({\mathcal{T}}^{\prime}\) triangles fluctuates. When \(| t| \sim V^{\prime}\), the perturbation theory considered above breaks down, and the electrons minimise their energy by forming bound “molecular” states. As a result, the original model in Eq. (1) can be reformulated as a three-orbital extended Hubbard model on the triangular lattice formed by the \({\mathcal{T}}\) triangles:

with Δ = 3*t*. As follows, \({{\mathcal{H}}}_{{\rm{CF}}}\) has the form of crystal field that splits the electronic states at the \({\mathcal{T}}\) triangle into the single *a*_{1} and double degenerate *e*_{1} states with energy levels \(\frac{2{{\Delta }}}{3}\) and \(-\frac{{{\Delta }}}{3}\), respectively: \(\left|{a}_{1}\right\rangle =\frac{1}{\sqrt{3}}\left(\right.\left|1\right\rangle +\left|2\right\rangle +\left|3\right\rangle \left)\right.\), \(|{e}_{1}^{(1)}\rangle =\frac{1}{\sqrt{3}}(w|1\rangle +\bar{w}|2\rangle +|3\rangle ).\) and \(\left|{e}_{1}^{(2)}\right\rangle =\frac{1}{\sqrt{3}}\left(\right.\bar{w}\left|1\right\rangle +w\left|2\right\rangle +\left|3\right\rangle \left)\right.\) with *w* = *e*^{2πi/3}. Importantly, the *a*_{1} state is occupied when Δ < 0 (*t* < 0). Despite the weak interaction limit, the electrons are localised at the \({\mathcal{T}}\) triangles due to the large splitting of molecular levels. We refer to this state as a cluster Mott insulator as opposed to the PCO phase, where the electron delocalisation, in turn, is entirely driven by intersite Coulomb interactions.

When both *t* < 0 and \(t^{\prime}\, >\,0\) are large, the electrons localised at the \({\mathcal{T}}\) triangles can develop long-range magnetic order. In this limit, the on-site \(\widetilde{U}=\frac{U+2V}{3}\) comes back into play and forbids any double occupancy at the \({\mathcal{T}}\) triangles, and the corresponding spin model \({{\mathcal{H}}}_{\bigtriangleup }={\sum }_{\langle ij\rangle }{J}_{\bigtriangleup }{{\bf{S}}}_{i}\cdot {{\bf{S}}}_{j}\) on the triangular lattice can be derived to second order in \(t^{\prime} /U\) and \(t^{\prime} /{{\Delta }}\) (see Supplementary Note 3):

which can be both ferro- and antiferromagnetic, that can explain why some of the recently found Mo_{3}O_{8} systems are ferromagnetic insulators^{35}. Stability of the magnetic order is directly related to the strength of *t* and \(t^{\prime}\) in the sense that it can be suppressed by thermal or quantum fluctuations when *t* or \(t^{\prime}\) are not strong enough to avoid electron number fluctuations at the \({\mathcal{T}}^{\prime}\) triangles.

As shown above, Li_{2}ScMo_{3}O_{8} and Li_{2}InMo_{3}O_{8} have large values of Δ, and the ratio \(t/V^{\prime}\) favours electron localisation at the Mo_{3}O_{13} clusters stabilising a cluster Mott insulator phase. Indeed, having the largest \(t/V^{\prime}\) and \(t^{\prime} /V\), Li_{2}InMo_{3}O_{8} reveals an antiferromagnetic order with \(J_{\bigtriangleup}\) = 9.5 meV (109.8 K) in good agreement with the experimental value of 112 K^{23}. On the other hand, \(J_{\bigtriangleup}\) = 4.0 meV (46.7 K) in Li_{2}ScMo_{3}O_{8}, being consistent with the experimental value of 67 K, is close to the instability region where \(J_{\bigtriangleup}\) is small, as clearly seen in Fig. 6. Consequently, although the electrons tend to localise at the Mo_{3}O_{13} clusters, Li_{2}ScMo_{3}O_{8} is more likely to fall into the intermediate regime, where any long-range magnetic order is suppressed by quantum fluctuations down to low temperatures. Since the number of electrons at the \({\mathcal{T}}^{\prime}\) triangles is allowed to fluctuate when \(t/V^{\prime}\) and \(t^{\prime} /V\) are not strong, we conclude that the magnetic order in Li_{2}ScMo_{3}O_{8} is short range with possible QSL-like excitations.

## Discussion

Having considered an extended Hubbard model on the anisotropic kagomé lattice at 1/6 filling as the low-energy model for the Mo_{3}O_{8} cluster magnets, we showed that it features two different limits: a PCO of resonating hexagons with valence bond condensation and orphan spins, as realised in quantum paramagnet LiZn_{2}Mo_{3}O_{8}, and a cluster Mott insulator with the electrons localised at the kagomé triangles, as revealed in Li_{2}InMo_{3}O_{8} and Li_{2}ScMo_{3}O_{8} showing a Néel-type antiferromagnetic order and QSL behaviour, respectively. Based on first-principles calculations, we demonstrated that their manifestation can be attributed to the trimerisation of the kagomé lattice controlling the competition between kinetic energy and intersite Coulomb interactions and specifying the character of electron localisation at the Mo_{3}O_{13} clusters, that can help to unravel a largely speculated origin of magnetism in these systems.

We believe that the opposite sign of *t* and \(t^{\prime}\) is a peculiar aspect of the trimerised kagomé lattice inherent to the Mo_{3}O_{8} quantum magnets that reflects the bonding character of the Mo_{3}O_{13} clusters and plays an important role in their different magnetic behaviour. According to the general Jahn–Teller theorem, a lattice trimerisation should lift the ground-state degeneracy so that a single electron resides at the *a*_{1} orbital of the \({\mathcal{T}}\) triangle forming a one-dimensional representation of the point group, that only occurs when *t* < 0 and \(t^{\prime}\, >\,0\).

The key point of our study is that the magnetism of the Mo_{3}O_{8} systems can be described within a single model, where both strong localisation and itineracy appear as the opposing limits of competing interactions. Our calculations provide a good agreement with experimental data for both Li_{2}InMo_{3}O_{8} and Li_{2}ScMo_{3}O_{8}^{23,24} and can explain the difference in their magnetic properties. The results obtained for LiZn_{2}Mo_{3}O_{8} also agree with inelastic neutron scattering data for powder samples^{19,20}, reporting a gapless spectrum of magnetic excitations and short-range spin correlations, which are inherent to the PCO state, as well as the absence of detectable sharp excitations that suggests a disordered valence bond solid or a resonating valence bond state. Available momentum dependencies of the inelastic magnetic scattering intensity reveal scatterings concentrated at small momenta *q* < 1.0 Å^{−1} with the maximum at *q* ~ 0.41 Å^{−1} at 1.7 K^{19}, which is within the range of the first Brillouin zone (the length of the in-plane reciprocal lattice vector is ~1.25 Å^{−1} for the experimental structure of LiZn_{2}Mo_{3}O_{8}^{36}) and is also consistent with our calculations of the static spin structure factor indicating the maximum weight at the K point of the first Brillouin zone, as shown in Fig. 5c (~0.72 Å^{−1} for the experimental structure of LiZn_{2}Mo_{3}O_{8}). That being said, the possible origin of paramagnetism in LiZn_{2}Mo_{3}O_{8} can still be speculated.

Our first-principles calculations would possibly rule out some previous scenarios proposed within the same model for decoupling 2/3 of the spins in LiZn_{2}Mo_{3}O_{8} at low temperatures, such as a *U*(1) QSL state with the spinon Fermi surface^{22} and strong coupling between next-nearest neighbours^{34,37}. Cooperative rotations of the Mo_{3}O_{13} clusters resulting in an emergent hexagonal lattice^{21} assume electrons to be well localised at the clusters and are at variance with our results in the strong interaction limit. The presence of mobile Li ions at ~50 − 100 K can also locally contribute to the electron localisation^{18,19,20}. However, the peaks from the Zn/Li sites observed in Li NMR measurements have different maxima below the paramagnetic phase transition, and the Li NMR data from ref. ^{20} show a minimal spin concentration from impurities, indicating that the origin of magnetic response in LiZn_{2}Mo_{3}O_{8} arises from the Mo layers. Nevertheless, the dynamical and disorder effects cannot be unequivocally excluded from the consideration and should be carefully addressed in further studies. Importantly, the low-energy models derived in this study suggest that the delocalised nature of electrons in LiZn_{2}Mo_{3}O_{8} plays a very intricate role in defining magnetic properties advancing short-range spin dynamics over long-range ordering^{38}. We believe that further experimental studies on single crystals are important to verify the origin of unusual magnetic behaviour in LiZn_{2}Mo_{3}O_{8}, as was previously done to identify fractional spinon excitations of the quantum antiferromagnetic chains^{39} and continuous spin excitations of the triangular-lattice QSL^{40}.

Finally, it is known that spin-\(\frac{1}{2}\) systems with an odd number of electrons can reveal both long-range order and short-range correlations with topological excitations^{41}, and various scenarios with unusual magnetic properties at low temperature are expected to be realised in other cluster magnets. For example, Na_{3}A_{2}(MoO_{4})_{2}Mo_{3}O_{8} (A = In, Sc) was experimentally shown to feature a paramagnetic behaviour similar to LiZn_{2}Mo_{3}O_{8}^{42}, or Nb_{3}Cl_{8} with the same number of electrons and similar crystal structure was found to be completely nonmagnetic below 90 K, in contrast to the Mo_{3}O_{8} systems^{43,44}. The analysis presented in this work can be generally applied to these and other trimerised quantum systems, such as Li_{2}In_{1−x}Sc_{x}Mo_{3}O_{8}^{37} and ScZnMo_{3}O_{8}^{45}, and further experimental and theoretical studies of these materials hold promise for discovering novel quantum effects.

## Methods

### First-principles calculations

Electronic structure calculations were performed within local density approximation^{46} and projected augmented waves^{47}, as implemented in the Vienna ab initio simulation package (VASP)^{48}. Band structure calculations including spin-orbit coupling were carried out by using norm-conserving pseudopotentials as implemented in the Quantum ESPRESSO package (QE)^{49}. The plane wave cutoff was set to 600 and 1360 eV for VASP and QE, respectively, the Brillouin zone was sampled by a 10 × 10 × 5 Monkhorst–Pack *k*-point mesh^{50}, and the convergence criteria for the total energy calculations was 10^{−9} eV. High-symmetry *k*-points used for the band structure calculations are \({K}=(\frac{1}{3},\frac{1}{3},0)\), \({M}=(\frac{1}{2},0,0)\), \({A}=(0,0,\frac{1}{2})\), \({H}=(\frac{1}{3},\frac{1}{3},\frac{1}{2})\), \({L}=(\frac{1}{2},0,\frac{1}{2})\).

To calculate model parameters of Eq. (1), we employed the maximally localised Wannier functions as implemented in the wannier90 package^{51,52}. The parameters of Coulomb interactions were calculated within constrained random-phase approximation in the basis of Wannier functions^{53,54}.

The crystal structure parameters of Li_{2}InMo_{3}O_{8}, Li_{2}ScMo_{3}O_{8} and LiZn_{2}Mo_{3}O_{8} adopted from refs. ^{23,36} are given in Supplementary Tables 1 and 2. For LiZn_{2}Mo_{3}O_{8}, the disorder of the Li and Zn atoms was fixed, and their arrangement was checked to give a small effect on the final set of model parameters. It is worth noting that there is a small splitting of the *a*_{1} and *e*_{1} states corresponding to the inequivalent Mo layers due to the Li/Zn disorder. Both octahedral sites having a higher Li occupancy^{20} and tetrahedral sites can contribute to the local splitting of molecular levels leading to a small change of the model parameters.

The crystal structures were visualised with VESTA^{55}.

### Exact diagonalisation

Exact diagonalisation of the quantum dimer model \({{\mathcal{H}}}_{{D}}\) was performed on finite clusters containing *N* = 27 and 36 sites with periodic boundary conditions. The spin–spin correlation function is computed as:

where 〈...〉 is the thermal average. The corresponding structure factor is calculated on the extended Brillouin zone (double of the original one):

Specific heat and spin susceptibility for \({{\mathcal{H}}}_{{D}}\) on a single hexagon are calculated as:

respectively, where *β* is inverse temperature, and *m*_{z} is the total magnetic moment of the plaquette state.

## Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

## References

- 1.
Moessner, R. Magnets with strong geometric frustration.

*Can. J. Phys.***79**, 1283–1294 (2001). - 2.
Lee, P. A. An end to the drought of quantum spin liquids.

*Science***321**, 1306–1307 (2008). - 3.
Balents, L. Spin liquids in frustrated magnets.

*Nature***464**, 199–208 (2010). - 4.
Savary, L. & Balents, L. Quantum spin liquids: a review.

*Rep. Prog. Phys.***80**, 016502 (2017). - 5.
Misguich, G., Lhuillier, C., Bernu, B. & Waldtmann, C. Spin-liquid phase of the multiple-spin exchange Hamiltonian on the triangular lattice.

*Phys. Rev. B***60**, 1064–1074 (1999). - 6.
Motrunich, O. I. Variational study of triangular lattice spin-1/2 model with ring exchanges and spin liquid state in

*κ*− (ET)_{2}Cu_{2}(CN)_{3}.*Phys. Rev. B***72**, 045105 (2005). - 7.
Bauer, B. et al. Chiral spin liquid and emergent anyons in a kagome lattice Mott insulator.

*Nat. Commun.***5**, 5137 (2014). - 8.
Gong, S.-S., Zhu, W. & Sheng, D. N. Emergent chiral spin liquid: fractional quantum hall effect in a kagome Heisenberg model.

*Sci. Rep.***4**, 6317 (2014). - 9.
Zhou, Y., Kanoda, K. & Ng, T.-K. Quantum spin liquid states.

*Rev. Mod. Phys.***89**, 025003 (2017). - 10.
Hiroi, Z. et al. Spin-1/2 kagomé-like lattice in volborthite Cu

_{3}V_{2}O_{7}(OH)_{2}⋅ 2H_{2}0.*J. Phys. Soc. Jpn.***70**, 3377–3384 (2001). - 11.
Helton, J. S. et al. Spin dynamics of the spin-1/2 kagome lattice antiferromagnet ZnCu

_{3}(OH)_{6}Cl_{2}.*Phys. Rev. Lett.***98**, 107204 (2007). - 12.
Olariu, A. et al.

^{17}O NMR study of the intrinsic magnetic susceptibility and spin dynamics of the quantum kagome antiferromagnet ZnCu_{3}(OH)_{6}Cl_{2}.*Phys. Rev. Lett.***100**, 087202 (2008). - 13.
Shimizu, Y. et al. Spin liquid state in an organic Mott insulator with a triangular lattice.

*Phys. Rev. Lett.***91**, 107001 (2003). - 14.
Itou, T. et al.

^{13}C NMR study of the spin-liquid state in the triangular quantum antiferromagnet EtMe_{3}Sb[Pd(dmit)_{2}]_{2}.*J. Phys. Conf. Ser.***145**, 012039 (2009). - 15.
Itou, T. et al. Quantum spin liquid in the spin-1/2 triangular antiferromagnet EtMe

_{3}Sb[Pd(dmit)_{2}]_{2}.*Phys. Rev. B***77**, 104413 (2008). - 16.
McCarroll, W. H. Structural relationships in ARMo3O8 metal atom cluster oxides.

*Inorg. Chem.***16**, 3351–3353 (1977). - 17.
Cotton, F. A. Metal atom clusters in oxide systems.

*Inorg. Chem.***3**, 1217–1220 (1964). - 18.
Sheckelton, J. P., Neilson, J. R., Soltan, D. G. & McQueen, T. M. Possible valence-bond condensation in the frustrated cluster magnet LiZn

_{2}Mo_{3}O_{8}.*Nat. Mater.***11**, 493–496 (2012). - 19.
Mourigal, M. et al. Molecular quantum magnetism in LiZn

_{2}Mo_{3}O_{8}.*Phys. Rev. Lett.***112**, 027202 (2014). - 20.
Sheckelton, J. P. et al. Local magnetism and spin correlations in the geometrically frustrated cluster magnet LiZn

_{2}Mo_{3}O_{8}.*Phys. Rev. B***89**, 064407 (2014). - 21.
Flint, R. & Lee, P. A. Emergent honeycomb lattice in LiZn

_{2}Mo_{3}O_{8}.*Phys. Rev. Lett.***111**, 217201 (2013). - 22.
Chen, G., Kee, H.-Y. & Kim, Y.-B. Cluster Mott insulators and two Curie-Weiss regimes on an anisotropic kagome lattice.

*Phys. Rev. B***93**, 245134 (2016). - 23.
Yu, H. et al. Spin-liquid behavior in the spin-frustrated Mo

_{3}cluster magnet Li_{2}ScMo_{3}O_{8}in contrast to magnetic ordering in isomorphic Li_{2}InMo_{3}O_{8}.*Phys. Rev. B***92**, 014409 (2015). - 24.
Iida, K. et al. Quantum magnetisms in uniform triangular lattices Li

_{2}AMo_{3}O_{8}(A = In, Sc).*Sci. Rep.***9**, 1826 (2019). - 25.
Harrison, W. A.

*Elementary Electronic Structure*(World Scientific, 1999). - 26.
Runge, E. & Fulde, P. Charge degrees of freedom in frustrated lattice structures.

*Phys. Rev. B***70**, 245113 (2004). - 27.
Nishimoto, S., Nakamura, M., O’Brien, A. & Fulde, P. Metal-insulator transition of fermions on a kagome Lattice at 1/3 filling.

*Phys. Rev. Lett.***104**, 196401 (2010). - 28.
Moessner, R., Sondhi, S. L. & Chandra, P. Phase diagram of the hexagonal lattice quantum dimer model.

*Phys. Rev. B***64**, 144416 (2001). - 29.
Schlittler, T. M., Mosseri, R. & Barthel, T. Phase diagram of the hexagonal lattice quantum dimer model: order parameters, ground-state energy, and gaps.

*Phys. Rev. B***96**, 195142 (2017). - 30.
O’Brien, A., Pollmann, F. & Fulde, P. Strongly correlated fermions on a kagome lattice.

*Phys. Rev. B***81**, 235115 (2010). - 31.
Pollmann, F., Fulde, P. & Shtengel, K. Kinetic ferromagnetism on a kagome lattice.

*Phys. Rev. Lett.***100**, 136404 (2008). - 32.
Pollmann, F., Roychowdhury, K., Hotta, C. & Penc, K. Interplay of charge and spin fluctuations of strongly interacting electrons on the kagome lattice.

*Phys. Rev. B***90**, 035118 (2014). - 33.
Ferhat, K. & Ralko, A. Phase diagram of the 1/3-filled extended Hubbard model on the kagome lattice.

*Phys. Rev. B***89**, 155141 (2014). - 34.
Chen, G. & Lee, P. A. Emergent orbitals in the cluster Mott insulator on a breathing kagome lattice.

*Phys. Rev. B***97**, 035124 (2018). - 35.
Sinclair, R. P. The synthesis and characterization of new triangular lattice compounds with exotic magnetic ground states (PhD diss., University of Tennessee, 2018). https://trace.tennessee.edu/utk_graddiss/4953.

- 36.
Sandvik, K. E. et al. Controlling the stoichiometry of the triangular lattice antiferromagnet Li

_{1+x}Zn_{2−y}Mo_{3}O_{8}.*J. Solid State Che.***271**, 216–221 (2019). - 37.
Akbari-Sharbaf, A. et al. Tunable quantum spin liquidity in the 1/6th-filled breathing kagome lattice.

*Phys. Rev. Lett.***120**, 227201 (2018). - 38.
Moessner, R. & Sondhi, S. L. Resonating valence bond liquid physics on the triangular lattice.

*Prog. Theor. Phys. Suppl.***145**, 37–42 (2002). - 39.
Mourigal, M. et al. Fractional spinon excitations in the quantum Heisenberg antiferromagnetic chain.

*Nat. Phys.***9**, 435–441 (2013). - 40.
Paddison, J. A. M. et al. Continuous excitations of the triangular-lattice quantum spin liquid YbMgGaO

_{4}.*Nat. Phys.***13**, 117–122 (2017). - 41.
Hastings, M. B. Lieb-Schultz-Mattis in higher dimensions.

*Phys. Rev. B***69**, 104431 (2004). - 42.
Haraguchi, Y., Michioka, C., Ueda, H. & Yoshimura, K. Highly spin-frustrated magnetism in the topochemically prepared triangular lattice cluster magnets Na

_{3}A_{2}(MoO_{4})_{2}Mo_{3}O_{8}(A = In, Sc).*Chem. Eur. J.***27**, 15879–15883 (2017). - 43.
Yu, H. et al. Magnetic-nonmagnetic phase transition with interlayer charge disproportionation of Nb

_{3}trimers in the cluster compound Nb_{3}Cl_{8}.*Inorg. Chem.***56**, 3483–3488 (2017). - 44.
Sheckelton, J. P. et al. Rearrangement of van der Waals stacking and formation of a singlet state at T = 90 K in a cluster magnet.

*Inorg. Chem. Front.***4**, 481–490 (2017). - 45.
Torardi, C. C. & McCarley, R. E. Synthesis, crystal structures, and properties of lithium zinc molybdenum oxide (LiZn

_{2}Mo_{3}O_{8}), zinc molybdenum oxide (Zn_{3}Mo_{3}O_{8}), and scandium zinc molybdenum oxide (ScZnMo_{3}O_{8}), reduced derivatives containing the Mo_{3}O_{13}cluster unit.*Inorg. Chem.***24**, 476–481 (1985). - 46.
Kohn, W. & Sham, L.-J. Self-consistent equations including exchange and correlation effects.

*Phys. Rev. A***140**, A1133–1138 (1965). - 47.
Blochl, P. E. Projector augmented-wave method.

*Phys. Rev. B***50**, 17953 (1994). - 48.
Kresse, G. & Hafner, J. Ab initio molecular dynamics for liquid metals.

*Phys. Rev. B***47**, 558–561 (1993). - 49.
Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials.

*J. Phys. Condens. Matter***21**, 395502 (2009). - 50.
Monkhorst, H. J. & Pack, J. D. Special points for Brillouin-zone integrations.

*Phys. Rev. B***13**, 5188–5192 (1976). - 51.
Mostofi, A. A. et al. An updated version of wannier90: a tool for obtaining maximally-localised Wannier functions.

*Comput. Phys. Commun.***185**, 2309–2310 (2014). - 52.
Marzari, N. et al. Maximally localized Wannier functions: theory and applications.

*Rev. Mod. Phys.***84**, 1419–1475 (2012). - 53.
Springer, M. & Aryasetiawan, F. Frequency-dependent screened interaction in Ni within the random-phase approximation.

*Phys. Rev. B***57**, 4364–4368 (1998). - 54.
Aryasetiawan, F. et al. Frequency-dependent local interactions and low-energy effective models from electronic structure calculations.

*Phys. Rev. B***70**, 195104 (2004). - 55.
Momma, K. & Izumi, F. VESTA 3 for three-dimensional visualization of crystal, volumetric and morphology data.

*J. Appl. Crystallogr.***44**, 1272–1276 (2011).

## Acknowledgements

The authors thank Wei Ren, Yuya Haraguchi, V. Yu. Irkhin, Yu. N. Skryabin and D.I. Khomskii for various discussions. I.V.S. and S.V.S. were supported by projects RFBR 20-32-70019, programs AAAA-A18-118020190095-4 (Quantum) and contract No. 02.A03.21.0006.

## Author information

### Affiliations

### Contributions

S.A.N. conceived the study. All model and first-principles calculations were performed by S.A.N. All authors participated in analysing the data and discussions. S.A.N. wrote the paper.

### Corresponding author

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Nikolaev, S.A., Solovyev, I.V. & Streltsov, S.V. Quantum spin liquid and cluster Mott insulator phases in the Mo_{3}O_{8} magnets.
*npj Quantum Mater.* **6, **25 (2021). https://doi.org/10.1038/s41535-021-00316-7

Received:

Accepted:

Published: