Introduction

Mott insulator and twisted Moiré lattice are two systems at the center of strongly correlated phenomena. The former is generally characterized by metal-insulator transition triggered by local interactions. The latter is a correlated state tuned manually with effective long-range interactions. It is also famous for the flat bands and the accompanied spontaneous symmetry breaking. The concurrence of electronic correlation and magnetic couplings characterizes these systems as the most appealing platforms for studying emergent phenomena. Further equipping them with flat bands in the electronic structure adds additional instabilities, making these systems more exotic and fascinating. Compared to conventional Mott insulators and the twisted Moiré systems, pristine materials with correlated flat bands and nonlocal interactions that are free of spontaneous symmetry breaking are extremely rare. Such systems would interpolate the atomic-type Mott insulators and the Moiré correlated insulators, such that they would provide a direct competition of local and nonlocal interactions free of other types of order, which opens a new avenue to realize the quantum spin liquid (QSL) states1,2,3.

Since the first proposal of P.W. Anderson4 and the further elaboration by Kivelson5 and Laughlin6, the search for QSL has become one central topic of research in contemporary condensed matter physics. Despite the experimental challenge for a firm convincing, the spin-1/2 transition-metal (TM) Kagome lattice7,8,9,10,11 and the triangular lattice12,13,14,15 are believed to be the most likely hosts of QSL among the many candidate systems. Geometrical frustration inherently contained in these systems prompts the competition of many magnetic configurations of similar energy leading to the large degeneracy in the ground state. Each configuration may break spin rotational symmetry, while the coexistence of many different kinds of spatially oscillating magnetic configurations prevents the formation of long-range magnetic order down to very low temperatures. Compared to the search of the spin-1/2 QSL states in conventional TM Kagome compounds16,17,18 and spin-orbital coupled j = 1/2 pyrochlore systems19, the recent advances in the Kagome lattice with Mo3O8 trimerized cluster20,21,22,23,24,25,26,27,28 and the star of David Ta cluster in 1T-TaS229 provide a platform to realize an effective spin-1/2 QSL in molecular-type solid systems.

In this letter, we propose that monolayer Nb3Cl8 is a cluster Mott insulator with correlated flat bands and possesses great potential to realize a 2D molecular QSL state. Among the different systems in Nb3X8 (X=Cl, Br, I) family30,31,32,33,34,35, Nb3Cl8 is unique that it has strong intralayer bonding and weak interlayer coupling30, 36, 37. Each layer electronically decouples from neighboring layers, leading to a quasi-2D Kagome lattice with breathing mode38. The breathing of Nb gives rise to a trimer cluster of three neighboring Nb atoms. Different trimers tile in the lattice forming an effective 2D triangular system. The three Nb atoms in a trimer equally contribute to a μB/3, making monolayer Nb3Cl8 an effective spin-1/2 triangular system (see Fig. 1b), which highly resembles the trimerization of Mo3O8 cluster. The high-temperature Curie-Weiss susceptibility of Nb3Cl8 abruptly drops to Pauli-like paramagnetic behavior below ~ 100 K which is then followed by another Curie-Weiss increase at a further lower temperature. The Pauli-like paramagnetism and the small Curie-Weiss increase are remarkably similar to 1T-TaS2/TaSe239, which are believed to be 2D QSL candidate29, 40. This high degree of similarities encourages us to propose that Nb3Cl8 is a cluster Mott insulator and potentially a 2D molecular QSL. This flat band system interpolates the conventional Mott system with local interactions and the Moiré systems with long-range interactions. It is a system with short-range correlation and flat bands, but free of spontaneous symmetry breaking.

Fig. 1: Band selective renormalization and molecular nature of monolayer Nb3Cl8.
figure 1

a The density plot of the density functional theory (DFT) electronic structure shows several nearly flat bands around the Fermi level (EF), while those bands at higher binding energies are more dispersive. \({\epsilon }_{i}^{cal}\) with i ∈ [1, 4] are the binding energy of the flat bands in DFT calculations. \({\epsilon }_{i}^{exp}\) with i ∈ [1, 3] denote the location of the first three bands detected in angle-resolved photoemission spectroscopy. Here DOS refers to the density of states. b Molecular orbitals of the flat bands \({\epsilon }_{i}^{cal}\) which potentially induce a quantum spin liquid state with effective spin-1/2 Nb-trimer organized in effective triangular lattice.

Full size image

Results

Molecular trimerization

We start our argument by examining the inconsistency recently observed in density functional theory (DFT) calculations (see Supplementary Note 141) and the angle-resolved photoemission spectroscopy (ARPES) for Nb3Cl842, 43. We emphasize that such an inconsistency indicates a remarkably correlated and molecular nature of Nb3Cl8, whose low-energy gap corresponds to a cluster Mott insulator.

Two ARPES measurements were recently conducted at the high-temperature phase of Nb3Cl8 in which Nb trimerizes uniformly. The two neighboring layers shift horizontally and stack alternatively along the c-direction (see Supplementary Note 241 for more details of the monolayer structure). A standard DFT calculation yields the electronic structure shown in Fig. 1(a), where we plot the momentum-dependent density of states in the background and overlay the band structure as red dashed lines. The Bloch bands between [−2, 1] eV are considerably flat compared to the rest high-energy ones, strongly indicating their charge localized character. The presence of flat bands is a hallmark of the Kagome lattice. These flat bands mainly consist of Nb-d states as expected, with all five d orbitals contributing to a certain degree. The Cl-p states are well separated and stay below −3 eV. The flat bands \({\epsilon }_{2}^{cal}\), \({\epsilon }_{3}^{cal}\), and \({\epsilon }_{4}^{cal}\) mostly consist of Nb-\({d}_{3{z}^{2}-{r}^{2}}\) and \({d}_{{x}^{2}-{y}^{2}}\) orbitals. They are the states relevant in ARPES. The discrepancy between DFT and ARPES are two folds. First, the three flat bands observed in ARPES locate at higher binding energies indicated by \({\epsilon }_{i}^{exp}\). \({\epsilon }_{2-4}^{cal}\) agree nicely with \({\epsilon }_{1-3}^{exp}\) if one shift the former ones downwards by 1 eV42. However, as such \({\epsilon }_{1}^{cal}\) will simultaneously move to the Fermi level which was unfortunately not resolved in ARPES. Second, if one does not shift \({\epsilon }_{2}^{cal}\) and view it as a half-filled Hubbard band, it would split and form a Mott gap under the atomic Coulomb repulsion of Nb-d electrons. The lower Hubbard band, together with \({\epsilon }_{3}^{cal}\) and \({\epsilon }_{4}^{cal}\), would correspond to \({\epsilon }_{1-3}^{exp}\) and can nicely explain the insulating nature observed in ARPES43. While \({\epsilon }_{2}^{exp}\) and \({\epsilon }_{3}^{exp}\) would still need to be shifted around to match better with calculations. So far, a consistent theory that explains both the Mott gap and the rigid shift of the flat bands at higher binding energies is lacking. In particular, why the electronic correlation selectively operates only on \({\epsilon }_{2}^{cal}\) but not on the other bands, which consist of Nd-d orbitals as well, is not yet discussed.

We argue that the above discrepancy between DFT and ARPES strongly points to the fact that Nb3Cl8 is not a conventional TM Kagome lattice governed by atomic orbitals. It is a molecule with strong intersite hybridization and Coulomb interaction, which cannot be adequately described by local approximations, such as the dynamical mean-field theory (DMFT) on atomic orbitals44. To make this point clear, in the following we take monolayer Nb3Cl8 as an example (see Supplementary Note 241 for details of the structure), whose low-energy excitation is similar to the bulk ones due to the small interlayer coupling43. We further extract the low-energy flat bands with the Wannier tight-binding technique45. We average the momentum of the tight-binding model \({\hat{H}}_{m{m}^{{\prime} }}({{{{{{{\bf{k}}}}}}}})\) to obtain the local Hamiltonian \({\hat{H}}_{m{m}^{{\prime} }}\) (see Supplementary Note 341), whose eigenvalues are the local energy levels denoted as blue dashed lines. The small deviation of the DFT bands from the local energy levels indicates, on the one hand, the coupling between different unit cells is negligibly small, i.e., the Nb-trimer isolates from each other. On the other hand, the atomic orbital decomposition of the local energy levels would largely resemble that of those flat bands, from which we can better understand the loss of atomic orbital character in Nb3Cl8.

The six Wannier orbitals (see Supplementary Note 341) display two distinct characters. Three of them reside at Nb sites. While the other three locate at the middle of the Nb-Nb bond. Every eigenstate mixes the two types of Wannier orbitals, yielding a molecular state, see Fig. 1b. Due to the delocalized nature of some Nd-d states, a study of Nb3Cl8 with local approximation, such as the DMFT, will not be able to capture the molecular nature of it. In Fig. 2a we showed a calculation of the DFT + DMFT of monolayer Nb3Cl8 with the EDMFT package46,47,48. As DMFT solves the impurity problem individually at each Nb site but neglects the correlation between different Nb atoms, there is no surprise that the resulting electronic structure significantly differs from ARPES. In particular, DMFT predicts a metallic state and smaller binding energies for both bands below the Fermi level, contradicting the experiment.

Fig. 2: Nb3Cl8 as a cluster Mott insulator.
figure 2

a Electronic structure from a density functional theory + dynamical mean-field theory (DFT + DMFT) calculation with Hubbard interaction strength U = 4.0 eV, and Hund coupling strength J = 0.8 eV. b Renormalization of the flat bands from a DFT + molecular-orbital DMFT (MO-DMFT) calculation with U = 0.75, V = 0.6 eV. The position of the chemical potential (EF) is arbitrary inside the Mott gap and has been shifted to the bottom of the upper Hubbard band in this plot.

Full size image

Cluster Mott insulator

In the following, we show that to get the desired experimental binding energies \({\epsilon }_{1-3}^{exp}\), an extended Hubbard model naturally emerges which coincides with the molecular nature of the Nd-trimer. As indicated by the DFT calculations, six electrons fully occupy energy levels ϵ3 and ϵ4, and one electron stays at ϵ2, as shown in Fig. 3a. Starting from this state, we examine three different scenarios of the interacting Hamiltonian and analyze their single particle excitations (see Supplementary Fig. 341) to resolve the most appropriate description.

$$H={H}_{0}+U\mathop{\sum}\limits_{i=1}{n}_{i\uparrow }{n}_{i\downarrow }\ ,$$
(1)
$$H={H}_{0}+U{n}_{2\uparrow }{n}_{2\downarrow }+V\mathop{\sum}\limits_{i\ne 2,\sigma }{n}_{2,\sigma }{n}_{i,\bar{\sigma }}\ .$$
(2)
$$H={H}_{0}+U\mathop{\sum}\limits_{i}{n}_{i\uparrow }{n}_{i\downarrow }+\frac{V}{2}\mathop{\sum}\limits_{i\ne j,\sigma ,{\sigma }^{{\prime} }}{n}_{i,\sigma }{n}_{j,{\sigma }^{{\prime} }}\ .$$
(3)

H0 = ∑i,σ(ϵi − μ)niσ is the energy levels. i denotes the energy level index instead of the real-space coordinate as in the conventional Hubbard model, i.e. the three Hamiltonians are defined in the eigenbasis. U is the Coulomb repulsion between two electrons with opposite spins occupying the same band. V is the repulsion between two electrons occupying two different bands. \(\bar{\sigma }=-\sigma\) denotes the opposite spin orientation with respect to σ. Here we also introduce the chemical potential μ to ensure the correct filling, i.e., seven electrons on six states. To reproduce the experimental \({\epsilon }_{1-3}^{exp}\), the desired shift of energy levels would be the following: ϵ2 splits into lower and upper Hubbard bands, meanwhile, ϵ3 and ϵ4 move to higher binding energies.

Fig. 3: Single-particle excitations of various interacting Hamiltonian.
figure 3

a The ground state configuration of seven electrons occupying six states. b–d display the single-particle excitation energies corresponding to model Eq. (1)–(3). In each plot, we put the chemical potential below the upper Hubbard band. But its location inside this gap is arbitrary.

Full size image

First, model Eq. (1) correctly predicts the Mott gap but cannot describe the shift of ϵ3 and ϵ4, as shown in Fig. 3b. In Eq. (1), the electron excitations for ϵ1 and ϵ2 are ϵ1 − μ and ϵ2 − μ + U. Removing one electron from ϵ2−4 would lead to the hole excitations ϵ2 − μ and ϵ3,4 − μ + U. To keep the correct filling, we put μ right below the upper Hubbard band, which leads to the three occupied states ϵ2 − U, ϵ3 and ϵ4. That is to say that, if we consider Coulomb repulsion at all energy levels as in Hamiltonian Eq. (1), ϵ2 will develop a Mott gap but ϵ3 and ϵ4, unfortunately, will not move to the higher binding energy, disapproving this model (see Supplementary Note 441 for more details).

Next, we move to model Eq. (2) that seems artificial as it indicates a band-selective renormalization in this system, i.e., it considers the Coulomb repulsion only on energy level ϵ2 but neglects interactions on all the others. Surprisingly, model Eq. (2) correctly describes both the Mott gap and the shift of ϵ3,4, which nicely agrees with ARPES43. It becomes the same model considered in Gao’s work.43 when V = 0. Without repeating the same calculations above, we show the single-particle excitation energy in Fig. 3c. Setting the chemical potential μ below the upper Hubbard band, we immediately see that the lower Hubbard band and ϵ3,4 all shift to higher binding energy in the same amount of U42, 43. We note that, although it looks artificial in eigenspace, model Eq. (2) becomes realistic and well-defined on the Wannier basis. Transforming Eq. (2) back to the Wannier basis, it becomes

$$H \approx \mathop{\sum}\limits_{i,j,\sigma }{c}_{i\sigma }^{{{{\dagger}}} }({H}_{i,j}-\mu {\delta }_{i,j}){c}_{j,\sigma }\\ +\tilde{U}{n}_{loc,\uparrow }{n}_{loc,\downarrow }+\tilde{V}\mathop{\sum}\limits_{\sigma ,{\sigma }^{{\prime} }}{n}_{loc,\sigma }{n}_{bond,{\sigma }^{{\prime} }}\\ +{\tilde{V}}^{{\prime} }\mathop{\sum}\limits_{\sigma }({n}_{1\alpha ,\sigma }{n}_{2\beta ,\sigma }+{n}_{1\alpha ,\sigma }{n}_{3\alpha ,\sigma }+{n}_{2\beta ,\sigma }{n}_{3\alpha ,\sigma })\ .$$
(4)

Here we only keep the leading terms and neglect all other ones that are at least an order of amplitude smaller. Hi,j is the non-interacting local Hamiltonian in Supplementary Eq. (1)41. nloc,σ = n1α,σ + n2β,σ + n3α,σ represents the particle number of the atomic-like Wannier orbitals, and nbond,σ = n1β,σ + n2α,σ + n3β,σ for that of the molecular-like Wannier orbitals. The notation “site + orbital" refers to the orbital basis function of the local Hamiltonian. For example, 1α corresponds to an orbital associated with the first Nb atom. α is one of the two Nd-eg orbitals we took for the initial Wannier projection. After the maximal localization procedure, it may move away from the Nb site. See Supplementary Note 341 for more details. \(\tilde{U}=0.094U+0.43V\), \(\tilde{V}=0.32V\), and \({\tilde{V}}^{{\prime} }=0.43V\) are the renormalized Coulomb interactions. Different from the U and V parameters in Eqs. (1)–(3) which are defined for the repulsions between electrons in the band basis, here we use \(\tilde{U}\), \(\tilde{V}\), and the \({\tilde{V}}^{{\prime} }\) to denote electron repulsions on the Wannier orbitals. Mathematically, the two different sets of parameters are related by the transformation discussed in Supplementary Note 541. \(\tilde{U}\) and \(\tilde{V}\) include the interaction on the same Nb site, as well as the interaction between different Nb sites, which is a characteristic feature of the cluster Mott insulators. Additionally, there exists Coulomb interaction between the atomic-like and the molecular-like Wannier orbitals as described by \(\tilde{V}\) term. Equation (4) reveals that monolayer Nb3Cl8 is a cluster Mott insulator and the intersite interaction is as strong as the intrasite ones. Comparing this model to the experimental energy levels \({\epsilon }_{1}^{exp} \sim -0.75\) eV, \({\epsilon }_{2}^{exp} \sim -1.75\) eV, and \({\epsilon }_{3}^{exp} \sim -2.5\) eV, we roughly estimate U and V as 0.75 eV and − 37 meV. The negligibly small V agrees well with the previous study43. We, thus, conclude that the mysterious band-selective renormalization observed in ARPES results from a peculiar orbital composition yielding a complete molecularization of Nb-trimer.

Model Eq. (2) becomes a more realistic model Eq. (3) if one further includes the interaction of the molecular-like Wannier orbitals. As shown in Fig. 3(d), in model Eq. (3)U determines the shift of the lower Hubbard band, and V controls the shift of ϵ3,4. Thus, U and V are of similar amplitude in this model and they are estimated as U ~ 0.75 eV and V ~ 0.6 eV. On the Wannier basis, this model contains all the intrasite and intersite Coulomb repulsions, which is a standard extended Hubbard model for cluster Mott insulators. By solving the DFT + molecule DMFT (MO-DMFT) equation with the interacting Hamiltonian in Eq. (3) and the Hubbard-I approximation, we correctly recovered the experimental spectra including both the formation of the Mott gap at the Fermi level and the rigid band shift at higher binding energies (see Fig. 2b). We note that, in this calculation, we only considered the six correlated orbitals and neglected all other free electron bands for simplicity. The MO-DMFT equation for solving Eq. (4) does not introduce supercell and the three-site extended Hubbard model can be viewed as a multiorbital Hubbard model defined in the Nb3Cl8 primitive cell. Compared to the single impurity DMFT defined on the atomic orbitals, the nonlocal Coulomb interaction within the Nb-trimer gives rise to the cluster Mott insulating phase. Model Eq. (2), as the simplified version of Eq. (3), is also adequate to resolve the inconsistency of DFT and ARPES. From the comparison in Fig. 2a and b, we clearly see the critical role of the intersite interaction within each trimer, thus, we confirm monolayer Nb3Cl8 is a cluster Mott insulator. Our findings agree well with the observations of cluster/molecule Mott insulating phase in GaV4S849, where the difference in applying DMFT methodology on atomic- and molecular-orbitals was clearly shown, emphasizing the importance of nonlocal Coulomb interactions.

Possible molecular quantum spin liquid state

After convincingly demonstrating monolayer Nb3Cl8 as a cluster Mott insulator with strong intratrimer Coulomb interactions as the Mo3O8 family materials20,21,22,23,24,25,26,27,28 and 1T-TaS229, we want to further argue that it may realize a molecular QSL phase. The precondition for a QSL is the presence of randomly distributed local moments. They do not form long-range order but the ground state wavefunction demonstrates a long-range entanglement. The latter requires a more sophisticated Parton theory which can be formulated based on the electronic models we proposed in the previous section. Here we want to shed light on the former and show that, in each Nb-trimer, there exists a spin-1/2 local moment that does not belong to any specific Nb atom but is contributed collectively by them. This conclusion is well anticipated and again resembles that of Mo3O8 family compounds. As there are seven electrons inside each trimer, the effective spin is 1/2 contributing to each Nb-trimer 1μB local moment, which is further confirmed in our spin-polarized DFT calculations. Furthermore, we argue from the examination of several magnetic configurations that these local moments are unlikely to form any long-range order. Various magnetic structures have competing energies hindering anyone to stabilize, which leads to a magnetically disordered ground state. We also examined the bare spin susceptibility whose peak locations in Q-space imply the spin correlation in real space. Both the DFT and DFT + MO-DMFT spin susceptibility \({\chi }_{{{{{{{{\bf{Q}}}}}}}}}^{spin}\) of the ϵ2-band does not show any obvious peak but a continuum of the maximum along the Brillouin zone boundary (see Fig. 4a and b) for DFT and MO-DMFT results, respectively). Including more Nb-d bands does not change the behavior of \({\chi }_{{{{{{{{\bf{Q}}}}}}}}}^{spin}\). We note that we calculated the correlated spin susceptibility with the MO-DMFT lattice Green’s function without vertex contribution (see Supplementary Note 641). The absence of the Fermi surface in the DMFT electronic structure of Fig. 2b results in a flat spin susceptibility. The slow varying \({\chi }_{{{{{{{{\bf{Q}}}}}}}}}^{spin}\) indicates that it is unlikely for any spin long-range order to form in this system. Thus, we conclude that in monolayer Nb3Cl8, each Nb-trimer contributes 1μB but these local moments do not organize with long-range order as different magnetic orders strongly compete in energy. The competing magnetic correlation yields a paramagnetic ground state. We further simulate this paramagnetic state in DFT with the special quasi-random structure (SQS) method50 and obtain an electronic structure nicely resembling that of ARPES as shown in Fig. 4c. This calculation reproduces all essential features including the development of gap and the shift of bands towards higher binding energy. The energy shift of ϵ4 is less pronounced than in ARPES possibly due to the insufficient treatment of the true many-body correlation in spin-polarized DFT.

Fig. 4: Competing magnetic interactions.
figure 4

a Momentum-dependent spin susceptibility \({\chi }_{{{{{{{{\bf{Q}}}}}}}}}^{spin}\) calculated with the density functional theory (DFT). b DFT + molecular orbital-dynamical mean-field theory (MO-DMFT) electronic structures. The colorbar indicates the amplitude of \({\chi }_{{{{{{{{\bf{Q}}}}}}}}}^{spin}\). c The paramagnetic electronic structure from the DFT- special quasi-random structures (SQS) calculations with DFT + U (Hubbard term strength U = 3 eV) and spin-orbit coupling.

Full size image

After demonstrating the lack of long-range magnetic order in monolayer Nb3Cl8, we further compare it to three Mo3O8 family compounds, i.e., LiZn2Mo3O8, Li2InMo3O8, and Li2ScMo3O820,21,22,23,24,25,26,27,28. We show that monolayer Nb3Cl8 stays deeper in the cluster Mott phase than the three compounds and is more likely to be U(1) QSL. The three Mo3O8 compounds have similar two-dimensional crystal structures, electron configurations, and molecular orbitals as Nb3Cl8, but they demonstrate different low-temperature magnetic behaviors. LiZn2Mo3O8 and Li2ScMo3O8 show strong antiferromagnetic couplings but no long-range order, while Li2InMo3O8 exhibits clear antiferromagnetic order. The bandwidth W of each isolated band becomes smaller from Li2InMo3O8, Li2ScMo3O8, LiZn2Mo3O8, to Nb3Cl8, indicating a more localized nature of the Nb-trimer in Nb3Cl8 than the Mo-trimer in Mo3O8 systems. The ratio of intertrimer/intratrimer hoppings t2/t1 has the smallest value in Nb3Cl8. As a consequence, Nb3Cl8 stays at the large limit of V1/t2 and t1/t2 in the phase diagram Fig. 1a of Yao’s work51, where V1 is the intratrimer Coulomb interaction strength and t1, t2 are the intratrimer and intertrimer hopping amplitude. Further given the possible absence of long-range magnetic order, it is likely that monolayer Nb3Cl8 is a cluster Mott insulator with a U(1) QSL. Furthermore, the energy spacing Δ between the states at the Fermi level and the first conduction bands is also much larger in Nb3Cl8 than in the Mo3O8 family systems. The effective spin exchange coupling between Nb-trimers is thus expected to be smaller in Nb3Cl852. Thermal or quantum fluctuations are then easier to suppress the long-range magnetic ordering in Nb3Cl8 and favor a QSL state.

Conclusions

We discuss the implications of our findings for the experiment. The breathing mode of the Nb-trimer is similar to the Mo-trimer in Mo3O8 family compounds but it has a simpler van der Waals structure that makes it easier to prepare a monolayer form. Due to the large interlayer coupling, bulk Nb3Br8 and Nb3I8 are different from monolayer Nb3Cl8. They are band insulators in the bulk form. However, in the monolayer limit, the removal of the interlayer coupling would result in a similar half-filled flat band, and thus our conclusions presented in this work would apply as well.

Though the nature of the cluster Mott insulator has been explained in the affirmative, the verification of the molecular QSL state in Nb3Cl8 further requires more experimental efforts, especially the low-temperature thermal conductivity and inelastic neutron scattering measurement. The broad peak in magnetic specific heat below the Curie-Weiss temperature and the broad continuum of spin excitations in the inelastic neutron scattering are two common features of QSL discovered in many candidate systems. Additionally, the local probe measurements such as NMR and muon spin resonance could be helpful to confirm more convincingly the lack of magnetic long-range order. Theoretically, if one neglects the small V term, the effective model Eq. (4) becomes a single-band cluster model. Such a single-band extended Hubbard cluster model on breathing Kagome lattice is known to support U(1) QSL phase51. We note that, besides the correlated orbitals, model Eq. (4) contains non-interacting molecular-like orbitals hybridizing with the atomic-like orbitals, providing additional ingredients to engineering the QSL state.

In summary, we consistently explain the discrepancy between DFT calculations and ARPES measurement. We validate the single-band Hubbard model in energy space as the minimal model correctly describing its low-energy excitations. We find that the novel band-selective normalization observed in this system is a consequence of the coexistence of atomic and molecular Nb-d orbitals in this system. We conclude that monolayer Nb3Cl8 is a cluster Mott insulator embedded in an emerging triangular lattice. Each Nb-trimer contributes 1μB local moments, which do not order in long-range due to the degeneracy of various competing magnetic states. We argue that it may constitute a candidate material for molecular quantum spin liquid state with flat bands.

Methods

DFT calculations

The first-principle study of Nb3Cl8 was conducted within the Kohn-Sham scheme of density-functional theory (DFT). We adopted three different implementations of DFT in different calculations, including the Quantum ESPRESSO (QE) package53, 54, the Vienna Ab initio Simulation Package (VASP)55, 56, and the full-potential augmented plane-wave + local orbitals (APW+lo) program, WIEN2k57.

We used QE to calculate the DFT spectral function and the site-orbital resolved density of states shown in Fig. 1b. The flat bands between [−3:1] eV have less intensity than other bands with higher binding energy, but these bands consist of similar orbital components and have similar intensity. As long as the flat bands are within the binding-energy range accessible for ARPES, their signal should be captured.

We used VASP with GGA-PBE electronic exchange-correlation functional58 in the crystal structure relaxation, the phonon dispersion, and the slab calculations. The plane-wave cut-off energy is 400 eV. An 8 × 8 × 1 k-mesh with the Monkhorst-Pack sampling scheme59 is used for the Brillouin zone summation. We set the vacuum space to be larger than 15 Å in the monolayer slab. We include the spin-orbit coupling effect in the electronic structure calculation but neglect is in the crystal structure relaxation and the phonon calculations.

SQS method for simulating paramagnetic state

To study the electronic structures of paramagnetic Nb3Cl8, we used two different methods. One is the DFT calculation based on the concept of randomly distributed local moments. We used the MCSQS code50 of the Alloy Theoretic Automated Toolkit (ATAT)60,61,62 to generate SQS supercell. SQS is optimized according to the criterion that a specified set of correlations between neighboring site occupations in the supercell match the corresponding correlation of the true disordered system50, 63. In this study, we choose two-body and three-body correlations to construct the objective function with the cutoff distance of clusters Nb-trimers set to 9.5 Å and 6 Å. The SQS contains 176 atoms, which is a 4 × 4 × 1 supercell in comparison with the unit cell of Nb3Cl8 monolayer. The electronic structures of PM Nb3Cl8 calculated on the SQS supercell were unfolded back to the primitive cell by the BANDUP code64, 65.

DMFT calculations

To study how the electronic correlations and magnetism interplay for the electronic structure of Nb3Cl8, we performed the all-electron charge self-consistent DFT+DMFT calculations by the embedded-DMFT package66. The DFT part for the DFT+DMFT calculations is prepared by Wien2k. We set RKmax = 7.0 and use a 13 × 13 × 1 k-mesh for Brillouin zone sampling. GGA-PBE functional was still used in WIEN2k calculations. DMFT method maps the strongly correlated Hubbard model to the quantum impurity model. In this study, we set the on-site Hubbard interaction U = 4 eV and the Hund’s exchange interaction J = 0.8 eV for Nb-4d orbitals. The quantum impurity problem was solved by the continuous-time quantum Monte Carlo (CT-QMC) method67,68,69,70.

To account for the intersite correlations beyond the simple-impurity approximation, we have constructed an extended Hubbard model on three Nb-sites and six orbitals. It does not require a supercell calculation, thus, it can be viewed as a multi-flavor DMFT model with favor being both site and orbital indices. We solve this model, i.e., Eq. (4), with the Hubbard-I approximation without full charge self-consistency.