Introduction

Two-dimensional van der Waals materials constitute a versatile platform to realize quantum states by design, as they can be synthesized in many different stacking conditions1, offer a wide variety of chemical compositions, and are easily manipulated by back gates, strain and the like. Stacking two sheets of van der Waals materials atop each other at a relative twist has recently emerged as a vibrant research direction to enhance the role of electronic interactions, with first reports on twisted bilayer graphene2,3,4,5,6 and another van der Waals materials stacked atop each other at a twist7,8,9,10,11,12,13,14,15,16,17 displaying features of correlated physics that afford a high level of control. In particular, bi-, tri-, and quadruple-layer graphene18 as well as twisted few-layer transition metal dichalcogenides (TMDs)19,20 are currently under intense experimental scrutiny13,21,22,23,24,25,26,27,28,29. By forming a moiré supercell at small twist angles, a large unit cell in real space emerges for twisted systems, which due to quantum interference effects leads to a quasi-two-dimensional system with strongly quenched kinetic energy scales. This reduction in kinetic energy scale, signaled by the emergence of flat electron bands, in turn enhances the role of electronic interactions in these systems. Therefore, twisted systems enable the realization of new correlated condensed matter models, establishing a solid-state quantum simulator platform30.

Whereas the flatting of band dispersions in two-dimensional moiré superlattices results mainly from the localization of charge density distributions by the moiré potential, a well-known alternate pathway to flat bands can occur in certain lattices such as the Lieb and the Kagome lattices. Here, purely geometric considerations lead to the formation of perfectly localized electronic states that have weight only on single plaquettes or hexagons, respectively, and that are eigenstates of the kinetic Hamiltonian due to destructive interference between lattice hopping matrix elements31. To put it differently, linear combinations of the macroscopically degenerate extended Bloch states in these systems allows to form localized Wannier-like eigenstates (living on single plaquettes or hexagons in the examples above) with no dispersion (for a review on the subject see, e.g,32). Such flat band systems can give rise to many interesting phenomena, such as the formation of nontrivial topology when time-reversal symmetry is broken, or other exotic quantum phases of matter due to their susceptibility to quantum fluctuations and electronic correlations32.

Here, we demonstrate that both flat band mechanisms can be engineered to coexist in twisted bilayers of MoS2 (tbMoS2): a TMD of direct experimental relevance that has been extensively studied from synthesis to applications33,34. We confirm that families of flat bands emerge when two sheets of MoS2 in the 2H structure are stacked at a twist12,35 due to moiré potentials. Our large-scale ab initio based simulations show that while the first set of engineered flat bands closest to the edge of the bandgap with twist angles close to Θ ≈ 0∘ can be used to effectively engineer a non-degenerate electronic flat band in analogy to a single layer of graphene at meV energy scales, more intriguingly, the next set of flat bands instead realizes a strongly asymmetric flat band px–py honeycomb lattice36,37. Both of these families of bands should be accessible experimentally via gating. The strongly asymmetric nature of this px–py honeycomb lattice is in marked contrast to the much-discussed case of twisted bilayer graphene, where an approximately symmetric version of such a Hamiltonian is now believed to describe the low-energy flat band structures found at small twist angle38,39,40,41,42. The strongly asymmetric px–py honeycomb model itself features two almost entirely dispersionless flat bands that touch the top and the bottom of graphene-like Dirac bands at the Gamma point, respectively. These flat bands in this model originate from destructive interference, in analogy to flat bands in the Lieb and the Kagome lattices31 discussed above, and will be referred to as ultra-flat bands in the following discussion. On top of that, the total bandwidth of the strongly asymmetric px–py honeycomb effective model realized here (all four bands) can be further flattened by decreasing the twist angle. In addition, these ultra-flat bands can be topologically nontrivial in the presence of spin-orbital coupling (SOC)43. Although all the flat bands discussed here originate from the Γ-point states of MoS2 and are not affected by intrinsic SOC (see Supplementary Fig. 3), we expect that substrate engineering44 can be used to introduce SOC coupling into these bands and invoke topologically nontrivial behavior of the ultra-flat band states. Previously, the px-py model was studied in the context of cold gases where exotic correlated phases were predicted36,45,46, as well as in semiconductor microcavities47 and certain 2D systems such as organometallic frameworks48,49 and Bismuth deposited on SiC50 with a focus on their nontrivial topology properties. Our findings elevate tbMoS2 to an interesting platform where effects of ultra-flat bands can be studied systematically in a strongly correlated solid-state setting.

Notably, in the strong-coupling regime, the px–py model amended by Hubbard and Hund’s interactions gives rise to a spin-orbital honeycomb model which – depending on the specific parameters and symmetries of the model – hosts magnetic, orbital as well as valence-bond orderings, or even more exotic quantum spin-orbital liquid phases51,52,53. With this, our work adds an interesting type of lattice model – the highly asymmetric px–py Hubbard model – to the growing list of systems that can effectively be engineered using the twist angle between multiple layers. This is particularly intriguing as we maintain the full advantages that come with two-dimensional van der Waals materials, such as relative simplicity of the chemical composition and controllability of the material properties; e.g. of the filling (by a back gate), electric tunability (by displacement fields) or the bandwidth of the model (by the twist angle).

Results

Ab initio characterization of twisted MoS2

We first characterize the low-energy electronic properties of twisted bilayer MoS2 using density functional theory (DFT) calculations (see Methods). DFT in particular has established itself as a reliable tool to provide theoretical guidance and to predict the band structure of many twisted bi- and multilayer materials8,13,15. However, such a first-principles characterization becomes numerically very demanding as the twist angle Θ approaches small values and the unit cell becomes very large entailing many atoms (of the order of a few thousands and more). Nevertheless, it is that limit in which strong band-narrowing effects and as a consequence prominent effects of correlations are expected. The results of such a characterization are summarized in Fig. 1. Note that atomic relaxation has been shown to affect the electronic properties of twisted 2D materials12,35,54. While for twisted bilayer graphene this effect is only significant at twist angles smaller than 1 degree54, it noticeably alters the low-energy band dispersions and charges density localization for twisted transition metal dichalcogenides bilayer (such as MoS2) even with relatively large twist angles above 1 degree12,35. Therefore, we relax all the systems in our DFT calculations. Panel (a) shows the relaxed atomic structure of two sheets of MoS2 in real space, twisted with respect to each other. A moiré interference pattern forms at a small twist angle yielding a large unit cell, within which we identify different local patterns of stacking of the two sheets of MoS2, indicated via areas framed by cyan, magenta or purple dashed lines. The local stacking arrangements of the respective areas are given in the right sub-panels of the panel (a). Note that the BMo/S and the BS/Mo regions are equivalent as they are related by C2 symmetry. These equivalent BMo/S/BMo/S regions form a hexagonal network. In panel (b) we show the ab initio band structure of the twisted material after relaxation, where we find two families of bands that will become increasingly flat and start to detach from all other bands, as the twist angle is lowered. We mark these bands by blue and red color in panel (b), which shows results for decreasing angles from Θ = 3.16∘–2.28∘. The bandwidth of these two energetically separated groups of bands is summarized in panel (c) of Fig. 1. We find that the bandwidth of these two bands shrinks drastically as the angle is decreased, yielding bandwidths of the order of 10 meV as the angle approaches Θ ≈ 2°. Similar features are also shown in the work of Naik et al.35. The bandwidth and the shape of the flat bands (in particular for the second set) in our calculations are slightly quantitatively different from the previous work probably because we relax the structure directly with DFT while the authors of ref. 35 use a force-field approach. Note that these flat bands near the top of the valence bands originate from the states around the Γ point in the Brillouin zone of the primitive unit cell of untwisted MoS2, with both S pz and Mo d\({}_{{{{{{{{\rm{{z}}}}}}}^{2}}}}\) characters (see Supplementary Fig. 2 for a DFT characterization of the orbital contribution to the different bands). This is different to the case of twisted WSe2, where the top valence flat bands originate from the states around the K point in the Brillouin zone of the primitive unit cell (dominated by W d\({}_{{x}^{2}-{y}^{2}}\) and dxy orbitals), which experience different interlayer moiré potentials compared with those of the Γ-point flat bands discussed here leading to an effective triangular lattice Hubbard model13. Since also in other TMDs, such as MoSe2 and WS2, the top of the valence band in the untwisted bilayer is also located at the Γ point in the Brillouin zone55,56, the physics we discussed here transfers also to those materials being twisted.

Fig. 1: Atomic and electronic structures of twisted bilayer MoS2.
figure 1

a Atomic structure of tbMoS2 at Θ = 3.15°. Local atomic arrangements of the three different regions in the moiré unit cell are indicated in the right panels. The Mo (S) atoms are indicated with purple (yellow) balls. b Evolution of low-energy band structures at the top of the valence bands of tbMoS2 with decreasing small twist angles. The first set and the second set of valence bands are highlighted with blue and red lines, respectively. c Evolution of the bandwidth of the first set and the second set of valence bands with decreasing twist angles. Inset: twist angle dependence of the ratio of the hopping amplitudes tπ and tσ in the px–py honeycomb lattice.

Full size image

The upper bands in Fig. 1 (marked in blue) show a Dirac cone at the K point and behave very similar to the bands found for monolayer graphene (with the exception of a reduced bandwidth). They are spin degenerate in nature, but feature no additional degeneracy except at certain high symmetry points. Instead, the next set of bands (marked in red) is essential to our work. They too feature a Dirac cone at the K point, but also feature two additional ultra-flat bands at the top and bottom in addition to a band structure similar to graphene. The ratio between the width of the ultra-flat and the flat bands decreases as the angle is decreased, but saturates in our calculations as a twist angle of Θ ≈ 2.28° is approached. We attribute this saturation to lattice relaxation effects; note however that the overall bandwidth keeps decreasing. To access this second set of bands we need to empty the bands marked in blue first. The effects of this doping are of minor quantitative nature (see Supplementary Fig. 5).

Remarkably, this second family of flat bands is well-described by an effective px–py tight-binding model on a honeycomb lattice, depicted schematically in Fig. 2a, and conveniently described by the following Hamiltonian:

$${H}_{0}= \, \mathop{\sum}\limits_{\langle i,j\rangle ,s}({t}_{\sigma }{{{{{{{{\bf{c}}}}}}}}}_{i,s}^{{{{\dagger}}} }\cdot {{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }\cdot {{{{{{{{\bf{c}}}}}}}}}_{j,s}-{t}_{\pi }{{{{{{{{\bf{c}}}}}}}}}_{i,s}^{{{{\dagger}}} }\cdot {{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }\cdot {{{{{{{{\bf{c}}}}}}}}}_{j,s})\ \\ + \mathop{\sum}\limits_{\langle \langle i,j\rangle \rangle ,s}({t}_{\sigma }^{N}{{{{{{{{\bf{c}}}}}}}}}_{i,s}^{{{{\dagger}}} }\cdot {{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }\cdot {{{{{{{{\bf{c}}}}}}}}}_{j,s}-{t}_{\pi }^{N}{{{{{{{{\bf{c}}}}}}}}}_{i,s}^{{{{\dagger}}} }\cdot {{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }\cdot {{{{{{{{\bf{c}}}}}}}}}_{j,s}),$$
(1)

where \({{{{{{{{\bf{c}}}}}}}}}_{i,s}={({c}_{i,{{{{{{{\rm{x}}}}}}}},s},{c}_{i,{{{{{{{\rm{y}}}}}}}},s})}^{T}\) with ci,x(y),s annihilating an electron with px(y)-orbital at site i and with spin s = ↑, ↓. \(\left\langle i,j\right\rangle\) (\(\left\langle \left\langle i,j\right\rangle \right\rangle\)) denotes (next) nearest neighbors. For each sum in Eq. (1), the first term describes the σ hopping (head to tail) between the p-orbitals and the second term denotes the π hopping (shoulder to shoulder). Furthermore, \({{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }=({{{{{{{{\bf{r}}}}}}}}}_{i}-{{{{{{{{\bf{r}}}}}}}}}_{j})/| {{{{{{{{\bf{r}}}}}}}}}_{i}-{{{{{{{{\bf{r}}}}}}}}}_{j}|\), with ri being the position of site i and \({{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }=U{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }\) with U being the two-dimensional 90 degree rotation matrix \(U=\left(\begin{array}{ll}0&-1\\ 1&0\end{array}\right)\). Finally, tσ and tπ (\({t}_{\sigma }^{N}\) and \({t}_{\pi }^{N}\)) are the nearest neighbor (next-nearest neighbor) hopping amplitudes for the σ-bonding term and π-bonding term, respectively. Figure 2b, c depict the corresponding dispersions, density of states, and wave functions in comparison to model predictions, illustrating that the four moiré bands at low energies are well captured by Eq. (1) upon the choice of hopping parameters tπ = 0.25tσ, \({t}_{\sigma }^{N}=0.07{t}_{\sigma }\) and \({t}_{\pi }^{N}=-0.04{t}_{\sigma }\). The density of states exhibits a characteristic four van Hove singularities structure, with two originating from the Dirac bands and two stemming from the additional two ultra-flat bands. The small ratio between the nearest neighbor hopping amplitudes tπ/tσ determines the residual small dispersion in the ultra-flat bands we report. This ratio is controllable by the twist angle, which is summarized in the inset of Fig. 1c. All these parameters are related to the interlayer moiré potential and are thus expected to be also affected and controllable by the uniaxial pressure perpendicular to the layers as demonstrated for twisted bilayer graphene4.

Fig. 2: px-py honeycomb model for twisted bilayer MoS2.
figure 2

a Illustration of the model: in a honeycomb lattice composed of sublattices A and B, there are two orthogonal orbitals (px and py) at each of the two sublattice sites. The solid and the dashed lines denote the py and the px orbitals, respectively, and the red and the blue color denotes the positive and the negative side of the orbital, respectively. b Fitting the dispersion of the px-py model to the second set of valence bands of tbMoS2 calculated with DFT for tbMoS2 at 2.65°. The left panel shows the corresponding density of states displaying the signature four-peak structure. c Charge density, real and imaginary parts of the wave function calculated with DFT for the states in the two quasi-flat bands 1 and 4 shown in (b). The isosurface of the charge density is colored yellow. The positive and the negative parts of the isosurfaces of the wave function are colored in pink and purple, respectively. The solutions of the corresponding states from the px–py model are indicated with the blue and red ovals and agree with the DFT results.

Full size image

The flat band wavefunctions consist of atomic wavefunctions from the pz orbital on S atoms and the d\({}_{{{{{{{{\rm{{z}}}}}}}^{2}}}}\) orbital on Mo atoms. Modulated by the moiré potential, the weighting of the atomic wavefunctions and their modulus square (i.e., charge density) vary at different atomic sites across the whole supercell, showing distinct patterns for different flat band states at the K point in the supercell Brillouin zone as shown in Panel (c) of Fig. 2. These patterns of the charge density as well as the real and the imaginary part of the total wavefunctions obtained from DFT show features consistent with those of the px–py Hamiltonian of Eq. (1). Note, that we call this the px-py Hamiltonian to connect to established literature on the subject; whereas the actual moiré wave functions are composed of pz and dz orbitals, they transform like px, py orbitals according to the irreps of the reduced symmetry group of the moiré supercell. Interestingly, the charge density distribution of the top ultra-flat band state displays a Kagome lattice structure. We have thus unambiguously established twisted MoS2 to be a candidate system to realize a px–py model on the honeycomb lattice with strongly asymmetric hoppings tσ and tπ, giving rise to a new set of ultra-flat bands.

Correlations and magnetic properties

We now study the role of electronic interactions. As the highly-anisotropic px-py orbital structure constitutes the essential novelty of twisted bilayer MoS2, we focus on quarter filling (one electron per sublattice in the Moié unit cell) where orbital fluctuations can be expected to be crucial. This filling fraction is straightforwardly accessible in the experiment via back gating, and we defer a discussion of the half-filled case to Supplementary Note 1. To proceed, we assume purely local electronic interactions, which can be generically parameterized in terms of the Hubbard-Kanamori Hamiltonian:

$${H}_{U}=\, U\mathop{\sum}\limits_{i,\alpha }{n}_{i\alpha \uparrow }{n}_{i\alpha \downarrow }+(U-2J)\mathop{\sum}\limits_{i}{n}_{ix}{n}_{iy}+J\mathop{\sum}\limits_{i,s,s^{\prime} }{c}_{ixs}^{{{{\dagger}}} }{c}_{iys^{\prime} }^{{{{\dagger}}} }{c}_{ixs^{\prime} }^{}{c}_{iys}^{}\\ +J\mathop{\sum}\limits_{i,\alpha \ne \beta }{c}_{i\alpha \uparrow }^{{{{\dagger}}} }{c}_{i\alpha \downarrow }^{{{{\dagger}}} }{c}_{i\beta \downarrow }^{}{c}_{i\beta \uparrow }^{}$$
(2)

for two orbitals with rotational symmetry. More realistic modelling should include long-range interactions. However, for our choice of commensurate quarter filling, any longer-ranged component of the Coulomb interaction at strong-coupling will serve merely to renormalize the effective spin-orbital interactions of the resulting Kugel-Khomskii model and we therefore concentrate on purely local interactions for simplicity. Furthermore, our DFT calculations suggest tπ ≈ 0.25tσ and only weak next-nearest neighbor hopping at small twist angles; we therefore neglect next-nearest neighbor hopping in the analysis below (see Supplementary Fig. 4 for a comparison of the band structures with and without next-nearest neighbor hopping). An ab initio based characterization of the values of U and J requires numerically expansive Wannierzation of the wave functions and is unfortunately beyond the scope of this work. However, by substrate engineering22 it is likely that a whole range of values can be accessed and therefore it is useful to vary these parameters to explore all possible phases accessible in experiments to make concrete predictions. Vice versa given a future experimental observation our results can be used to estimate the strength of correlations.

Figure 3c depicts the local density of states as a function of Hubbard U and Hund’s exchange J interactions, calculated via an exact diagonalization study of Eqs. (1) and (2) for a cluster depicted schematically in (a). Clear evidence of a charge gap beyond U/tσ ~ 4 at small J signifies the onset of a correlated insulator which could be directly observed via transport and scanning tunnelling microscopy. The behavior of the gap is depicted in Fig. 3b as a function of U, J and signifies that charge fluctuations are strongly suppressed for large U. Establishing the existence of a charge gap motivates to set up a strong-coupling Hamiltonian routinely employed for the types of systems under scrutiny here.

Fig. 3: Charge gap and correlations for twisted bilayer MoS2 at vanishing temperature.
figure 3

a depicts the 16-orbital cluster geometry employed for exact diagonalization of the Hubbard-Kanamori Hamiltonian. b depicts the charge gap as a function of Hubbard U and Hund’s exchange J interactions, calculated for the 16-orbital cluster and extracted from (c) the local density of states, which is readily accessible via scanning tunnelling microscopy. A well-defined charge gap develops beyond U/tσ ~ 4 at small J that scales linearly with the Hubbard interaction U. Vertical gray dotted lines indicate phase transitions to charge-ordered states at large J/U, coinciding with a closing of the charge gap.

Full size image

In this regime, a natural follow-up questions concerns possible orderings of the orbital and magnetic degrees of freedom. The corresponding strong-coupling Kugel-Khomskii Hamiltonian57,58,59 for the px-py model at quarter filling is given in refs. 51,52,53 and reads:

$$H= \, \mathop{\sum}\limits_{\langle ij\rangle }\frac{1}{U-3J}{\xi }_{ij}^{1}\left[{t}_{\sigma }{t}_{\pi }{\bar{Q}}_{ij}-({t}_{\sigma }^{2}+{t}_{\pi }^{2})({P}_{ij}^{xy}+{P}_{ij}^{yx})\right]\\ -\,\frac{1}{U+J}{\xi }_{ij}^{0}\left[{t}_{\sigma }{t}_{\pi }{Q}_{ij}+2{t}_{\sigma }^{2}{P}_{ij}^{xx}+2{t}_{\pi }^{2}{P}_{ij}^{yy}\right]\\ +\,\frac{1}{U-J}{\xi }_{ij}^{0}\left[{t}_{\sigma }{t}_{\pi }({Q}_{ij}-{\bar{Q}}_{ij})-2{t}_{\sigma }^{2}{P}_{ij}^{xx}-2{t}_{\pi }^{2}{P}_{ij}^{yy}-({t}_{\sigma }^{2}+{t}_{\pi }^{2})({P}_{ij}^{xy}+{P}_{ij}^{yx})\right].$$
(3)

Here, \({\xi }_{ij}^{1}=3/4+{{{{{{{{\bf{S}}}}}}}}}_{i}{{{{{{{{\bf{S}}}}}}}}}_{j}\) denotes the projector onto triplet states, whereas \({\xi }_{ij}^{0}=1/4-{{{{{{{{\bf{S}}}}}}}}}_{i}{{{{{{{{\bf{S}}}}}}}}}_{j}\) selects the singlet spin states instead. Note that the orbital operators, for example Qij, are bond dependent, giving rise to a strong spatial anisotropy of the resulting spin-orbit model. To be more precise following ref. 52, the operators Qij and \({\bar{Q}}_{ij}\) describe processes where orbital occupations of sites i and j are reversed, that is they are defined as \({Q}_{ij}=({\tau }_{i}^{+}{\tau }_{j}^{+}+{\tau }_{i}^{-}{\tau }_{j}^{-})/2\) and \({\bar{Q}}_{ij}=({\tau }_{i}^{+}{\tau }_{j}^{-}+{\tau }_{i}^{-}{\tau }_{j}^{+})/2\), with \({\tau }_{i}^{\pm }={{{{{{{{\bf{n}}}}}}}}}_{ij}^{\perp }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i}\pm i{\tau }_{i}^{y}\) where \({{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i}={({\tau }_{i}^{z},{\tau }_{i}^{x},{\tau }_{i}^{y})}^{T}\). The orbital projection operators can then be expressed as \({P}_{ij}^{xx}=(1+{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i})(1+{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{j})/4\), \({P}_{ij}^{yy}=(1-{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i})(1-{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{j})/4\), \({P}_{ij}^{xy}=(1+{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i})(1-{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{j})/4\) and \({P}_{ij}^{yx}=(1-{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i})(1+{{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{j})/4\), where e.g. \({P}_{ij}^{xx}\) selects states where the superposition \(({p}_{x}{{{{{{{{\bf{e}}}}}}}}}_{x}+{p}_{y}{{{{{{{{\bf{e}}}}}}}}}_{y}){{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }\) is occupied on nearest neighbor sites i and j connected by the bond \({{{{{{{{\bf{n}}}}}}}}}_{ij}^{\parallel }\).

To study its ground state phase diagram using the ab initio parameters found in the previous section, we employ a mean-field analysis of competing for orbital orderings with ferromagnetic and antiferromagnetic spin order. Note, that the simplifying assumption of vanishing temperature – a standard one in condensed matter research – still allows to draw conclusions for the low-temperature physics accessible in experiments as fingerprints of the phases we discuss extend into this regime as well. To this end, we note that on the bipartite honeycomb lattice the SU(2) invariant spin sector would, on its own, order either ferro- or antiferromagnetically, depending on the sign of the exchange couplings. As an Ansatz, we therefore assume that one of the respective states is stabilized and decouple the spin from the orbital degrees of freedom by replacing SiSj with its expectation value 〈SiSj〉 = ±1/4 such that \({\xi }_{ij}^{1}=1,{\xi }_{ij}^{0}=0\) for ferromagnetic spin order and \({\xi }_{ij}^{1}={\xi }_{ij}^{0}=1/2\) for Neél order.

After such a mean-field decoupling corresponding to the ground state in the spin sector, we analyze the ground states of the resulting Hamiltonian for the orbital degrees of freedom, which we approximate as classical vectors. We use an iterative energy minimization combined with simulated annealing techniques (see Methods) to converge the mean-field equations and find the phase diagram summarized in Fig. 4. Panel (a) shows the energy of ferromagnetic and antiferromagnetic spin configurations from which the magnetic phase diagram can be read off. This is given in the upper part of the plot and we find antiferromagnetic ordering with an intermittent ferromagnetic phase at intermediate ratios of 0.1 < J/U < 1/3. In the lower part of the plot, we show the corresponding subsidiary orbital order. From our simulations, we identify three different configurations of orbital vectors τ, which can be classified according to their projection on a single definite plane in space, shown in the lower left of the plots: (1) ferro-orbital (FO) nematic order5,6,60,61,62, where the vectors on all lattice sites align in parallel to the xz-plane. Quantum mechanically, finite values of \(\langle {\tau }_{i}^{x/z}\rangle\) indicate an imbalance of the occupation of px and py orbitals, breaking rotation symmetry and thereby motivating the notion of a nematic state. (2) AFO nematic order; each vector is aligned anti-parallel with its nearest neighbors corresponding to \(\langle {\tau }_{i}^{x/z}\rangle \,\ne\, 0\) on each sublattice, but without finite projections \({\tau }_{i}^{y}\) on individual sites. (3) FO magnetic order; all vectors order along the y-axis, such that \(\langle {\tau }_{i}^{y}\rangle \,\ne\, 0\), which, in the quantum mechanical system, would indicate time-reversal symmetry breaking. The inclusion of quantum fluctuations can change this picture and more exotic ground states may emerge. For example, for our ab initio band structure parameters, a noncollinear spin dimer phase is predicted in a certain range of interaction couplings and even a quantum spin-orbital liquid is found in its proximity53. Since these exotic phases primarily occur for weak Hund’s coupling and strong orbital anisotropies, the assumptions made for our calculations can therefore be justified for sizable JH and modest distances to the isotropic tσ = tπ point.

Fig. 4: Magnetic phase diagram for twisted bilayer MoS2.
figure 4

a Classical ground state energy per orbital in units of \({{{{{{{\rm{{{\Delta }}}}}}}}}}={t}_{\sigma }^{2}/U\), assuming ferro- (blue) or antiferromagnetic (red) order for the spin degrees of freedom. We take the ab initio parameters, tπ = 0.25tσ and use an iterative energy minimization. The lower panel determines the phase boundaries for the orbital degrees of freedom given the energetically more favorable spin order shown in the top panel. At J/U = 0.1 we find the spin order to change from AFM to FM, with AFO nematic order for the orbital degrees of freedom remaining stable in agreement with ref. 53. b Configurations of orbital vectors are found at the end of iterative minimization. Note that we display the projection of τ to the plane in \({{\mathbb{R}}}^{3}\) (indicated by the axis shown in the bottom left), such that nematic states with finite contributions only in xz direction ((1) & (2)) can be distinguished from magnetic states (3) which point perpendicular, i.e along the y-axis.

Full size image

Discussion

We have established that twisted bilayer MoS2 is a promising platform to realize the orbital anisotropic px–py Hubbard model by employing large-scale ab initio calculations. We find that families of flat bands emerge where the first family of flat bands shows s-orbital character and the second family is an intriguing realization of a strongly asymmetric px–py Hubbard model both on a honeycomb lattice, adding a lattice with nontrivial almost perfectly-flat bands due to destructive interference to the growing list of systems that can be engineered in twisted heterostructures. The symmetry of these flat bands is inherited from the hexagonal lattice formed by the equivalent BMo/S and BS/Mo regions. At an even smaller angle, the sequence in the family of flat bands found with respect to their orbital character continues. Our analysis shows that the low-energy DFT band structures in this system can be well captured by a free electron gas model modulated by a simple harmonic potential that has hexagonal (D6) symmetry, which is consistent with a recent study63. This simple model further shows that the next family would exhibit a d-orbital character on the honeycomb lattice. Such a lattice would effectively realize a multi-orbital generalization of a Kagome lattice – a prototypical model for quantum spin liquids. However, at such small angles strong relaxation is likely to become dominant, prohibiting access to this regime and potentially spoiling its experimental realization. Currently, the ab initio characterization of such small angles is numerically too exhaustive and this work sparks a direct need for novel computational methods to tackle this question.

Furthermore, our combined exact diagonalization and strong-coupling expansion approaches classify the magnetic and orbital phase diagrams, however, the inclusion of quantum fluctuations stipulates an intriguing avenue of future theoretical research.

Indeed, previous theoretical works provide some evidence for a quantum spin liquid in the SU(4)-symmetric Kugel-Khomskii model on the honeycomb lattice64, the square lattice as a related system without frustration65 and studied the role of perturbations that break SU(4) symmetry and isotropy53 In twisted MoS2, this regime would in fact map to larger twist angles, where the anisotropy of the px-py model is less pronounced, as well as to a regime of vanishing Hund’s coupling, placing such a putative quantum spin liquid at the transition between FO nematic and AFO nematic phases.

In addition, by proximity or variations in the chemical composition of the twisted bilayer, it might be possible to induce spin-orbit coupling splitting of the ultra-flat bands at the top and bottom of the asymmetric px–py dispersion. Such a bandgap opening would induce interesting topological properties66 in a highly tunable materials setting.

Methods

Details on ab initio calculations

We calculate the electronic properties of twisted bilayer MoS2 with ab initio methods based on density functional theory (DFT) as implemented in the Vienna ab initio Simulation Package (VASP)67. We employ plane-wave basis sets with an energy cutoff of 550 eV and pseudopotentials as constructed with the projector augmented wave (PAW) method68. The exchange-correlation functionals are treated at the generalized gradient approximations (GGA) level69. The supercell lattice constants are chosen such that they correspond to 3.161 Å for the 1 × 1 primitive cell of MoS2. Vacuum spacing larger than 15 Å is introduced to avoid artificial interaction between the periodic images along the z-direction. Because of the large supercells, a 1 × 1 × 1 k-grid is employed for the ground state and the relaxation calculations. For all the calculations, all the atoms are relaxed until the force on each atom is less than 0.01 eV/Å. Van der Waals corrections are considered with the method of Tkatchenko and Scheffler70. We extract the real and the imaginary parts of the DFT wavefunctions with the VASPKIT code71.

Details on exact diagonalization

Exact diagonalization calculations were performed for the electronic tight-binding model in Eq. (1) with Hubbard-Kanamori interactions defined in Eq. (2). All calculations were performed for a two-orbital eight-site cluster with periodic boundary conditions at quarter filling, corresponding to eight spin-1/2 particles in sixteen orbitals. Rotationally symmetric Kanamori interactions are adopted, with \(U^{\prime} =U-2J\). As the magnitudes of the Hubbard U and Hund’s exchange J interactions cannot be reliably predicted for a Moié supercell from first principles, all presented results are shown as a function of U, J. Calculations of the single-particle Green’s functions and local density of states are performed starting from the ground state in the total momentum Ktot = 0 and total spin Sz = 0 sectors, using the Lanczos method and continued-fraction representation, and a spectral broadening (imaginary part of the self-energy) of η = 0.1 is imposed.

Details on minimization procedure for classical Hamiltonian

Metropolis Monte Carlo simulations are a prime tool for the investigation of classical spin models, since they allow for off-diagonal, spatially anisotropic spin couplings to be included, even when one-spin terms, such as magnetic fields, are involved. Here we employ a special variant of the algorithm to the mean-field version of (3), keeping in mind that the ‘spins’ used in the simulation are approximations to orbital operators τ. First, a lattice site i is randomly chosen, and its respective gradient field hi = ∇iH is computed for the current spin configuration {τi}. Second, a random orientation \({{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i}^{\prime}\) for the vector at site i is proposed and the weight

$$g=\min \left({e}^{-\beta ({{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i}^{\prime}-{{{{{{{{\boldsymbol{\tau }}}}}}}}}_{i}){{{{{{{{\bf{h}}}}}}}}}_{i}},1\right),$$
(4)

is computed for an effective inverse temperature β. Performing several Metropolis updates with increasing values of β we are able to efficiently lower the energy of a random initial configuration, minimizing the odds to converge to a local minimum by only allowing optimal updates (i.e. τi = −hi) right from the start. After Na sweeps over the full lattice, the so-obtained configuration is ameliorated by No optimization sweeps, where the randomly selected spin is rotated anti-parallel to the local gradient field such that the energy is deterministically lowered in every step and we converge as close to the global energy minimum as possible. Hence, this algorithm is reminiscent of Monte Carlo simulations with simulated annealing, but at zero temperature where thermal fluctuations are frozen out.

To benchmark our implementation we have carried out the minimization procedure in the isotropic limit tσ = tπ for Na = No = 105, where the optimization sweeps are terminated when the energy change after one sweep, ϵ, becomes small (usually ϵ ≤ 10−10). Mapping out the phase diagram for both the FM, 〈SiSj〉 = 1/4, as well as the AFM, 〈SiSj〉 = −1/4, spin sector on a lattice with N = 1250 spins subject to periodic boundary conditions we find the result in Fig. 5, which is consistent with the one presented in ref. 52. For J < 0 the AFM spin sector has lower energy, with the orbitals forming a ferro-orbital (FO) nematic state where \(\langle {\tau }_{i}^{x/z}\rangle \,\ne\, 0\) and \(\langle {\tau }_{i}^{y}\rangle =0\). For J > 0 one finds the FM spin sector (for which the orbital degrees of freedom restore their rotation invariance) to dominate as long as J < 1/3, where the AFM sector takes over again and establishes a FO magnetic state, i.e. \(\langle {\tau }_{i}^{x/z}\rangle =0\) and \(\langle {\tau }_{i}^{y}\rangle \,\ne\, 0\).

Fig. 5: Magnetic phase diagram of the strong-coupling Hamiltonian (3) in the isotropic limit tσ = tπ.
figure 5

Our results from iterative minimization are in agreement with ref. 52, stabilizing FO nematic order (1) for J/U < 0 (see main text) and FO magnetic order (3) (see main text) for J/U > 1/3. In the intermediate range of parameters 0 < J/U < 1/3 the ferromagnetic spin sector is selected, such that, due to vanishing \({\xi }_{ij}^{0}\), rotation invariance is restored for the orbital vectors, giving rise to AFM order (2) but without any preferred axis in euclidean space.

Full size image