Magic in twisted transition metal dichalcogenide bilayers

Abstract

The long-wavelength moiré superlattices in twisted 2D structures have emerged as a highly tunable platform for strongly correlated electron physics. We study the moiré bands in twisted transition metal dichalcogenide homobilayers, focusing on WSe₂, at small twist angles using a combination of first principles density functional theory, continuum modeling, and Hartree-Fock approximation. We reveal the rich physics at small twist angles θ < 4^∘, and identify a particular magic angle at which the top valence moiré band achieves almost perfect flatness. In the vicinity of this magic angle, we predict the realization of a generalized Kane-Mele model with a topological flat band, interaction-driven Haldane insulator, and Mott insulators at the filling of one hole per moiré unit cell. The combination of flat dispersion and uniformity of Berry curvature near the magic angle holds promise for realizing fractional quantum anomalous Hall effect at fractional filling. We also identify twist angles favorable for quantum spin Hall insulators and interaction-induced quantum anomalous Hall insulators at other integer fillings.

Introduction

In condensed matter physics, simple and elegant models have often brought new ideas and started new paradigms. Celebrated examples include the Hubbard model for strongly correlated electron system¹, the Tomonaga-Luttinger model for one-dimensional electron liquid^2,3, and the Kitaev model for non-Abelian quantum spin liquid⁴, to name a few. As toy models are designed to illustrate key concepts in the simplest form, they are rarely realized directly in real materials, whose atomic-scale electronic structures are inevitably more complex. The recent advent of long-wavelength moiré superlattices based on 2D van der Waals structures provides a new and promising venue for physical realization and quantum simulation of model Hamiltonians. In magic-angle twisted bilayer graphene⁵ (TBG), experiments have discovered a variety of correlated electron states^{6,7,8,9,10,11,12} facilitated by flat moiré bands.

More recently, moiré superlattices of semiconducting transition metal dichalcogenides (TMD) have attracted interest as a potentially simpler and more robust platform for simulating the Hubbard model on an emergent lattice^{13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32}. Each lattice site represents a low-energy electronic orbital in the moiré unit cell that spreads over many atoms. These semiconductor moiré systems can thus be viewed as artificial 2D solids—a periodic array of “magnified atoms”¹⁹. The atomic potential depth and interatomic bonding are highly tunable by the choice of TMD materials, the twist angle¹³ and the displacement field^20,23. Thus, TMD-based moiré materials provide a favorable platform for simulating idealized models in two dimensions.

In this work, we predict the realization of generalized Kane−Mele models with topological flat band, interaction-driven Haldane insulator and Mott insulators in twisted TMD homobilayers at small twist angles. Contrary to current thoughts, we show by band structure calculation and analytical derivation that a magic twist angle exists in twisted TMD homobilayers, where the topmost valence miniband from the ±K-valleys is almost perfectly flat and well separated from other bands. This band carries a spin/valley Chern number and is well described by a generalized Kane−Mele model³³.

At half filling of this topological flat band, we show that repulsive interactions drives spontaneous spin/valley polarization leading to Haldane’s quantum anomalous Hall insulator³⁴. We further find an out-of-plane displacement field drives a transition from the Haldane insulator into a Mott insulator. Depending on the twist angle, this Mott state is either a spin/valley polarized ferromagnet or features intervalley coherence that spontaneously breaks the spin/valley U(1) symmetry. Thus our work reveals a rich phase diagram of topological, correlated and broken-symmetry insulators enabled by the flat band in TMD homobilayers at small twist angles below the 4^∘–5^∘ range in current experimental studies^23,35.

Due to spin-valley locking³⁶, monolayer TMDs such as WSe₂ and MoTe₂ feature top valence bands with spin-↑ at +K valley and spin-↓ at −K. We study TMD homobilayers with a small twist angle θ starting from AA stacking, where every metal (M) or chalcogen (X) atom on the top layer is aligned with the same type of atom on the bottom layer. In such twisted structure, the K points of the two layers are slightly displaced and form the two corners of the moiré Brillouin zone, denoted as κ_±. A set of spin-↑ (↓) moiré bands is formed from hybridized +K (−K) valley bands of the two layers. The complete filling of a single moiré band including spin degeneracy thus requires 2 holes per moiré unit cell.

Results

In order to obtain accurate moiré band structures, we perform large-scale density functional theory calculations with the SCAN+rVV10 van der Waals density functional³⁷, which captures the intermediate-range vdW interaction through its semilocal exchange term. Focusing on twisted bilayer WSe₂, we find that lattice relaxation has a dramatic effect on moiré bands. Our DFT calculations at θ = 5.08^∘ with 762 atoms per unit cell show a significant variation of the layer distance d in different regions on the moiré superlattice, as shown in Fig. 1b. d = 6.7 Å is smallest in MX and XM stacking regions, where the metal atom on top layer is aligned with chalcogen atom on the bottom layer and vice versa, while d = 7.1 Å is largest in MM region where metal atoms of both layers are aligned. With the fully relaxed structure, the low-energy moiré valence bands of twisted bilayer WSe₂ are found to come from the ±K valley (shown in Fig. 1c), as opposed to the Γ valley in previous computational studies³⁸ and consistent with recent works^35,39,40.

Fig. 1: Comparison with large-scale DFT calculations.

a The κ_± points of the moiré Brillouin zone are formed from the K points of the monolayer Brillouin zones, which are rotated by ±θ/2. b The interlayer distance of the twisted WSe₂ structure obtained from DFT is shown, demonstrating a large variation between the MM and XM/MX regions. c The continuum band structure (blue lines) is plotted in comparison with large-scale DFT calculations (black dots) at twist angle θ = 5.08^∘, showing excellent agreement.

At small twist angles, the large size of moiré unit cell makes it difficult to perform DFT calculations directly on twisted TMD homobilayers. An alternative and complementary approach, introduced by Wu et al.¹⁴, is the continuum model based on an effective mass description, which models the formation of moiré bands using spatially modulated interlayer tunneling Δ_T(r) and layer-dependent potential Δ_1,2(r). The continuum model Hamiltonian for ±K valley bands is given by

$${{{{{{{{\mathcal{H}}}}}}}}}_{\uparrow }=\left(\begin{array}{ll}-\frac{{\hslash }^{2}{({{{{{{{\bf{k}}}}}}}}-{{{{{{{{\boldsymbol{\kappa }}}}}}}}}_{+})}^{2}}{2{m}^{* }}+{{{\Delta }}}_{1}({{{{{{{\bf{r}}}}}}}})&{{{\Delta }}}_{\rm {T}}({{{{{{{\bf{r}}}}}}}})\\ {{{\Delta }}}_{\rm {T}}^{{{{\dagger}}} }({{{{{{{\bf{r}}}}}}}})&-\frac{{\hslash }^{2}{({{{{{{{\bf{k}}}}}}}}-{{{{{{{{\boldsymbol{\kappa }}}}}}}}}_{-})}^{2}}{2{m}^{* }}+{{{\Delta }}}_{2}({{{{{{{\bf{r}}}}}}}})\end{array}\right)$$

(1)

and ${{{{{{{{\mathcal{H}}}}}}}}}_{\downarrow }$ as its time-reversal conjugate.

The continuum model approach is valid at small twist angle where the moiré wavelength is large enough. In this case, the atom configuration within any local region of a twisted bilayer is identical to that of an untwisted bilayer with one layer laterally shifted relative to the other by a corresponding displacement vector d₀. For example, ${{{{{{{{\bf{d}}}}}}}}}_{0}=0,-\left({{{{{{{{\bf{a}}}}}}}}}_{1}+{{{{{{{{\bf{a}}}}}}}}}_{2}\right)/3,\left({{{{{{{{\bf{a}}}}}}}}}_{1}+{{{{{{{{\bf{a}}}}}}}}}_{2}\right)/3$, with a_1,2 the primitive lattice vector of a monolayer, correspond to the MM, MX and XM regions respectively. Therefore, the moiré potentials for twisted TMD bilayers Δ_T(r) and Δ_1,2(r) as a function of coordinate on the moiré superlattice can be determined from the valence band edges of the untwisted bilayer as a function of the corresponding shift vector²⁰. In the lowest harmonic approximation, Δ_T(r) and Δ_1,2(r) are sinusoids that interpolate between MM, MX and XM regions¹⁴:

$${{{\Delta }}}_{1,2}({{{{{{{\bf{r}}}}}}}})=2V\mathop{\sum}\limits_{j=1,3,5}\cos ({{{{{{{{\bf{g}}}}}}}}}_{j}\cdot {{{{{{{\bf{r}}}}}}}}\pm \psi )$$

(2)

$${{{\Delta }}}_{\rm {T}}({{{{{{{\bf{r}}}}}}}})=w(1+{e}^{-i{{{{{{{{\bf{g}}}}}}}}}_{2}\cdot {{{{{{{\bf{r}}}}}}}}}+{e}^{-i{{{{{{{{\bf{g}}}}}}}}}_{3}\cdot {{{{{{{\bf{r}}}}}}}}})$$

(3)

where g_j are (j − 1)π/3 counter-clockwise rotations of the moiré reciprocal lattice vector ${{{{{{{{\bf{g}}}}}}}}}_{1}=(4\pi \theta /\sqrt{3}{a}_{0},0)$, and a₀ is the monolayer lattice constant. Up to an overall energy scale, the continuum model depends only on the dimensionless parameters $\alpha \equiv V{\theta }^{2}/({m}^{* }{a}_{0}^{2})$, w/V and ψ.

From our DFT calculation for untwisted bilayers with relaxed layer distance and using the vacuum level as an absolute reference energy for the band edge, we obtain the continuum model parameters V = 9.0 meV, ψ = 128^∘, and w = 18 meV. Importantly, the interlayer tunneling strength w is twice larger than previously reported¹⁴. To demonstrate the accuracy of the continuum model method, we compare in Fig. 1c the band structures computed by large-scale DFT directly at θ = 5.08^∘ and by the continuum model with the above parameters, finding excellent agreement. Details on the DFT calculation can be found in Supplementary Note 1 (see supplemental material for details on DFT calculations and continuum model).

We remark that different approaches^31,32,40 can lead to different conclusions on topology. Thus, we utilize a method to determine band topology directly from our large-scale DFT band structure based on symmetry eigenvalues. As detailed in Supplementary Note 2 (see supplemental material for details of topology of DFT wavefunctions), we are able to isolate bands from the ±K valley and compute their C_3z eigenvalue at the high symmetry momenta γ, κ_±, which determine their Chern number (mod 3)⁴¹. The C_3z eigenvalues for the first two bands, summarized in Table 1, are consistent with the first two bands having non-trivial valley Chern number ${{{{{{{{\mathcal{C}}}}}}}}}_{K,1}={{{{{{{{\mathcal{C}}}}}}}}}_{K,2}=1$.

Table 1 C_3z eigenvalues of the first two bands from each valley, computed from large-scale DFT wavefunctions at high symmetry momentum points.

Using the new continuum model parameters established above, along with the lattice constant a₀ = 3.317 Å⁴² and the effective mass m^* = 0.43m_e^43,44, we calculate the band structure of twisted bilayer WSe₂, E_i(k), at various twist angles, as shown in Fig. 2a. The bandwidth of the first band, $W=\mathop{\max }\nolimits_{{{{{{{{\bf{k}}}}}}}}}{E}_{1}({{{{{{{\bf{k}}}}}}}})-\mathop{\min }\nolimits_{{{{{{{{\bf{k}}}}}}}}}{E}_{1}({{{{{{{\bf{k}}}}}}}})$, as well as the (direct or indirect) band gaps ε_ij between pairs of bands (i, j) = (1, 2) and (2, 3), ${\varepsilon }_{ij}=\mathop{\min }\nolimits_{{{{{{{{\bf{k}}}}}}}}}{E}_{i}({{{{{{{\bf{k}}}}}}}})-\mathop{\max }\nolimits_{{{{{{{{\bf{k}}}}}}}}}{E}_{j}({{{{{{{\bf{k}}}}}}}})$, is shown in Fig. 3. Focusing on topological features of the first two valence bands, we can divide the moiré band structure into three main regimes divided by θ₁ ≈ 1.5^∘ and θ₂ ≈ 3.3^∘.

Fig. 2: Continuum model band structure and Berry curvature at various twist angles.

a The band structure E_i(k) along with the Chern numbers of the first two bands and (b) the (scaled) Berry curvature $| {\kappa }_{+}{| }^{2}{{{{{{{\mathcal{F}}}}}}}}({{{{{{{\bf{k}}}}}}}})$ of the first band is shown for the continuum model at θ = 1^∘, 1.43^∘, 1.67^∘, and 2.5^∘. At θ = 2.5^∘, the first band maxima is located at the κ_± points and the Berry curvature is peaked at γ. At θ = 1^∘, the band maximum is instead at γ and ${{{{{{{\mathcal{F}}}}}}}}$ is peaked at κ_±. During the crossover region between these angles, E₁(k) and ${{{{{{{\mathcal{F}}}}}}}}({{{{{{{\bf{k}}}}}}}})$ both become very flat. We find that the band dispersion E₁(k) is flattest at θ ≈ 1.43^∘ and the Berry curvature ${{{{{{{\mathcal{F}}}}}}}}({{{{{{{\bf{k}}}}}}}})$ is most uniform at θ ≈ 1.67^∘, both shown.

Fig. 3: The bandwidth of the first band W, and indirect band gap between the first two pairs of bands ε₁₂ and ε₂₃.

The bandwidth is minimized at θ_m, while being well separated from the remaining bands. The Chern numbers of the first two bands, $[{{{{{{{{\mathcal{C}}}}}}}}}_{K,1},{{{{{{{{\mathcal{C}}}}}}}}}_{K,2}]$, is shown before and after the ε₂₃ gap closing at θ = θ₁. For θ ≥ θ₂, ε₁₂ vanishes.

First, for θ < θ₁, the top two bands are well separated from the rest of the spectrum, and carry opposite Chern number $[{{{{{{{{\mathcal{C}}}}}}}}}_{K,1},{{{{{{{{\mathcal{C}}}}}}}}}_{K,2}]=[+1,-1]$. The bandwidth of the first band W < 1 meV remains very small throughout. In this regime of very small twist angles, the character of the top two valence bands can be understood from an effective tight binding model on a moiré honeycomb lattice that takes the form of a Kane−Mele model, as suggested in the insightful work of Wu et al.¹⁴ As we will later show, the original Kane−Mele description with up to second nearest neighbor hopping terms only describes the band structure well for very small angles θ ≲ 1^∘. As θ increases towards θ₁, longer range hopping terms become more important.

At θ = θ₁, the band gap ε₂₃ closes and the Chern number of the top two bands changes to [+1, +1]. In this second regime, θ₁ < θ < θ₂, both top bands have same Chern number [+1, +1] and are still all separated by a sizable gap ε₁₂, ε₂₃ > 0. The bandwidth of the first band increases rapidly with θ, reaching around W ≈ 20 meV at θ₂ (not shown). Finally, in the third regime, θ₂ < θ ≲ 5.4^∘, the indirect gap ε₁₂ vanishes, but the direct gap remains open. The Chern number of the top two bands remains well defined at [+1, +1], but the bands now overlap in energy and are highly dispersive. In both the second and third regimes, ε₂₃ > 0; thus, the top two bands together form a gapped ${{{{{{{\mathcal{C}}}}}}}}=2$ manifold. Beyond θ ≳ 5.4^∘, the gap ε₂₃ also vanishes and the top two bands are no longer isolated (Supplementary Note 3; see supplemental material for derivation of the analytic magic angle). Topology of the continuum model at θ ≈ 5^∘ is consistent with that determined directly from large-scale DFT in Table 1, further strengthening our confidence in the continuum model description even up to larger angles.

For θ < θ₂, especially near θ₂ where the first band is more dispersive, the spin Chern number ${{{{{{{\mathcal{C}}}}}}}}=1$ and ε₁₂ > 0 is favorable for a quantum spin Hall insulator at a filling of n = 2 holes per moiré unit cell. Also, for the wide range of angles θ₁ < θ ≲ 5.4^∘, ε₂₃ > 0 and the top two bands both carry spin Chern number ${{{{{{{\mathcal{C}}}}}}}}=1$, giving rise to a double quantum spin Hall state with two sets of counter-propagating spin-polarized edge modes at n = 4.

We now address the bandwidth W, which shows a sharp minimum at θ = θ_m ≈ 1.43^∘ reminiscent of the magic angle in TBG. To understand this, notice that the top band, shown in Fig. 2a, has two qualitatively different behaviors in the small and large θ limit. For θ ≳ 2.5^∘, E₁(k) reaches its maximum at κ_± and minimum at γ, which can be understood from the weak moiré effects at small α. For small θ ≲ 1^∘, the opposite holds and E₁(k) is maximum is at γ and minimum at κ_±, which can be understood from the effective Kane−Mele model, which we will derive explicitly. At the crossover between these two limits, the band maxima and minima must switch locations in the moiré Brillouin zone, potentially leading to a flat band. As can be clearly seen, W achieves a minimum at a particular magic angle θ_m during this crossover. At θ_m the gap to the next state ε₁₂ ≈ 3.7 meV is much larger than the bandwidth W ≈ 0.1 meV. The band structure at θ_m is shown in Fig. 2a, which shows that the first band is almost completely flat and separated from the next band. For even smaller θ, both ε₁₂ and W vanish, but the ratio W/ε₁₂ diverges. Thus, we may view θ_m as the angle at which the top band is flattest while still being well isolated from the rest of the spectrum.

Analytic progress can be made in estimating the magic angle by considering the dispersion near γ. Assuming that the bandwidth will be minimized near the angle at which E₁(γ) changes from minima to maxima, expanding ${E}_{1}({{{{{{{\boldsymbol{\gamma }}}}}}}}+{{{{{{{\bf{k}}}}}}}})\approx {E}_{1}({{{{{{{\boldsymbol{\gamma }}}}}}}})+\frac{{{{{{{{{\bf{k}}}}}}}}}^{2}}{2{m}_{\gamma }}+{{{{{{{\mathcal{O}}}}}}}}({{{{{{{{\bf{k}}}}}}}}}^{3})$, the effective mass m_γ should diverge near the crossover. Let ${\tilde{\theta }}_{m}$ to be the angle at which ${m}_{\gamma }^{-1}=0$. Then, considering only the six most relevant states at γ, we have (Supplementary Note 4; see supplemental material for continuum model band structure at higher twist angles)

$${\tilde{\theta }}_{m}^{-2}=\frac{8{\pi }^{2}}{9{m}^{* }{a}_{0}^{2}}\left(\frac{1}{{{{{{{{{\mathcal{E}}}}}}}}}_{{n}_{0}}-{{{{{{{{\mathcal{E}}}}}}}}}_{{n}_{0}+1}}+\frac{1}{{{{{{{{{\mathcal{E}}}}}}}}}_{{n}_{0}}-{{{{{{{{\mathcal{E}}}}}}}}}_{{n}_{0}-1}}\right)$$

(4)

where ${{{{{{{{\mathcal{E}}}}}}}}}_{n}=2w\cos (\pi n/3)+2V\cos (2\pi n/3-\psi )$, and n₀ is the integer (mod 6) which maximizes ${{{{{{{{\mathcal{E}}}}}}}}}_{n}$ (n₀ = 1 for WSe₂ parameters). We find that Eq. (4) provides a decent estimate for θ_m in the cases considered. In WSe₂, we have ${\tilde{\theta }}_{m}=1.4{7}^{\circ }$, compared to θ_m = 1.43^∘ at which the bandwidth is minimized.

Next, we turn to the Berry curvature ${{{{{{{\mathcal{F}}}}}}}}({{{{{{{\bf{k}}}}}}}})$ of the top band, shown in Fig. 2b. In all cases, the first band has Chern number ${{{{{{{{\mathcal{C}}}}}}}}}_{K,1}=\frac{1}{2\pi }{\int}_{{{{{{{{\rm{BZ}}}}}}}}}{{{{{{{\mathcal{F}}}}}}}}{\rm {d}}{{{{{{{\bf{k}}}}}}}}=1$; however, the distribution changes very drastically as θ is varied. At θ ≳ 2^∘, ${{{{{{{\mathcal{F}}}}}}}}$ is peaked around the band minimum at γ. At θ ≲ 1^∘, ${{{{{{{\mathcal{F}}}}}}}}$ is instead sharply peaked at the κ_± points. Near the crossover region, the distribution of ${{{{{{{\mathcal{F}}}}}}}}$ shifts from γ to κ_±, and can become very evenly distributed. We find that ${{{{{{{\mathcal{F}}}}}}}}$ is most evenly distributed near θ = 1.67^∘, shown in Fig. 2b, where ${{{{{{{\mathcal{F}}}}}}}}$ becomes almost uniform in the Brillouin zone. The uniform distribution of ${{{{{{{\mathcal{F}}}}}}}}$ is reminiscent to that of Landau levels. Time-reversal symmetry forces the corresponding spin-↓ bands from the −K valley to have opposite Chern number.

We emphasize that the physics of the magic angle arises due to the crossover between two qualitatively different behaviors of the first band at low and high angles. Additional factors unaccounted for by the continuum model may result in, for example, angle-dependent model parameters. However, as long as the qualitative behaviors at small and large angles are unchanged, there will still be crossover regime at which the band becomes flat. Even when the bands are not perfectly flat, a diverging mass can still give rise to a diverging higher-order van Hove singularity²³.

Recall that for θ < θ₁, the top two bands carry opposite Chern number and are separated from the rest of the spectrum, suggesting a description in terms of an effective tight binding model. We now focus on θ < θ₁ and directly derive an effective tight binding model for the first two moiré bands by explicitly constructing a basis of localized Wannier states. These Wannier states are constructed via a simple procedure which manifestly preserves the symmetries of the twisted homobilayer. Given the single-particle eigenstates $\{|{\phi }_{n,{{{{{{{\bf{k}}}}}}}}}\rangle \}$ of the continuum Hamiltonian (1) for each k in the mBZ, we first construct a superposition of the first two (n = 1, 2) eigenstates, $|{\tilde{\phi }}_{n,{{{{{{{\bf{k}}}}}}}}}\rangle ={\sum }_{m = 1,2}{U}_{nm}^{({{{{{{{\bf{k}}}}}}}})}\left|{\phi }_{m{{{{{{{\bf{k}}}}}}}}}\right\rangle$ using a 2 × 2 unitary matrix ${U}_{nm}^{({{{{{{{\bf{k}}}}}}}})}$, which maximizes the layer polarization at every k:

$${P}_{{{{{{{{\bf{k}}}}}}}}}=\mathop{\sum}\limits_{n=1,2}{(-1)}^{n}\left\langle {\tilde{\phi }}_{n,{{{{{{{\bf{k}}}}}}}}}| ({{{{{{{{\mathcal{P}}}}}}}}}_{-}-{{{{{{{{\mathcal{P}}}}}}}}}_{+})| {\tilde{\phi }}_{n,{{{{{{{\bf{k}}}}}}}}}\right\rangle ,$$

(5)

where ${{{{{{{{\mathcal{P}}}}}}}}}_{\pm }$ is the projector on to the top/bottom layer, so that $|{\tilde{\phi }}_{1,{{{{{{{\bf{k}}}}}}}}}\rangle$ is chosen to mostly consist of states in the top layer, and similarly for $|{\tilde{\phi }}_{2,{{{{{{{\bf{k}}}}}}}}}\rangle$ on the bottom layer. This uniquely specifies $|{\tilde{\phi }}_{n,{{{{{{{\bf{k}}}}}}}}}\rangle$ up to a phase, which we choose to be real and positive at the XM (n = 1) or MX (n = 2) stacking regions (Supplementary Note 5; see supplemental material for details on the derivation of localized Wannier functions and tight binding model). The Wannier states at moiré lattice vector R is then defined $\left|{W}_{{{{{{{{\bf{R}}}}}}}}}^{n}\right\rangle =\frac{1}{\sqrt{{N}_{k}}}{\sum }_{{{{{{{{\bf{k}}}}}}}}}{e}^{-i{{{{{{{\bf{k}}}}}}}}\cdot {{{{{{{\bf{R}}}}}}}}}\left|{\tilde{\phi }}_{n{{{{{{{\bf{k}}}}}}}}}\right\rangle$. They are localized about their centers with a root-mean-square distance a_W ≈ 5 nm, and are also mostly composed of states in one layer: $\left\langle {W}_{{{{{{{{\bf{R}}}}}}}}}^{1}| {{{{{{{{\mathcal{P}}}}}}}}}_{+}| {W}_{{{{{{{{\bf{R}}}}}}}}}^{1}\right\rangle \approx 0.83$ is mostly in the top layer, and vice versa for $\left|{W}_{{{{{{{{\bf{R}}}}}}}}}^{2}\right\rangle$.

It is straightforward to obtain the hopping matrix elements of the effective tight binding model in the Wannier basis for the top two bands as a function of θ. Figure 4b shows the nth nearest neighbor hopping matrix elements t_n obtained in this way, up to n = 5. As anticipated, the effective tight binding model at θ < θ₁, including the spin/valley degrees of freedom, is a generalized Kane−Mele model with sites centered on the honeycomb lattice formed by MX and XM stacking regions¹⁴.

Fig. 4: Wannier functions and tight binding model parameters.

a Wannier functions at the magic angle, b tight binding parameters as a function of θ, and c the bandwidth of the top band in the effective tight binding model TB_n keeping up to nth nearest neighbor hopping terms, compared to that of the continuum model.

The tight binding Hamiltonian is found to be

$${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{TB}}}}}}}}}={t}_{1}\mathop{\sum}\limits_{\langle i,j\rangle ,\sigma }{c}_{i\sigma }^{{{{\dagger}}} }{c}_{j\sigma }+| {t}_{2}| \mathop{\sum}\limits_{\langle \langle i,j\rangle \rangle ,\sigma }{e}^{i\phi \sigma {\nu }_{ij}}{c}_{i\sigma }^{{{{\dagger}}} }{c}_{j\sigma }+\cdots$$

(6)

where ${c}_{i\sigma }^{{{{\dagger}}} },{c}_{i\sigma }$ are fermionic creation/annihilation operators, σ = ± is the spin/valley degree of freedom, the sum 〈i, j〉 (〈〈i, j〉〉) is over (next) nearest neighboring sites i, j of the honeycomb lattice, and ν_ij = ±1 depending on whether the path i → j turns right (+) or left (−). The parameter t₁ is real, while t₂ ≡ ∣t₂∣e^iϕ is complex, and ⋯ contain longer range hopping terms. We find that ∣t_n∣ quickly reduce in magnitude with hopping distance n, and only the second neighbor hopping has a significant imaginary component. In Fig. 4c, we show the bandwidth of the top band, W, in the effective tight binding model TB_n including up to t_n hopping terms, compared to that of the continuum model. For θ ≲ 1^∘, TB₂ already captures the band structure very well. Near the magic angle, higher range hoppings become more important in capturing the flatness of the band.

For the small twist angles θ < θ₁ considered, since the size of the Wannier orbitals are small compared to the moiré unit cell, the dominant interaction is a simple on-site Hubbard term ${{{{{{{{\mathcal{H}}}}}}}}}_{U}=U{\sum }_{i}{n}_{i\uparrow }{n}_{i\downarrow }$. We also estimate U ~ e²/(ϵa_W) ≈ 70 meV at θ_m, using a realistic relative dielectric constant ϵ = 4 and a_W =  5nm, which is significantly larger than the tight binding parameters t_n. Therefore at such small twist angle, twisted WSe₂ homobilayers are in the strong-coupling regime, in contrast with θ ~ 4^∘–5^∘ where the bandwidth is comparable to the interaction strength^23,35.

In the following, we shall focus on the strongly correlated regime θ < θ₁ at a filling of n = 1 holes per moiré unit cell, where we expect the flat bands will favor the quantum anomalous Hall (QAH) insulator due to spontaneous spin/valley polarization. The reason is as follows. First, spin/valley polarized states filling the top band of ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{TB}}}}}}}}}$ with σ = ± are exact eigenstates of our interacting model, because the spin-orbit coupling in ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{TB}}}}}}}}}$ conserves the z-spin/valley component. Then, these spin/valley polarized states avoid the Hubbard interaction, and also minimize the kinetic energy in the case of a completely flat top band. Minimizing both parts of the Hamiltonian, they necessarily are many body ground states of the model at n = 1 filling. The complete polarization of ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{TB}}}}}}}}}$ exactly corresponds to Haldane’s model for QAH insulator.

To include corrections coming from the finite bandwidth W of the flat band, we solve our interacting problem within the Hartree−Fock (HF) approximation, where the Hubbard interaction is decoupled as ${n}_{i\uparrow }{n}_{i\downarrow }\simeq \langle {n}_{i\uparrow }\rangle {n}_{i\downarrow }+\langle {n}_{i\downarrow }\rangle {n}_{i\uparrow }-\langle {c}_{i\uparrow }^{{{{\dagger}}} }{c}_{i\downarrow }\rangle {c}_{i\downarrow }^{{{{\dagger}}} }{c}_{i\uparrow }-\langle {c}_{i\downarrow }^{{{{\dagger}}} }{c}_{i\uparrow }\rangle {c}_{i\uparrow }^{{{{\dagger}}} }{c}_{i\downarrow }$, up to a constant term, and where the expectation values for the spin and density at each site are determined self-consistently by iteration. Our numerical solutions of the HF equations are shown in Fig. 5a as a function of twist angle and interaction strength. As expected, we observe a transition from a metallic state to a ferromagnetic QAH insulator polarized along z when U increases. Within HF, this transition can be understood as follows. The fully polarized states yield a rigid shift of the bands by σU/2. When U is larger than the non-interacting bandwidth W, a full gap opens and the ferromagnetic state fully fills one of Chern bands of ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{TB}}}}}}}}}$, which leads to a QAH phase. We remark that the appearance of the QAH phase relies on both the non-trivial Chern number as well as the fact that the band is flat and isolated, features which are maximized at the magic angle, as illustrated by a dip of the insulating phase above θ ≃ 1.4^∘.

Fig. 5: Numerical solution of the self-consistent Hartree−Fock approximation.

The phase diagram is shown as a function of the twist angle and (a) interaction strength at fixed displacement field Δ = 0, or b displacement field at fixed interaction strength U = 10∣t₁∣, including up to fifth nearest neighbor hopping terms. The insulating phases denoted by QAH, FM_z, 120^∘ AFM_xy, and FM_xy are described in the main text. The dashed blue line shows the boundary of the FM_z phases determined by the first magnon instability. The colors indicate the charge gap. Hartree−Fock calculations were done with a $\sqrt{3}\times \sqrt{3}$ unit cell of 6 atom sites, and a 30 × 30 grid of (k) points.

To precisely locate the transition between QAH insulator and the metallic phase, we compute the magnon excitation spectrum above the fully ferromagnetic state by exact diagonalization (ED) of the interacting Hamiltonian projected on the spin-1 excitation subspace⁴⁵. For large U, this spectrum is gapped and the QAH is robust against spin flips. Decreasing U eventually brings one magnon at zero energy, which destabilizes the ferromagnetic states and drives the transition to a metal. As shown in Fig. 5a, the ED results almost perfectly agrees with the HF boundaries, putting them on firmer grounds.

For the large values of U relevant to WSe₂, the magnons have a large gap, and the lowest excitation corresponds to an interband transition between two bands with same spin. The QAH phase is thus protected by a gap ε₁₂ ≈ 3.7 meV near the magic angle, leading to quantum Hall effect at elevated temperature.

We also highlight that the QAH may also be observed for larger twist angles, where the first band still carries a non-zero Chern number (Fig. 2), and its bandwidth remain small compared to the estimated U (Fig. 3). Likewise, the second band is topological and quite flat for θ ~ 2^∘–3^∘ (Supplementary Note 3), and therefore QAH may also be observed at a filling of n = 3. Twisted TMD bilayers with ±K-valley bands are thus expected to be an intrinsically robust platform for interaction-induced QAH phases.

It is interesting and worthwhile to compare the QAH phase in twisted TMD and graphene bilayers. Anomalous Hall effect and its quantization have been experimentally observed in magic-angle TBG^11,12, where the alignment with hBN substrate is likely the origin of valley Chern number⁴⁶ and both spin and valley degeneracy are lifted due to repulsive interaction in the flat band^{47,48,49,50,51}. Due to the presence of SU(2)-invariant spin degrees of freedom, QAH in TBG is subject to the adverse effect of gapless thermal fluctuation, which forbids long-range order at finite temperature in the thermodynamic limit. In contrast, spin-valley locking in TMD systems enables robust Ising-type spin/valley order that leads to QAH effect at lower temperature.

Another great advantage of twisted TMD bilayer is their high degree of tunability, in particular with respect to applied electric fields^19,35,52. Due to the layer polarization of the Wannier basis states, the displacement field can be modeled as a sublattice symmetry breaking term ${{{{{{{{\mathcal{H}}}}}}}}}_{{{\Delta }}}=\frac{{{\Delta }}}{2}{\sum }_{i}{s}_{i}{c}_{i\sigma }^{{{{\dagger}}} }{c}_{i\sigma }$, where s_i is (−)1 for i in the A (B) sublattice. Including this term in our HF treatment, we can investigate which phases should neighbor the QAH ferromagnet in experiments. We present our solutions of the HF equations as a function of twist angle and displacement field in Fig. 5b. There, we fix U = 10∣t₁∣, a tradeoff between the large U of the homobilayer system and the convergence rate of the HF self-consistent iteration algorithm. We find it necessary to consider an enlarged $\sqrt{3}\times \sqrt{3}$ unit cell in order to describe all ordered phases of the model.

At small displacement fields, the topmost moiré band remain relatively flat and our earlier arguments for spin/valley polarization still apply. This is confirmed by our HF solutions for Δ ≲ 2t₁ (Fig. 5b), which exhibit full spin polarization along the z axis. In this region, a transition nevertheless occurs at ${{\Delta }}=6\sqrt{3}| {t}_{2}| \sin \phi$ (up to t_n≥3 terms), where the single-particle gap between the two moiré bands closes, and their Chern numbers change from [+1, −1] to [0, 0]. This gap closing line marks the transition between a QAH insulator and a topologically trivial ferromagnet with spin/valley polarization (FM_z)³⁴. As the displacement field further increases, the bandwidth W also grows, which decreases the magnon gap (see discussion above). The spin/valley polarized phases eventually become unstable when the magnon gap closes, which can be seen with the very good agreement between the phase boundaries determined with HF and ED (Fig. 5b).

Beyond this spin-wave instability line, our results show the emergence of two new Mott insulating phases, where holes are mostly localized on the A sublattice, and their spin either form an antiferromagnetic pattern (120^∘ AFM_xy), or ferromagnetically align in the xy plane (FM_xy). Their appearance is most easily understood for large displacement fields, where the physics becomes analogous to that of localized moments on the triangular A sublattice. In the regime t₂ ≲ t₁ ≪ Δ, U relevant for our system, their coupling is described by an effective XXZ model with Dzyaloshinskii−Moriya (DM) interactions

$${{{{{{{{\mathcal{H}}}}}}}}}_{S}=\,\mathop{\sum}\limits_{{\langle i,j\rangle }_{B}}\,{J}_{\parallel }{s}_{i}^{z}{s}_{j}^{z}+{J}_{\perp }({s}_{i}^{x}{s}_{j}^{x}+{s}_{i}^{y}{s}_{j}^{y})+D\left[({{{{{{{{\bf{s}}}}}}}}}_{i}\times {{{{{{{{\bf{s}}}}}}}}}_{j})\cdot {{{{{{{\bf{z}}}}}}}}\right],$$

(7)

which is derived in Supplementary Note 6 (see supplemental material for the derivation of the effective spin Hamiltonian). The parameters of this effective spin model are given by

$${J}_{\parallel }=\frac{4| \tilde{t}{| }^{2}}{U}+{{{{{{{\rm{Re}}}}}}}}\left(\frac{4{t}_{1}^{2}{t}_{2}}{{{{\Delta }}}^{2}}\right),$$

(8a)

$${J}_{\perp }={{{{{{{\rm{Re}}}}}}}}\left(\frac{4{\tilde{t}}^{2}}{U}+\frac{4{t}_{1}^{2}{t}_{2}}{{{{\Delta }}}^{2}}\right),$$

(8b)

$$D={{{{{{{\rm{Im}}}}}}}}\left(\frac{4{\tilde{t}}^{2}}{U}+\frac{4{t}_{1}^{2}{t}_{2}}{{{{\Delta }}}^{2}}\right),$$

(8c)

with $\tilde{t}={t}_{2}+{t}_{1}^{2}/{{\Delta }}$. In Eq. 8, we have separated exchange terms coming from different physical processes. The first ones $\propto {\tilde{t}}^{2}/U$ arise from nearest neighbor tunneling on the triangular A sublattice, while the others $\propto {t}_{2}{({t}_{1}/{{\Delta }})}^{2}$ originate from loop-exchange on the honeycomb lattice that do not involve any double occupancy.

For twist angles θ ≲ 1^∘, t₂ ≪ t₁ and Eq. (7) reduces to an antiferromagnetic (AFM) Heisenberg model, where J_∥ = J_⊥ > 0 are dominated by the nearest neighbor tunneling on the triangular lattice. This simplified triangular lattice description, valid for very small twist angles, has been proposed in earlier studies of Mott insulators in twisted TMDs^24,26,53. It was shown to yield an antiferromagnetic phase that the small residual DM interaction pins in the xy plane. This is the origin of the AFM_xy phase observed in Fig. 5b. We also note that the weak-coupling version of AFM_xy phase—an intervalley-coherent $\sqrt{3}\times \sqrt{3}$ density wave²³—has been proposed for the correlated insulating state at n = 1 in twisted bilayer WSe₂ at θ ~ 4^∘–5^∘35.

For larger twist angles, t₂ becomes substantial and we observe that J_⊥ becomes negative for the realistic parameter U ≫ Δ, dominated by a third-order exchange process $\propto {t}_{1}^{2}{t}_{2}$ on the honeycomb lattice without double occupancy. Then, the FM_xy phase is favored as shown in Fig. 5b. The competition between AFM_xy and FM_xy phases can be analyzed by solving Eq. (7) for classical spins. This approach, detailed in the Supplementary Note 6 (see supplemental material for the derivation of the effective spin Hamiltonian), gives a transition between the two phases when $| D| =-\sqrt{3}{J}_{\perp }$. For Δ = 5t₁, this criterion yields a critical twist angle θ = 0.95^∘, which roughly agrees with our HF results. We note that the ferromagnetic phase due to J_⊥ < 0 does not appear in twisted TMDs based on simplified triangular lattice descriptions.

Finally, we comment on the effect of nearest neighbor repulsion V∑_〈i, j〉n_in_j to our HF phase diagram. This term favors the layer polarized phases, such as the FM_xy and AFM_xy which appear at large ∣Δ∣. Small V therefore narrows the range in Δ at which the QAH phase appears. For large V, there is a sharp transition at Δ = 0 between layer polarized Mott insulating phases, which can lead to the strong hysteretic behavior of Mott ferroelectricity^20,54. The long-range component of interactions can be controlled by screening from nearby metallic layers. Multiple recent experiments^55,56 on twisted WSe₂ homobilayers in the presence of a nearby WSe₂ monolayer report strong screening effects when the monolayer is doped. This raises the interesting possibility of a screening-induced transition between the QAH and Mott insulating phases.

Discussion

Our phase diagram demonstrates the high experimental tunability of TMD twisted homobilayers, where the applied displacement field can tune between quantum anomalous Hall phase and Mott insulators involving three types of magnetic orders: FM_z, FM_xy, and AFM_xy. Despite being electrically insulating, the xy-ordered Mott insulators support coherent magnon transport²³, which can be detected by optical spin injection and spatial-temporal mapping recently developed for TMD bilayers⁵⁷. The experimental feasibility of tuning and distinguishing between topologically different insulators at the same filling adds to the attractiveness and desirability of TMD-based moiré systems.

In parallel to our work on twisted TMD homobilayers, a breakthrough experiment led by Kin Fai Mak and Jie Shan discovered unexpectedly a QAH phase with spontaneous spin/valley polarization in a TMD heterobilayer MoTe₂/WSe₂ at the filling of n = 1 tuned by displacement field^58,59. Large-scale DFT calculation and wavefunction analysis reveal two dispersive moiré bands forming the Kane−Mele model, suggestive of a similar origin of QAH as described here.

Looking forward, the remarkable flat Chern band we found, combined with the uniformity of Berry curvature, suggests that twisted TMD homobilayers near magic angle may be an ideal setting for observing a fractional quantum anomalous Hall state at zero magnetic field.