Introduction

Recent research interests have focused on the moiré superlattice in 2D Van der Waals heterostructures, including graphene1,2,3,4,5,6,7,8 and transition metal dichalcogenide (TMD) multilayers9,10,11,12,13,14,15,16,17, due to the strong correlation effect in the presence of flat bands. The flat bands formed by low-energy gapless Dirac fermions in magic angle twisted bilayer graphene typically have a bandwidth ~ 5 meV, much smaller than the band gap 25 ~ 35 meV that separates flat bands from higher energy bands and the Coulomb interaction of order 30 meV2,3. In contrast, the flat bands in TMD moiré heterostructures are formed by electrons with parabolic dispersion and have a typical bandwidth ~ 10 meV, separated by a comparable gap from other energy bands, and a huge on-site Coulomb interaction ~ 100 meV10,11,18. Besides the above materials, moiré superlattice has also been found in another family of van der Waals heterostructures consisting of topological insulators (TIs)19,20,21,22,23,24,25,26,27,28. These TI-based moiré heterostructures show different features. TIs have the anomalous gapless surface bands that connect the bulk conduction and valence bands due to non-trivial bulk topology. The spin splitting of surface bands has a typical energy scale of hundreds meV due to the strong spin-orbit coupling (SOC). Previous studies29,30,31,32 show that a single surface state remains gapless upon the moiré superlattice potential, leading to satellite Dirac cones and van Hove singularities, instead of isolated flat bands. Furthermore, the moiré superlattice in magnetic TI materials, e.g. MnBi2Te4, was predicted to host Chern insulator phase33.

In this work, we studied a model of the TI thin film (e.g. (Bi,Sb)2Te3 film) with the moiré superlattice potential (See Fig. 1). Different from a bulk TI, a strong hybridization between two surface states is expected for the TI thin film. The hybridization between two surface states can create isolated minibands that possess non-trivial \({{\mathbb{Z}}}_{2}\) topological invariant, denoted by ν below, in the low-energy moiré spectrum in a wide parameter space, particularly when the moiré potential approximately has six-fold rotation symmetry. In the presence of inversion symmetry, an emergent chiral symmetry in the low energy sector of surface states gives rise to νCB1 + νV B1 = 1 for the lowest Kramers’ pair of conduction minibands, denoted as CB1, and the highest Kramers’ pair of valence minibands, denoted as VB1, in Fig. 1d. We find νCB1 = 1, νV B1 = 0 (νCB1 = 0, νV B1 = 1) when the minima (maxima) of the moiré potential approximately form a hexagonal lattice. In the case of non-trivial CB1 (νCB1 = 1, νV B1 = 0), the lowest two Kramers’ pairs of conduction minibands (CB1 and CB2 in Fig. 1d) together can be adiabatically connected to the Kane-Mele model34 when increasing quadratic terms, and thus CB2 is also topologically non-trivial, νCB2 = 1. An asymmetric potential between two surface states can be generated by external gate voltages to break inversion but preserve six-fold rotation and generally induce the gap closing between different conduction minibands, leading to nodal phases. In the parameter regions where the conduction minibands are gapped from other minibands (parameter regions I, II, III in Fig. 2c), the CB1 is always topologically non-trivial, νCB1 = 1. We further study the influence of the Coulomb interaction via Hartree-Fock mean field theory when the CB1 carries νCB1 = 1 and is half filled, and find that the quantum anomalous Hall (QAH) state competes with a trivial insulator state in region I of Fig. 2c and it can be robustly energetically favored by the asymmetric potential in region II. Finally, we propose a possible experimental realization of the TI-based moiré heterostructure consisting of a monolayer Sb2 layer on top of Sb2Te3 thin films based on the first principles calculations.

Fig. 1: TI-based moiré heterostructures and moiré minibands.
figure 1

a A schematic figure for the twisted 2D materials (black) on top of a topological insulator thin film (cyan) with an angle θ. b Schematic illustration of the moiré potentials from twisted 2D materials on the top and bottom surface of a TI thin film. The blue Dirac cones represent the top and bottom surface states coupled by m. An out-of-plane external electrical field Ee creates the potential V0. c The moiré potential Δ(r) with ϕ = 0. \({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}},\, {{{{{{{{\bf{a}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}}\) are primitive vectors for a moiré unit cell. 1a, 1b, 1c are Wyckoff positions under the point group C3v. d Schematic view of the spectrum. The orange (blue) lines are top (bottom) surface Dirac cones at \({{\Gamma }}\). Inset is the moiré BZ with the first shell moiré reciprocal lattice vectors.

Full size image
Fig. 2: Topological moiré minibands and phase diagram.
figure 2

a, b The topological phase diagrams of the lowest conduction bands CB1 for different moiré potentials with V0/E0 = 0 for (a) and V0/E0 = 1.2 for (b). The three phases for CB1 are topological insulator (TI) phase with νCB1 = 1, normal insulator (NI) phase with νCB1 = 0 and semi-metal (SM) phase with CB1 connected to higher energy bands. The 1a, 1b, 1c are the Wyckoff positions of corresponding atomic orbitals for CB1 in NI phases. c The phase diagram for different uniform asymmetrical potentials with ϕ = 0. Regions I, II and III are three parameter regimes with νCB1 = 1 for CB1. d, e Examples of spectra with nontrivial CB1 in the regions I and II, respectively. The spectrum in (d) has both TR and inversion, and is thus doubly degenerate. The inset of (e) is the Wannier center flow for CB1.

Full size image

Results

Model Hamiltonian

We show a schematic of a heterostructure consisting of TI thin films and another 2D material (e.g. 2D Sb thin films) in Fig. 1a, b, and the moiré potential induced by the 2D material can affect both the top and bottom surface states with different strength. We assume the Fermi energy is within the bulk gap of the TI thin film, and thus model this system with the Hamiltonian

$${H}_{0}({{{{{{{\bf{r}}}}}}}})= {H}^{{{{{{{{\rm{TI}}}}}}}}}+{H}^{{{{{{{{\rm{M}}}}}}}}}({{{{{{{\bf{r}}}}}}}}),\\ {H}^{{{{{{{{\rm{TI}}}}}}}}}= v{\tau }_{z}(-i{\partial }_{y}{s}_{x}+i{\partial }_{x}{s}_{y})+m{\tau }_{x}{s}_{0},\\ {H}^{{{{{{{{\rm{M}}}}}}}}}({{{{{{{\bf{r}}}}}}}})= \frac{1+\alpha }{2}{{\Delta }}({{{{{{{\bf{r}}}}}}}}){\tau }_{0}{s}_{0}+\frac{1-\alpha }{2}{{\Delta }}({{{{{{{\bf{r}}}}}}}}){\tau }_{z}{s}_{0}+{V}_{0}{\tau }_{z}{s}_{0}.$$
(1)

HTI denotes two surface states of a TI thin film with the inter-surface hybridization \(m={m}_{0}+{m}_{2}(-{\partial }_{x}^{2}-{\partial }_{y}^{2})\), and \({h}_{D}^{t/b}({{{{{{{\bf{r}}}}}}}})=\pm v(-i{\partial }_{y}{s}_{x}+i{\partial }_{x}{s}_{y})\) is the top/bottom surface Dirac Hamiltonian35. s0,x,y,z(τ0,x,y,z) are the identity and Pauli matrices for spin (surfaces) and v is the Fermi velocity. HM denotes the potential term, in which the V0 term is the uniform asymmetric potential between two surfaces by gate voltages, the Δ(r) term is the moiré potential, and the α parameter (0 ≤α ≤1) represents the asymmetry between top and bottom surfaces. Δ(r) is real, spin-independent29, and assumed to possess the C3v symmetry coinciding with the atomic crystal symmetry of TI thin films. With the basis of the Hamiltonian, the corresponding symmetry operators are \({C}_{3z}=\exp (-i\pi {\tau }_{0}{s}_{z}/3)\) for three-fold rotation, \({{{{{{{{\mathcal{M}}}}}}}}}_{y}={\tau }_{0}{s}_{y}\) for y-directional mirror, and \({{{{{{{\mathcal{T}}}}}}}}=i{\tau }_{0}{s}_{y}{{{{{{{\mathcal{K}}}}}}}}\) with \({{{{{{{\mathcal{K}}}}}}}}\) as complex conjugate for time-reversal (TR). The moiré superlattice potential can be expanded as

$${{\Delta }}({{{{{{{\bf{r}}}}}}}})=\mathop{\sum}\limits_{{{{{{{{\bf{G}}}}}}}}}{{{\Delta }}}_{{{{{{{{\bf{G}}}}}}}}}{e}^{i{{{{{{{\bf{G}}}}}}}}\cdot {{{{{{{\bf{r}}}}}}}}},$$
(2)

where \({{{{{{{\bf{G}}}}}}}}={n}_{1}{{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}+{n}_{2}{{{{{{{{\bf{b}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}}\) is the moiré reciprocal lattice vectors with \({{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}=\frac{4\pi }{\sqrt{3}| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}| }(1/2,\sqrt{3}/2),\, {{{{{{{{\bf{b}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}}=\frac{4\pi }{\sqrt{3}| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}| }(-1/2,\sqrt{3}/2)\) and n1,2 as integers. \({{{{{{{{\bf{a}}}}}}}}}_{1,2}^{{{{{{{{\rm{M}}}}}}}}}\) are the primitive vectors for moiré superlattice (see Fig. 1c). The uniform part ΔG=0 can be absorbed into the chemical potential μ and the asymmetric potential V0. To the lowest order, we only keep the first shell reciprocal lattice vectors \(\pm {{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}},\pm {{{{{{{{\bf{b}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}},\pm ({{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}-{{{{{{{{\bf{b}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}})\), as shown in Fig. 1d. The values of ΔG for different Gs are connected by three-fold rotation C3z and \({{{{{{{\mathcal{T}}}}}}}}\), so there is only one independent complex parameter, chosen to be \({{{\Delta }}}_{{{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}}={{{\Delta }}}_{1}{e}^{i2\pi \phi }\), where Δ1 is real and ϕ is the phase that tunes the relative strengths of potentials at three Wyckoff positions 1a, 1b, 1c in one moiré unit cell. Figure 1c shows the moiré potential at ϕ = 0 with an additional six-fold rotation symmetry \({C}_{6z}=\exp (-i\pi {\tau }_{0}{s}_{z}/6)\), and the corresponding potential minima form the multiplicity-2 Wyckoff positions of the hexagonal lattice. The parameters used in our calculations below are \(| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}|=28\) nm, \({E}_{0}=v| {{{{{{{{\bf{b}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}|=38.5\) meV36, m0 = 0.4E0, Δ1 = 0.24E0. The m2 term and other quadratic terms are negligible for the low energy minibands in realistic materials as the relevant energy scale is around 1 meV with a typical moiré momentum 10−2 Å−1, much smaller than other terms in HTI. But we still keep this term in low energy Hamiltonian as it plays an important role for connecting this model to the Kane-Mele model discussed below.

\({{\mathbb{Z}}}_{2}\) nontrivial moiré minibands

We first illustrate the crucial role of inter-surface hybridization in inducing isolated moiré minibands in TI thin films through the schematic view of the spectrum in Fig. 1d. For a single Dirac surface state, it is known29,31,37 that moiré potential can fold the Dirac dispersion and the band touchings at the TR-invariant momenta, e.g. Γ and M, in the moiré Brillouin zone (BZ) remain gapless due to the Kramers’ theorem of TR symmetry. This leads to satellite Dirac cones, but prevents the formation of gaps and hence isolated moiré minibands. For TI thin films, the inter-surface hybridization m can directly produce a gap at Γ while its combined effect with the moiré potential Δ(r) can lead to a gap (proportional to mΔ1) at M (Fig. 1d). The gap openings at both Γ and M lead to the isolated moiré minibands, as demonstrated in Fig. 2d, e for the moiré spectrum of the model Hamiltonian Eq. (1) with different sets of parameters. The bandwidth of isolated bands can be significantly reduced by increasing m, Δ1 and the length of moiré unit cells. (See Supplementary Note 1A)

We are interested in the possibility of realizing \({{\mathbb{Z}}}_{2}\)-nontrivial moiré minibands, particularly the low-energy Kramers’ pairs of conduction (valence) minibands, labelled by CB1, CB2 (VB1, VB2) in Fig. 2d, e. For the parameters in Fig. 2d, CB1 and CB2 are topologically non-trivial while VB1 and VB2 are trivial (νCB1 = νCB2 = 1, νVB1 = νVB2 = 0). For the parameters in Fig. 2e, only CB1 is non-trivial while other minibands are trivial (νCB1 = 1, νCB2 = νV B1 = νV B2 = 0). Figure 2a, b show the \({{\mathbb{Z}}}_{2}\)-invariant νCB1 for CB1 as a function of α and ϕ for a fixed V0/E0 = 0 and 1.2, respectively. The blue regions correspond to νCB1 = 1 while the white regions to νCB1 = 0, and these two regions are separated by metallic lines (orange color). For both V0 values, the νCB1 = 1 blue regions appear around ϕ = 0, 1/3, 2/3. At these ϕ values, there is an additional C6z rotation symmetry, leading to a hexagonal lattice with the C6v group. Moreover, ϕ and ϕ + 1/3 are equivalent up to a translation with the vector \({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}/3+2{{{{{{{{\bf{a}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}}/3\) (See Supplementary Note 1B). Thus, we only show νCB1 as a function of α and V0 for ϕ = 0 in Fig. 2c and find three different parameter regions I, II, III with νCB1 = 1. These topologically non-trivial regions are separated by semi-metal phases that have band touchings between CB1 and CB2. νCB1 for other ϕ is discussed in Supplementary Note 1B and normal insulator phases are discussed in Supplementary Note 1D.

The region I can be adiabatically connected to the parameter set α = 1, V0/E0 = 0 with the band dispersion shown in Fig. 2d, where the inversion symmetry \({{{{{{{\mathcal{I}}}}}}}}={\tau }_{x}{s}_{0}\) and the horizontal mirror symmetry \({{{{{{{{\mathcal{M}}}}}}}}}_{z}=-i{\tau }_{x}{s}_{z}\) are present (D6h group). The \({{{{{{{{\mathcal{C}}}}}}}}}_{6z}\) symmetry leads to the existence of the inversion symmetry by \({{{{{{{\mathcal{I}}}}}}}}={{{{{{{{\mathcal{C}}}}}}}}}_{6z}^{3}{{{{{{{{\mathcal{M}}}}}}}}}_{z}\). From the Fu-Kane parity criterion38, the \({{\mathbb{Z}}}_{2}\)-invariant ν can be determined by \({(-1)}^{\nu }={\prod }_{i}{\lambda }_{{{{\Gamma }}}_{i}}\) and \({\lambda }_{{{{\Gamma }}}_{i}}\) is the parity of eigenstates at the TR invariant momenta Γi=1,...,4, which correspond to one Γ point and three M points in 2D moiré BZ. The values of parities \({\lambda }_{{{{\Gamma }}}_{i}}\) can be derived analytically in the weak Δ1 limit and depend on the hybridization m and the moiré potential strength Δ1 (See Supplementary Note 1A). From the analytical solutions, we find CB1 and VB1 have the same parity at M (\({\lambda }_{M}^{{{{{{{{\rm{CB}}}}}}}}1}={\lambda }_{M}^{{{{{{{{\rm{VB1}}}}}}}}}\)) but opposite parities at Γ (\({\lambda }_{{{\Gamma }}}^{{{{{{{{\rm{VB1}}}}}}}}}=-{\lambda }_{{{\Gamma }}}^{{{{{{{{\rm{CB}}}}}}}}1}\)), resulting in νCB1 + νVB1 = 1 mod 2, implying that one of them is \({{\mathbb{Z}}}_{2}\)-nontrivial while the other is trivial. As discussed in Supplementary Note 1A, the relation of \({{\mathbb{Z}}}_{2}\) invariant between the CB1 and VB1 minibands can be understood as the consequence of the emergent chiral symmetry operator \({{{{{{{\mathcal{C}}}}}}}}={\tau }_{z}{s}_{z}\) of HTI.

At ϕ = 0 and α = 1 in Fig. 2d, we notice that the CB2 minibands are also topologically non-trivial (νCB2 = 1), so νCB1 + νCB2 = 0 mod 2. According to the irreducible representations of CB1 and CB2 at high-symmetry momenta (See Supplementary Note 1C), these two minibands can together form an elementary band representation (EBR) \({\bar{E}}_{1}^{2b}\uparrow G\) induced in the space group P6mm39,40, which corresponds to the atomic limit with two s-wave atomic orbitals at the symmetry-related Wyckoff positions 1b and 1c in Fig. 1(c). Once CB1 is isolated from CB2, CB1 itself does not have an atomic limit preserving \({{{{{{{{\mathcal{C}}}}}}}}}_{6z}\) symmetry and is thus topological. If \({{{{{{{{\mathcal{C}}}}}}}}}_{6z}\) symmetry is relaxed, CB1 and CB2 instead correspond to the reducible band representation \({}^{1}{\bar{E}}^{2}\bar{E}+{\bar{E}}_{1}\uparrow {G}^{{\prime} }\) with \({G}^{{\prime} }\) to be P3m1, and correspondingly, they can be gapped. Indeed, as demonstrated in Supplementary Note 1C, when the m2 term is tuned to dominate over other terms in H0, we can adiabatically connect the CB1 and CB2 together in Fig. 2d to the effective Kane-Mele model34. This provides an alternative explanation of non-trivial \({{\mathbb{Z}}}_{2}\) numbers for both CB1 and CB2 in Fig. 2d.

For the nontrivial region II in Fig. 2c, we consider the parameter set ϕ = 0, α = 0, V0/E0 = 1.2 with the energy dispersion shown in Fig. 2e. The Fu-Kane criterion cannot be applied as inversion is broken, so we directly calculate the Wannier center flow41 for the CB1 in the inset of Fig. 2e, which corresponds to νCB1 = 1. Different from the case of Fig. 2d, CB2 is now topologically trivial νCB2 = 0. We also examine the band evolution with respect to m2 in the model, which is quite different from the case with inversion symmetry, as discussed in Supplementary Note 1C. When the m2 term dominates in H0, CB1 and CB2 can be mapped to the Kane-Mele model with a Rashba SOC term from the inversion symmetry breaking, which leads to the gap closing between CB1 and CB2 around K in moiré BZ. The overall \({{\mathbb{Z}}}_{2}\) number νCB1 + νCB2 mod 2 is 0 because CB1 and CB2 together also form a EBR coming from s-wave atomic orbitals located at the two potential minima of the \({{{{{{{{\mathcal{C}}}}}}}}}_{6z}\) symmetric moiré potential. When reducing m2, a Dirac type of gap closing between CB2 and higher-energy conduction minibands occurs at certain critical value of m2 and changes νCB1 + νCB2 mod 2 to 1, which is persisted to m2 = 0 (νCB1 = 1 and νCB2 = 0). The other \({{\mathbb{Z}}}_{2}\) non-trivial minibands are found to appear in a much higher energy when m2 is small (See Supplementary Fig. 6 in Supplementary Note 1C). This is in sharp contrast to the inversion-symmetric case in which CB1 and CB2 together have νCB1 + νCB2 = 0 mod 2 when varying m2.

Interaction-driven QAH state

The Coulomb interaction of electrons in the moiré superlattice can be estimated as \({U}_{0}={e}^{2}/4\pi {\varepsilon }_{0}{\varepsilon }_{r}| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}| \, \approx \, 5.11\) meV ~ 0.13E0, in which e is the electron charge, ε0 is vacuum permittivity, and dielectric constant εr is about 10.42 The value of U0 is comparable to both the moiré miniband width ~ 0.1E0 ≈ 3.85 meV and miniband gaps ~ 0.1E0. We next study the effects of the Coulomb interaction with the Hartree-Fock mean-field theory42,43,44,45,46,47. We first project the moiré Hamiltonian and the Coulomb interaction into the low-energy subspace spanned by either CB1 (a two-band model) or both CB1 and CB2 (a four-band model). By treating the density matrix \({\rho }_{{n}_{1}{n}_{2}}({{{{{{{\bf{k}}}}}}}})=\langle {c}_{{n}_{1}}^{{{{\dagger}}} }({{{{{{{\bf{k}}}}}}}}){c}_{{n}_{2}}({{{{{{{\bf{k}}}}}}}})\rangle\) as the order parameter with \({c}_{n}^{{{{\dagger}}} }({{{{{{{\bf{k}}}}}}}})\) for the creation operator of the nth eigenstate in the two-band or four-band subspace, we can decompose the Coulomb interaction Hamiltonian into two-fermion terms so that the order parameter ρ(k) can be solved self-consistently (See Supplementary Note 2).

In the two-band model, we generally consider two types of order parameters, (1) ρz(k) ∝ fz(k)σz and (2) ρxy(k) ∝ fx(k)σx + fy(k)σy, where the σ matrix is for the Kramers’ pair of CB1 and fx,y,z(k) represents the momentum-dependent part of the order parameter. The order parameter ρ0 ∝ σ0 is directly related to the band occupation and we always consider half-filling for the Kramers’ pair bands of CB1. At ϕ = 0, α = 1, V0/E0 = 0, the horizontal mirror symmetry \({{{{{{{{\mathcal{M}}}}}}}}}_{z}\) is present and the non-interacting Hamiltonian possesses D6h group symmetry, so two spin states of CB1 can be labelled by the mirror eigen-values ± i, and the σ matrices of the order parameter ρ is written under the \({{{{{{{{\mathcal{M}}}}}}}}}_{z}\) eigenstates. The two mirror-eigenstates carry nonzero mirror Chern number ± 1 from the nontrivial \({{\mathbb{Z}}}_{2}\) topology. Thus, ρz(k) and ρxy(k) correspond to the mirror-polarized and mirror-coherent ground states.

The mirror polarized state (ρz(k)) spontaneously breaks \({{{{{{{{\mathcal{C}}}}}}}}}_{2z}{{{{{{{\mathcal{T}}}}}}}}\) symmetry relating two mirror eigenstates, while the mirror coherent state (ρxy(k)) breaks \({{{{{{{{\mathcal{M}}}}}}}}}_{z}\) by superposition of two mirror eigenstates with a fixed relative U(1) phase between them (See Supplementary Note 2C). The self-consistent calculations suggest that both ρz(k) and ρxy(k) can be non-zero solutions when the Coulomb interaction exceeds certain critical values Uc ~ 0.05E0 ≈ 1.92 meV, as shown in Fig. 3c, where the ground state energies of self-consistent ρz(k) and ρxy(k) are shown as a function of interaction strength \(U({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}})\), which is treated as a tuning parameter and equal to U0 for the realistic moiré superlattice. Our estimate of Coulomb interaction 0.13E0 in TI moiré systems is larger than this critical value. From Fig. 3c, we also see that the mirror-polarized state ρz(k) has a lower ground state energy than the mirror-coherent state ρxy(k). The energy spectrum of the CB1 before (blue lines) and after (orange lines) taking into account the ρz(k) order parameter is shown in Fig. 3a, in which the metallic state of CB1 (blue lines) is fully gapped out by ρz(k) at half-filling. With the order parameter ρz(k) (orange lines in Fig. 3a), the ground state only fills the lower energy band while the higher band is the excited spectrum within the Hartree-Fock approximation. Due to non-zero mirror Chern number of non-interacting CB1 states, the mirror-polarized state ρz(k) carries Chern number + 1 of the lower band and thus gives rise to the QAH state. As shown in Fig. 3b (also Supplementary Note 2C), the mirror coherent state ρxy(k) opens gaps at TR invariant momenta Γ and M by spontaneously breaking \({{{{{{{\mathcal{T}}}}}}}}\) but has nodes at K due to the \({C}_{2z}{{{{{{{\mathcal{T}}}}}}}}\) symmetry. This explains why the mirror-polarized state has a lower ground state energy than the mirror-coherent state. Besides all the uniform order parameters, a nonuniform magnetic order parameter32 is also examined in Supplementary Note 2F, and is found to possess a larger critical interaction strength compared to the QAH phase at the half filling of CB1. Thus, the mirror-polarized QAH state can be driven by Coulomb interaction in this system.

Fig. 3: Hartree-Fock ground energies and ground states.
figure 3

a The spectra (orange) for the Hartree-Fock mean-field Hamiltonian with the order parameter ρz(k) at half filling of CB1 for the case with ϕ = 0, α = 1, V0/E0 = 0. C is the Chern number of each band. b The Hartree-Fock spectra (black) for the order parameter ρxy(k) for the same parameter as (a). In (a, b), the blue lines are single-particle spectra. c The difference in energy per particle between the self-consistent Hartree-Fock states Ei and the non-interacting state En as a function of Coulomb interaction strengths for the order parameters ρz(k) (orange) and ρxy(k) (black). d The spectra (orange) for the Hartree-Fock mean-field Hamiltonian with the order parameter ρz(k) at half filling of CB1 for the case with ϕ = 0, α = 0, V0/E0 = 1.2. e The Hartree-Fock spectra (black) for the order parameter ρxy(k) for the same parameter as (d). In (d, e), the blue lines are single-particle spectra. f The energy difference Ei − En for the order parameters ρz(k) (orange) and ρxy(k) (black).

Full size image

We also study the case of ϕ = 0, α = 0, V0/E0 = 1.2 within the two-band model, in which the mirror \({{{{{{{{\mathcal{M}}}}}}}}}_{z}\) is broken at the single-particle level and six-fold rotation remains, in Supplementary Note 2C and find the ρz(k) is still energetically favored, as shown in Fig. 3f. The spectra with the order parameter ρz(k), ρxy(k) are shown in Fig. 3d, e, respectively. The ground state is a Chern insulator.

As the miniband gap is comparable to Coulomb interaction, one may ask if the inter-miniband mixing due to Coulomb interaction can change the topological nature of the ground state. Thus, we study the Coulomb interaction effect in a four-band model including both CB1 and CB2, as discussed in Supplementary Note 2D. For the inversion-symmetric case ϕ = 0, α = 1, V0/E0 = 0, the ground state of the four-band model is still the mirror polarized C = ± 1 state in regime B (blue) of Fig. 4a, when \(U({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}})=0.08{E}_{0}\) is smaller than the miniband gap ~ 0.1E0, with the spectra shown in Fig. 4c. When \(U({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}})=0.13{E}_{0}\) is larger than the miniband gap (regime C (brown) of Fig. 4a), the strong Coulomb interaction can induce mixing between CB1 and CB2 within one mirror parity sector and drive a topological phase transition to the C = 0 state shown in Fig. 4d (More details in Supplementary Note 2D). However, the situation for the inversion-asymmetric case ϕ = 0, α = 0, V0/E0 = 1.2 is different as νCB1 = 1 and νCB2 = 0. For the realistic estimated value \(U({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}) \, \approx \, 0.13{E}_{0}\) that is larger than miniband gap, the interacting ground state of the four-band model carries C = ± 1 and thus remains the same as that of the two-band model, as shown by the regime B (blue) in Fig. 4b. The energy spectra in this case is shown in Fig. 4e. By comparing the phase diagrams for the inversion symmetric and asymmetric cases, we conclude that the asymmetric potential V0 stabilizes the interaction-driven QAH state in TI moiré heterostructures.

Fig. 4: Hartree-Fock interacting phase diagrams.
figure 4

a The energy difference per particle Ei − En at 1/4 filling of the four-band model with both CB1 and CB2 for the case ϕ = 0, α = 1, V0/E0 = 0. Here Ei and En is the interacting ground state energy and non-interacting metallic state energy, respectively. The orange (black) line is for the \({C}_{2}{{{{{{{\mathcal{T}}}}}}}}\) symmetry breaking (preserving) density matrix. The interacting ground states in the regime A, B, C correspond to a metallic phase, an insulating phase with C = ± 1, and an insulating phase with C = 0, respectively. b Ei − En for the case with ϕ = 0, α = 0, V0/E0 = 1.2. c, d The spectra (orange) of the Hartree-Fock mean-field Hamiltonian for the Coulomb interaction strength in regime B and C of (a). C is the Chern number of each band. e The spectra (orange) of the mean-field Hamiltonian for the Coulomb interaction strength in regime B of (b). In (c–e), the blues lines are single-particle spectra.

Full size image

Sb2/Sb2Te3 moiré heterostructure

We propose a possible experimental realization of a TI based moiré heterostructure with twisted Sb2 monolayer on top of Sb2Te3 thin film. The moiré lattice structure is shown in Fig. 5a. Sb2Te3 is a prototype of three dimensional TI with layered structures. Within one quintuple layer (QL, see the red and blue dots in Fig. 5a), there is strong chemical binding formed by the sequential Te-Sb-Te-Sb-Te atomic layers and the van der Waals coupling is between adjacent QLs48. Precise control of layer thickness of the Sb2Te3 thin film has been achieved via molecular beam epitaxy (MBE) method experimentally49,50. On the top of Sb2Te3 thin film, Sb2 monolayer could be deposited19,51,52, forming a Sb2/Sb2Te3 heterostructure. By using density functional theory (DFT) calculations, we confirm that Sb2 monolayer with buckled honeycomb structure marked as the gray in Fig. 5a is a semiconductor with a band gap larger than that of Sb2Te3 thin films. Furthermore, we put Sb2 monolayer on the top of 2QL Sb2Te3 thin films with different stackings, including the AA, AB, and BA stackings (see Fig. 5a). The corresponding electronic band structures are shown in Fig. 5c. The work function of monolayer Sb2 and Sb2Te3 thin film matches with each other, forming the type I semiconductor hetero-junction. Around the Fermi level, the conduction and valence bands are both mainly contributed by two strongly hybridized surface states of the 2QL Sb2Te3 thin film. The role of Sb2 monolayer is to provide a potential along the out-of-plane direction, leading to a Rashba type of spin-split bands. Thus, the twisted Sb2/Sb2Te3 moiré heterostructure satisfies the requirements mentioned above for the \({{\mathbb{Z}}}_{2}\) nontrivial moiré minibands.

Fig. 5: DFT calculations of Sb2 on top of 2QL Sb2Te3 films.
figure 5

a Side view of the Sb2/2QL Sb2Te3 heterostructure with AA stacking (left panel) and the moiré pattern for twisted Sb2 on top of Sb2Te3 thin film (right panel). To show the moiré pattern clearly, we only plot atoms in the region marked by black dashed lines in the left panel. The regions with green, purple, and yellow background label structures with AA, AB, and BA stacking respectively. The primitive vectors for moiré supercell \({{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}\) and \({{{{{{{{\bf{a}}}}}}}}}_{2}^{{{{{{{{\rm{M}}}}}}}}}\) are marked by black arrows. b Top views of configurations with AA, AB, BA stacking, respectively. The atomic primitive lattice vectors of the 2QL Sb2Te3 thin film are labeled as \({\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1}\ {{{{{{{\rm{and}}}}}}}}\ {\tilde{{{{{{{{\bf{a}}}}}}}}}}_{2}\). The green arrow labels the shift dR between the Sb2Te3 layer and Sb2 monolayer in each stacked configuration. c Band structures (blue lines) around the \(\tilde{{{\Gamma }}}\) point for heterostructures with AA, AB, and BA stacking from DFT calculations. The orange lines are fitted spectra from the two-surface-state atomic Hamitlonian in Eq. (3). The Brillouin zone is plotted for the slab model used in DFT calculations with atomic primitive lattices. The Fermi levels are set as zero. d The superlattice potential \(\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})\) as a function of dR shown in the moiré superlattice. \({\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1,2}\) are marked by the black arrows. e Energy spectrum for twisted monolayer Sb2 and 2QL Sb2Te3 with the superlattice potential shown in (d).

Full size image

To connect the theoretical moiré model Hamiltonian in Eq. (1) to electronic band structure from DFT calculations, we first introduce a uniform shifting vector dR between monolayer Sb2 and 2QL Sb2Te3 thin film, and AA, AB, and BA stackings correspond to \({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}}=0,\ {\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1}/3+2{\tilde{{{{{{{{\bf{a}}}}}}}}}}_{2}/3,\ {{{{{{{\rm{and}}}}}}}}\ 2{\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1}/3+{\tilde{{{{{{{{\bf{a}}}}}}}}}}_{2}/3\), respectively (Fig. 5b). \({\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1,2}\) are atomic primitive lattice vectors for the Sb2Te3 lattice shown in Fig. 5b. The spectrum from DFT calculations with different stackings is fitted by the dispersion of two-surface-state atomic Hamiltonian

$${H}^{{{{{{{{\rm{DFT}}}}}}}}} ({{{{{{{\bf{k}}}}}}}},{{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})= {H}^{{{{{{{{\rm{TI}}}}}}}}}({{{{{{{\bf{k}}}}}}}})\\ + \frac{1+\alpha }{2}\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}}){\tau }_{0}{s}_{0}+\frac{1-\alpha }{2}\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}}){\tau }_{z}{s}_{0},$$
(3)

where s0(τ0,z) are the Pauli matrices for the spin (surfaces). \(\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})\) is a uniform atomic potential induced by the Sb2 monolayer for a fixed dR and different dR values correspond to different stacking configurations, shown in Fig. 5b. For the dR values corresponding to the AA, AB, BA and several other stackings in Supplementary Note 3, we fit the energy dispersion of the model Hamiltonian HDFT(k, dR) to that from the DFT calculations as shown in orange lines in Fig. 5c, which fits well with the conduction bands of the surface states. From fitting, we can extract \(\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})\), which can be further interpolated as a continuous function of dR shown in Fig. 5d. \(\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})\) has the periodicity of the atomic unit-cell defined by \({\tilde{{{{{{{{\bf{a}}}}}}}}}}_{1,2}\). All other parameters in HDFT(k, dR) are treated as constants and can also obtained by fitting to the DFT bands. After obtaining the parameters for HDFT(k, dR), the next step is to connect them to those of the moiré Hamiltonian H0 in Eq. (1). For the moiré TI with the twist angle θ, the local shift between two layers at the atomic lattice vector R of the Sb2Te3 layer is \({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}}={{{{{{{\mathcal{R}}}}}}}}(\theta ){{{{{{{\bf{R}}}}}}}}-{{{{{{{\bf{R}}}}}}}}\), where \({{{{{{{\mathcal{R}}}}}}}}(\theta )\) is the rotation operator, so we can obtain the potential

$${{\Delta }}({{{{{{{\bf{R}}}}}}}}) \, \approx \, \widetilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})$$
(4)

at the location R. The last step is to treat Δ(r) as a function of continuous r by interpolating the function Δ(R) (See Supplementary Note 4), and Δ(r) serves as the morié superlattice potential for the model Hamiltonian H0(r). Besides, all the other parameters in H0 are chosen to be the same as those in HDFT. In Fig. 5d, the potential maximum of \(\tilde{{{\Delta }}}({{{{{{{{\bf{d}}}}}}}}}_{{{{{{{{\bf{R}}}}}}}}})\) appears at the AB stacking while two local minima exist at the BA and AA stackings and are close in energy. The parameters for the moiré potential at θ = 0. 5° is given by Δ1/E0 = 0.22, α = 0.16, and ϕ = 0.68π, close to ϕ = 2π/3 for the C6z-rotation symmetric potential. Figure 5e shows the energy dispersion of moiré minibands for V0/E0 = 1.2, in which the lowest conduction bands (cyan) indeed are isolated minibands with nontrivial νCB1=1.

Discussion

In summary, we demonstrate that the superlattice potential in a TI thin film can give rise to \({{\mathbb{Z}}}_{2}\) non-trivial isolated moiré minibands and Coulomb interaction can drive the system into the QAH state when the Kramer’s pair of non-trivial minibands are half filled. Besides the twisted Sb2 monolayer on top of the Sb2Te3 thin film, our model can be generally applied to other TI heterostructures with the in-plane superlattice potential, which can come from either the moiré pattern of another 2D insulating material or gating a periodic patterned dielectric substrate53,54,55,56,57. The 2D TI thin films can be in a quantum spin Hall state or trivial insulator state, depending on the relative sign between m0 and m2 in the model Hamiltonian (see Eq. (1))58. Our calculations suggest that the moiré potential can lead to \({{\mathbb{Z}}}_{2}\) non-trivial minibands no matter the sign of m2, once this term is negligible compared to the linear term in the moiré scale. Such a result implies the possibility of realizing isolated \({{\mathbb{Z}}}_{2}\) non-trivial minibands in other 2D topologically trivial systems with strong Rashba SOC. In our calculation, a large moiré superlattice constant (\(| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}| \sim 28\) nm) leads to small energy scales, around a few meV, for miniband widths, miniband gaps and Coulomb interactions, which may be disturbed by disorders. The miniband topological property is robust against disorder when the disorder strength is smaller than the miniband width ( ~ 2.2 meV). In Supplementary Note 2E, we reduce \(| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}|\) to ~ 14 nm, which yields larger energy scales (around 10 meV) of minibands and Coulomb interaction, and our Hartree-Fock calculations suggest the estimated Coulomb interaction is still strong enough to drive the system into the QAH state. For a smaller moiré lattice constant \(| {{{{{{{{\bf{a}}}}}}}}}_{1}^{{{{{{{{\rm{M}}}}}}}}}|\), it is desirable to reduce the bandwidth of moiré minibands while keeping the Coulomb energy, and this can be achieved by twisting two identical TIs or with in-plane magnetization, as proposed recently30,59. Moreover, at larger moiré superlattice constant, the lattice reconstruction may occur in the moiré superlattice (See Supplementary Note 3).