Introduction

Flat-band systems have been proposed as interesting theoretical models to prove the existence of ferromagnetic ordering with itinerant electrons1,2,3,4. Theoretical developments in such flat-band systems have been made almost in parallel with those in the widely discussed topological insulators (TIs)5,6,7,8. The nontrivial topology of electronic bands in a kagome lattice, one of those flat-band systems, has been extensively studied9,10,11,12,13,14,15,16,17,18.

The experimental quests for topological materials with kagome lattice have also been carried out. Many of such experimental efforts were stimulated by the prediction of Weyl semimetals19,20, including intermetallic compounds involving Co21,22,23,24,25, Fe26,27,28,29, Mn30,31,32, and van-der-Waals compounds33, as well as optical lattices34,35. More recently, the coexistence of superconductivity and nontrivial band topology was reported in a kagome compound36,37,38,39.

When a flat band is partially occupied by electrons, the Coulomb repulsive interactions could become dominant over the electronic kinetic energy. This situation is already realized in two-dimensional electron gases under applied magnetic fields, where flat bands correspond to Landau levels. Fractional quantum Hall (FQH) effects were thus discovered40,41. An exact numerical analysis made an important contribution by demonstrating that quantum fluctuations are essential to stabilize FQH states over charge density wave states42. Once the charge excitation gap is induced at a fractional filling, the property of FQH states is elegantly explained using effective theory43.

Recently, further intriguing proposals were put forward by considering flat bands with nontrivial topology and repulsive interactions, whereby FQH states could be generated without having Landau levels, called fractional Chern insulators (FCIs). These proposals considered single-band models on a kagome lattice44,45, checkerboard lattices46,47,48,49,50, a Haldane model on a honeycomb lattice, and a ruby lattice45, as well as multi-band models on a buckled honeycomb lattice51, a triangular lattice52, and a square lattice for the mercury-telluride TI45. It was later revealed that quantum Hall states realized in flat band systems and those realized under an applied magnetic field are adiabatically connected53. When realized in real materials, FCI states in flat band systems could become a vital element of topological quantum computing54,55. Based on numerical results44,45,46,47,48,49,50,51,52, the possibility of FCI states was suggested in some flat band systems12,15,17. However, the material realization of such FCI states has yet to be demonstrated as theoretical proposals often focus on simple one-band models and other proposed systems have small band gaps.

Motivated by the recent experimental realization of kagome materials, where multiple transition-metal d orbitals are active near the Fermi level, we consider in this work a multiorbital itinerant-electron model on a two-dimensional kagome lattice. With the atomic spin–orbit coupling (SOC), this model shows multiple topological phases, including spin Hall insulators when spin splitting is absent and Chern insulators when spin splitting is induced. Furthermore, this model exhibits flat bands having nonzero Chern number as in a single-band kagome system10. We found that non-local Coulomb interactions induce FCI states when such a flat band is fractionally occupied by electrons. Note that our approach employs the original on-site local source of the SOC, while most of simplified models widely employed in other efforts assume the form of SOC simply based on symmetry considerations. Thus, our work relies on more fundamental foundations. Our model calculation is particularly relevant to CoSn-type intermetallic compounds when a single kagome layer becomes available.

Results

Theoretical model

To begin with, we set up a multi-orbital tight-binding model on a kagome lattice

$${H}_{{{{{{{{\rm{t}}}}}}}}}=-\mathop{\sum}\limits_{\left\langle {{{{{{{\bf{r}}}}}}}}\,{{{{{{{{\bf{r}}}}}}}}}^{\prime}\right\rangle }\mathop{\sum}\limits_{\alpha \beta \sigma }\left({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{{\bf{r}}}}}}}}}^{\prime}}^{\alpha \beta }{c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }^{{{{\dagger}}} }{c}_{{{{{{{{{\bf{r}}}}}}}}}^{\prime}\beta \sigma }+{{{{{{{\rm{H.c.}}}}}}}}\right),$$
(1)

as schematically shown in Fig. 1. Here, \({c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }^{({{{\dagger}}} )}\) is the annihilation (creation) operator of an electron at site r, orbital α, and with spin σ =  or . As discussed by Meier et al.25, CoSn-type kagome systems have several flat bands with {yz, xz}, {xy, x2 − y2}, or 3z3 − r2 character. We focus on a {yz, xz} subset for simplicity and use α = a for the yz orbital and b for the xz orbital. With this basis, nearest-neighbor hopping intensities \({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{\alpha \beta }\) can be parameterized using Slater integrals56. Between site 1 and site 2, \({\hat{t}}_{{{{{{{{\bf{1}}}}}}}}{{{{{{{\bf{2}}}}}}}}}\) is diagonal in orbital indices as \({t}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{aa}={t}_{\delta }\) and \({t}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{bb}={t}_{\pi }\), corresponding to (ddδ) and (ddπ), respectively, by Slater and Koster56. Other components are obtained by rotating the basis a and b as shown in the Methods section. From now on, tπ is used as the unit of energy.

Fig. 1: Schematics of our theoretical model.
figure 1

a Kagome lattice with three sublattices, labeled 1, 2, and 3. The two arrows are lattice translation vectors a1,2. b Local orbitals a = yz and b = xz. Colored ellipsoids indicate regions of electron wave functions, where the sign is positive. c Nearest-neighbor hopping integrals. yz(xz) orbitals between site 1 and site 2 are hybridized via diagonal hopping tδ(π), i.e., δ(π) bonding. Other hopping integrals between site 2 and site 3 and between site 1 and site 3 are obtained via the Slater rule56 as shown in the Methods section.

Because yz and xz are written using the eigenfunctions of angular momentum lz = ±1 for l = 2 as \(\left|yz\right\rangle =\frac{{{{{{{{\rm{i}}}}}}}}}{\sqrt{2}}(\left|1\right\rangle +\left|-1\right\rangle )\) and \(\left|xz\right\rangle =-\frac{1}{\sqrt{2}}(\left|1\right\rangle -\left|-1\right\rangle )\), respectively, the SOC \(\lambda \overrightarrow{l}\cdot \overrightarrow{s}\) in the {yz, xz} subset is written as

$${H}_{{{{{{{{\rm{soc}}}}}}}}}=\frac{\lambda }{2}\mathop{\sum}\limits_{{{{{{{{\bf{r}}}}}}}}\,\sigma }\left({{{{{{{\rm{i}}}}}}}}{\sigma }_{\sigma \sigma }^{z}{c}_{{{{{{{{\bf{r}}}}}}}}a\sigma }^{{{{\dagger}}} }{c}_{{{{{{{{\bf{r}}}}}}}}b\sigma }+{{{{{{{\rm{H.c.}}}}}}}}\right),$$
(2)

where \({\hat{\sigma }}^{z}\) is the z component of the Pauli matrices.

As shown in Supplementary Note 1, an effective model for the {xy, x2 − y2} doublet has the same form as the above Ht + Hsoc. By symmetry, there is no hopping matrix between the {yz, xz} doublet and the other orbitals xy, x2 − y2, and 3z2 − r2, but the {xy, x2 − y2} doublet and the 3z2 − r2 singlet could be hybridized. As discussed briefly later, the degeneracy in the {yz, xz} doublet and in the {xy, x2 − y2} doublet could be lifted by a crystal field. Such band splitting is also induced by the difference between tδ and tπ. Furthermore, all d orbitals could in principle be mixed by the SOC. Including these complexities is possible but depends on the material and they usually induce smaller perturbations, therefore, here they are left for future analyses.

Non-interacting band topology

By diagonalizing the single-particle Hamiltonian Ht + Hsoc, one obtains dispersion relations as shown in Fig. 2. In the simplest case, where the hopping matrix \({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{\alpha \beta }\) does not distinguish tδ and tπ and the SOC is absent, the dispersion relation is identical to the one for the single-band tight-binding model, consisting of flat bands and graphene-like bands as shown by gray lines in Fig. 2a. Note that each band is fourfold degenerate because of two orbitals and two spins per site. Including SOC does not change the dispersion curve but simply shifts \(\overrightarrow{l}\cdot \overrightarrow{s}=\pm 1/2\) bands (see Supplementary Note 1).

Fig. 2: Dispersion relations of the non-interacting model.
figure 2

Bulk dispersion relations without the spin-orbit coupling (SOC) (a) and with the SOC λ = 0.2 (b). In both cases, energy E is scaled by the π-bond hopping integral tπ. Gray lines in a indicate the dispersion with the δ-bond hopping integral tδ = 1, which realizes the ideal dispersion in a kagome lattice. Blue lines in (a) and red lines in (b) are dispersions with tδ = 0.5. The inset shows the first Brillouin zone with high-symmetry lines used in (a), (b). The Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\) for the spin up component of each band is also shown in (b). c Dispersion relations with tδ = 0.5 and λ = 0.2 in the ribbon geometry, which is periodic along the a1 direction and contains 20 unit cells along the perpendicular direction. Gapless edge modes are indicated by red and blue lines.

Including orbital dependence as tδ ≠ tπ without SOC instead splits the fourfold degeneracy except for two points at the Γ point and two points at the K point. Quite intriguingly, Dirac dispersions emerge from the topmost flat bands as shown as blue lines in Fig. 2a (see Supplementary Note 1 for more discussion). Turning on the SOC further splits such fourfold degeneracy, leading to nontrivial band topology. In this particular example, the spin component along the z axis is conserved giving unique characteristics to this case. As shown in Fig. 2b, the spin up component of each band is characterized by a nonzero Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\). Because of the time-reversal symmetry, spin down bands have opposite Chern numbers. The topological property is also confirmed by gapless modes in the dispersion relation with the ribbon geometry, as shown in Fig. 2c. Here, there appear one (two) pair of gapless modes between the highest and the second highest (between the second lowest and the third lowest) bands, shown as red (blue) curves, corresponding to the sum of Chern numbers below the gap, −1(−2). The other edge states are invisible because of the overlap with the bulk continuum.

A multi-orbital kagome model thus naturally shows quasi flat bands with nontrivial topology. However, close inspection revealed that, with tδ = 0.5 and λ = 0.2, the minimum of the highest band at the K point is slightly lower than the maximum of the second highest band at the Γ point. Thus, instead of a TI, a topological semimetal is realized when the Fermi level is located between the highest band and the second highest band. In fact, there are ways to make the gap positive. Here, we consider second-neighbor hopping matrices \({\hat{t}}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{(2)}\). As explained in Supplementary Note 1, these are also parametrized by π-bonding (ddπ) and δ-bonding (ddδ), \({t}_{\pi }^{(2)}\) and \({t}_{\delta }^{(2)}\), respectively. For simplicity, we fix the ratio between tπ and \({t}_{\pi }^{(2)}\) and between tδ and \({t}_{\delta }^{(2)}\) as \({t}_{\pi }^{(2)}/{t}_{\pi }={t}_{\delta }^{(2)}/{t}_{\delta }={r}_{2}\), and analyze the sign and magnitude of the band gap Δgap between the highest band and the second highest band, as well as the flatness of the highest band defined by \({{\Delta }}\varepsilon \equiv {\varepsilon }_{1,\max }-{\varepsilon }_{1,\min }\).

Figure 3a plots Δε as a function of tδ and r2 with λ = 0.2. As mentioned previously, the perfectly flat band with Δε = 0 is realized at tδ = 1 and r2 = 0, but band gap Δgap is zero. The flatness is immediately modified by reducing tδ from 1. As indicated by an open square in the plot, tδ = 0.5 and r2 = 0 gives Δε ~ 0.88 and negative band gap Δgap ~ −0.027. Nonzero r2 controls the relative energy between the zone center and the zone boundary. In particular, negative r2 pushes up the energy at the K point, hereby the flatness is recovered. Naturally, the flatness and the positive gap are correlated as indicated by red loops in the second and forth quadrants because the separation between the highest band and the second highest band is fixed by the SOC strength. As indicated by a filled circle, tδ = 0.5 and r2 = −0.2 gives Δε ~ 0.22 and positive band gap Δgap ~ 0.17. Corresponding dispersion relation is shown in Fig. 3b. The Chern numbers remain unchanged by this r2.

Fig. 3: Control of the band flatness.
figure 3

a Color map of the flatness Δε of the highest band as a function of tδ and the ratio between the nearest-neighbor and the second-neighbor hopping r2 with λ = 0.2. Open square (filled circle) locates tδ = 0.5 with r2 = 0 (−0.2). Red closed loops show the areas where the band gap is positive Δgap > 0. b Bulk band structure with tδ = 0.5, r2 = −0.2, and λ = 0.2. Band-dependent Chern number is also shown.

Many-body effects

Having established the topological properties at the single-particle level, we turn our attention to many-body effects focusing on the highest-energy flat band. A unique property of the current model is that the topmost quasi flat band has Chern number \(| {{{{{{{\mathcal{C}}}}}}}}| =1\). Thus, a large spin polarization can be induced by many-body interactions57 or by a small magnetic field. Further intriguing possibilities are FCI states when a topological flat band has a fractional filling and the insulating gap is induced by correlation effects44,46,47,48,49,50,51,52. We examine such a possibility in our kagome model. Assuming the spin polarization in the highest band, we introduce local and nearest-neighbor Coulomb repulsive interactions as \({H}_{{{{{{{{\rm{U}}}}}}}}}=U{\sum }_{{{{{{{{\bf{r}}}}}}}}}{n}_{{{{{{{{\bf{r}}}}}}}}a\uparrow }{n}_{{{{{{{{\bf{r}}}}}}}}b\uparrow }+V{\sum }_{\left\langle {{{{{{{\bf{r}}}}}}}}{{{{{{{\bf{r}}}}}}}}^{\prime} \right\rangle }{\sum }_{\alpha \beta }{n}_{{{{{{{{\bf{r}}}}}}}}\alpha \uparrow }{n}_{{{{{{{{\bf{r}}}}}}}}^{\prime} \beta \uparrow }\), where \({n}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }={c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }^{{{{\dagger}}} }{c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }\). Here U is the effective Coulomb interaction given by \(U=U^{\prime} -J\) with the interorbital Coulomb repulsion \(U^{\prime}\) and the interorbital exchange interaction J. These interactions are then projected onto the highest band, leading to the effective Hamiltonian Heff = Ht + Hsoc + HU.

Note that the Sz conservation is not essential to realize FCI. For our case and most of others, including complexities which break Sz conservation does not destroy FCI as long as the flat band has the nontrivial topology and is well separated from other bands, justifying projecting interaction terms onto the flat band and allowing for an accurate Lanczos calculation. While the computational cost would be expensive, direct calculations of multiband models with Sz-non-conserving terms would show FCI if the appropriate condition is fulfilled, but this possibility has not been fully explored yet.

The effective Hamiltonian Heff is diagonalized in momentum space. For this purpose, we discretize the momentum space into N1 × N2 patches and express the Hamiltonian in the occupation basis, i.e., the Hilbert space is built up by \(\left|{\varphi }_{l}\right\rangle ={\prod }_{{{{{{{{\bf{k}}}}}}}}\in l}{\psi }_{1{{{{{{{\bf{k}}}}}}}}}^{{{{\dagger}}} }\left|0\right\rangle\), where ψ1k is the single-particle wave function for the highest flat band at momentum k, and the combination of k is specified by l. Due to the translational symmetry and the momentum conservation of many-body interaction terms, Heff is subdiagonalized according to the total momentum ktot = ∑kl k modulo b1 and b2, with b1,2 being two reciprocal lattice vectors. In this study, we take N1 = 4 and N2 = 6 and consider ν = 1/3 filling, that is, the number electrons in the highest flat band is Ne = 8. Momentum sector will be specified using integer index (k1, k2) corresponding to the total momentum ktot = b1k1/N1 + b2k2/N2.

Figure 4a, b show the low-energy spectra of the interacting model with tδ = 0.5 and r2 = 0 and tδ = 0.5 and r2 = −0.2, respectively, with U = 2 and V = 1 as a function of total momentum. In (a), the energy spectrum has a unique ground state at total momentum (k1, k2) = (0, 0) (note that this is to show the competition between the wide band width and the correlation effects on the highest band). When r2 is introduced as −0.2, the highest band becomes flatter, leading to a drastic change in the energy spectrum. There appear three energy minima at (k1, k2) = (0, 0), (0, 2), and (0, 4), forming a threefold degenerate ground state manifold (GSM), which is separated from the other states by an energy gap ~0.03. As shown in Fig. 4c, the three sectors evolve with each other by inserting magnetic fluxes without having overlap with higher energy states (energy separation is slightly reduced to ~ 0.02). These results strongly suggest a ν = 1/3 FCI state.

Fig. 4: Emergence of a ν = 1/3 fractional Chern insulating state.
figure 4

Low-energy spectra of an interacting model with tδ = 0.5 with r2 = 0 (a) and r2 = −0.2 b Other parameter values are λ = 0.2, U = 2 (local Coulomb interaction), and V = 1 (nearest-neighbor Coulomb interaction). Ground state energy is indicated by black squares, and excited state energies are indicated by different symbols. c Spectral flow of the ground state manifold upon flux insertion with tδ = 0.5 with r2 = −0.2. Red solid lines, green dashed lines, and blue dash-dotted lines are for sector (k1, k2) = (0, 0), (0, 2), and (0, 4), respectively.

To confirm that this threefold degenerate ground state really represents a FCI state instead of trivial states such as charge density waves, we compute Chern numbers \({{{{{{{{\mathcal{C}}}}}}}}}_{({k}_{1},{k}_{2})}\) by introducing twisted boundary conditions5,58. Here, we discretize the boundary phase unit cell into 20 × 20 meshes, and numerically evaluate the Berry curvature \({F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})\) as detailed in the Methods section as well as in Supplementary Note 2. Figure 5 shows \({F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})\) in a discretized grid (n1, n2) for the GSM with tδ = 0.5, r2 = −0.2, λ = 0.2 with U = 2 and V = 1. Along the n1 direction, these plots are periodic. Along the n2 direction, plot (a) is continuously connected to plot (b), plot (b) is connected to plot (c), and plot (c) is connected back to plot (a). This also confirms the threefold GSM, where inserting one flux quantum along the b2 direction shifts the sector (k1, k2) = (0, 0) to (0, 2), (0, 2) to (0, 4), and (0, 4) to (0, 0). By adding up the discretized values of \({F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})\), we obtain \({{{{{{{{\mathcal{C}}}}}}}}}_{(0,0)}=0.331489\), \({{{{{{{{\mathcal{C}}}}}}}}}_{(0,2)}=0.330318\), \({{{{{{{{\mathcal{C}}}}}}}}}_{(0,4)}=0.338193\), and the sum of the three Chern numbers is exactly 1 within the numerical accuracy. The slight deviation from the ideal value \({{{{{{{\mathcal{C}}}}}}}}=1/3\) is ascribed to finite-size effects. This proves the existence of a ν = 1/3 FCI phase with a quantized fractional Hall response \({\sigma }_{{{{{{{{\rm{H}}}}}}}}}=\frac{1}{3}{e}^{2}/h\), where e is the electron charge and h is the Planck constant. In our numerical analyses, we did not find a ground state with the threefold degeneracy and Chern number zero, thus excluding the charge density wave states. This is probably because the quantum effects make such states unstable, as discussed in ref. 42.

Fig. 5: Many-body Berry curvature as a function of discretized boundary phases.
figure 5

a Sector (k1, k2) = (0, 0), b sector (0, 2), and c sector (0, 4). Parameter values are tδ = 0.5, r2 = −0.2, λ = 0.2 with U = 2 and V = 1.

Discussion

In this work, we have considered an itinerant electron model on a kagome lattice with twofold degenerate orbitals per site. However, each site has C2 rotational symmetry, rather than C3 or C4. Thus, the degeneracy between the two orbitals (yz and xz) can be lifted. In our tight-binding model, a difference in the hopping amplitude between tπ and tδ in fact lifts such degeneracy, leading to the splitting of the band structure. Thus, adding local crystal field splitting, which respects the underlying lattice symmetry, would not fully destroy the topological property found in this work, while the position of topological or flat bands would be modified depending on model parameters. As a number of kagome materials have already displayed a nontrivial band topology21,22,23,24,26,27,28,30,31,32,36,37,38,39, reducing the thickness of such materials down to a few unit cells, or growing thin films of such materials and tuning the Fermi level to a topological flat band by chemical substitution or gating, might be a promising route to observe the phenomena predicted here. The sign and the magnitude of the parameter r2 could depend on details of the material, such as the species of ligand ions, and might be further controlled by compressive or tensile strain. First principles calculations would help to construct realistic material-dependent models12,15,17. It is anticipated that the separation between {yz, xz}, {xy, x2 − y2}, or 3z3 − r2 subsets will be enhanced by reducing the film thickness compared with that in the bulk so that one can focus on one of the subsets only. In addition to a kagome lattice, topological flat bands appear in dice and Lieb lattices59,60,61. Study of FCI states in such lattice geometries and material search is another important direction.

To summarize, we have demonstrated the close interplay between the spatial frustration and the orbital degree of freedom in a kagome lattice. With the relativistic SOC, such an interplay not only affects the band dispersion, but also induces nontrivial topology. Specifically, we showed that the original flat bands in a kagome lattice become dispersive and topologically nontrivial. When such topological bands are fractionally occupied by electrons, many-body interactions drive further intriguing phenomena, i.e., fractional Chern insulating states. Our work may bridge the gap between idealized theoretical studies and real materials.

Methods

Non-interacting {y z, x z} model

Here we deduce the hopping matrices of the {yz, xz} model in the Slater–Koster approximation.

For nearest-neighbor bonds, in addition to the diagonal matrix \({\hat{t}}_{{{{{{{{\bf{1}}}}}}}}{{{{{{{\bf{2}}}}}}}}}\) presented in the main text, we have

$${\hat{t}}_{{{{{{{{\bf{13}}}}}}}}}=\, \frac{1}{4}\left[\begin{array}{cc}3{t}_{\pi }+{t}_{\delta }&\sqrt{3}({t}_{\pi }-{t}_{\delta })\\ \sqrt{3}({t}_{\pi }-{t}_{\delta })&{t}_{\pi }+3{t}_{\delta }\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{23}}}}}}}}}=\, \frac{1}{4}\left[\begin{array}{cc}3{t}_{\pi }+{t}_{\delta }&-\sqrt{3}({t}_{\pi }-{t}_{\delta })\\ -\sqrt{3}({t}_{\pi }-{t}_{\delta })&{t}_{\pi }+3{t}_{\delta }\end{array}\right].$$
(3)

Similarly, second neighbor hopping matrices can be written as

$${\hat{t}}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{(2)}=\, \left[\begin{array}{cc}{t}_{\pi }^{(2)}&0\\ 0&{t}_{\delta }^{(2)}\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{1\,3}}}}}}}}}^{(2)}=\, \frac{1}{4}\left[\begin{array}{cc}{t}_{\pi }^{(2)}+3{t}_{\delta }^{(2)}&-\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})\\ -\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})&3{t}_{\pi }^{(2)}+{t}_{\delta }^{(2)}\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{2\,3}}}}}}}}}^{(2)}=\, \frac{1}{4}\left[\begin{array}{cc}{t}_{\pi }^{(2)}+3{t}_{\delta }^{(2)}&\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})\\ \sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})&3{t}_{\pi }^{(2)}+{t}_{\delta }^{(2)}\end{array}\right],$$
(4)

where subscript (2) is introduced to highlight the difference from the nearest-neighbor bonds. These are schematically shown in Fig. 6. \({t}_{\pi }^{(2)}\) and \({t}_{\delta }^{(2)}\) correspond to (ddπ) and (ddδ), respectively, by Slater and Koster56.

Fig. 6: Second neighbor hopping matrices.
figure 6

yz(xz) orbitals between site 1 and site 2 are hybridized via diagonal hopping \({t}_{\pi (\delta )}^{(2)}\), i.e., π(δ) bonding. Other hopping integrals between site 1 and site 3 and between site 2 and site 3 are given by \({\hat{t}}_{{{{{{{{\bf{1\,3}}}}}}}}}^{(2)}\) and \({\hat{t}}_{{{{{{{{\bf{2\,3}}}}}}}}}^{(2)}\), respectively, obtained via the Slater rule56.

Non-interacting Berry curvature

The band-dependent Berry curvature of non-interacting electrons is given as a function of momentum k as

$${{{\Omega }}}_{n{{{{{{{\bf{k}}}}}}}}}={{{{{{{\rm{i}}}}}}}}\mathop{\sum}\limits_{m(\ne n)}\frac{\left\langle n\right|{\hat{v}}_{x{{{{{{{\bf{k}}}}}}}}}\left|m\right\rangle \left\langle m\right|{\hat{v}}_{y{{{{{{{\bf{k}}}}}}}}}\left|n\right\rangle -({\hat{v}}_{x{{{{{{{\bf{k}}}}}}}}}\leftrightarrow {\hat{v}}_{y{{{{{{{\bf{k}}}}}}}}})}{{({\varepsilon }_{m{{{{{{{\bf{k}}}}}}}}}-{\varepsilon }_{n{{{{{{{\bf{k}}}}}}}}})}^{2}},$$
(5)

where, using the Hamiltonian matrix in momentum space \({\hat{H}}_{{{{{{{{\bf{k}}}}}}}}}\), \({\hat{v}}_{\eta {{{{{{{\bf{k}}}}}}}}}\) is given by \({\hat{v}}_{\eta {{{{{{{\bf{k}}}}}}}}}=\partial {\hat{H}}_{{{{{{{{\bf{k}}}}}}}}}/\partial {k}_{\eta }\). With this Berry curvature, the band dependent Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\) is given by

$${{{{{{{{\mathcal{C}}}}}}}}}_{n}=\frac{1}{2\pi }{\int}_{{{{{{{{\rm{BZ}}}}}}}}}{{{{{\rm{d}}}}}}^{2}k\,{{{\Omega }}}_{n{{{{{{{\bf{k}}}}}}}}},$$
(6)

where the momentum integral is taken in the first Brillouin zone.

Many-body Chern number

The many-body Chern number is computed by introducing a twist boundary condition to a single-particle wave function as \(\psi ({{{{{{{\bf{r}}}}}}}}+{N}_{j}{{{{{{{{\bf{a}}}}}}}}}_{j})={{{{{\rm{e}}}}}}^{{{{{{{{\rm{i}}}}}}}}{\theta }_{j}}\psi ({{{{{{{\bf{r}}}}}}}})\), where Nj=1,2 are the numbers of unit cells along lattice translation vectors aj=1,2, with phase factors θj=1,2. This corresponds to inserting magnetic fluxes. When one flux quantum is inserted, θj changes from 0 to 2π and discretized momentum k moves from its original position to its neighbor along the bj direction with the momentum shift given by Δk = bj/Nj.

Many-body Chern number of the ground state (k1, k2) is computed via \({{{{{{{{\mathcal{C}}}}}}}}}_{({k}_{1},{k}_{2})}=\frac{1}{2\pi }\int\nolimits_{0}^{2\pi }{{{{{\rm{d}}}}}}{\theta }_{1}\int\nolimits_{0}^{2\pi }{{{{{\rm{d}}}}}}{\theta }_{2}{F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})\)58 where F(θ1, θ2) is the Berry curvature given by

$${F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})={{{{{{{\rm{Im}}}}}}}}\left\{\left\langle \frac{\partial {{{\Phi }}}_{({k}_{1},{k}_{2})}}{\partial {\theta }_{2}}\left|\frac{\partial {{{\Phi }}}_{({k}_{1},{k}_{2})}}{\partial {\theta }_{1}}\right.\right\rangle -({\theta }_{1}\leftrightarrow {\theta }_{2})\right\}.$$
(7)

Here, \(|{{{\Phi }}}_{({k}_{1},{k}_{2})}\rangle\) is the many-body wave function constructed using single-particle wave functions with a twist boundary condition ψ(r) after the Fourier transformation to momentum space. The momentum index (k1, k2) will be omitted in the following discussion for simplicity.

Partial derivative of a wave function with respect to θj is approximated by a finite difference as \(\left|\partial {{\Phi }}/\partial \theta \right\rangle \approx \frac{1}{| {{\Delta }}{{{{{{{\boldsymbol{\theta }}}}}}}}| }[\left|{{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}}+{{\Delta }}{{{{{{{\boldsymbol{\theta }}}}}}}})\right\rangle -\left|{{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}})\right\rangle ]\). Here, the vector notation is used for θ = (θ1, θ2), and Δθ = (Δθ1, 0) or (0, Δθ2). Then, it is required to compute a product of two wave functions as \(\left\langle {{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}})| {{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}}^{\prime} )\right\rangle\) with \({{{{{{{\boldsymbol{\theta }}}}}}}}\,\ne\, {{{{{{{\boldsymbol{\theta }}}}}}}}^{\prime}\). Because we are using a multiorbital model projected onto the flat band, special care is needed, as detailed in Supplementary Note 2.