Fragile topologically flat band in the checkerboard antiferromagnetic monolayer FeSe

By means of the first-principles calculations and magnetic topological quantum chemistry, we demonstrate that the low energy physics in the checkerboard antiferromagnetic (AFM) monolayer FeSe, very close to an AFM topological insulator that hosts robust edge states, can be well captured by a double-degenerate fragile topologically flat band just below the Fermi level. The Wilson loop calculations identify that such fragile topology is protected by the $S_{4z}$ symmetry, which gives rise to an AFM higher-order topological insulator that support the bound state with fractional charge $e/2$ at the sample corner. This is the first reported $S_{4z}$-protected fragile topological material, which provides a new platform to study the intriguing properties of magnetic fragile topological electronic states. Previous observations of the edge states and bound states in checkerboard AFM monolayer FeSe can also be well understood in our work.

As a wonderful platform to study the emergent phenomena among superconductivity, magnetism and topological bands, FeSe has generated much attentions recent years, in which high-T c superconductivity [36][37][38], topological superconductivity [39][40][41], topological states [42][43][44][45] have been reported. In particular, Wang et al. have reported that the monolayer FeSe in the checkerboard antiferromagnetic (cb-AFM) phase is a 2D AFM topological insulator (TI) [44]. After that, edge states are observed at the nematic domain boundaries of monolayer FeSe [45]. A bound state near Fermi level is further observed at the end of boundaries. However, the topological origin of the bound state has not been discussed, and the topological nature in the cb-AFM monolayer FeSe remains elusive until now.
In this work, by using the first-principles calculations and magnetic topological quantum chemistry [20][21][22], we systematically study the topological properties of the cb-AFM monolayer FeSe. We demonstrate that their low energy physics can be well captured by a doubledegenerate fragile topologically flat band just below the Fermi level, which gives rise to an AFM HOTI phase that support the bound state with fractional charge e/2 at the sample corner. The winding of square Wilson loop confirms that such fragile topology is protected by the S 4z symmetry. This makes cb-AFM monolayer FeSe the first reported S 4z protected fragile topological material, which may lead to intriguing responds to disorder and magnetic field. Further analysis find that cb-AFM monolayer FeSe is very close to a well-defined AFM TI with a nontrivial S z protected spin Chern number. These results can well explain the previous observations of topological edge states and bound state near the Fermi level [44,45], and also provide a new platform to study the intriguing properties of the magnetic fragile topology.
Crystal structure and computational method -As illustrated in Fig. 1a and 1b, both bulk and monolayer FeSe adopt the tetragonal lattice structure with the inversion symmetric space group P 4/nmm (No. 129), in which Fe atoms occupy the Wyckoff position 2a (0, 0.5, 0), and Se atoms occupy the Wyckoff position 2c (0, 0, z). In this work, we mainly focus on the band structures and topological properties in the cb-AFM phase of monolayer FeSe as shown in Figs. 1a,c-d. In this magnetic configuration, the nearest neighboring Fe ions are antiferromagnetically coupled to each other and the inversion symmetry is broken, leading to a magnetic space group P 4 /n m m (No. 129.416 [46]). P 4 /n m m is generated from a space-time inversion PT , a twofold screw C 2x ({Ĉ 2x | 1 2 , 0, 0}), and an improper fourfold rotation S 4z ({Ĉ 4zMz |0, 1 2 , 0}). Our first-principles calculations are performed by the Vienna ab initio simulation package (VASP) [47,48]. Similar as in Ref. [45], Perdew-Burke-Ernzerhof (PBE) type of the generalized gradient approximation + Hubbard U (GGA+U ) [49,50] is used as the exchange-correlation potential with U = 1.0 eV and J = 0.2 eV for the Fe 3d-orbitals. The crystal parameters a = 3.85Å and z = 0.255 are used in all calculations. The cut-off energy for the wave function expansion is set to 400 eV, and a 9×9×1 k-mesh in the first Brillouin zone (BZ) is used for self-consistent calculations. The SOC is considered selfconsistently. We then construct the Wannier functions without performing maximally localized procedure for the Fe 3d-orbitals using WANNIER90 [51].
Electronic structures and fragile topology -First, we carry out the non-SOC calculations of the cb-AFM monolayer FeSe, and plot the corresponding projected density of states (DOS) of the 3d-orbitals on Fe 2 ion (see Fig. 1a majority spin components as shown in Fig. 2a. These feature can also be represented by the spin-polarized band structures as shown in Fig. 2b, which indicates that five majority spin bands are fully occupied and only one minority band is occupied. Such occupancy corresponds to the 3d 6 configuration of the Fe 2+ ion very well. Further detailed orbital components analysis demonstrates that the one occupied minority band is mainly contributed by the d z 2 -orbital, except a band inversion with the d yz/xz -orbitals at the M point near the Fermi level (see Fig. 2c). The band inversion leading to a semimetallic feature with a parabolic touching at the M point as shown in Fig. 2b. When the SOC interaction is included, the gapless touching of d yz -and d xz -orbitals at M point is opened that arise an insulating gap about of 50 meV. Since the dispersion of the d z 2 band is very weak, the gap opening also leads to a very flat isolated band just locate below the Fermi level, as shown in Fig. 2c. We further note that each band is double-degenerate in the presence of PT symmetry.
To investigate the topological properties of the cb-AFM monolayer FeSe, we employ the symmetry analysis under the theory of magnetic topological quantum chemistry [20][21][22], in which the magnetic elementary band representations (MEBRs) as the basis of magnetic atomic insulators (AIs) are defined. Within this theory, if the irreducible co-representations (co-irreps) of the  [21,46]. Since the monolayer FeSe is 2D, only the z = 0 Wyckoff positions and kz = 0 high symmetry momenta points are considered. The MEBRs are denoted as D@w, which represent a band representation induced by the D orbital at Wyckoff position w.

MEBRs
Γ(0,0) occupied bands at high symmetry momenta points can be decomposed into the linear combination of MEBRs with positive integer, the system is equivalent to a magnetic AI. Otherwise, the system must be topologically nontrivial. We have calculated the co-irreps of all occupied bands in Fig. 2c, which indicate that cb-AFM monolayer FeSe is a magnetic AI, but not the AFM TI as reported in Refs. [44,45]. As pointed out in Refs. [21,28], the band structures of AI have two combination of manners: i) AI = AI +AI, ii) AI = AI + fragile topological bands. When we focus on the one isolated flat band just blow the Fermi level, the calculated co-irreps give rise to {Γ 7 , X 2 X 4 , M 6 }, and its fragile topological characteristic will be discussed below. Comparing to the MEBRs of the magnetic space group P 4 /n m m as tabulated in Table I, one can find that the co-irreps of the isolated flat band can only be expressed as a combination of MEBRs with negative integer: the negative coefficient manifests that it is a fragile topological band [28,52]. Such fragile topological characteristic can be usually revealed by the nontrivial winding of Wilson loop along some symmetric path over the whole BZ [53][54][55]. Here, we design a series of square loops l i that satisfy S 4z symmetry as shown by the upper right of Fig. 2d, with l 0 at the M point and l end corresponding to the boundary of the first BZ. The calculated Wilson loop spectra of the fragile topological band are plotted as red lines in Fig. 2d, which give rise to a conjugate pairs {e iθ , e −iθ } for each l i yielding to the PT symmetry [56]. More importantly, each branch of the Wilson loop spectra evolves a 2π phase from the M point to the whole BZ, leading to a nontrivial winding number 1. We also design the other series of rectangle loops that break S 4z symmetry as shown by the lower right of Fig. 2d. The corresponding Wilson loop spectra are plotted as blue lines in Fig. 2d, which obviously show that the crossing at θ = ±π is opened, resulting in a trivial winding number 0. Therefore, we conclude that the fragile topological band in cb-AFM monolayer FeSe is protected by the S 4z symmetry, which is first reported in this work. It may leads to different responds to disorder, comparing to the inversion symmetry protected fragile topological band [57,58].
In addition to the fragile topologically flat band, we would like to point out that the cb-AFM monolayer FeSe is very close to a stable AFM TI. Such stable AFM TI phase is protected by the spin conserving symmetry S z , which decouples the interactions between spin-up and spin-down flavor. In general, the S z symmetry is broken in realistic electron systems [59,60]. However, we demonstrate that the S z -breaking interaction in cb-AFM monolayer FeSe is very tiny, which can be verified by two bodies of evidence. First, we have symmetrized the Wannier HamiltonianĤ 0 of cb-AFM monolayer FeSe by an additional S z symmetry, i.e.Ĥ S = S zĤ0 S −1 z , and plot the band structures ofĤ S (red dash line) as a comparison to the bands ofĤ 0 (blue solid line) in Fig. 3a. The red bands in Fig. 3a match the blue bands very well, which strongly implys that theĤ 0 is approximate to having S z symmetry. The approximately S z symmetry is also consistent with our first-principles calculations, which display the weak hybridization between majority spin states and minority spin states as shown in Fig. 2a  and 2b. Second, one can quantitatively estimate the S z -breaking process as a competition between the SOC interaction λ soc and the on-site magnetic polarization splitting ∆, where λ soc regards the spin mixing ability and ∆ preserves spin conserving [54]. Reminding that the magnetic splitting ∆ in cb-AFM monolayer FeSe is estimated as ∼2.5 eV from our first-principles calculations, and the λ soc of Fe is usually considered as ∼0.03 eV. Such a huge energy difference makes that the S z symmetry is a good approximation in cb-AFM monolayer FeSe.
When S z symmetry is restored, the system can be considered as two copies of spin-Chern insulators that are connected by the PT symmetry. In Fig. 3b, we calculate and plot the edge states ofĤ S on the AFM boundary (cut #1 in Fig. 1c) as red lines, which shows that two edge states crossing each other exactly and form a massless Dirac cone at the Γ point. As a result, we confirm that theĤ S describes an AFM TI protected by the S z symmetry. As a comparison, we also plot the calculated edge spectra ofĤ 0 on #1 boundary as the blue lines in Fig. 3b. As expected, the edge spectra ofĤ 0 are very similar to those ofĤ S , except a very tiny massive gap about of 0.5 meV exists at the Γ point yielding to the S z symmetry breaking. These results confirm again that the S z -breaking is very weak in the cb-AFM monolayer FeSe, which is very close to a well-defined AFM TI. This is why the quantum spin Hall like edge states have been reported and observed in Refs. [44,45]. Apart from the edge states, the topological properties can also be revealed by the Wilson loop spectra of all occupied bands [61]. In Fig. 3c and 3d, we plot the Wilson loop spectra along the Γ − M path ofĤ S andĤ 0 , respectively. The spectra in Fig. 3c obviously show gapless crossing at the timereversal momenta points, reflecting a well-defined spin-Chern number |C s | = 1. However, the Wilson loop spectra ofĤ 0 are generally gaped, indicating that cb-AFM monolayer FeSe is not a well-defined AFM TI.
Corner States -The fragile topology usually relates to a HOTI and leads to filling anomaly [14]. Such filling anomaly could give rise to partially occupied corner states at the sample corner of a 2D HOTI [12,14]. In the system with S 4z symmetry, one can identify a HOTI by the S 4z eigenvalues of occupied bands [11,15]. We develop a Fu-Kane-like formula to compute the HOTI indicator Z as following, where ζ n (Γ) and ζ n (M ) are the S 4z eigenvalues of the nth-band at Γ and M point, respectively, ζ n (X) is the C 2z eigenvalue of the nth-band at X point. FM boundary (cut #2 and cut #3 in Fig. 1d) [62]. We calculate and plot the spin polarized edge states along the #2 and #3 boundary in Fig. 4a and 4b, respectively. In both of which, a massive gap can be observed between two opposite spin polarized edge states. Further analysis shows that the gap opening on the spin-up boundary (#2) is induced by a mass term m #2 ∝ σ y , while the gap opening on the spin-down boundary (#3) is induced by an opposite mass term m #3 ∝ −σ y , as shown in the inset of Fig. 4a and 4b, respectively. The opposite mass term is consistent with previous work that two boundaries are related by the C 4z T symmetry [11]. Therefore, one can expect that the mass term could cancel each other at the intersection of #2 and #3 boundary, and left zero-energy corner states surviving at 2c position.
In Fig. 4c, we plot the low energy spectra of a square sample with the center at 2c position constructed by 20 × 20 unit cells by using the the Wannier Hamiltonian H 0 . As expected, four corner states including two occupied and two unoccupied (blue dots in Fig. 4c) appear near the Fermi level. However, becauseĤ 0 has no chiral symmetry, the four corner states are not pinned at zero energy exactly. In Fig. 4d, we plot the spatial distribution of the two occupied corner states, which are localized at four corners and each corner has e/2 charge, corresponding to previous conclusions of the HOTI protected by C 4z T symmetry very well [15]. Finally, we emphasize that such calculated corner states are strongly supported by the previous experimental observations, in which bound state near the Fermi level has been observed at the crossing point of four domain walls by scanning tunneling microscopy [45]. We would like to suggest the experimenters to measure the charge of each bound state, which will be very helpful to confirm its topological origin.
Discussion -At last, we would like to discuss that the HOTI phase and corner states can only exist in the cb-AFM phase, but cannot be realized in other magnetic phases such as the nonmagnetic or stripe AFM phase. As we all know, the corner states are associated with the mismatch between the atomic sites and the charge centers [12]. As shown in Fig. 1c and 1d, the electrons on the nearest neighboring Fe ions are oppositely polarized in the cb-AFM configuration, which could strongly suppress the nearest neighbor hopping t 1 (see Fig. 1c), and makes the next nearest neighbor hopping t 2 domination. The dominant t 2 will lead to the charge centers move to the 2c site, and give rise to corner states at the sample corner. However, t 1 is always dominant in the NM or stripe AFM configuration, it makes the charge centers locate at the 4d site (see Fig. 1b), where the electrons can smoothly move to 2a site or 2c site. Therefore, our calculations, combining with the observation of the corner states [45], strongly suggest that the cb-AFM phase is realized in the few layers of the tensile FeSe.