Magnetic wallpaper Dirac fermions and topological magnetic Dirac insulators

Abstract

Topological crystalline insulators (TCIs) can host anomalous surface states which inherits the characteristics of crystalline symmetry that protects the bulk topology. Especially, the diversity of magnetic crystalline symmetries indicates the potential for novel magnetic TCIs with distinct surface characteristics. Here, we propose a topological magnetic Dirac insulator (TMDI), whose two-dimensional surface hosts fourfold-degenerate Dirac fermions protected by either the ${p}_{c}^{{\prime} }4mm$ or $p{4}^{{\prime} }{g}^{{\prime} }m$ magnetic wallpaper group. The bulk topology of TMDIs is protected by diagonal mirror symmetries, which give chiral dispersion of surface Dirac fermions and mirror-protected hinge modes. We propose candidate materials for TMDIs including Nd₄Te₈Cl₄O₂₀ and DyB₄ based on first-principles calculations, and construct a general scheme for searching TMDIs using the space group of paramagnetic parent states. Our theoretical discovery of TMDIs will facilitate future research on magnetic TCIs and illustrate a distinct way to achieve anomalous surface states in magnetic crystals.

Introduction

The surface states of topological insulators (TIs) have anomalous characteristics that are unachievable in ordinary periodic systems¹. A representative example is the twofold-degenerate gapless fermion on the surface of three-dimensional (3D) TIs protected by time-reversal symmetry (TRS)^2,3,4,5. Contrary to the case of ordinary two-dimensional (2D) crystals with TRS in which gapless fermions appear in pairs, a single gapless fermion can exist on the surface of TIs through its coupling to the bulk bands. Such a violation of fermion number doubling^6,7,8 is a representative way in which the anomalous characteristics of surface states are manifested at the boundary of TIs.

In topological crystalline insulators (TCIs)^9,10, crystalline symmetries enrich the ways in which anomalous surface states are realized. For example, in systems with rotation symmetry and TRS, variants of the fermion doubling theorem enabled by symmetries can be anomalously violated on the surface of TCIs¹¹. Additionally, in the case of mirror-protected TCIs¹², although the number of surface gapless fermions can be even, the surface band structure exhibits a chiral dispersion along mirror-invariant lines such that anomalous chiral fermions appear in the one-dimensional (1D) mirror-resolved subspace of the 2D surface Brillouin zone (BZ). More recently, studies showed that in crystals with glide mirrors, the anomalous surface states can have an hourglass-type band connection¹³. Moreover, when the surface preserves two orthogonal glide mirrors, a single fourfold-degenerate Dirac fermion¹⁴ was shown to be achievable as an anomalous surface state¹⁵.

In magnetic crystals, there is great potential to achieve a new type of magnetic TCI with distinct anomalous surface states^{16,17,18,19,20,21,22} because there are abundant magnetic crystalline symmetries described by 63 magnetic wallpaper groups (MWGs) and 1421 magnetic space groups (MSGs)^23,24, which are overwhelmingly larger than the 17 wallpaper groups and 230 space groups of nonmagnetic crystals^{25,26,27,28,29}. Very recently, exhaustive studies of magnetic topological phases and their classification have been performed^16,17,18, and various novel magnetic topological phases have been systematically categorized. However, as far as we can tell, all the surface states of magnetic TCIs reported up to now appear in the form of twofold-degenerate gapless fermions, whose detailed band connection depends on the surface symmetry.

Here, we propose a magnetic TCI with fourfold-degenerate gapless fermions on the surface, coined the topological magnetic Dirac insulator (TMDI). A fourfold-degenerate gapless fermion, a Dirac fermion for short hereafter, can appear on the surface of a magnetic insulator when the MWG of the surface is one of the three MWGs $p{4}^{{\prime} }{g}^{{\prime} }m$, ${p}_{c}^{{\prime} }mm$, and ${p}_{c}^{{\prime} }4mm$, among 63 possible MWGs. Contrary to the surface Dirac fermion in nonmagnetic crystals protected by two orthogonal glides, our surface Dirac fermion is protected by symmorphic symmetries combined with either an antiunitary translation symmetry or an antiunitary glide mirror.

In particular, in magnetic crystals whose (001)-surface MWG is either $p{4}^{{\prime} }{g}^{{\prime} }m$ or ${p}_{c}^{{\prime} }4mm$, the bulk topology is characterized by the mirror Chern number (MCN) ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ about the diagonal mirror planes normal to either the [110] or $[1\bar{1}0]$ direction. Because of this, in TMDIs, the way in which the surface anomaly is realized is different from the case of the nonmagnetic Dirac insulator¹⁵ and more similar to the case of mirror-protected nonmagnetic TCIs¹². Namely, along the mirror-invariant line on the surface BZ, the Dirac fermion develops a chiral dispersion relevant to the MCN. Moreover, the MCN of TMDIs also induces hinge modes at open boundaries along the x and y directions, which respect diagonal mirrors.

Using first-principles calculations, we propose candidate materials for TMDIs, including Nd₄Te₈Cl₄O₂₀ and DyB₄. Since the database for magnetic materials only has a limited number of materials, we construct a systematic way to find candidate magnetic materials for TMDIs using the space group of paramagnetic parent compounds.

Results

Dirac fermions and magnetic wallpaper groups

In 2D magnetic crystals, Dirac fermions with fourfold-degeneracy can be symmetry-protected at the BZ corner, M = (π, π), by three MWGs, i.e., Type-III MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ and Type-IV MWGs ${p}_{c}^{{\prime} }mm$ and ${p}_{c}^{{\prime} }4mm$ [see Fig. 1a–c]. Here, we use the notation of Belov and Tarkhova (BT)²³ for denoting MWGs and the notation of Belov, Neronova, and Smirnova (BNS)³⁰ for denoting MSGs. Note that Type-III MWGs have antiunitary spatial symmetries combining TRS T with spatial symmetries, while Type-IV MWGs have antiunitary translation symmetries combining T and fractional lattice translations. All three MWGs have mirror-invariant lines, whose normal directions are $\hat{x}$, $\hat{y}$, or $\hat{x}\pm \hat{y}$.

Fig. 1: MWGs and fourfold-degenerate Dirac fermions.

a–c MWGs that protect fourfold degeneracy. Type-IV MWGs a ${p}_{c}^{{\prime} }mm$ and b ${p}_{c}^{{\prime} }4mm$, and c Type-III MWG $p{4}^{{\prime} }{g}^{{\prime} }m$. Black arrows represent spin configurations located at generic positions. The styles of lines indicate the types of symmetry elements: glides (blue dashed lines), antiunitary glides (red dashed lines), mirrors (blue solid lines), and antiunitary mirrors (red solid lines). d Typical band structure of 2D crystals belonging to MWG $p{4}^{{\prime} }{g}^{{\prime} }m$. Γ = (0, 0), X = (π, 0), M = (π, π), and ${\Gamma }^{{\prime} }=(2\pi ,2\pi )$ denote high-symmetry points. A fourfold-degenerate Dirac fermion appears at M near E = 0 (green arrow). Red (blue) lines denote states with eigenvalue +i ( − i) of the diagonal mirror. Along Γ-M-${\Gamma }^{{\prime} }$, each mirror eigenvalue sector of the Dirac fermion has a nonchiral dispersion. e Left: a Dirac fermion appearing as an anomalous surface state of a topological magnetic Dirac insulator (TMDI) with a chiral dispersion in each mirror sector. Right: schematic depiction of a TMDI, which hosts a fourfold-degenerate Dirac fermion on the (001) surface, and mirror-protected hinge modes on the sides invariant under diagonal mirrors.

In 2D systems belonging to the Type-III MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ described in Fig. 1c, a Dirac fermion is protected at M by twofold rotation about the z-axis C_2z, antiunitary glide mirror $T{G}_{y}\equiv T\{{m}_{y}| \frac{1}{2},\frac{1}{2}\}$, and off-centered diagonal mirror ${\widetilde{M}}_{x\overline{y}}=\{{m}_{x\overline{y}}| \frac{1}{2},{{{\rm{-}}}}\frac{1}{2}\}$. Here, the notation {g∣t} denotes the point group symmetry g followed by a partial lattice translation t. ${m}_{x\overline{y},y}$ are mirror symmetries that act on real-space coordinates as ${m}_{x\overline{y}}:(x,y,z)\to (y,x,z)$ and m_y: (x, y, z) → (x, − y, z). [See the conventions in Supplementary Note (SN) 1.] As detailed in SN 3, the fourfold degeneracy is formed by four states ψ_±, TG_yψ_±, ${\widetilde{M}}_{x\overline{y}}{\psi }_{\pm }$, and $T{G}_{y}{\widetilde{M}}_{x\overline{y}}{\psi }_{\pm }$, where ψ_± is an energy eigenstate with C_2z eigenvalue ±i.

In contrast, 2D systems belonging to the Type-IV MWGs ${p}_{c}^{{\prime} }mm$ and ${p}_{c}^{{\prime} }4mm$, described in Fig. 1a, b, respectively, have common symmetry elements, i.e., antiunitary translation ${T}_{G}=\{T| \frac{1}{2},\frac{1}{2}\}$ and two mirrors M_x = {m_x∣0} and M_y = {m_y∣0}, where m_x: (x, y, z) → ( − x, y, z). At M, these symmetry elements anticommute with each other, and ${T}_{G}^{2}=-1$. These relations protect the fourfold degeneracy formed by ψ_±, T_Gψ_±, M_yψ_±, and T_GM_yψ_±, where ψ_± has M_x eigenvalue ± i (see SN3). Note that the same symmetry representation was also studied in ref. ³¹.

A typical band structure supported by MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ is shown in Fig. 1d. Since MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ has diagonal mirror ${\widetilde{M}}_{x\overline{y}}$, the energy bands can be divided into two different mirror eigensectors along the $\Gamma M{\Gamma }^{{\prime} }$ direction. Here, Γ = (0, 0), and ${\Gamma }^{{\prime} }=(2\pi ,2\pi )$. Focusing on the band structures in each mirror sector, we find that the numbers of upward (chiral) and downward (antichiral) bands crossing the Fermi level [E = 0 in Fig. 1d] at M are the same. Otherwise, the mirror-resolved band structure cannot be periodic along $\Gamma M{\Gamma }^{{\prime} }$. Hence, a Dirac fermion in 2D crystals belonging to MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ is nonchiral in each mirror sector. Similar phenomena also occur in MWGs ${p}_{c}^{{\prime} }mm$ and ${p}_{c}^{{\prime} }4mm$. In general, the mirror-resolved dispersion of Dirac fermion in 2D crystals protected by MWGs is nonchiral. Although the local dispersion near the Dirac point may exhibit either chiral or nonchiral dispersion, the full band dispersion along the mirror invariant line is always nonchiral in 2D crystals.

Chiral surface Dirac fermions

A 2D Dirac fermion, which is nonchiral in 2D systems, can be chiral on the surface of 3D magnetic TCIs, as illustrated in Fig. 1e. Here, we systematically search for 3D magnetic insulators that can host a Dirac fermion on the (001) surface. As a 2D fourfold-degenerate Dirac fermion can be protected by one of the three MWGs $p{4}^{{\prime} }{g}^{{\prime} }m$, ${p}_{c}^{{\prime} }mm$, and ${p}_{c}^{{\prime} }4mm$, we focus on the MSGs whose (001) surface has one of these three MWGs. By studying the MSG symbols and the detailed surface symmetries, we find that there are at least 31 MSGs that can be generated from such MWGs and additional generators compatible with the MWGs. (See Supplementary Table 1.)

All 31 MSGs have mirror planes whose normal vectors are orthogonal to the (001) direction. Thus, the corresponding MCNs can give chiral dispersions along the mirror-invariant lines on the (001) surface. First, MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ has off-centered diagonal mirrors³²${\widetilde{M}}_{x\overline{y}}=\{{m}_{x\overline{y}}| \frac{1}{2},{{{\rm{-}}}}\frac{1}{2}\}$ and ${\widetilde{M}}_{xy}=\{{m}_{xy}| \frac{1}{2},\frac{1}{2}\}$ [see Fig. 1c]. For the 11 MSGs relevant to MWG $p{4}^{{\prime} }{g}^{{\prime} }m$, we define four MCNs ${{{{\mathcal{C}}}}}_{\pm }^{{k}_{x} = {k}_{y}}$ and ${{{{\mathcal{C}}}}}_{\pm }^{{k}_{x} = -{k}_{y}}$, which are defined in the k_x = k_y and k_x = − k_y planes, respectively. Here, the ± sign denotes the mirror eigenvalues of occupied bands. All the MCNs are equivalent up to sign because of the symmetry relations among ${\widetilde{M}}_{x\overline{y}}$, ${\widetilde{M}}_{xy}$, and TC_4z. Hence, the bulk topology can be classified by ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}\equiv {{{{\mathcal{C}}}}}_{+}^{{k}_{x} = {k}_{y}}=-{{{{\mathcal{C}}}}}_{-}^{{k}_{x} = {k}_{y}}$. Similarly, we can define MCNs for the 5 MSGs related to MWG ${p}_{c}^{{\prime} }4mm$, which have four mirrors, M_x, M_y, ${M}_{x\overline{y}}$, and M_xy. Among them, only two MCNs, ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and ${{{{\mathcal{C}}}}}_{m}^{y}\equiv {{{{\mathcal{C}}}}}_{+}^{{k}_{y} = 0}=-{{{{\mathcal{C}}}}}_{-}^{{k}_{y} = 0}$, are independent, and serve as bulk topological invariants. Finally, for the 15 MSGs related to MWG ${p}_{c}^{{\prime} }mm$, the relevant MCNs are ${{{{\mathcal{C}}}}}_{m}^{x}\equiv {{{{\mathcal{C}}}}}_{+}^{{k}_{x} = 0}=-{{{{\mathcal{C}}}}}_{-}^{{k}_{x} = 0}$ and ${{{{\mathcal{C}}}}}_{m}^{y}\equiv {{{{\mathcal{C}}}}}_{+}^{{k}_{y} = 0}=-{{{{\mathcal{C}}}}}_{-}^{{k}_{y} = 0}$. For more detailed discussions on the MCNs, see SN4.

Now, we classify the Wilson loop spectra^{33,34,35,36,37} according to the MCNs. We consider the k_z-directed Wilson loop ${{{{\mathcal{W}}}}}_{z}({{{{\bf{k}}}}}_{\perp })$,

$${{{{\mathcal{W}}}}}_{z}{({{{{\bf{k}}}}}_{\perp })}_{nm}=\left\langle {u}_{n}({{{{\bf{k}}}}}_{\perp },\pi )\left\vert \mathop{\prod }\limits_{{k}_{z}}^{\pi \leftarrow -\pi }{P}_{{{{\rm{occ}}}}}({{{{\bf{k}}}}}_{\perp },{k}_{z})\right\vert {u}_{m}({{{{\bf{k}}}}}_{\perp },-\pi )\right\rangle ,$$

(1)

where ${P}_{{{{\rm{occ}}}}}({{{\bf{k}}}})\equiv \mathop{\sum }\nolimits_{n = 1}^{{n}_{{{{\rm{occ}}}}}}\left\vert {u}_{n}({{{\bf{k}}}})\right\rangle \left\langle {u}_{n}({{{\bf{k}}}})\right\vert$ is a projection operator for occupied bands $\left\vert {u}_{n}({{{\bf{k}}}})\right\rangle$ and k_⊥ = (k_x, k_y). Since the Wilson loop is unitary, its eigenvalue can be collectively denoted as $\{{e}^{i\theta ({{{{\bf{k}}}}}_{\perp })}\}=\{{e}^{i{\theta }_{a}({{{{\bf{k}}}}}_{\perp })}| {\theta }_{a}({{{{\bf{k}}}}}_{\perp })\in (-\pi ,\pi ],a=1,\ldots ,{n}_{{{{\rm{occ}}}}}\}$. Then, {θ(k_⊥)} defines the Wilson loop spectrum, or equivalently, the Wilson bands. Wilson loop spectra and surface band structures have the same spectral features^35,38. Thus, the band structure on the (001) surface can be systematically classified based on Wilson loop analysis. (The details on tight-binding notation and Wilson loop is provided in SN2 and SN5.)

First, let us consider the MSGs related to MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ [see Fig. 2a–e]. At $\overline{M}=(\pi ,\pi )$, four Wilson bands form a fourfold degeneracy, which can be identified as a Dirac fermion on the (001) surface. The connectivity of Wilson bands is classified by the MCN ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$, which is encoded in the slope of Wilson bands in each mirror-sector crossing a horizontal reference line [a green dashed line in Fig. 2b] along the $\overline{\Gamma }$-$\overline{M}$ direction. In Fig. 2d, for example, as two Wilson bands with mirror eigenvalue − i intersect the reference line with a negative slope, we obtain ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=2$. See SN5 for the details on the counting rules for ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$.

Fig. 2: Wilson loop spectra of 3D magnetic TCIs with (001)-surface MWGs $p{4}^{{\prime} }{g}^{{\prime} }m$ and ${p}_{c}^{{\prime} }4mm$.

Classification of Wilson loop spectra, whose connectivity is equivalent to the (001)-surface band structure, based on the MCNs ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and ${{{{\mathcal{C}}}}}_{m}^{y}$. a–e Type-III TMDIs with MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ on the (001) surface. a (001)-surface BZ. ${\widetilde{M}}_{x\overline{y}}$ is the diagonal mirror used to define ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$. b–e Wilson loop spectra corresponding to b ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=0$, c ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=1$, d ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=2$, and e ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=4$. For convenience, the position of the Dirac fermion at $\overline{M}$ is adjusted to be located at θ(k) = 0. The red (blue) lines correspond to Wilson bands with mirror eigenvalue + i ( − i). The green dashed lines are the reference lines used to count the MCN ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$. In c–e, in which ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}\,\ne\, 0$, the dispersion of the Dirac fermion can be chiral in each mirror sector along $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$. Note that $\overline{{\Gamma }^{{\prime} }}=(2\pi ,2\pi )$. In b, where ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=0$, the Dirac fermion is nonchiral. In d, where ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=2$, the reference line is crossed by two chiral (upward) modes with mirror eigenvalue + i. In c, where ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=1$, the dispersion is chiral along the entire $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$ line but locally looks nonchiral near the Dirac point. The Wilson loop structure in c can be deformed into that in h by pushing the twofold crossing at $\overline{\Gamma }$ upward, which gives a locally chiral dispersion at $\overline{M}$. When $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}| \,>\, 2$, a Dirac fermion must appear with other gapless surface states along the $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$ line [black arrows in e]. f–j Type-IV TMDIs with MWG ${p}_{c}^{{\prime} }4mm$ on the (001) surface. f (001)-surface BZ. g–j Wilson loop spectra corresponding to g $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})=(0,0)$, h $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})=(1,-1)$, i $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})=(2,0)$, and j $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})=(0,2)$. Note that ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}={{{{\mathcal{C}}}}}_{m}^{y}$ (mod 2) holds for insulators.

Now, we compare the Wilson loop spectra of topological phases with nonzero ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and the trivial phase with zero ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ by focusing on the region near the fourfold degeneracy at $\overline{M}$. Along the $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$ line, the four bands are divided into two different mirror sectors. When ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=0$, as in Fig. 2b, the dispersion in each mirror sector is nonchiral, similar to that of Dirac fermions in 2D crystals in Fig. 1d. In Fig. 2b, chiral and antichiral modes in the same mirror sector (e.g., the +i sector) cross the green dashed reference line with opposite signs of the group velocity. In contrast, their numbers are not equal in Fig. 2d, where ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=2$.

The Dirac fermions in Fig. 2c, e, which correspond to ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=1$ and 4, respectively, appear with additional surface states (black arrows). When ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=1$, the dispersion is chiral along the entire $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$ line but locally looks nonchiral near the Dirac point. However, if the dispersion along $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$ is deformed such that the additional surface states near $\overline{\Gamma }$ are pushed away from the Fermi level (which corresponds to θ = 0 in Wilson loop spectra), the Dirac fermion in Fig. 2c becomes chiral, as in Fig. 2h. In contrast, such a deformation is impossible in Fig. 2e, where ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=4$. In general, one can show that a nonzero MCN $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}| \le 2$ manifests as a chiral dispersion of the surface Dirac fermion along $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$, provided that there is no additional surface state other than the Dirac fermion at the Fermi level. In contrast, when $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}| \,>\, 2$, additional surface states always appear along $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$. Hence, the chiral dispersion of the Dirac fermion when $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}| \le 2$ and the coexistence of additional surface states when $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}| \,>\, 2$ are signatures of the nontrivial bulk topology of 3D TMDIs with nonzero ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$. An exact formulation of the relation among the chiral dispersion of the Dirac fermion, MCN ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$, and number of additional surface states is given in SN6.

Similarly, one can analyze the Wilson loop spectra of the MSGs related to MWG ${p}_{c}^{{\prime} }4mm$ [see Fig. 2f–j]. The MCN ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ (${{{{\mathcal{C}}}}}_{m}^{y}$) can be determined by examining the ${M}_{x\overline{y}}$ (M_y) eigenvalues and the slopes of Wilson bands crossing a reference line along $\overline{\Gamma }$-$\overline{M}$ ($\overline{\Gamma }$-$\overline{X}$). The relation between ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and the chiral dispersion of the Dirac fermion is identical to the case of the MSGs with MWG $p{4}^{{\prime} }{g}^{{\prime} }m$. The only additional feature is that ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and ${{{{\mathcal{C}}}}}_{m}^{y}$ must be equivalent up to modulo 2, i.e., ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}={{{{\mathcal{C}}}}}_{m}^{y}$ (mod 2), in insulating phases.

Finally, in the 15 MSGs related to MWG ${p}_{c}^{{\prime} }mm$, the Wilson loop spectra can be classified by the MCNs on the k_x,y = 0 planes, ${{{{\mathcal{C}}}}}_{m}^{x}(0)$ and ${{{{\mathcal{C}}}}}_{m}^{y}(0)$, related to M_x,y mirrors. The MCNs on the k_x,y = π planes, ${{{{\mathcal{C}}}}}_{m}^{x}(\pi )$ and ${{{{\mathcal{C}}}}}_{m}^{y}(\pi )$, are always trivial, while surface Dirac fermions can be chiral only for nonzero ${{{{\mathcal{C}}}}}_{m}^{x,y}(\pi )$. Thus, the surface Dirac fermion in the MSGs with MWG ${p}_{c}^{{\prime} }mm$ is nonchiral and trivial.

Topological magnetic Dirac insulators

According to the Wilson loop analysis, the bulk band topology of the 11 MSGs with Type-III MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ and the 5 MSGs with Type-IV MWG ${p}_{c}^{{\prime} }4mm$ can be characterized by ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ and $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})$, respectively. In these 16 MSGs, when ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}\,\ne\, 0$, a Dirac fermion whose mirror-resolved dispersion is chiral can appear on the (001) surface. Based on this, we define TMDIs as 3D magnetic TCIs with nonzero ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ hosting a 2D chiral Dirac fermion on the (001) surface. Additionally, according to the (001)-surface MWG, TMDIs can be divided into Type-III and Type-IV TMDIs such that Type-III (Type-IV) TMDIs have (001)-surface MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ (${p}_{c}^{{\prime} }4mm$).

Interestingly, TMDIs also exhibit higher-order topology^{11,29,39,40,41,42,43,44,45,46} with hinge modes at open boundaries along the x and y directions when the entire finite-size systems respect the diagonal mirror symmetries. The MCN for the diagonal mirror ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ follows the higher-order bulk-boundary correspondence^41,42,45. The dispersion of hinge modes can be both chiral and helical depending on the details of the systems. Note that the number of chiral and antichiral hinge modes at each hinge can be changed by a mirror-symmetric and bulk-gap-preserving perturbation because such a perturbation can close and reopen a surface gap^42,44,47. However, the MCN ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ protects at least $| {{{{\mathcal{C}}}}}_{m}^{x\overline{y}}|$ hinge modes at each mirror-invariant hinge. (See SN7 for more details.) We provide a tight-binding model for a Type-III TMDI in SN10, which confirms the bulk-boundary correspondence described above.

Candidate materials

Using first-principles calculations, we propose DyB₄ and Nd₄Te₈Cl₄O₂₀ as candidate materials for a Type-III TMDI and a Type-IV TMDI, respectively, whose electronic and topological properties are summarized in Figs. 3 and 4, respectively. Although their band structure are metallic, as the systems have nonzero direct gap at all momenta, their mirror Chern numbers are well defined. We note that the two candidate materials, DyB₄ and Nd₄Te₈Cl₄O₂₀, are not available in the existing material databases^{20,48,49,50,51,52,53,54,55}. In general, searching for candidate magnetic materials is more challenging than searching for nonmagnetic materials because the number of available materials in magnetic material databases^20,54 is limited to approximately 1600⁵⁴, which is much smaller than the number of nonmagnetic materials. To overcome this limitation, we will also propose a general scheme to systematically search for candidate materials for TMDIs, whose magnetic structures and MSGs are derived from their parent paramagnetic states.

Fig. 3: DyB₄, a Type-III TMDI candidate.

a Tetragonal crystal structure of DyB₄. b, c Magnetic spin structures in the b Γ₂ and c Γ₄ states. The Γ₄ state corresponds to Type-III MSG 127.392 $P{4}^{{\prime} }/{m}^{{\prime} }{b}^{{\prime} }m$ with (001)-surface MWG $p{4}^{{\prime} }{g}^{{\prime} }m$. d Density functional theory (DFT) total energy differences between the Γ₄ and Γ₂ states. The absolute values are adjusted to be in the range between 0 eV and 1 eV. The magnetic phase transition is indicated by the brown solid line. The Γ₄ state has lower energy than the Γ₂ state when U≤ 6.5 eV. e, f Bulk band structure from DFT+U calculations (U = 6.5 eV and J = 1 eV) for the Γ₄ state. The band gap near the Fermi level is indicated by the red dashed lines. g, h Wilson loop spectra below the red dashed line along $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$. The winding structure exhibits g ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-2$ at U = 6.5 eV and h ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-1$ at U = 6.0 eV. i, j (001)-surface spectra for the i top (B-terminated) and j bottom (Dy-terminated) surfaces obtained using the surface Green’s function method. The white arrows guide where surface states appear. k, l (001)-surface band structures from a 60-layer slab calculation, which are drawn to show surface states indicated by the white arrows in i, j. Bulk and surface bands are represented by gray and black, respectively. Surface Dirac fermions are moved inside the gap by applying a surface potential of −0.2 eV on the top. i–l are calculated using the Wannierized tight-binding model from the DFT+U calculation (U = 6.5 eV).

Fig. 4: Nd₄Te₈Cl₄O₂₀, a Type-IV TMDI candidate.

a, b Crystal structures of NdTe₂ClO₅ and Nd₄Te₈Cl₄O₂₀ (2 × 2 × 1 supercell). c Magnetic structure compatible with Type-IV MSG 129.421 P_C4/nmm and (001)-surface MWG ${p}_{c}^{{\prime} }4mm$. d Bulk band structure from DFT+U calculations (U = 6 eV). The band gap near the Fermi level is indicated by the red dashed line. e, f (001)-surface spectra for the e top (O-terminated) and f bottom (Nd-terminated) surfaces obtained using the surface Green’s function method. The surface Dirac fermion is indicated by the white arrow in e. g k_z-directed Wilson loop spectrum along the diagonal path $\overline{\Gamma }$-$\overline{M}$-$\overline{{\Gamma }^{{\prime} }}$. The winding structure exhibits ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-1$. h–i (001)-surface for the h top and i bottom surfaces from a 30-layer slab calculation. Bulk and surface bands are represented by gray and black, respectively. The fourfold-degenerate Dirac fermions are located approximately 1 eV from the Fermi level on the top surface. A surface potential of −0.544 eV is introduced to push the Dirac point on the bottom surface into the gap.

Let us first consider DyB₄, a member of the rare-earth tetraborides family, whose crystal structure is shown in Fig. 3a. The paramagnetic parent phase of DyB₄ has space group 127 P4/mbm. In experiments⁵⁶, this material was reported to have two competing spin configurations that correspond to the Γ₂ (with MSG 127.395 $P4/{m}^{{\prime} }{b}^{{\prime} }{m}^{{\prime} }$) and Γ₄ (with MSG 127.392 $P{4}^{{\prime} }/{m}^{{\prime} }{b}^{{\prime} }m$) magnetic states, as shown in Fig. 3b, c. Between them, the Γ₂ state was reported to be more favored⁵⁶. Note that the Γ₄ phase can support a Type-III TMDI with (001)-surface MWG $P{4}^{{\prime} }{g}^{{\prime} }m$. Here, we investigate the conditions in which the Γ₄ state becomes the magnetic ground state in DyB₄ based on DFT+U(where U is the onsite Coulomb interaction) calculations. We examine the total energies of the Γ₂ and Γ₄ states as a function of U with fixed Hund’s coupling J = 1 eV. As shown in Fig. 3d, DyB₄ undergoes a magnetic phase transition from the Γ₄ to Γ₂ state when U > 6.5 eV.

An indirect gap near the Fermi level E_F=0.0 eV exists in the DFT+ U result (U = 6.5 eV, J=1.0 eV), as shown in Fig. 3e. Because the gap along the Γ-Z line is tiny, as shown in Fig. 3f, a topological phase transition between phases with different MCNs ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ can be easily induced by a small perturbation. (See SN9 for the band structures between U=6.0 eV and 6.5 eV and SN12 for a detailed analysis of this topological phase transition.) Fig. 3g, h show the k_z-directed Wilson loop spectra for U = 6.5 eV and 6.0 eV, respectively. Because of gap closing and reopening at Z, a topological phase transition from the ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-1$ state to the ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-2$ state occurs as U increases from 6.0 eV to 6.5 eV. Hence, we identify DyB₄ at U = 6.5 eV (6.0 eV) as a Type-III TMDI with ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-2$ ( − 1).

More specifically, let us consider DyB₄ at U = 6.5 eV. Figure 3i–j show the surface Green’s function calculations for the top (B-terminated) and bottom (Dy-terminated) surfaces. A surface Dirac fermion with fourfold degeneracy is clearly identified near E = − 0.35 eV on the bottom surface [Fig. 3j, l]. In contrast, the top surface state is buried in the bulk conduction band and thus not visible in the gap. However, the hidden surface state can be brought into the gap by applying a surface potential E_surf = − 0.2 eV, as shown in Fig. 3k.

The second material Nd₄Te₈Cl₄O₂₀ is a candidate for a Type-IV TMDI with $({{{{\mathcal{C}}}}}_{m}^{x\overline{y}},{{{{\mathcal{C}}}}}_{m}^{y})=(-1,1)$. Since its magnetic properties are unknown, this material is found by using our general scheme to search for candidate materials, as briefly explained below. MSG 129.421 P_C4/nmm (123.16.1014 ${P}_{P}4/{m}^{{\prime} }mm$ in the OG setting) is one of the MSGs that can host Type-IV TMDIs, which is derived from nonmagnetic space group (SG) 123 P4/mmm. The crystal structure of NdTe₂ClO₅⁵⁷ in the paramagnetic phase has SG 123 P4/mmm, as shown in Fig. 4a. Introducing a 2 × 2 × 1 supercell, Nd atoms can form a spin configuration compatible with MSG 129.421 P_C4/nmm, as shown in Fig. 4b, c. Assuming the magnetic ordering corresponding to MSG 129.421 P_C4/nmm, we study the electronic and topological properties of Nd₄Te₈Cl₄O₂₀. The bulk band structure in Fig. 4d shows a semimetallic state with an indirect gap between − 0.5 eV and 0.0 eV. For the bands below this gap, we compute the k_z-directed Wilson loop spectrum, which exhibits a nonzero MCN, ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=-1$, as shown in Fig. 4g. Hence, we identify Nd₄Te₈Cl₄O₂₀ as a candidate for a Type-IV TMDI with (001)-surface MWG ${p}_{c}^{{\prime} }4mm$.

The surface Green’s function calculations for the top (O-terminated) and bottom (Nd-terminated) surfaces are shown in Fig. 4e, f. Within a gap with a magnitude of 1.5 eV at $\overline{M}$, a surface Dirac fermion with fourfold degeneracy is clearly identified near E = 1 eV on the top surface. As shown in Fig. 4h, the four states of the surface Dirac fermion are split into two nondegenerate bands and accidentally nearly degenerate bands, in a similar way as in Fig. 2h. Although bottom surface states buried in bulk states are not visible in the gap [Fig. 4f], the hidden surface state can be pushed into the gap by applying a surface potential of − 0.544 eV, as shown in Fig. 4i.

Our material search scheme used to find Nd₄Te₈Cl₄O₂₀ can generally be applied to all 16 MSGs (11 Type-III and 5 Type-IV MSGs) that can host TMDIs, as follows. For each MSG, (i) determine the SG of paramagnetic phases, (ii) obtain the transformation matrix of basis vectors between the magnetic unit cell and crystal unit cell, and (iii) build up the magnetic unit cell from the paramagnetic unit cell according to the transformation matrix as well as restrict the spin configuration of magnetic atoms. (See SN8 for details.) Additionally, using this material search scheme, we studied other candidate materials including two TMDI candidates with nonzero ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ (Ce₂Pd₂Pb, Ce₂Ge₂Mg) and four materials with ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}=0$ (HoB₄, FeSe, FeTe, AlGeMn, N₃TaTh). (See SN9 for details.)

Discussion

Let us discuss the symmetry-breaking effect on the 2D Dirac fermion, either in 2D bulk crystals or on the surface of 3D TMDIs. When a perturbation that preserves diagonal mirrors in MWGs $p{4}^{{\prime} }{g}^{{\prime} }m$ and ${p}_{c}^{{\prime} }4mm$ is applied, the Dirac fermion in 2D crystals becomes gapped while that on the surface of 3D TMDIs remains gapless because of the nonzero MCN. In contrast, a mirror-breaking perturbation can induce various phase transitions in both cases because a 2D Dirac fermion corresponds to a multicritical point in magnetic systems³¹. For example, MWG $p{4}^{{\prime} }{g}^{{\prime} }m$ is reduced to $p2{g}^{{\prime} }{g}^{{\prime} }$ when ${\widetilde{M}}_{x\overline{y}}$ and TC_4z are broken by tensile strain (see SN13). In this case, a mass term is allowed; thus, the Dirac fermion can be gapped. The Chern number of the resulting gapped phase changes by 2 when the sign of the mass is reversed. This, in turn, indicates the appearance of chiral edge channels at domain walls between two gapped domains with opposite signs of the mass.

Finally, we discuss the bulk topological responses of TMDIs. Since mirror symmetry reverses the spatial orientation, it quantizes the axion angle θ to 0 or π, which is related to the MCN by $\theta /\pi ={{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ (mod 2)^18,58. Hence, TMDIs with odd ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ correspond to an axion insulator exhibiting quantized magnetoelectric effects⁵⁹. In contrast, TMDIs with an even ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$ have a vanishing axion angle. However, according to a recent theoretical proposal⁶⁰, even helical higher-order topological insulators (HOTIs) with θ = 0 can exhibit spin-resolved magnetoelectric effects. In the case of 3D TMDIs, spin-resolved bands are ill-defined because of noncollinear magnetic orderings inherent in Type-III and Type-IV MSGs. However, by combining the spin and sublattice degrees of freedom, the pseudospin-resolved magnetoelectric effect can be defined in 3D TMDIs with an even ${{{{\mathcal{C}}}}}_{m}^{x\overline{y}}$, which is demonstrated using a tight-binding model in SN11. A more systematic study of bulk topological responses in magnetic TCIs with noncollinear magnetic ordering is an important subject for future research.

Methods

For the ab initio calculation, the Vienna Ab initio Simulation Package (VASP) is employed with the projector augmented-wave method (PAW)⁶¹. We employ the generalized gradient approximation (PBE-GGA) for exchange-correlation potential⁶². The default VASP potentials, an energy cutoff with 400 eV, and a 8 × 8 × 14 Monkhorst-pack k-point mesh are used. The spin–orbit coupling is considered because of the presence of the heavy rare-earth atoms in the unit cell. On-site Coulomb interaction is taken into account with U = 6.5 eV and J (Hund’s coupling) = 1 eV in DyB₄ and U = 6 eV and J = 0 eV in NdTe₂ClO₅. Wannier Hamiltonians were constructed by WANNIER90⁶³ and symmetrized by WannSymm code⁶⁴. The WannierTools package⁶⁵ was used to produce the slab band structure and Wilson loop spectra based on the symmetrized Hamiltonian. The experimental crystal structure of DyB₄⁵⁶ are used without structural relaxation. Details on the ab-initio calculation can be found in SN S9.