Spectral signatures of the surface anomalous Hall effect in magnetic axion insulators

Abstract

The topological surface states of magnetic topological systems, such as Weyl semimetals and axion insulators, are associated with unconventional transport properties such as nonzero or half-quantized surface anomalous Hall effect. Here we study the surface anomalous Hall effect and its spectral signatures in different magnetic topological phases using both model Hamiltonian and first-principles calculations. We demonstrate that by tailoring the magnetization and interlayer electron hopping, a rich three-dimensional topological phase diagram can be established, including three types of topologically distinct insulating phases bridged by Weyl semimetals, and can be directly mapped to realistic materials such as MnBi₂Te₄/(Bi₂Te₃)_n systems. Among them, we find that the surface anomalous Hall conductivity in the axion-insulator phase is a well-localized quantity either saturated at or oscillating around e²/2h, depending on the magnetic homogeneity. We also discuss the resultant chiral hinge modes embedded inside the side surface bands as the potential experimental signatures for transport measurements. Our study is a significant step forward towards the direct realization of the long-sought axion insulators in realistic material systems.

Introduction

Magnetic topological systems have drawn significant attention recently due to their unconventional bulk transport properties and surface states¹. The topological properties of these magnetic topological phases are typically described by nontrivial bulk topological indices². On the other hand, the topological nature of the surface states also implies that there would be nontrivial surface transport behavior in these magnetic topological systems, which is closely related to their exotic bulk response properties. Typical examples are three-dimensional (3D) insulators with non-vanishing Chern–Simons orbital magnetoelectric coupling exhibiting effective axion electrodynamics, which are characterized by fractionalized surface anomalous Hall effect^3,4,5,6. If either time-reversal (T) or inversion symmetry (I) is present, the coupling phase angle θ must be quantized as 0 or π (modulo 2π), the latter of which (θ = π) with an energy gap at the surface is also known as “axion insulators” defined in 3D systems^3,4,7. Thus, the axion insulator exhibits a quantized bulk magnetoelectric coupling coefficient, which is equivalent to a half-quantized surface anomalous Hall conductivity (AHC) e²/2h^3,4.

3D T-preserved topological insulators (TIs) possess the θ = π condition. However, the resulting half-quantized surface AHC is totally compensated by the gapless surface Dirac cones of TIs⁸. Therefore, the T-preserved axion insulator phase was typically realized by introducing extrinsic magnetic dopants to the top and bottom surfaces of a 3D TI to gap the surface states^9,10,11,12, while keeping the bulk still nonmagnetic. On the other hand, bulk magnetic TIs with inversion symmetry are categorized as I-preserved axion insulators¹³. Examples include the recently discovered superlattice-like stoichiometric compounds MnBi₂Te₄/(Bi₂Te₃)_n in either ferromagnetic (FM) or antiferromagnetic (AFM) phases^{14,15,16,17,18,19,20,21}. Besides, this Van der Waals (VdW) layered material family also provides an ideal platform for realizing fruitful topological phases, such as Chern insulator, quantum spin Hall (QSH) insulator, and high-order TI^{22,23,24,25,26,27,28,29}.

Compared with the quantum anomalous Hall state where the topological nature is well established by the chiral edge states carrying a nonzero Chern number, the direct evidences of an axion insulator, such as the topological magnetoelectric effect or the resultant surface AHC, are much more challenging to measure^30,31. Conventional quantum transport measurements inevitably count the top and bottom surface together, giving rise to a (½ + ½) or (½ − ½) quantized AHC, depending on the relative magnetic orientation of the two surfaces. To avoid indistinguishable signature with quantum anomalous Hall effect, a T-preserved axion insulator typically requires different magnetic doping to the top and bottom surfaces. In a magnetic hysteresis loop, this slab setup would give rise to two quantum anomalous Hall states with opposite Chern numbers connected by an intermediate insulating phase with zero Hall plateau³². In even-layer MnBi₂Te₄ slabs, zero Hall plateau is observed as an indirect evidence of the I-preserved axion insulator phase²³. However, such zero Hall plateau can also be presented by a trivial case where both surface gaps are dominated by finite-size effect and thus do not contribute any surface anomalous Hall conductivity at all^33,34,35. Therefore, despite being a fascinating theoretical concept, some key issues about the surface AHC of axion insulators, such as the locality and the device design, still remain elusive. More importantly, a clear prediction of the unique signature of axion insulators that can be experimentally detected is still lacking.

In this work, by tuning the interlayer coupling and magnetization of a generic model Hamiltonian, we construct a topological phase diagram with direct mappings to 3D MnBi₂Te₄/(Bi₂Te₃)_n compounds, including axion insulators, Weyl semimetals, 3D Chern insulators, and 3D QSH insulators. Their distinctive surface AHC features are comprehensively studied. In addition to the model study, we then construct atomistic Hamiltonians from density-functional theory (DFT) with close reliance on realistic attributes of materials, yielding a direct comparison with the angle-resolved photoemission spectroscopy (ARPES) measurements for the band dispersions. By projecting the Chern number of a thick slab onto each VdW layer, we find that such real-space, local Chern marker in the axion insulator phase is well localized at the surface and results in a surface AHC either saturated at or oscillating around e²/2h, depending on the magnetic homogeneity. In comparison, the 3D Chern insulator phase, as well as the Weyl semimetal phase, does not manifest a well-defined surface AHC. Remarkably, we propose that the surface anomalous Hall effect in the axion insulator phase of MnBi₂Te₄/(Bi₂Te₃)_n leads to an unusual chiral hinge mode embedded in the side surface Bloch states, which clearly distinguishes from that in a trivial insulator. Such a clear signature is supposed to be detectable by appropriate surface transport measurements.

Multiple topological phases from model Hamiltonian calculations

To begin with, we tune the hopping parameter between the VdW layers in MnBi₂Te₄/(Bi₂Te₃)_n to realize different 3D topological phases, and illustrate that a well-localized, half-quantized surface AHC is the signature of the axion insulator phase. Recall that a MnBi₂Te₄ monolayer can be effectively described by a Bi₂Te₃ monolayer under a FM exchange field²⁷, we consider a 3D-layered structure composed by vertically stacking 2D TIs, i.e., bilayer Bi₂Te₃, with variable separation between bilayers (d) and magnetization (M). Thanks to the successful synthesis of single-crystal Mn–Bi–Te family and molecular beam epitaxy technique, such multilayer heterostructure could be realized by intercalating an atomic or VdW buffer layer, e.g., BN or In₂Se₃, into MnBi₂Te₄/(Bi₂Te₃)_n. The model Hamiltonian is written as

$$H= \ {{\hslash }}{v}_{f}{\tau }_{z}{\left({\boldsymbol{\sigma }}\times {\mathbf{k}}\right)}_{z}+{m}_{{\mathbf{k}}}{\tau }_{x}+M{\sigma }_{z}\\ +{t}_{\mathrm{IB}}^{0}\left({\upsilon }_{+}{\tau }_{-}+{\upsilon }_{-}{\tau }_{+}\right)+{t}_{\mathrm{IB}}\left({\upsilon }_{+}{\tau }_{+}{e}^{i{k}_{z}D}+{\upsilon }_{-}{\tau }_{-}{e}^{-i{k}_{z}D}\right),$$

(1)

where ${\bf{k}}=\big({k}_{x},{k}_{y},{k}_{z}\big)=\big({k}_{\parallel },{k}_{z}\big)$ is the momentum, $\sigma$, $\tau$ and $\upsilon$ are Pauli matrices acting on spin, surface, and layer, respectively, with ${s}_{\pm }=\big({s}_{x}\pm i{s}_{y}\big)/2$ ($s=\tau ,\upsilon$; here “surface” refers to the two surface states of each TI monolayer, which are coupled together by the hybridization term ${m}_{\mathbf{k}}{\tau }_{x}$). The first two terms describe a monolayer Bi₂Te₃ with Fermi velocity ${v}_{f}$ and Dirac mass ${m}_{{\bf{k}}}=\big(\Delta -B{k}_{\parallel }^{2}\big)$^36,37, while the third term denotes the FM exchange coupling between the magnetization M and electrons’ spin. The last two terms describe the intra-bilayer and inter-bilayer tunneling, respectively, with ${t}_{\mathrm{IB}}^{0}$ the intra-bilayer hopping integral and $D$ the superlattice period. For the inter-bilayer hopping, we introduce an exponentially decaying scaling, i.e., ${t}_{\mathrm{IB}}={t}_{\mathrm{IB}}^{0}\cdot {e}^{-\alpha \left(d-{d}_{0}\right)/{d}_{0}},$ where ${d}_{0}$ and $d$ are the intra- and inter-bilayer spacing. The inversion symmetry $I={\tau }_{x}{\upsilon }_{x}$ is preserved in this Hamiltonian. The model parameters and the analytical solutions of Eq. (1) are provided in Supplementary Note 1.

As shown in Fig. 1, the origin of the phase diagram represents a 3D Bi₂Te₃ TI phase. Varying the inter-bilayer coupling with preserved T leads to a ${{\mathbb{Z}}}_{2}$ classification, denoted by the horizontal axis. When the inter-bilayer coupling is weakened, the phase transition between strong TI (${{\mathbb{Z}}}_{2}$ = 1) and weak TI (${{\mathbb{Z}}}_{2}$ = 0) occurs. With increasing magnetization, the two TI phases evolve to axion insulator and 3D QSH insulator, respectively, the latter of which can be considered as a trivial stacking of T-broken QSH insulator³⁸. When the exchange field is strong enough, a 3D Chern insulator phase with chiral side surface states emerges, which is adiabatically connected to a vertical stacking of 2D Chern insulators^39,40,41. Under a finite exchange field, the transition between these three topologically distinct insulating phases inevitably passes an intermediate region, i.e., the Weyl semimetal phase⁴².

Fig. 1: Topological phase diagram.

Phase diagram of the multilayer topological heterostructure in terms of relative spacing $\left(d-{d}_{0}\right)/{d}_{0}$ and magnetization $M$ (rescaled as the number of Mn layers per VdW layer) derived from Eq. (1). Insets show the sketch of the heterostructure and ${{\mathbb{Z}}}_{4}$ indices with parities. Four pristine topological materials in FM phase Bi₂Te₃ ($M=0$), MnBi₄Te₇ ($M=1/2$), MnBi₂Te₄ ($M=1$), and Mn₂Bi₂Te₅ ($M=2$) are mapped at the vertical axis.

Depending on the magnetization per VdW layer, the different pristine FM MnBi₂Te₄/(Bi₂Te₃)_n compounds can be mapped onto the vertical axis of the phase diagram with d = d₀, as marked by the red stars in Fig. 1. The mapping follows two assumptions: (i) the exchange field in different MnBi₂Te₄/(Bi₂Te₃)_n materials is homogeneous; such assumption works well previously for the mapping of MnBi₂Te₄/(Bi₂Te₃)_n slabs to the 2D topological phase diagram²⁷. (ii) The band inversion only occurs at $\Gamma =\left(0,0,0\right)$and $Z=\left(0,0,\pi /D\right)$, which means that the band orders at the other inversion-invariant momenta remain the same as that of the nonmagnetic 2D limit ($M=0,D\to \infty$), i.e., bilayer Bi₂Te₃. Since the Hamiltonian in Eq. (1) has I, one can compute the symmetry indicator of I, a ${{\mathbb{Z}}}_{4}$ invariant, to determine their topological nature^43,44,45:

$${{\mathbb{Z}}}_{4}=\mathop{\sum }\limits_{{\mathbf{k}}=1}^{8}\frac{{n}_{{\mathbf{k}}}^{+}-{n}_{{\mathbf{k}}}^{-}}{2}\,{\mathrm{mod}}\ 4=\mathop{\sum}\limits_{{\mathbf{k}}=\Gamma,Z}{n}_{{\mathbf{k}}}^{-}\,{\mathrm{mod}}\ 4,$$

(2)

where ${n}_{{\bf{k}}}^{+}$/${n}_{{\bf{k}}}^{-}$ is the number of occupied states with even/odd parity at one of the eight inversion-invariant momenta k. By adding a small magnetization, each doubly degenerate band of the weak TI and strong TI phases splits into two bands with the same parity, leading to ${{\mathbb{Z}}}_{4}$ = 0 and ${{\mathbb{Z}}}_{4}$ = 2, respectively. The former, i.e., 3D QSH insulator, is equivalent to the vertical stacking of 2D T-broken QSH insulators with the parities shown in Fig. 1. Such a phase cannot be described by symmetric Wannier functions, while the Wannier obstruction can be removed upon adding a set of trivial elementary band representations (see Supplementary Note 2 for details). Therefore, it corresponds to a distinct type of “fragile topology”, manifesting a novel twisted bulk-boundary correspondence^46,47. The latter corresponds to an axion-insulator phase, as predicted by previous studies using a single ${{\mathbb{Z}}}_{4}$ invariant^18,19,28. Nevertheless, we find that the 3D Chern insulator phase also yields ${{\mathbb{Z}}}_{4}$ = 2 but a different parity distribution compared with the axion insulator. Therefore, the full indicator group of inversion symmetry ${{\mathbb{Z}}}_{4}\times {{\mathbb{Z}}}_{2}\times {{\mathbb{Z}}}_{2}\times {{\mathbb{Z}}}_{2}$ is required to further distinguish these two phases⁴⁸, where the ${{\mathbb{Z}}}_{2}$ indicators can be chosen as the parity of the Chern number in the k_i = π (i = x, y, z) plane. As shown in Fig. 1, the full symmetry indicator of the axion insulator phase and the 3D Chern insulator phase are 2:(000) and 2:(111), respectively.

In the phase diagram derived from Eq. (1), T symmetry is always broken, except for the horizontal line with M = 0. We find that that the three magnetic insulating phases are isolated from each other by an intermediate Weyl-semimetal phase, which is also consistent with the parity analysis. Our symmetry analysis based on DFT calculations shows that all the pristine FM MnBi₂Te₄/(Bi₂Te₃)_n (n = 0–3) compounds are I-preserved axion insulators. For comparison, we also calculate Mn₂Bi₂Te₅ where the magnetic moments per VdW layer are twice as that in MnBi₂Te₄, and obtain an unambiguous Weyl semimetal phase, as marked in Fig. 1.

DFT-calculated surface AHC of magnetic topological phases in MnBi₂Te₄/(Bi₂Te₃)_n

Having established the phase diagram, we next calculate the profile of the local Chern marker⁴⁹ of the topological phases with nontrivial ${{\mathbb{Z}}}_{4}$ numbers. In principle, one can define a local AHC ${\sigma }_{\mathrm{AHC}}={C}_{z}\cdot {e}^{2}/{h}$, with ${C}_{z}$ being the real-space projected Chern number in the z-direction. Using the Wannier-representation tight-binding Hamiltonians obtained by DFT-calculated Bloch eigenstates, we compute the local Chern marker ${C}_{z}\left(l\right)$ projected onto each VdW layer^50,51, expressed as

$${C}_{z}\left(l\right)=\frac{-4\pi }{A}{\mathrm{Im}}\frac{1}{{N}_{k}}\mathop {\sum }\limits _{k}\mathop {\sum }\limits _{v{v}^{{\prime} }c}{X}_{{vck}}{Y}_{{v}^{{\prime} }{ck}}^{\dagger }{\rho }_{v{v}^{{\prime} }k}\left(l\right),$$

(3)

where $X$ and $Y$ are the position operators along the x and y directions, respectively. ${\rho }_{{{vv}}^{{\prime} }}(l)$ is the projection matrix on to the corresponding layer $l$, which implies a summation over all atoms within a VdW layer (see Methods).

The results for the pristine FM MnBi₂Te₄ slabs are shown in Fig. 2a. We find that the integrated layer-projected Chern marker ${\mathbb{C}}\left(l\right)={\sum }_{l}{C}_{z}\left(l\right)$ is stabilized at 1/2 for $l \, > \, 2$, and rises up to 1 when it passes over the last two layers, giving rise to a Chern insulator as a whole with ${\mathbb{C}}=1$. In this sense, the penetration depth of the surface AHC is about two SLs, while the internal layers do not contribute to the AHC due to the homogeneous spin alignment. Such behavior is similar to the prediction of the axion insulator phase in a nonmagnetic bulk TI with a gapped surface⁷. In addition, the slabs with more than 4 SLs are thick enough to reveal the half-quantized AHC localized at both surfaces, indicating an I-preserved, FM axion insulator phase.

Fig. 2: Local Chern marker of various topological phases.

a–c Integrated local Chern marker ${\mathbb{C}}\left(l\right)$ for MnBi₂Te₄ slabs in a axion insulator, b 3D Chern insulator, and c Weyl semimetal phases. In panel a–c, the color lines terminate at different positions, denoting the total thickness of the calculated slabs up to 16 VdW layers. d, e Integrated local Chern marker ${\mathbb{C}}\left(l\right)$ as a function of layer index $l$ for a 16-layer slab of d MnBi₂Te₄ and e MnBi₄Te₇.

The FM MnBi₂Te₄ turns into a 3D Chern insulator when $d\ge 1.7{d}_{0}$, for which the Chern number of a 2D slice within the k_x–k_y plane at an arbitrary k_z equals to 1 (Supplementary Note 3). Despite sharing the same ${{\mathbb{Z}}}_{4}$ invariant with the axion insulator, the integrated local Chern number ${\mathbb{C}}\left(l\right)$ of the 3D Chern insulator behaves quite differently. As shown in Fig. 2b, each bilayer contributes to an exact quantized Chern number 1, giving rise to a ${\mathbb{C}}\left(l\right)$ proportional to $l$. Therefore, there is no well-defined surface AHC, instead, the total AHC of the slab is proportional to the total number of primitive cells in the slab.

We note in Fig. 1 that bulk FM MnBi₂Te₄ falls in the vicinity of the boundary between axion insulator and Weyl semimetal⁴², which is thus sensitive to numerical details such as the choice of exchange-correlation functionals and lattice constants^24,26. Figure 2c shows the corresponding ${\mathbb{C}}\left(l\right)$ for the Weyl semimetal phase obtained by applying a 1% lattice expansion. It is found that for the insulating slabs (Supplementary Note 3), the surface AHC is no longer quantized to 1/2 due to the bulk contribution. Especially for the slabs thicker than five VdW layers, the surface AHC contribution from the top and bottom two layers ranges from 0.42 to 0.35, while the internal-layer contribution increases linearly with the number of layers due to the bulk AHC⁵². For all these slabs, the distribution of the surface AHC is not localized at one surface, but extends to the center of the slab.

Half-quantized surface AHC in the axion insulator phases

We now focus on the axion insulator phase and its surface AHC in FM and AFM MnBi₂Te₄/(Bi₂Te₃)_n compounds. The magnetic ground states for MnBi₂Te₄, MnBi₄Te₇, and MnBi₆Te₁₀ (n = 0–2) are A-type AFM along the z-axis with the local moments of Mn FM ordered within each MnBi₂Te₄ layer^17,53, while MnBi₈Te₁₃ (n = 3) is a ferromagnet^18,21. From the perspective of ${{\mathbb{Z}}}_{4}$ indices, all of the above-mentioned compounds, no matter FM or AFM states, are I-preserved axion insulators according to our DFT calculations. For slab calculations, the total Chern number depends on whether I is preserved in the slab geometry. Therefore, for FM and odd-layer AFM MnBi₂Te₄ slabs with I, the total Chern number reaches to 1 because both top and bottom surfaces contribute the same AHC e²/2h. On the other hand, as shown in Fig. 2d, the topmost layer of the even-layer AFM phase (with broken I) contributes almost half-quantized AHC, i.e., ${C}_{z}\left(1\right)=0.49$, while the bottom layer contributes an opposite AHC ${C}_{z}\left(16\right)=-0.49$, leading to a zero Hall plateau state and a zero total Chern number. Starting from the second layer, ${\mathbb{C}}\left(l\right)$ no longer saturates but oscillates around 1/2 with a period of the unit cell in the z-direction, i.e., two VdW layers. Every additional layer contributes reversely to ${\sigma }_{\mathrm{AHC}}$ due to the flipping spin direction, leading to the oscillation with the amplitude as large as 0.21e²/h. Such behavior is in sharp contrast to T-preserved axion insulators proposed before. For the case of FM and AFM MnBi₄Te₇, the layer-projected AHC at MnBi₂Te₄ termination reaches the half-quantized value after three or four VdW layers, then oscillates around e²/2h, again, due to the inhomogeneity of the magnetic moments. The oscillation period is also determined by the thickness of a unit cell, i.e., 2 (4) VdW layers for FM (AFM) phase. The details are provided in Supplementary Note 4.

It is worthwhile to note that the half-quantized AHC of an axion insulator is a local property at the gapped surface⁵⁴. To demonstrate this, we consider a thick slab of FM MnBi₄Te₇, which is insulating for the MnBi₂Te₄ termination but metallic for the Bi₂Te₃ termination. We find that as long as the Fermi level (E_f) locates within the surface gap of the MnBi₂Te₄ termination, the corresponding surface AHC would stay around e²/2h. On the other hand, the surface AHC with the metallic Bi₂Te₃ termination varies with different choices of E_f (see Supplementary Note 4). Overall, the locality of the surface AHC does not rely on the metallicity of the whole slab, but is due to the vanishing contribution of the local Chern marker from the bulk state, i.e., ${\sum }_{l\in {internal}}^{l+u.c.}C\left(l\right)=0$. This can also be used as a criterion to distinguish the “surface” and “bulk” layers.

Spectral signatures of the surface AHC—hinge states

It is of great importance to consider how the computed local Chern marker and surface AHC corresponds to a measurable physical quantity², thus providing direct evidence for the axion-insulator phase. To be specific, one might wonder what kind of “mode” carries the half-quantized AHC in a realistic material. Unlike the 2D system with the well-defined 1D edge, 2D surfaces of a 3D material are terminated by hinges between different surfaces. While the top and bottom surfaces of MnBi₂Te₄/(Bi₂Te₃)_n are both gapped by the out-of-plane magnetization, the side surfaces are either gapless or gapped depending on the AFM or FM configuration, respectively. Combining DFT calculations and recursive Green’s function approaches (see Methods), we calculate the real-space local density of states (LDOS) of the surface and hinge states of a semi-infinite sample, as shown in Fig. 3. Fortunately, the hinge states of both AFM and FM axion-insulator phases provide effective signatures that could be detected by experiments. We next discuss the two cases separately.

Fig. 3: Spectra of surface and hinge states.

a Schematic of an axion insulator, with its top surface, top hinge, side surface, bottom hinge, and bottom surface marked as ①-⑤, respectively. b,e Schematic of the topological band spectra at spots ①-⑤ for b AFM and e FM MnBi₂Te₄. The orange and yellow lines denote in-band hinge and in-gap hinge states, respectively. c, f The hinge and the side surface LDOS of c AFM MnBi₂Te₄ (② and ③) and f FM MnBi₂Te₄ (② and ④). d The degree of chirality of the in-band hinge states with the Fermi level marked by the blue dashed lines in c and f.

For the side surface of AFM MnBi₂Te₄, a gapless Dirac cone occurs due to the combined symmetry $T{\tau }_{1/2}$, where ${\tau }_{1/2}$ denotes the half-cell translation along the stacking axis⁵⁵. Similar to the nonmagnetic TI, the manifestation of the bulk magnetoelectric response at the side surface is compensated by the opposite contribution from the surface Dirac cone, leading to vanishing side-surface AHC. Instead, there exists helical modes with the opposite spin channels propagating through opposite directions. The terminated AFM MnBi₂Te₄ sample and the calculated LDOS of the top hinge and side surface states are shown in Fig. 3a–c (position ② and ③). Compared with the helical gapless side surface states, we find that a remarkable feature of the hinge state is the asymmetric spectral weight between the left-and right-moving modes, indicating its chiral nature. We denote such chiral hinge modes embedded inside the side surface bands as “in-band hinge” states, and attribute them to the difference of the surface AHCs of the top and the side surfaces³¹ (see Supplementary Note 6 for details). For even-layer AFM slabs where the top and bottom surfaces have opposite magnetizations, the top and bottom in-band hinge states manifest opposite chiralities accordingly.

For FM MnBi₂Te₄, the side surface is gapped by the z-direction magnetization and a lower crystal symmetry, stemming from a high k-order effect of spin-momentum locking. In such a situation, the in-band hinge states still exist, with the top and bottom hinges having the same chirality, as shown in Figs. 3e and 3f. In addition, such a side surface exhibits a half-quantized local Chern marker, leading to a single chiral mode (½ + ½) at the top hinge and no chiral modes (½ − ½) at the bottom hinge, inside the side surface gap (position ② and ④, see Fig. 3f). Such a chiral hinge mode with integer AHC denoted as in-gap hinge mode, is equivalent to the chiral domain wall state in high-order topological insulators^56,57, which is also predicted in FM MnBi₂Te₄/(Bi₂Te₃)_n by model calculations²⁸. However, we note that such in-gap hinge mode only exists within a small energy range, i.e., 6 meV for FM MnBi₂Te₄ because its side surface gap is a high-order magnetization gap. On the other hand, the in-band hinge modes for both of the top and bottom hinge, originated from the half-quantized surface AHC, remain robust within a much larger energy range, i.e., the top/bottom surface gap (52 meV), which is favorable for experimental detection.

Besides the non-integer and integer topological charge, another feature to compare the in-band hinge states and the in-gap hinge states is their decay length in the real space, which clearly reflects their distinct topological origins. Figure 4 shows the LDOS of the hinge states projected onto different positions of the top surface (along the x-direction, Fig. 4a–e) and the side surface (along the z-direction, Fig. 4f–j), respectively. We find that the in-gap hinge state decays rapidly along the horizontal (x) direction at the top surface, while it decays much more slowly along the vertical (z) direction at the side surface. Specifically, five-unit-cell along the x-direction at the top surface is long enough for the in-gap hinge states to drop to below 1/10 of the maximal spectral intensity at the hinge, while at the side surface, the in-gap hinge states only decay to 1/7 of the maximal spectral weight within a thickness of 30 VdW layers along the z-direction. This seems counterintuitive due to the weak VdW interaction between layers along the z-direction. However, it can be understood by taking into account the surface bandgap: the side surface gap is much smaller than the top surface gap due to the magnetization direction, leading to a much longer wavefunction decay length⁵⁸. On the other hand, the in-band hinge states reverse their chiralities on different sides of the top surface (see Figs. 4b, 4d and 4e), and decay rapidly through both of the top and the side surfaces (blue curves in Figs. 4e and 4j), which agrees well with the locality of the surface AHC shown in Fig. 2d. Such consistency again demonstrates that the in-band hinge state is an ideal physical quantity to verify the existence of the surface AHC of the axion insulator phase. Note that the in-band hinge profile in Fig. 4j also distinguishes that of a FM trivial insulator and a Chern insulator on top of a trivial insulator.

Fig. 4: Real-space evolution of the hinge spectra.

Surface states and the evolution of the hinge states through the top surface a–e and the side surface f–j for FM axion insulator MnBi₂Te₄. a Top surface states. b–d LDOS of a sample with 20-unit-cell thick in the x-direction, including the right hinge b, center of the top surface c, and the left hinge d. e Intensity of the in-band (blue) and in-gap (red) hinge states projected onto different positions of the top surface as a function of the unit-cell index (#1 and #20 are the hinges, while #10 is the center of the computed cell). f Side surface states. g–i LDOS of a sample with 30-unit-cell thick in the z-direction (i.e., 30 VdW layers), including the top hinge g, center of the side surface h, and the bottom hinge i. j Similar to e but for the side surface.

The in-band hinge states will contribute to unique transport signatures when E_f crosses the side surface bands. As shown in Fig. 3d, compared with nonchiral top and side surface states, these localized chiral in-band hinge states exhibit imbalanced spectral weight for the left-moving and right-moving modes. We note that even for FM axion insulators, one can probe a specific hinge with only in-band hinge contribution (position ④ in Fig. 3f). We thus propose a device setup with a thick-enough sample and multiterminal leads attached to one surface covering only a few VdW layers. While the signal of in-band hinge modes is buried by the metallic side surface states for typical two-terminal transport measurements, one can expect nonzero signal through nonlocal surface transport measurements⁵⁹. Although the exact number of e²/2h conductance is not topologically protected and not immune from subtle device structure and disorder effects, the chiral in-band hinge states still give rise to unambiguous transport signature as the direct evidence of axion insulators, which is in sharp contrast to the case of a trivial insulator⁶⁰. We also provide a spectral comparison between a FM axion insulator and a FM trivial insulator in Supplementary Note 7.

Discussion

A gapped surface state is the kernel to realize the axion insulators. While recent ARPES measurements unexpectedly show an almost gapless surface Dirac cone in MnBi₂Te₄^60,61,62, we briefly provide two promising candidates with surface gaps in MnBi₂Te₄/(Bi₂Te₃)_n family in Supplementary Fig. 8. By comparison of our ARPES and DFT results, we can distinguish two different types of surface gaps depending on the specific terminations. Type I originates from the typical surface magnetization that introduces a ${M}_{z}{\sigma }_{z}$ term to a gapless Dirac fermion, exemplified by the (001) surface of MnBi₂Te₄, and the MnBi₂Te₄ termination of MnBi₂Te₄/(Bi₂Te₃)_n. Although the surface states of MnBi₂Te₄ are reported to be gapless due to the possible reconstruction of the geometric or magnetic configurations at the surface^60,61,62, there are still unambiguous surface gaps observed in various conditions, including Sb-doped MnBi₂Te₄ and the MnBi₂Te₄ termination of MnBi₈Te₁₃^21,63,64. Type II, on the other hand, is caused by the hybridization effect between the Dirac cone and the bulk valence band, exemplified by the Bi₂Te₃ termination of MnBi₄Te₇⁶⁵. Compared with the magnetization gap, the hybridization gap exchanges the orbital characters of the surface band and bulk band. Nevertheless, the broken T also gives rise to an imbalanced Berry curvature with opposite momenta and thus a half-quantized AHC, which is also provided in Supplementary Note 8.

To summarize, we demonstrate that the half-quantized surface AHC in MnBi₂Te₄/(Bi₂Te₃)_n series is well localized at a few layers from the top/bottom surfaces, which could be an experimental observable. The surface AHC is represented by the chiral hinge mode embedded inside the side surface bands, providing guidelines for nonlocal transport measurements. Our finding establishes an ideal platform to realize the long-sought axion states and the related topological magnetoelectric phenomena. Besides, the fruitful topological phase diagram, including 3D Chern insulators, Weyl semimetals, and fragile 3D QSH insulators, attributes new possibility to this family in the search of novel quantum materials.

Methods

DFT calculations

DFT calculations are performed using Vienna Ab-initio Simulation Package (VASP) ⁶⁶ to provide insights on electronic structures in the MnBi₂Te₄/(Bi₂Te₃)_n system. The generalized gradient approximation developed by Perdew, Burke, and Ernzerhof (PBE)⁶⁷ is used to describe the exchange-correlation energy in our calculation. The projector augmented wave (PAW) method is used to treat the core and valence electrons using the following electronic configurations: 3p⁶4s²3d⁷ for Mn, 5d¹⁰6s²6p³ for Bi, and 5s²5p⁴ for Te. The electron correlation effects of Mn-3d states are considered by the inclusion of the Hubbard U (PBE+U), with U(Mn) = 5 eV. The Brillouin zone is sampled by an 8 × 8 × 1 Γ-centered Monkhorst–Pack k-point mesh. Once the electronic structure is converged, the Bloch states are projected to the Wannier functions^68,69 of the Mn-3d, Bi-6p, and Te-5p orbitals to build the tight-binding Hamiltonian.

The calculation of local Chern number for a particular VdW layer ($l$) is systematically derived by Varnava et al. in ref. ⁵¹, and expressed as Eq. (3) in the main text, where $A$ is the unit cell area, ${X(Y)}_{{ijk}}=\frac{\left\langle {\psi }_{{ik}} \right| {i\hslash v}_{x\left(y\right)} \left| {\psi }_{{jk}}\right\rangle }{{E}_{{ik}}-{E}_{{jk}}}$ is the matrix element for the position operator along the $x$ or $y$ directions, and the band indices go through the conduction $(c)$ and valence $(v)$ bands as denoted for the summation. ${N}_{k}$ is the number of $k$-points and ${{\rho }}(l)$ the projection matrix onto the orbitals of the atoms within the corresponding VdW layer.

Calculation of the hinge states

We have employed two DFT-based methods to compute the hinge states, as plotted in Fig. 3 and Fig. 4. Both methods show consistent chiral states at the hinge of the MnBi₂Te₄ system. In Fig. 3, the hinge states are calculated using a bi-semi-infinite open-boundary geometry condition, which is also used in ref. ⁷⁰. The structure is semi-infinite along the $x$- and $z$- directions, while the periodic boundary condition (PBC) is maintained along the $y$-direction so that ${k}_{y}$ remains a good quantum number. The Hamiltonian can be written in terms of a quasi-block-tridiagonal form as follows:

$$H=\left(\begin{array}{ccccccccc}{H}_{0} & {H}_{1}^{x} & 0 & 0 & \cdots & {H}_{1}^{z}& 0 & \cdots & \ \\ {{H}_{1}^{x}}^{\dagger } & {H}_{0} & {H}_{1}^{x} &0 & \cdots & & & & \\ 0 & {{H}_{1}^{x}}^{\dagger } & {H}_{0} & {H}_{1}^{x} & \ddots & & & & \\ 0 & 0 & {{H}_{1}^{x}}^{\dagger } & {H}_{0} & \ddots & & & & \\ \vdots & \vdots & \ddots &\ddots & \ddots & & & & \\ {{H}_{1}^{z}}^{\dagger } & & & & & H_0 &{{H}_{1}^{z}} & 0 &\cdots \\ 0 & & & & & {{H}_{1}^{z}}^{\dagger } &H_0 &{{H}_{1}^{z}} &\ddots \\ \vdots & & & & & 0 & \ddots & \ddots & \ddots \end{array}\right),$$

(4)

where ${H}_{0},{H}_{1}^{x}$ and ${H}_{1}^{z}$ are the ground-state Hamiltonian and hopping matrices along the x- or z-directions, respectively. Such a tight-binding Hamiltonian is obtained from the maximally localized Wannier functions constructed by the wannier90 package interfaced to the VASP code. Then the Hamiltonian can be written into

$$H=\left(\begin{array}{cc}{H}_{0} & {H}_{I}\\ {H}_{I}^{\dagger } & {H}_{R}\end{array}\right).$$

(5)

One can immediately define a Green’s function for such Hamiltonian following the traditional scheme, i.e.,

$$G=\left(\begin{array}{cc}{G}_{0} & {G}_{I}\\ {G}_{I}^{\dagger } & {G}_{R}\end{array}\right).$$

(6)

Note that ${G}_{R}$ includes the surface Green’s function along both the x- and z-directions, which can be computed iteratively.

In Fig. 4, the evolution of the hinge states is investigated as a function of distance away from the hinge. Then a supercell with finite width along the x- (for the top surface) or the z-direction (for the side surface) is needed. The system geometries are a) For the top surface, finite along x (20-unit-cell-thick), semi-infinite along z, PBC along y; b) For the side surface, finite along z (30-unit-cell-thick), semi-infinite along x, PBC along y. Surface states are computed iteratively as the retarded Green’s function along the semi-infinite direction. More details can be found in Supplementary Note 5.