Nonmagnetic band gap at the Dirac point of the magnetic topological insulator (Bi_1−xMn_x)₂Se₃

Abstract

Magnetic doping is expected to open a band gap at the Dirac point of topological insulators by breaking time-reversal symmetry and to enable novel topological phases. Epitaxial (Bi_1−xMn_x)₂Se₃ is a prototypical magnetic topological insulator with a pronounced surface band gap of ∼100 meV. We show that this gap is neither due to ferromagnetic order in the bulk or at the surface nor to the local magnetic moment of the Mn, making the system unsuitable for realizing the novel phases. We further show that Mn doping does not affect the inverted bulk band gap and the system remains topologically nontrivial. We suggest that strong resonant scattering processes cause the gap at the Dirac point and support this by the observation of in-gap states using resonant photoemission. Our findings establish a mechanism for gap opening in topological surface states which challenges the currently known conditions for topological protection.

Introduction

The topological surface state (TSS) of the three-dimensional topological insulator (3D TI) Bi₂Se₃ forms a Dirac cone in the band dispersion E(k_||) of electron energy E versus wave vector component k_|| parallel to the surface¹. Differently from graphene, the electron spin is nondegenerate and locked to k_||, and the Dirac crossing point is protected by time-reversal symmetry (TRS) against distortions of the system^1,2. This stability has been theoretically investigated demonstrating that nonmagnetic impurities at the surface and in the bulk can form a resonance in the surface-state density of states (DOS) without opening a band gap^3,4,5. A magnetic field breaks TRS lifting the Kramers degeneracy E(k_||, ↑)=E(−k_||, ↓) between up (↑) and down (↓) spins, and can open a band gap at the Dirac point¹. For the two-dimensional quantum-spin-Hall system HgTe, the effect of a perpendicular magnetic field on the topologically protected edge state has been demonstrated successfully⁶, and similar effects are expected from magnetic impurities in this system⁷. In subsequent studies on HgTe, the effect of the magnetic field was much smaller, and recently in an inverted electron-hole bilayer from InAs/GaSb, helical edge states proved robust in perpendicular magnetic fields of 8 T (ref. 8). At the surface of a 3D TI, calculations show that magnetic impurities can open a gap at the Dirac point and exhibit ferromagnetic order with perpendicular anisotropy mediated by the TSS through Ruderman–Kittel–Kasuya–Yosida (RKKY) exchange coupling^4,9,10.

3D TIs with magnetic impurities have been studied by angle-resolved photoelectron spectroscopy (ARPES). The magnetic impurities have been incorporated into the bulk¹¹ and at the surface of Bi₂Se₃ (ref. 10). For Fe impurities, the opening of large surface band gaps of ∼100 meV was reported in both cases^10,11. In a previous work, however, we found that Fe deposited on Bi₂Se₃ does not open a gap for a wide range of preparation conditions, revealing a surprising robustness of the TSS towards magnetic moments¹². This conclusion also extends to Gd/ Bi₂Se₃ (ref. 13) and Fe/Bi₂Te₃ (ref. 14).

The absence of a surface band gap is consistent with the recent finding that Fe on Bi₂Se₃ does not favour a magnetic anisotropy perpendicular to the surface, at least in the dilute limit¹⁵. In this respect, bulk impurities in TIs have an advantage over surface impurities in that ferromagnetic order and perpendicular magnetic anisotropies have been achieved in the bulk systems^{11,16,17,18,19,20,21}. Fe incorporated in bulk Bi₂Te₃ is known to order ferromagnetically with a Curie temperature (T_C) of ∼12 K for concentrations of x=0.04 showing an easy axis perpendicular to the base plane^16,17. On the other hand, (Bi_1−xFe_x)₂Se₃ has not been found to be ferromagnetic at temperatures above 2 K (ref. 17). For x=0.16 and 0.25, ferromagnetic order was found at 2 K (ref. 11). By ARPES²², surface-state band gaps in (Bi_1−xFe_x)₂Se₃ were reported and assigned to a magnetic origin for Fe concentrations from x=0.05 to 0.25 including nonferromagnetic concentrations such as x=0.12 with a band gap of 45 meV (ref. 11). These band gaps have been attributed to short-range magnetic order. Concerning Mn incorporation, bulk (Bi_1−xMn_x)₂Se₃ with x=0.02 has been shown to exhibit a surface band gap with an occupied width of 7 meV (ref. 11), which was suggested as an indication of ferromagnetic order induced by the TSS¹¹. However, much larger surface band gaps were observed for n-doped (Bi_1−xMn_x)₂Se₃ films²⁰, where ferromagnetic order of Mn at the surface was found to be strongly enhanced with T_C up to ∼45 K. The ferromagnetic order resulted in an unusual spin texture of the TSS. The T_C at the surface was much higher than in the bulk²⁰, which was partially attributed to Mn surface accumulation. A strong enhancement of the surface T_C was also predicted by mean-field theory for this system (for example, from 73 K (bulk) to 103 K (surface))²³. Magnetically doped TIs with ferromagnetic order are important as realizations of novel topological phases. When spin degeneracy is lifted by the exchange splitting, bulk band inversion can occur selectively for one spin. If also the Fermi level is in the band gap, as predicted for Cr and Fe in Bi₂Se₃ (ref. 24), this gives rise to an integer quantized Hall conductance σ_xy in thin films termed the quantized anomalous Hall effect^7,24. This has recently been reported for Cr in (Bi,Sb)₂Te₃ (ref. 25). When a perpendicular magnetization breaks TRS at the surface of a bulk TI, the resulting mass and surface band gap give rise to quantized edge states. In this case σ_xy is half-integer quantized yielding a half quantum Hall effect²⁶ which can be probed at ferromagnetic domain boundaries at the surface²⁷ and leads to exotic topological magnetoelectric effects such as the point-charge-induced image magnetic monopole^28,29. Another topological phase predicted for magnetically doped Bi₂Se₃ is the realization of a 3D Weyl fermion system in which the Dirac-like dispersions become a property of the bulk³⁰.

Here we present a detailed investigation on the correlation between the TSS and magnetic properties in the Bi-Mn-Se system, where the magnetic impurity Mn is introduced in the bulk of ∼0.4 μm thick (Bi_1−xMn_x)₂Se₃ epilayers with concentrations up to x=0.08. By ARPES we find remarkably large band gaps of ∼200 meV persisting up to temperatures of 300 K far above T_C, which is found to be ∼10 K at the surface showing only a limited enhancement by ≤4 K over the bulk T_C. We find that the magnitude of the band gaps by far exceeds that expected from TRS breaking due to magnetic order. By ARPES from quantum-well states we exclude that Mn changes the inverted bulk band gap. Using resonant photoemission and ab initio calculations we conclude that, instead, impurity-induced resonance states destroy the Dirac point of the TSS. We further support our conclusion by showing that extremely low-bulk concentrations of nonmagnetic In do also open a surface band gap in Bi₂Se₃. Our findings have profound implications for the understanding of the conditions for topological protection.

Results

Concentration and temperature dependence of surface band gap

Figure 1 shows Mn-concentration-dependent high-resolution ARPES dispersions of the TSS and bulk valence-band (BVB) states of (Bi_1−xMn_x)₂Se₃ with x up to 0.08 measured at 12 K and 50 eV photon energy. At this energy, contributions from the bulk-conduction band (BCB) do not appear due to the dependence of the photoemission transitions on the component of the electron wave vector perpendicular to the surface k_⊥ (ref. 22), but the BCB is partially occupied as data at 21 eV photon energy reveal (Supplementary Fig. 1). The pure Bi₂Se₃ film (Fig. 1a) is n-doped and exhibits a well-resolved Dirac point with high photoemission intensity at a binding energy of E_D ∼0.4 eV that is seen as an intense peak in the energy-distribution curve at zero momentum superimposed as red curve on the right-hand side of the panel. The intact and bright Dirac point marked by a horizontal dashed line in Fig. 1a demonstrates that the TSS is gapless in Bi₂Se₃. At larger binding energies, the lower half of the Dirac cone overlaps with the BVB, which is observed for all (Bi_1−xMn_x)₂Se₃ samples (Figs 1b–d). Increasing Mn concentration, we find a gradual upward shift of the band edges in energy, revealing a progressive p-type doping (hole doping). Most notably, a surface band gap opens at the Dirac point with increasing Mn content. In each energy-distribution curve, the opening of the gap is also evident from the development of an intensity dip at the binding energy of the original Dirac point. Thus, the energy separation between the upper Dirac band minimum and the energy position of the intensity dip is about half of the gap size. Note that the horizontal dashed lines in Figs 1b–d highlight the minimum limit of the gap size after taking into account the contribution from the linewidth broadening to the ARPES spectra (see Supplementary Note 1 and Supplementary Fig. 2). The surface band gap rapidly increases with Mn content and exceeds 200 meV for x=0.08. For x=0.02, the suppression of the Dirac point intensity as compared with the undoped Bi₂Se₃ film indicates the existence of a small gap. This is consistent with the fact that the TSS dispersion becomes slightly parabolic and is characterized by an increased effective mass of m* ≈ 0.09m_e for x=0.02. This value increases further to ≈0.15m_e for x=0.08, where m_e is the free-electron mass.

Figure 1: Effect of Mn doping on the electronic band dispersions of (Bi_1−xMn_x)₂Se₃.

(a–d) Mn-concentration-dependent high-resolution ARPES dispersions measured at 50 eV photon energy and a temperature of 12 K for x values of (a) 0, (b) 0.02, (c) 0.04 and (d) 0.08. The red lines represent EDCs obtained in normal emission (k_||=0). A surface band gap opens at the Dirac point of the TSS which increases in size with increasing Mn content. The horizontal white-dashed lines highlight the minimum limit of the gap size after taking into account the contribution from the linewidth broadening. The opening of the gap leads to an intensity dip in the EDCs around the energy region of the original Dirac point. For x=0.02, the suppression of intensity at the Dirac point and the more parabolic surface-state dispersion are signatures of a small gap. Besides the increased linewidth broadening due to the Mn impurities, these effects become more pronounced with increasing Mn content.

The observed surface band gaps are similar to those previously reported²⁰, which have been attributed to long-range ferromagnetism and TRS breaking of the topologically protected surface state²⁰. Figures 2a–d show high-resolution ARPES dispersions of the TSS, as well as the normal-emission spectra through the Dirac point at temperatures of 12 and 300 K for x=0.08 (see also Supplementary Fig. 3). Strikingly, there is no significant change as temperature is raised, and very clearly the band gap in the TSS persists up to room temperature. Moreover, we find a similar behaviour for the whole sample series independently of the Mn content, which challenges the dominant role of ferromagnetic order in inducing the band gap in the TSS, unless the surface T_C is above room temperature.

Figure 2: Temperature dependence of the large energy scale surface band gap.

(a,b) Energy-momentum ARPES dispersions obtained for Mn-doped Bi₂Se₃ epilayers with 8% Mn concentration. The red lines are the energy-distribution curves obtained in normal emission (k_||=0). The horizontal white-dashed lines highlight the surface gap. Measurements are taken at 50 eV photon energy and a temperature of (a) 12 K and (b) 300 K. The surface band gap does not show a remarkable temperature dependence.

Bulk and surface magnetism

We begin in Fig. 3 with characterizing the bulk magnetic properties using superconducting quantum-interference device (SQUID) magnetometry. The comparison of measurements with in-plane and out-of-plane applied magnetic field in Fig. 3a shows that at a temperature of 2 K the bulk easy axis lies in the surface plane. This holds irrespective of the Mn concentration. Since the opening of a gap at the Dirac point requires a magnetization perpendicular to the surface⁹, we concentrate in the following on the out-of-plane component of the magnetization. In the hysteresis loops displayed in Fig. 3b measured with an out-of-plane applied magnetic field at different temperatures, we observe a narrow magnetic hysteresis at 4.2 K, whereas paramagnetic behaviour emerges at 7 K once the ferromagnetic transition has been crossed (inset in Fig. 3b). For a better determination of T_C we present in Fig. 3c modified Arrott plots according to a 3D Heisenberg ferromagnet, normalized to the mass of the sample, with exponents β=0.348 and γ=1.41, from which we deduce T_C=5.5 K in the bulk (see also Supplementary Fig. 4). This is well below the temperature of our ARPES measurements shown in Figs 1 and 2.

Figure 3: Characterization of the bulk magnetic properties of Mn-doped Bi₂Se₃ films.

(a) In-plane (ip) and out-of-plane (oop) magnetization curves for Mn concentrations of x=0.04 and x=0.08, measured at a temperature of 2 K using SQUID magnetometry. The magnetic anisotropy lies in the surface plane irrespective of the Mn concentration. (b) Corresponding hysteresis loops for x=0.08 measured at a temperature of 4.25 K (squares) and 7 K (triangles), respectively. The applied magnetic field is perpendicular to the (111) sample surface (see sketch). Inset: a zoom-in around zero magnetic field showing hysteresis at 4.25 K and a paramagnetic state at 7 K. (c) Arrott plots at various temperatures from which a Curie temperature T_C of 5.5 K is deduced. Between 4.25 K and 6.75 K, data are shown for temperature increments of δT=0.25 K.

The only remaining possibility that long-range ferromagnetism opens the gap at the Dirac point is an enhanced surface magnetism with magnetization perpendicular to the surface^9,20,23. This cannot be verified directly by the bulk-sensitive SQUID magnetometry. Therefore, we perform X-ray magnetic circular dichroism (XMCD) measurements at the Mn L_2,3-edges to probe the near-surface ferromagnetic order. The detection by total electron yield leads to a probing depth in the nanometre range. Figure 4a shows for x=0.04 the normalized intensity of the Mn L_2,3 absorption edges obtained upon reversal of the photon helicity at a temperature of 5 K with an out-of-plane applied magnetic field of 3 T. The XMCD difference spectrum is shown in Fig. 4b, with the normalized XMCD difference signal following one part of the out-of-plane hysteresis as a function of the applied magnetic field as inset. The temperature dependence of the XMCD signal measured in remanence (0 T applied magnetic field) is presented in Fig. 4c for Mn concentrations of x=0.04 and 0.08. This signal is obtained after switching off an applied magnetic field of 5 T perpendicular to the surface, and represents the remanent XMCD, which is proportional to the remanent magnetization of the film surface. We clearly observe a ferromagnetic Mn component in the remanent XMCD appearing around 640 eV which becomes increasingly more pronounced at lower temperatures. The lineshape of the XMCD spectrum is similar to that of Mn in GaAs³¹, indicating a predominant d⁵ configuration. More specifically, the lineshape compares rather well with Mn in Bi₂Te₃ for which the comparison with an Anderson impurity model recently gave a superposition of 16% d⁴, 58% d⁵ and 26% d⁶ character³². The ferromagnetic Mn component shows surface ferromagnetic order in the remanent XMCD only below 10 K for Mn concentrations of x=0.04 and 0.08 (inset in Fig. 4c). On one hand, this result is not inconsistent with the predicted strong-surface enhancement of T_C since the enhancement effect vanishes already when chemical potential and Dirac point differ by 0.2 eV (ref. 23). On the other hand, our XMCD result means that the magnetically induced gap would have to close above 10 K.

Figure 4: XMCD in total electron yield at the Mn L_2,3-edges.

(a) X-ray absorption spectra measured for opposite helicities of the circularly polarized incident light at 5 K with an out-of-plane applied magnetic field of 3 T. A sketch of the experimental geometry is also shown. (b) Corresponding XMCD difference spectrum. The inset shows XMCD measurements for different applied magnetic fields. (c) Temperature dependence of the remanent XMCD for a Mn concentration of x=0.04. Inset: detailed temperature dependence for x=0.04 and 0.08. (d) To re-examine the bulk magnetic properties, field-cooling measurements are obtained under an in-plane (ip) and out-of-plane (oop) applied magnetic field of 10 mT using SQUID magnetometry. The temperature dependence for x=0.04 and 0.08 compares well with the surface-sensitive measurements shown in the inset of (c). (e) XMCD-PEEM image obtained using photoelectron microscopy for x=0.08 at room temperature, revealing the absence of magnetic domains with partial or full out-of-plane magnetic order. Scale bar, 500 μm (horizontal white-solid line).

Having established that the bulk and surface magnetic properties of our samples are very similar, as a next step, we performed field-cooling experiments to re-examine the bulk magnetic properties. Figure 4d shows field-cooled SQUID data for an applied magnetic field of 10 mT parallel and perpendicular to the surface plane. Above T_C, there is no preferential orientation of the anisotropy axis perpendicular to the surface. Additional zero-field-cooled SQUID data compare rather well with these results (Supplementary Fig. 5). Further XMCD measurements under an in-plane and out-of-plane applied magnetic field are consistent with the SQUID data and are shown in Supplementary Fig. 6. These XMCD and SQUID results strongly suggest that above T_C also short-range static magnetic order does not play a role at the surface or in the bulk, respectively (see Supplementary Note 2), and that domains with partial or full out-of-plane magnetic order are absent. We further support this conclusion by additional XMCD measurements carried out by the means of X-ray photoelectron emission microscopy (XMCD-PEEM) at room temperature, with a lateral resolution of ∼20 nm. Figure 4e shows the XMCD image which due to the light incidence (16° grazing) is sensitve to both in- and out-of-plane components of the magnetization, with additional data given in Supplementary Figs 7,8. The XMCD image displays no magnetic domains at room temperature thus also excluding short-range static inhomogeneous magnetic order with magnetization partially or fully in or out of the surface plane, at least within our lateral resolution (see Supplementary Note 3 for details). A similar conclusion can be drawn regarding out-of-plane magnetic short-range fluctuations (see discussion in Supplementary Note 4 and Supplementary Fig. 9). All our observations taken together lead us to the important conclusion that the surface band gap is not due to magnetism and, thus, the novel topological phases cannot be realized with (Bi_1−xMn_x)₂Se₃.

Absence of topological phase transition with concentration

There exists an alternative nonmagnetic explanation for the surface band gap. It is in principle possible that the incorporation of Mn changes the bulk band structure and even reverts the bulk band inversion, rendering Mn-doped Bi₂Se₃ topologically trivial. This topological phase transition with concentration is, for example, the basis for the HgTe quantum spin-Hall insulator⁶. Interestingly, it has been argued that a gap in the TSS can be a precursor of a reversed bulk band inversion, as discussed for TlBi(S_1−xSe_x)₂ (ref. 33). Indium substitution in Bi₂Se₃ behaves similarly, and leads to a topological-to-trivial quantum-phase transition with an inversion point between 3 and 7% In in thin films³⁴. Figure 5 shows that the bulk band gap stays constant within 4% of its value for 8% Mn incorporation. This statement is possible because at low photon energies changes in the bulk band gap are traced most precisely from quantum-well states in normal emission (k_||=0). The simultaneous quantization of BCB and BVB is created by surface band bending after adsorption of small amounts of residual gas³⁵. This means that the scenario of reversed bulk band inversion does not hold here.

Figure 5: Tracing changes in the bulk band gap of Mn-doped Bi₂Se₃ films.

Changes in the bulk band gap are traced most precisely from quantum-well states (QWS) in normal emission (red curves on the right-hand side of each panel).(a–c) Energy-momentum dispersions acquired at a photon energy of hν=18 eV for (a) Bi₂Se₃, (b) 2% and (c) 8% Mn doping. The simultaneous quantization of bulk-conduction band (BCB) and valence-band (BVB) is created by surface band bending after adsorption of small amounts of residual gas³⁵. It is found that Mn doping does not change the bulk band gap of Bi₂Se₃. (d–f) Similar results as in a–c, respectively, but at a photon energy of hν=16 eV, where large apparent surface band gaps are observed. This unexpected photon-energy dependence of a surface state is interpreted as coupling to bulk states mediated by the Mn impurities.

Another remarkable feature is the measured size of the surface band gap. A perpendicular magnetic anisotropy has recently been predicted by density-functional theory (DFT) for Co in Bi₂Se₃ (ref. 36) as well as for 0.25 monolayer Co adsorbed on the surface in substitutional sites, leading to a surface band gap of ∼9 meV (ref. 36), which, however, is one order of magnitude smaller than the gaps observed here. For (Bi_1−xMn_x)₂Te₃, a surface band gap of 16 meV has been calculated under the condition of perpendicular magnetic anisotropy³⁷. For Mn in Bi₂Se₃ at the energetically favoured Bi-substitutional sites, an in-plane magnetic anisotropy is predicted³⁸. The size of the gap at the Dirac point is only 4 meV (ref. 38). Perpendicular anisotropy and ferromagnetic order are absent in our samples at the temperature of the ARPES measurements.

Role of the local magnetic moment and intercalation scenarios

If the large local magnetic moment of the Mn is responsible for the measured band gaps via TRS breaking, the effect should be equal or larger when Mn is deposited directly on the surface. Here we argue in the following way: If TRS is broken only due to the Mn magnetic moment, we do not require hybridization to open a gap. By depositing Mn we create a completely different environment for the Mn at the surface, allowing us to understand whether the Mn magnetic moment is solely responsible for the opening of large surface band gaps via TRS breaking and, ideally, without hybridization playing a role. Mn is very well suited for such comparison, as due to the high stability of its half-filled d⁵ configuration, its high magnetic moment is to a large extent independent of the atomic environment and the resulting hybridization. DFT obtains 4.0 μ_B for most substitutional sites (except for the hypothetical Se-substitutional site with still 3.0 μ_B) and only for interstitial Mn the moment peaks with 5.0 μ_B (ref. 38). Mn deposition was performed at ∼30 K in order to keep the Mn atoms isolated from each other and on the surface. As the XMCD showed a predominant d⁵ configuration for Mn bulk impurities, which is most stable, we can assume the same magnetic moment for Mn impurities deposited on the surface. Figure 6 shows that a similar p-doping occurs as with the Mn in the bulk. In another respect, the magnetic moments do not act in a similar way as in the bulk. Even 30% of a Mn monolayer on Bi₂Se₃ does not open a band gap at the Dirac point. This result is similar to what we have observed for Fe on Bi₂Se₃ (ref. 12) and Bi₂Te₃ (ref. 14).

Figure 6: Effect of Mn deposition on the surface of Bi₂Se₃ films.

(a) Energy-momentum ARPES dispersion showing a gapless Dirac cone in pure Bi₂Se₃ films after deposition of 0.3 monolayer Mn at 30 K on the surface. Measurements are performed at the same temperature using 50 eV photon energy. (b) EDCs extracted from the measurements shown in a. The red curves in a and b emphasize the EDC in normal emission (k_||=0).

Most recently, the appearance of surface band gaps in ARPES at the Dirac point of the TSS has been discussed based on different mechanisms. One is momentum fluctuations of the surface Dirac fermions in real space as observed by scanning tunnelling microscopy measurements of Mn-doped Bi₂Te₃ and Bi₂Se₃ bulk single crystals³⁹. However, these fluctuations amount to 16 meV (ref. 39), much less than the present band gaps. Another mechanism is top-layer relaxation, that is, breaking of the van-der-Waals bond between quintuple layers during cleavage of bulk crystals as proposed in Xu et al.²⁰ This would open a gap due to hybridization of TSS’s through the layer⁴⁰ but such an effect does not occur for epitaxial layers where no sample cleavage is used for surface preparation.

If such a separation of quintuple layers occurs, it is more likely caused by the Mn. Principally, intercalated Mn in the van-der-Waals gaps could separate quintuple layers electronically, although such effect has not been seen in experiments yet. At these new interfaces TSS’s could form and if they would behave like in ultra-thin Bi₂Se₃ films, they would hybridize and open a band gap⁴⁰. However, band gaps of the order of ∼100 meV would correspond to an unrealistic Mn intercalation pattern grouping three quintuple layers when compared with the films of Zhang et al.⁴⁰ In order to estimate the amount of Mn intercalation, we have analysed the change of the bulk lattice constant in our samples by X-ray diffraction (see Supplementary Note 5, Supplementary Figs 10–13 and Supplementary Tables 1,2). We find that the c-lattice constant increases by ∼0.15% for 4% Mn and by ∼0.45% for 8% Mn. If we would assume that this is completely due to an expansion of the van-der-Waals gaps, the interlayer distance would increase by only ∼1.8% for 4% Mn and by less than ∼5% for 8% Mn. According to DFT, intercalation of transition metals into the first van-der-Waals gap of Bi₂Se₃ leads to an expansion between 10 and 20% (ref. 41). New surface-state features can appear inside the bulk band gap but in order to be split remarkably off the BVB and BCB edges, expansions between 20 and 50% are required⁴¹. In addition, we point out that, despite our observation of a large surface band gap at room temperature, we observe a nearly zero spin polarization perpendicular to the surface (Supplementary Fig. 14). The absence of a hedgehog spin texture pinpoints a non-TRS breaking mechanism underlying the origin of the gap, a fact which is consistent with our conclusions derived from the magnetic properties. Moreover, it is understood that we cannot verify a small magnetic band gap opening of few meV as calculated in Schmidt et al.³⁶ and Henk et al.³⁷ because much larger gaps are present in the whole temperature range.

Topological protection beyond the continuum model

As we find that neither ferromagnetic order in the bulk or at the surface, nor the local magnetic moment of the Mn are causing the large band gaps that we observe, as we can exclude the nonmagnetic explanation of a reversal of the bulk band inversion and as we do not find sufficient indications for the nonmagnetic explanation of surface-state hybridization by van-der-Waals-gap expansion, we point out a different mechanism based on impurity-induced resonance states that locally destroy the Dirac point.

In fact, recently the treatment of the topological protection in the continuum model^3,4,5 has been extended for finite bulk band gaps^42,43. As a result, surface⁴² and bulk impurities⁴³ can mediate scattering processes via bulk states and the localized impurity-induced resonance states emerging at and around the Dirac point lead to a local destruction of the topological protection of the Dirac point. The resulting band gap opening depends on the resonance energy, impurity strength U as well as spatial location of the impurities^42,43. A typical gap size is of the order of 100 meV (ref. 42). Moreover, the gap does not rely on a particular magnetic property of the impurity and, therefore, the mechanism should apply for magnetic impurities as well. The model can also explain the absence of the gap when Mn impurities are placed on the surface, since bulk-like resonance states do not form when Mn is placed only on the surface. A definite assignment, however, requires realistic electronic structure calculations since a simple atomic Mn d⁵ configuration does not lead to states at the Dirac point.

To identify impurity resonances, we calculated for 8% Mn in Bi₂Se₃ at the Bi-substitutional sites the corresponding DOS by ab initio theory. The model structure used in the calculations is shown in Fig. 7a, where Bi atoms (yellow) acquire Mn character (blue wedges). The results of the calculation are shown in Fig. 7b, where it is seen that Mn impurity states (blue) strongly contribute to the total DOS (red) near the BVB maximum. The Mn impurity states in the gap are clearly identified by assuming ferromagnetic order and subtracting the minority from majority spin DOS. In experiment, impurity-band states are difficult to observe as the example of Ga_1−xMn_xAs shows⁴⁴, where—similarly to the present case—impurity states emerge near the BVB maximum. There are, in fact, indications for the impurity resonances in our data. If we look at the photon-energy dependence from 16 to 21 eV in the ARPES results shown in Supplementary Fig. 15, we see that the surface band gap is only well defined with respect to the minimum of the upper Dirac cone and consistent for 16–21 eV as well as 50 eV (for 50 eV see Figs 1, 2, where the BCB does not appear). The lower Dirac cone exhibits despite its two-dimensional nature a photon-energy (wave vector perpendicular to the surface, k_⊥) dependence when Mn is incorporated. The reason for this difference is that the lower Dirac cone strongly overlaps with the BVB while the upper Dirac cone does not overlap with the BCB. This can also be seen in Fig. 5, where an apparent band gap of ∼200 meV occurs for 2% Mn at hν=16 eV. At this small concentration, such an apparent gap seems to be nearly closed at 18 eV, where the suppression of the Dirac point intensity and the more parabolic TSS dispersion as compared with the undoped Bi₂Se₃ film indicates the existence of a small gap, in agreement with the results of Fig. 1. Note that the minimum band gap in the photon-energy dependence defines the gap size (see Supplementary Fig. 16). The minimum gap size observed at low photon energies agrees well with the one obtained at 50 eV. At 8% Mn, the surface band gap opens at all photon energies and reaches a minimum of ∼200 meV, that is, it is determined rather unambiguously, but some intensity appears in the surface band gap. Such intensity also appears in Xu et al.²⁰ The role of the impurity resonance is exactly to couple the TSS to bulk states. This 3D coupling is naturally different for the upper and lower Dirac cone due to the different bulk overlap and thus seen as the photon-energy dependence of the lower half of the Dirac cone.

Figure 7: Resonant photoemission and ab initio calculations.

(a) The structure of Bi₂Se₃ doped with 8% Mn at the Bi-substitutional sites used in the calculations. Within the coherent-potential approximation, the Bi atoms (yellow) acquire Mn character (blue wedges). Se atoms are shown in grey colour. (b) Calculated density of states (DOS). The total DOS (red) contains contributions from impurity resonances (vertical blue arrows) of strong d-character, a seen in the partial Mn DOS (blue). For the purpose of the calculation, ferromagnetic order at T=0 K was assumed. (c,d) Resonant photoemission at the Mn M-edge, focusing on the region of the surface gap (8% Mn). (c) Off-resonant (48 eV photon energy) and (d) on-resonant data (50 eV). (e) Schematics of the resonant process. At the resonance energy, the excitation occurs via 3p–3d transitions. The exciting photon energy (h ν) corresponds to a transition from an occupied core level to a valence-band empty state between the Fermi (E_F) and vacuum (E_vac) levels. The relaxation of excited electrons leads to enhanced photoemission from d-like Mn states. (f) Intensity difference obtained after subtracting off-resonant from on-resonant data. The resonances are seen in blue contrast and are marked with arrows, similar to the calculations shown in b.

We applied resonant photoemission which fortunately is comparatively strong for Mn due to its half-filled d-shell. Figures 7c and d show for 8% Mn off-resonant (hν=48 eV) and on-resonant photoemission (50 eV) measurements via the Mn 3p core level, focusing on the region of the surface gap, respectively. The spectra were normalized to the photon flux after taking into account the photon-energy dependence of the photoionization cross sections of Bi 6p and Se 4p, respectively. Subtracting off-resonant from on-resonant data allows us to visualize directly the contribution from impurity resonances in the ARPES spectrum, as there is enhanced photoemission from d-like Mn states through decay of electrons that are excited via 3p–3d transitions (Fig. 7e). The difference spectrum shown in Fig. 7f reveals the existence of impurity resonances inside the surface band gap and near the valence-band maximum. The resonances are seen in blue contrast and are marked with arrows, similar to the calculations shown in Fig. 7b. The unexpectedly strong dispersion of the resonant states with wave vector k_|| is in qualitative agreement with our one-step model photoemission calculations (see Supplementary Fig. 17 and Supplementary Note 6), and additionally supports the physical picture of Mn-induced coupling to the bulk states. We should emphasize that the Mn d⁵ configuration confirmed by our XMCD measurements offers much fewer states in the energy range of the Dirac cone than the other magnetic transition metals. The fact that already Mn breaks the Dirac cone indicates that the present result is of general importance for TIs doped with magnetic transition metals.

As nonmagnetic control experiment we have chosen thick In-doped Bi₂Se₃ bulk samples which give rise to a trivial phase close to ∼5% In concentration. Supplementary Fig. 18 shows that a large band gap of the order of ∼100 meV appears at the Dirac point already for a much smaller In concentration, namely 2%, far away from the inversion point of the topological quantum-phase transition (see Supplementary Note 7 and Supplementary Fig. 19 for more details). In addition, spin-resolved ARPES measurements around k_||=0 for In-doped samples (Supplementary Fig. 14) reveal that the out-of-plane spin polarization is zero, similarly to the result for the gapped Dirac cone in Mn-doped samples. Interestingly, the size of the band gap for 2% In is of the same order as the one for 8% Mn whereas no gap appears for 4% Sn (Supplementary Fig. 18). This indicates that the concentration range at which the large surface gap develops varies from dopant to dopant. On the basis of ideas proposed recently^42,43, this might be associated with the impurity-dependent strength U, regardless whether the dopant is magnetic or not. Our control experiments demonstrate the existence of a mechanism for surface band gap opening which is not directly connected to long-range or local magnetic properties. Although it is not possible to directly search for In impurity resonances in the photoemission experiment as there is no resonant photoemission condition available, for completeness we point out that our conclusion on the nonmagnetic origin of the surface band gap in Mn-doped Bi₂Se₃ films would not have been possible unless several findings in our experiment differed from previous experiments²⁰. This refers to the in-plane magnetic anisotropy, the absence of a temperature dependence of the gap, a much lower T_C, no out-of-plane spin polarization at the gapped Dirac point, no enhanced surface magnetism, a photon-energy dependence of the band gap, and an available resonant condition via the Mn 3p core level allowing us to directly observe the contribution from in-gap states.

To summarize, we revealed the opening of large surface band gaps in the TSS of Mn-doped Bi₂Se₃ epilayers that strongly increase with increasing Mn content. We find ferromagnetic hysteresis at low temperatures, with the magnetization oriented parallel to the film surface. Magnetic surface enhancement is absent. Even above the ferromagnetic transition, we observe surface band gaps which are one order of magnitude larger than the magnetic gaps theoretically predicted and, moreover, they do not show notable temperature dependence. No indication for a Mn-induced reversal of the bulk band inversion and no significant enhancement of the surface magnetic ordering transition with respect to the bulk are found. Control experiments with nonmagnetic bulk impurities, conducted in the topological phase, reveal that surface band gaps of the order of ∼100 meV can be created without magnetic moments. In line with recent theoretical predictions, we conclude that the band gap opening up to room temperature in Mn-doped films is not induced by ferromagnetic order and that even the presence of magnetic moments is not required. Our results are important in the context of topological protection and provide strong circumstantial evidence that Mn-doped Bi₂Se₃ is not suited for observing the quantized anomalous Hall effect or the half quantum Hall effect.

Methods

Sample growth and structural characterization

The samples were grown by molecular beam epitaxy on (111)-oriented BaF₂ substrates at a substrate temperature of 360 °C using Bi₂Se₃, Mn and Se effusion cells. The Mn concentration was varied between 0 and 8%, and a two-dimensional growth was observed by in situ reflection high-energy electron diffraction in all cases (see Supplementary Fig. 20 and Supplementary Note 8). All samples were single phase as indicated by X-ray diffraction. After growth and cooling to room temperature, the ∼0.4-μm-thick epilayers were in situ capped by an amorphous 50-nm thick Se layer, which was desorbed inside the ARPES and XMCD chambers by carefully annealing at ∼230 °C for ∼1 h. The Mn concentrations determined by core-level photoemission were in good agreement with those obtained from energy dispersive microanalysis, indicating no significant Mn accumulation at the surface of the samples (see Supplementary Fig. 21 and Supplementary Note 9).

High-resolution and spin-resolved ARPES

Temperature-dependent ARPES measurements were performed at the UE112-PGM2a beamline of BESSY II at pressures better than 1 × 10⁻¹⁰ mbar using p-polarized undulator radiation. Photoelectrons were detected with a Scienta R8000 electron energy analyser and the spin-resolved spectra were obtained with a Mott-type spin polarimeter coupled to the hemispherical analyser. For the spin-resolved measurements of Mn-doped Bi₂Se₃ samples, a magnetic field of +0.3 T was applied perpendicular to the surface plane at 20 K right before the acquisition of the spectra. Overall resolutions of ARPES measurements were 5 meV (energy) and 0.3° (angular). Resolutions for spin-resolved ARPES were 80 meV (energy) and 0.75° (angular).

Magnetic characterization

The characterization of the bulk magnetic properties was done by SQUID magnetometry. The bulk magnetization was recorded as a function of temperature and applied magnetic field applied either perpendicular (out-of-plane) or parallel (in-plane) to the surface of the films. The diamagnetic contribution of the BaF₂ substrate was derived from the field-dependent magnetization curves at room temperature and subtracted from all data. The characterization of the surface magnetic properties was done by XMCD and XMCD-PEEM measurements at the UE46-PGM1 and UE49-PGM1a beamlines of BESSY II, respectively. The experiments were performed using circularly polarized undulator radiation and under the same pressure conditions as the ARPES measurements. The XMCD spectra were taken using a high-field diffractometer as a function of temperature and applied magnetic field, and the XMCD-PEEM measurements were performed at room temperature and under grazing incidence (16°) with a lateral resolution of ∼20 nm.

Theoretical calculations

The one-step model photoemission calculations were performed using the results of ab initio theory as an input, and taking into account wave vector and energy-dependent transition matrix elements. The ab initio calculations were performed within the coherent-potential approximation using the Munich SPR-KKR program package^45,46, with spin-orbit coupling included by solving the four-component Dirac equation.