Two-component anomalous Hall effect in a magnetically doped topological insulator

Abstract

The anomalous Hall (AH) effect measurement has emerged as a powerful tool to gain deep insights into magnetic materials, such as ferromagnetic metals, magnetic semiconductors, and magnetic topological insulators (TIs). In Mn-doped Bi₂Se₃, however, the AH effect has never been reported despite a lot of previous studies. Here we report the observation of AH effect in (Bi,Mn)₂Se₃ thin films and show that the sign of AH resistances changes from positive to negative as the Mn concentration is increased. The positive and negative AH resistances are found to coexist in a crossover regime. Such a two-component AH effect and the sign reversal can also be obtained by electrical gating of lightly doped samples. Our results provide an important basis for understanding the puzzling interplay between the surface states, the bulk states, and various magnetic doping effects, as well as competing magnetic orders in magnetically doped TIs.

Introduction

Anomalous Hall (AH) effect is one of the most fundamental transport properties of magnetic materials, in which the interplay between magnetism and spin-orbit coupling produces a transverse Hall voltage perpendicular to the applied current and the magnetization^1,2. Despite more than a century of research, it was not until last decade the AH effect was associated with the Berry curvature of the occupied energy bands^3,4. Study of the AH effect has since become a valuable tool to understand the electronic properties of magnetic metals and semiconductors, although interpretations of the experimental results are often complicated by the details of band structures as well as the existence of extrinsic sources of AH effect^1,2, namely skew scattering⁵ and side jump⁶.

The sophisticated nature of AH effect and its connection to the band topology have recently been nicely illustrated in magnetically doped three-dimensional topological insulators (3D TIs, referred to as TIs below)^7,8. The surface of a TI hosts symmetry-protected gapless Dirac electrons with spin direction locked perpendicular to their momenta. When doped with magnetic impurities, TIs can host a large variety of magnetic phases, such as ferromagnetic, paramagnetic, noncolinear, and spin glass phases, arising from various types of exchange interactions between the magnetic impurities in presence of the surface and bulk states^9,10,11,12. The magnetically doped TIs have received intensive experimental efforts¹³, and the work on ferromagnetic (FM) order in Cr-doped (Bi,Sb)₂Te₃ thin films has led to the seminal discovery of quantum anomalous Hall effect (QAHE)¹⁴. In contrast, magnetic doping of Bi₂Se₃, a prototype TI isostructural to Bi₂Te₃ and Sb₂Te₃, has not allowed for the observation of QAHE, even though gapped surface states have been observed in angular resolved photoemission spectroscopy (ARPES) experiments^15,16. Among them, a particularly puzzling system is the Mn-doped Bi₂Se_3, in which even the AH effect has not been discovered despite compelling evidence for FM order in both the bulk and surface states^17,18,19,20. More surprisingly, a recent study combining ARPES and other techniques has strongly suggested a non-magnetic origin for the surface energy gap in the Mn-doped Bi₂Se₃, in contradiction to the earlier findings^15,20. It remains an open question how the non-magnetic gap, or more generally, the non-magnetic scattering effect of the magnetic impurities competes with magnetic interactions in the Mn-doped Bi₂Se₃ and other magnetically doped TIs. Unfortunately, very few experimental techniques can probe the magnetic order in both the surface and bulk states at very low temperatures. Therefore, a systematic study of the AH effect in the Mn-doped Bi₂Se₃ thin films will provide much needed information for understanding the complicated impurity-related physics in magnetically doped TIs.

Here, we report a detailed study of the AH effect in the Mn-doped Bi₂Se₃ thin films. By controlling the Mn doping level and tuning the chemical potential, the sign of the AH resistance can be varied from positive to negative. The positive and negative AH resistances are found to coexist in a wide range of parameters, and they exhibit qualitatively different dependences on the applied magnetic field and gate voltage. To the best of our knowledge, such a two-component AH effect has never been observed in any magnetically doped TI, despite that both the surface and bulk magnetic orders in TIs have been a subject of intense experimental study. Our results suggest that the non-magnetic scattering effects of the magnetic dopants can have a profound impact on the transport properties of magnetic TIs, and should be taken into account in seeking high quality TI-based magnetic materials.

Results

Basic characterization

The (Bi_1−xMn_x)₂Se₃ thin films studied in this work were grown with molecular beam epitaxy (MBE) on SrTiO₃(111) substrates. The samples were then patterned into ~200 μm wide Hall bars (Fig. 1a) for the electron transport measurements. Detailed description of the sample preparation process and the methods for magnetization and transport measurements are given in the Methods section. Figure 1b depicts an atomic force microscopy (AFM) image of a 10 quintuple-layer (QL) thick (Bi_1−xMn_x)₂Se₃ film with Mn concentration x = 0.02. The surface morphology of the film is characterized by triangular shaped terraces with sizes of the order 100 nm, which are commonly seen in Bi₂Se₃ thin films²¹. Figure 1c shows the magnetic field dependence of in-plane magnetization of a 70 nm thick (Bi_1−xMn_x)₂Se₃ (x = 0.088) thin film. The overall feature of the magnetization is consistent with those reported in refs.^17,20, in which the easy magnetization axes of the (Bi_1−xMn_x)₂Se₃ thin films are found to be in the film plane. Figure 1d displays the magnetoresistances (MRs) of a set of (Bi_1−xMn_x)₂Se₃ samples with Mn concentration x = 0–0.08 (samples A–G). All the samples exhibit positive MR and have a cusp-shaped minimum at B = 0 when the gate voltage V_G is set at zero. The corresponding Hall resistance (R_yx) curves are nearly linear and have negative slopes (Fig. 1e), indicating n-type charge carriers. The sheet electron densities extracted with n_s = −1/(eR_H) are in a range of 0.48–1.29 × 10¹³ cm⁻² for the (Bi_1−xMn_x)₂Se₃ thin films with x = 0.018–0.088 (samples B–G), where R_H is the (ordinary) Hall coefficient. The obtained electron densities are substantially lower than n_s ∼ 2.7 × 10¹³ cm⁻² for an undoped Bi₂Se₃ thin film prepared in the similar condition (sample A). The basic transport parameters of these samples are summarized in Table 1, and the corresponding band diagrams are schematically drawn in Fig. 1f. The lower Fermi level in the doped samples can be attributed to the substitution of Bi³⁺ ions by Mn²⁺ ions¹⁹.

Fig. 1

Basic characterization of (Bi_1−xMn_x)₂Se₃ films. a Sketch of a back-gated Hall bar device used in the electron transport measurements. b Atomic force microscopy (AFM) and in situ reflection high-energy electron diffraction (RHEED) (inset) images of a 10 quintuple-layer (QL) thick (Bi_1−xMn_x)₂Se₃ film (x = 0.02) grown on a STO(111) substrate. c In-plane magnetization as a function of applied magnetic field for a 70 nm thick (Bi_1−xMn_x)₂Se₃ film with x = 0.088 at T = 1.6 K. The inset shows the temperature dependence of in-plane magnetization. A linear background is subtracted to account for the paramagnetic signal from the STO substrate. Detailed discussion of the magnetic order in the (Bi,Mn)₂Se₃ thin films is given in Supplementary Note 3. d, e Magnetoresistance (MR) and Hall resistance R_yx at x = 0, 0.018, 0.0244, 0.055, 0.074, and 0.088 at T = 1.6 K. The MR is defined as [R_xx(B) − R_xx(0)]/R_xx(0). f Schematic sketch of band diagrams of the (Bi,Mn)₂Se₃ thin films without (left) and with (right) the surface energy gap induced by the Mn doping

Table 1 Basic transport characteristics of (Bi_1−xMn_x)₂Se₃ (x = 0.018–0.088). Here R_□, n_s, and μ denote the sheet resistivity (i.e., resistance per square), sheet electron density, and mobility, respectively. The data were extracted from low field magnetotransport measurements at T = 1.6 K

Two-component AH effect

Figure 2a–f shows the Hall resistance curves for a subset of (Bi_1−xMn_x)₂Se₃ thin films (x = 0–0.088), and the corresponding AH resistances (Fig. 2g–l) are obtained by subtracting the linear (ordinary) component from the total Hall resistances, i.e., $R_{{{\rm AH}}} \simeq R_{yx} - R_{{\rm H}}B$. For the undoped Bi₂Se₃ thin film (sample A), the nonlinear part of the Hall resistance is nearly zero for the entire field range. In contrast, the AH resistances are clearly visible for all the Mn-doped films (samples B, D−G). At low doping levels (e.g., x = 1.8%, sample B), the sign of the AH resistance (R_AH) above the (positive) magnetization saturation field is positive and opposite to that of the ordinary Hall resistance R_HB. Throughout this work, we refer to this as the positive R_AH, which has the same sign as the ordinary Hall resistance of the hole-type carriers. In contrast, the samples with high Mn doping levels (x = 0.074 and 0.088) exhibit the negative AH resistance. As shown in Fig. 2, the increase in the Mn concentration drives a crossover from the positive to negative R_AH. At intermediate doping levels, a kink structure begins to appear in the R_AH curves. This suggests coexistence of a positive R_AH component with a negative one. Such a two-component AH effect can be observed for a wide range of Mn concentrations (x ≥ 2.4%). Figure 2 further shows that these two components have different dependences on the Mn doping level: the magnitude of the negative R_AH component becomes larger with increasing Mn concentration, whereas the positive component has a nonmonotonic dependence. The overall trend is that negative component becomes more pronounced relative to the positive component with increasing Mn doping level, as evidenced by the reversal of the sign of R_AH in the high magnetic fields.

Fig. 2

Evolution of the Hall effect and the corresponding AH resistances with increasing Mn concentration. Shown in panels a–f and g–l are the magnetic field dependences of the Hall resistance R_yx and the nonlinear part of the Hall resistance (R_AH(B) = R_yx(B) − R_HB), respectively, for Mn concentrations x = 0–0.088 at T = 1.6 K. For doped samples (x > 0), the nonlinear Hall resistances predominantly originate from the AH effect. For the undoped sample (x = 0), it is nearly zero. Here the R_H values are obtained by linear fits of the raw R_yx(B) curves with magnetic field ranges 3–5 T for samples with x = 0–0.055 and 5–9 T for samples with x = 0.074 and 0.088. With the methods described in the Supplementary Information, the AH resistance R_AH is then separated into a positive component (orange line) and a negative one (blue line) for the samples with x > 0.02

Gate-voltage tuning of the AH effect

The sign reversal in R_AH and the two-component AH effect can also be obtained by gate-voltage tuning. This is illustrated with sample C (x = 0.02) in Fig. 3. As the gate voltage is decreased from V_G = +100 to −210 V, the sheet electron density is reduced from n_s = 0.91 × 10¹³ to 0.35 × 10¹³ cm⁻², based on the Hall effect measurements. The sheet resistance per square (i.e., sheet resistivity $\rho _{xx}$) increases from 2.7 to 6.0 kΩ in the same gate-voltage range. At high electron densities (V_G ≥ 100 V), R_AH only has the positive component. When the electron density is lowered by gating, a kink structure emerges at low magnetic fields and becomes more pronounced. At the lowest electron density, the magnitude of the negative component surpasses that of the positive one. Moreover, the negative R_AH component tends to saturate at a magnetic field lower than that of the positive component. Similar phenomena also take place for the ungated samples with higher Mn-doping levels (x = 0.024–0.088, Fig. 2).

Fig. 3

Gate-voltage tuning of the anomalous Hall effect in a (Bi_1−xMn_x)₂Se₃ thin film (x = 0.02). a Magnetic field dependences of R_yx at different gate voltages. b Sheet electron density n_s (red square) and mobility μ (green square) plotted as a function of gate voltage V_G. c Gate-voltage dependence of sheet resistivity R_□. d–i Magnetic field dependences of R_yx at gate voltages from V_G = 100 to −210 V. j–o Magnetic field dependences of R_AH for the same range of gate voltages as panels d–i. At V_G = −30 and −210 V, the positive and negative AH components separated from the R_AH curves (black) are shown in orange and blue lines, respectively. The Hall effect data were taken from Sample C at T = 1.6 K

The gate-voltage tuning of the two-component AH effect has also been performed for samples with higher Mn concentrations. Figure 4 shows the results of sample F (x = 0.074), in which the sheet electron density can be varied from n_s = 1.4 to 0.5 × 10¹³ cm⁻² and the sheet resistivity from 5 to 65 kΩ. At high electron densities (corresponding to $V_{\rm G} \ge 100$ V), the magnitude of the positive component is larger than that of the negative component. When the electron density is sufficiently low (n_s ≤ 0.7 × 10¹³ cm⁻², V_G ≤ 0 V), the magnitude of negative component increases rapidly with decreasing gate voltage and becomes considerably larger than the positive component. Like the other (Bi,Mn)₂Se₃ samples, the negative R_AH component in sample F always has a lower saturation field than the positive component, and the relative magnitude between the negative and positive components is significantly enhanced when the electron density is reduced.

Fig. 4

Gate-voltage tuning of the anomalous Hall effect in a (Bi_1−xMn_x)₂Se₃ thin film (x = 0.074). a R_□ curves at V_G = +210 to −210 V. b Gate-voltage dependence of n_s (red square) and μ (green square). c Gate-voltage dependence of R_□. d–i Magnetic field dependences of R_yx for gate voltages from 210 to −210 V. j–o Magnetic field dependences of R_AH for the same range of gate voltages as panels d–i. For V_G = ±210 V, the positive (orange) and negative (blue) AH components separated from the total R_AH are also shown. The Hall effect data were taken from Sample F at T = 1.6 K

Separation of the two AH components

The conspicuous differences in the two AH components suggest that they originate from different electronic states. Given that multiple parallel conduction channels coexist in the (Bi,Mn)₂Se₃ samples, it is more convenient to analyze the AH effect data in the form of conductivities. In general, the sheet longitudinal conductivity and Hall conductivity are given by $\sigma _{xx} = \mathop {\sum }\nolimits_i \sigma _{xx,i} = \rho _{xx}/(\rho _{xy}^2 + \rho _{xx}^2)$ and $\sigma _{xy} = \mathop {\sum }\nolimits_i \sigma _{xy,i} = \rho _{yx}/(\rho _{xy}^2 + \rho _{xx}^2)$, where the summation goes over all conduction channels and i is the channel index. Because of strong impurity scattering effect, most of the transport discussed in this work is in the low mobility regime (i.e., $\mu B \ll 1$), and correspondingly $\left| {\rho _{xy}} \right| \ll \rho _{xx}$. The Hall conductivity can thus be simplified to σ_xy ≃ ρ_yx/$\rho _{xx}^2$ (with ρ_yx ≃ ρ_xy) and the AH conductivity becomes σ_AH ≃ R_AH$/R_{xx}^2$. Following this definition, σ_AH has the same sign as the AH resistance R_AH.

As described in the Supplementary Notes 1 and 2, the AH conductivity can be separated into two components: $\sigma _{{{\rm AH}}}(B) = \sigma _{{{\rm AH}},1}(B) + \sigma _{{{\rm AH}},2}(B)$, where the first and second terms denote the positive and negative AH components, respectively. Figure 5 summarizes the results for sample C (x = 0.02). As shown in Fig. 5a, the sheet conductivity ${{\rm \sigma }}_{xx}$ drops from 9.1 to 4.7 e²/h as $n_{\rm s}$ is tuned from 0.82 to 0.35 × 10¹³ cm⁻². Figure 5b,c display the conductivity dependences of $\sigma _{{{\rm AH}},1}^{\rm s}$ and $\sigma _{{{\rm AH}},2}^{\rm s}$, the magnitudes of the extracted ${{\rm \sigma }}_{{{\rm AH}},1}$ and ${{\rm \sigma }}_{{{\rm AH}},2}$ above the saturation fields. The positive component $\sigma _{{{\rm AH}},1}^{\rm s}$ decreases nearly linearly with decreasing ${{\rm \sigma }}_{xx}$, whereas the negative component $\sigma _{{{\rm AH}},2}^{\rm s}$ become greater until ${{\rm \sigma }}_{xx}$ drops to ∼6 e²/h. The ratio between the two components, $\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$, increases monotonically from nearly zero to over 1.6 when n_s is varied from 0.82 to 0.35 × 10¹³ cm⁻² (Fig. 5d).

Fig. 5

Characteristics of the AH conductivity in a lightly doped (Bi_1−xMn_x)₂Se₃ sample (x = 0.02). a Sheet carrier density n_s as a function of sheet conductivity $\sigma _{xx}$. b,c $\sigma _{xx}$ dependences of the magnitudes of the positive and negative AH conductivities above the saturation fields, ${{\rm \sigma }}_{{{\rm AH}},1}$ (panel b) and ${{\rm \sigma }}_{{{\rm AH}},2}$ (panel c). d The ratio of the AH components, $\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$, plotted as a function of $\sigma _{xx}$. e Schematic band diagrams for high and low Fermi levels. The data shown in panels a–d were extracted from the Hall effect data taken from Sample C at T = 1.6 K

Figure 6 summarizes the results for sample F (x = 0.074). Figure 6a shows that σ_xx values of this sample are substantially lower than those of sample C at the same carrier densities. Similar to sample C, $\sigma _{{{\rm AH}},1}^{\rm s}$ of sample F also has an approximately linear dependence on σ_xx, as depicted in Fig. 6b. However, $\sigma _{{{\rm AH}},2}^{\rm s}$ of this sample has a drastically different dependence on σ_xx: it becomes smaller as σ_xx decreases (Fig. 6c). Nonetheless, the $\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$ ratio of sample F has a qualitatively similar dependence on σ_xx to sample C, namely increasing monotonically with decreasing σ_xx. As depicted in Fig. 6d, the $\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$ value increases from about 0.5 to 1.5 as σ_xx decreases from 5.6 e²/h to about 0.4 e²/h.

Fig. 6

Characteristics of the AH conductivity in a heavily doped (Bi_1-xMn_x)₂Se₃ sample (x = 0.074). Plotted in panels a–d are $\sigma _{xx}$ dependences of n_s, ${{\rm \sigma }}_{{{\rm AH}},1}$, ${{\rm \sigma }}_{{{\rm AH}},2}$, and the ratio $\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$, respectively. e Schematic band diagrams of the (Bi,Mn)₂Se₃ sample with high and low Fermi levels. The data shown in panels a–d were extracted from the Hall effect data taken from Sample F at T = 1.6 K

Discussion

The magnetism in the bulk of Mn-doped Bi₂Se₃ thin films have been studied with many techniques, including superconducting quantum interference device (SQUID) magnetometry¹⁷, ferromagnetic resonance¹⁸, polarized neutron reflectivity¹⁸, and X-ray absorption fine structure²². These measurements suggest a FM phase in the bulk with a Curie temperature less than 10 K (typically about 5 K) and an in-plane easy axis of magnetization. According to refs. ^19,22,23, the Mn impurities are mostly divalent and located in the substitutional Bi sites if the doping level is not too high. It has been proposed that the bulk FM order in (Bi,Mn)₂Se₃ arise from carrier-mediated exchange coupling of Mn²⁺ ions with 3d⁵ spin configuration¹⁸, similar to the FM order in the classic diluted magnetic semiconductor (Ga,Mn)As²⁴.

In contrast to the overall consistent results on the bulk magnetism, the surface magnetism in (Bi,Mn)₂Se₃ thin films has been controversial. In an early work combining ARPES and X-ray magnetic circular dichroism (XMCD) measurements, Xu et al. suggested a surface FM order with T_C above 100 K and an energy gap up to ∼0.1 eV in the surface states due to the magnetic exchange interaction¹⁶. The gap size is, however, much greater than theoretical values for the Mn-doped TIs^25,26. The nearly one order of magnitude enhancement of the surface T_C over the bulk value is also at odds with theory that only predicts modest enhancement for the surface magnetism²⁶. Recently, Sánchez-Barriga et al. showed with the XMCD measurements that the surface T_C in (Bi, Mn)₂Se₃ thin films is about 10 K, slightly higher than the FM ordering temperature in the bulk²⁰. They also found that in the same films the surface energy gap as large as ~0.2 eV can persist up to room temperature, and more strikingly similar energy gaps were also observed in Bi₂Se₃ thin films doped with non-magnetic impurities²⁰. Based on these findings and previous theoretical work^27,28, it is concluded that the energy gap seen in (Bi,Mn)₂Se₃ cannot be attributed to the magnetic exchange interaction, and it rather originates from resonant scatterings between the bulk impurities and the surface states²⁰. The ARPES measurements in ref. ²⁰ were, however, carried out at temperatures above T_C. It is therefore unclear that how the non-magnetic scatterings competes with the surface FM order in the ground states of (Bi,Mn)₂Se₃. It is necessary to address both experimentally and theoretically how the non-magnetic interactions introduced by the doping influences the spin structures and electron transport properties in magnetically doped TIs.

Given that the FM order has been confirmed for both the bulk and surface states in (Bi,Mn)₂Se₃ thin films, it is very puzzling that the AH effect has not been observed in previous transport experiments^17,29. The AH data presented in this article thus fill an important void in the literature. The distinctively different characteristics of the two AH components strongly suggest they are separately contributed by two sources. On one hand, the positive component $\sigma _{{{\rm AH}},1}$ always gets suppressed when the gate voltage is lowered. Since the (Bi,Mn)₂Se₃ samples remain n-type for the whole gate-voltage range, decreasing the gate voltage (or making it more negative) always lowers the bulk electron density until the bulk carriers are fully depleted. It is noteworthy that σ_xx decreases monotonically with decreasing V_G. Assignment of $\sigma _{{{\rm AH}},1}$ to the bulk states can thus provide a straightforward explanation of its dependence on σ_xx, since for all known mechanisms of the AH effect in bulk systems, reducing both σ_xx and n_s would lead to a drop in the AH conductivity^1,2. It is also very common to observe the AH effect which has the same sign as the ordinary Hall effect in a conventional magnetic system^1,2. On the other hand, for the samples with low Mn concentrations (e.g., x = 0.02), the negative component $\sigma _{{{\rm AH}},2}$ becomes larger with decreasing gate voltage. As will be detailed below, the sign and the chemical potential dependence of $\sigma _{{{\rm AH}},2}$ qualitatively agree with a previous experiment on Mn-doped Bi₂(Te,Se)₃ thin films, in which the AH effect is dominated by the intrinsic contribution of the surface states, at least at low carrier densities⁹. Assignment of $\sigma _{{{\rm AH}},2}$ to the surface states is also consistent with the prediction that perpendicular anisotropy is favored for the magnetization of the TIs surface states^26,30,31. As described in the previous section, $\sigma _{{{\rm AH}},2}$ has a saturation field much lower than the positive component $\sigma _{{{\rm AH}},1}$, which arises from the bulk FM order with the in-plane anisotropy (Please see Supplementary Note 3 and Supplementary Figures 9 and 10 for further discussion of the magnetic order in (Bi,Mn)₂Se₃ thin films).

The chemical-potential dependence of AH conductivity in the TI surface states has been explained with the theories based on the massive Dirac fermion model^32,33,34,35. On the simplest level, the ferromagnetically ordered magnetic impurities are treated as an exchange energy term $\Delta = J{\mathbf{S}} \cdot {\mathbf{s}}$ in the Hamiltonian of surface states, where $J$ is the exchange constant between the impurity spin ${\mathbf{S}}$ and surface electron spin ${\mathbf{s}}$. This term turns the gapless surface states into a gapped 2D Dirac band with ${\it{\epsilon }} =\pm \left[ {\left( {\hbar ck} \right)^2 + \Delta ^2} \right]^{\frac{1}{2}}$, where c is the Fermi velocity and k is the Fermi wavevector. The energy gap is thus 2Δ, corresponding to an effective mass m = $\frac{\Delta }{{{{c}}^2}}$, which could be positive or negative, depending on the sign of exchange constant ${{\rm J}}$. In the mean field level, the intrinsic AH conductivity is given by $\sigma _{{{\rm AH}}}^{{{\rm int}}} = - \frac{{e^2}}{{2h}}\frac{\Delta }{{{\it{\epsilon }}_{{\rm F}}}}$, where $\epsilon _{{\rm F}}$ is Fermi energy. Contributions from extrinsic sources, i.e., side jump and skew scatterings, can modify the AH conductivity substantially. According to Ado et al.³⁵, the total AH conductivity in the weakly disorder limit is given by

$$\sigma _{{{\rm AH}}}^{{{\rm tot}}} = - \frac{{8e^2}}{h}\frac{{{\it{\epsilon }}_{{\rm F}}\Delta ^3}}{{\left( {{\it{\epsilon }}_{{\rm F}}^2 + 3\Delta ^2} \right)^2}}$$

(1)

The magnitude of $\sigma _{{{\rm AH}}}^{{{\rm tot}}}$ decreases monotonically from $\frac{1}{2}\frac{{{{e}}^2}}{h}$ toward zero as ${\it{\epsilon }}_{{\rm F}}$ increases from $|\Delta |$ (corresponding to the upper gap edge) to larger values. In the limit of ${\it{\epsilon }}_{{\rm F}} \gg \Delta ,$ Eq. (1) reduces to $\sigma _{{{\rm AH}}}^{{{\rm tot}}} \propto \left( {\frac{\Delta }{{{\it{\epsilon }}_{{\rm F}}}}} \right)^3$. In the (Bi,Sb)₂(Te,Se)₃ family of magnetic TIs, the sign of m (and hence the sign of intrinsic surface AH effect) can be tuned by varying the chemical composition. For instance, the (Bi,Sb)₂Te₃ thin films doped with Cr¹⁴, V³⁶, or Mn³⁷ exhibit positive AH resistances, whereas the Mn-doped (Bi,Sb)₂(Te,Se)₃ has an opposite sign⁹. These results suggest that the sign difference may be attributed to the details of the p–d exchange interactions between the Mn ions and the anions (Te²⁻ or Se²⁻). For the samples with low Mn concentration, the gate-voltage dependence of the negative component $\sigma _{{{\rm AH}},2}$ is qualitatively in agreement with the above theory. As shown in Fig. 5, when ${\it{\epsilon }}_{{\rm F}}$ in sample C (x = 0.02) is lowered by gating, $\sigma _{{{\rm AH}},2}^{\rm s}$ increases by an order of magnitude. Based on the Hall effect data, as well as the ARPES results reported in ref. ²⁰, ${\it{\epsilon }}_{{\rm F}} \approx 0.3$ eV can be estimated for the large positive gate voltages. Following Eq. (1) and using the $\sigma _{{{\rm AH}},2}^{\rm s}$ values in Fig. 5, one obtains a magnetic gap Δ of the order 10 meV, which is comparable to the non-magnetic energy gap Δ_nm observed in (Bi_1.98Mn_0.02)₂Se₃ thin film with ARPES measurements²⁰. The fair agreement between the experiment and the massive Dirac fermion model can be attributed to ${\it{\epsilon }}_{{\rm F}} \gg \Delta _{{{\rm nm}}}$, which makes the non-magnetic effect of impurities insignificant.

For the samples with high Mn concentration or with very low Fermi levels, the surface AH conductivity can no longer be described with the massive Dirac fermion model. As shown in Fig. 6, $\sigma _{{{\rm AH}},2}^{\rm s}$ in sample F (x = 0.074) becomes smaller with decreasing electron density, opposite to the chemical potential dependence given by Eq. (1). According to ref. ²⁰, a Mn concentration of x = 0.074 would lead to a non-magnetic gap of 2Δ_nm ≈ 0.2 eV, which is comparable to the Fermi energy ${\it{\epsilon }}_F$ for the entire range of gate voltages. For sample C (x = 0.02), the observed saturation of $\sigma _{{{\rm AH}},2}^{\rm s}$ at large negative gate voltages can also be attributed to the chemical potential being comparable to Δ_nm. The strong resonance scatterings between the magnetic impurities in the bulk and the surface states have been identified to the origin of the robust non-magnetic gap observed in the ARPES experiment²⁰. The ground state spin structure is expected to deviate significantly from the standard hedgehog spin texture derived from the massive Dirac fermion model³², because of such impurity scattering effects of the bulk dopants. This can account for the suppression of the surface state AH conductivity ($\sigma _{{{\rm AH}},2}^{\rm s}$) shown in Figs. 5 and 6 (as well as in Supplementary Figure 8), when the $\epsilon _{{\rm F}}$ becomes comparable to Δ_nm. It is noteworthy that the non-magnetic impurity scatterings have also been invoked to explain the absence of the energy gap on the TI surfaces covered by magnetic dopants²⁷. According to recent theoretical studies, such processes can also cause the gap filling effect²⁷, and even the sign reversal of the AH conductivity in magnetically doped TIs³⁸. In short, magnetic doping can lead to many intriguing quantum phenomena that requires further investigation into the microscopic connection between the surface and bulk states in TIs.

Finally, it is worth noting that the sign reversal of AH conductivity has also been observed in other magnetic systems, such as field effect structures of (Ga,Mn)As thin films³⁹ and heavily doped TI thin films Bi_1.78Cr_0.22(Te_1−ySe_y)₃⁴⁰. In these systems, however, the AH conductivity has only one component, either positive or negative, for a certain gate voltage and/or chemical composition. The sign reversal of the AH effect in the (Bi,Mn)₂Se₃ thin film is caused by the gate-voltage or doping induced change in the relative weight of the two AH components, which rather originate from different sources, namely the bulk states and surface states. The observed chemical potential dependence of the negative AH component reveals, for the first time, an important role of the non-magnetic potential scatterings of the magnetic impurities in the transport properties of the surface states in the magnetically doped TIs. It will also be very interesting to explore whether the interplay between the drastically different surface and bulk magnetizations, along with the competition from various impurity effects, could lead to novel spin structures, such as spin canting, noncolinear, or topological spin textures^11,41.

Methods

Thin film growth and characterization

The (Bi_1−xMn_x)₂Se₃ thin films were grown on SrTiO₃ (111) substrates in a molecular beam epitaxy (MBE) system with base pressure of 3 × 10⁻¹⁰ mbar. For the growth of high quality thin films, it is important to have atomically clean and flat substrates. The STO substrates were cleaned with solvents and deionized water, followed by annealing at 1150 °C for 4 h in high pure oxygen (99.999%) before being transferred into the MBE chamber. During the growth, the substrate temperature was kept at 200 °C with a tungsten-filament heater mounted on the back of the sample holder. High-purity Bi, Mn, and Se sources (99.999%) were used and the evaporation rates were controlled by adjusting the temperatures of effusion cells. The flux ratio of Bi and Se was set at about 1:20, and the magnetic doping level was regulated by the flux ratio between Mn and Bi fluxes. In this work, the Mn concentrations in the (Bi_1−xMn_x)₂Se₃ thin films were limited to x ≤ 8.8%. According to ref. ¹⁹, such low Mn doping levels are necessary for obtaining samples free from secondary phases. The growth process was monitored with an in situ reflection high-energy electron diffraction (RHEED), and the high quality of the epitaxial growth was confirmed by streaky RHEED patterns. The thicknesses of the (Bi,Mn)₂Se₃ films were chosen to be 10 nm to obtain good gate-tunability while avoiding the hybridization effect of the top and bottom surfaces. The (Bi_1−xMn_x)₂Se₃ samples were characterized by AFM, X-ray diffraction, and vibrating sample magnetometry.

Transport measurements

After being taken out of the MBE chamber, the (Bi_1−xMn_x)₂Se₃ samples used for electron transport measurements were quickly patterned into Hall bars by hand scratching in order to minimize the surface degrading effects. Electrical contacts as well as the backgate electrodes were prepared with thermally deposited of Cr/Au bilayers. The transport measurements were carried out in a helium vapor flow cryostat with temperatures down to 1.6 K and magnetic fields up to 9 T with standard lock-in technique. The samples were typically excited by a 100 nA ac current with a frequency of ~13.7 Hz. The back-gate voltage was applied with a source-measurement unit with the leakage current limited below 2 nA.

Analyses of the Hall effect data

The AH resistances are obtained by subtracting the ordinary Hall resistances from the raw Hall effect data. The method is described in detail in Supplementary Note 1 and illustrated in Supplementary Figures 1 and 2. Its validity is further confirmed with the temperature dependent data shown in Supplementary Figures 2 and 3. Additional AH data extracted for two lightly doped samples (Samples D and H) are depicted in Supplementary Figure 4. Two methods for separating the two AH components are described in Supplementary Note 2 and illustrated in Supplementary Figures 5 and 6. The consistency between these two methods is shown in Supplementary Figures 7 and 8.

Data availability

The data that support the findings of this study are available from the corresponding author upon request.