Article | Open | Published:

# Two-component anomalous Hall effect in a magnetically doped topological insulator

Nature Communicationsvolume 9, Article number: 1282 (2018) | Download Citation

## Abstract

The anomalous Hall (AH) effect measurement has emerged as a powerful tool to gain deep insights into magnetic materials, such as ferromagnetic metals, magnetic semiconductors, and magnetic topological insulators (TIs). In Mn-doped Bi2Se3, however, the AH effect has never been reported despite a lot of previous studies. Here we report the observation of AH effect in (Bi,Mn)2Se3 thin films and show that the sign of AH resistances changes from positive to negative as the Mn concentration is increased. The positive and negative AH resistances are found to coexist in a crossover regime. Such a two-component AH effect and the sign reversal can also be obtained by electrical gating of lightly doped samples. Our results provide an important basis for understanding the puzzling interplay between the surface states, the bulk states, and various magnetic doping effects, as well as competing magnetic orders in magnetically doped TIs.

## Introduction

Anomalous Hall (AH) effect is one of the most fundamental transport properties of magnetic materials, in which the interplay between magnetism and spin-orbit coupling produces a transverse Hall voltage perpendicular to the applied current and the magnetization1,2. Despite more than a century of research, it was not until last decade the AH effect was associated with the Berry curvature of the occupied energy bands3,4. Study of the AH effect has since become a valuable tool to understand the electronic properties of magnetic metals and semiconductors, although interpretations of the experimental results are often complicated by the details of band structures as well as the existence of extrinsic sources of AH effect1,2, namely skew scattering5 and side jump6.

The sophisticated nature of AH effect and its connection to the band topology have recently been nicely illustrated in magnetically doped three-dimensional topological insulators (3D TIs, referred to as TIs below)7,8. The surface of a TI hosts symmetry-protected gapless Dirac electrons with spin direction locked perpendicular to their momenta. When doped with magnetic impurities, TIs can host a large variety of magnetic phases, such as ferromagnetic, paramagnetic, noncolinear, and spin glass phases, arising from various types of exchange interactions between the magnetic impurities in presence of the surface and bulk states9,10,11,12. The magnetically doped TIs have received intensive experimental efforts13, and the work on ferromagnetic (FM) order in Cr-doped (Bi,Sb)2Te3 thin films has led to the seminal discovery of quantum anomalous Hall effect (QAHE)14. In contrast, magnetic doping of Bi2Se3, a prototype TI isostructural to Bi2Te3 and Sb2Te3, has not allowed for the observation of QAHE, even though gapped surface states have been observed in angular resolved photoemission spectroscopy (ARPES) experiments15,16. Among them, a particularly puzzling system is the Mn-doped Bi2Se3, in which even the AH effect has not been discovered despite compelling evidence for FM order in both the bulk and surface states17,18,19,20. More surprisingly, a recent study combining ARPES and other techniques has strongly suggested a non-magnetic origin for the surface energy gap in the Mn-doped Bi2Se3, in contradiction to the earlier findings15,20. It remains an open question how the non-magnetic gap, or more generally, the non-magnetic scattering effect of the magnetic impurities competes with magnetic interactions in the Mn-doped Bi2Se3 and other magnetically doped TIs. Unfortunately, very few experimental techniques can probe the magnetic order in both the surface and bulk states at very low temperatures. Therefore, a systematic study of the AH effect in the Mn-doped Bi2Se3 thin films will provide much needed information for understanding the complicated impurity-related physics in magnetically doped TIs.

Here, we report a detailed study of the AH effect in the Mn-doped Bi2Se3 thin films. By controlling the Mn doping level and tuning the chemical potential, the sign of the AH resistance can be varied from positive to negative. The positive and negative AH resistances are found to coexist in a wide range of parameters, and they exhibit qualitatively different dependences on the applied magnetic field and gate voltage. To the best of our knowledge, such a two-component AH effect has never been observed in any magnetically doped TI, despite that both the surface and bulk magnetic orders in TIs have been a subject of intense experimental study. Our results suggest that the non-magnetic scattering effects of the magnetic dopants can have a profound impact on the transport properties of magnetic TIs, and should be taken into account in seeking high quality TI-based magnetic materials.

## Results

### Basic characterization

The (Bi1−xMn x )2Se3 thin films studied in this work were grown with molecular beam epitaxy (MBE) on SrTiO3(111) substrates. The samples were then patterned into ~200 μm wide Hall bars (Fig. 1a) for the electron transport measurements. Detailed description of the sample preparation process and the methods for magnetization and transport measurements are given in the Methods section. Figure 1b depicts an atomic force microscopy (AFM) image of a 10 quintuple-layer (QL) thick (Bi1−xMn x )2Se3 film with Mn concentration x = 0.02. The surface morphology of the film is characterized by triangular shaped terraces with sizes of the order 100 nm, which are commonly seen in Bi2Se3 thin films21. Figure 1c shows the magnetic field dependence of in-plane magnetization of a 70 nm thick (Bi1−xMn x )2Se3 (x = 0.088) thin film. The overall feature of the magnetization is consistent with those reported in refs.17,20, in which the easy magnetization axes of the (Bi1−xMn x )2Se3 thin films are found to be in the film plane. Figure 1d displays the magnetoresistances (MRs) of a set of (Bi1−xMn x )2Se3 samples with Mn concentration x = 0–0.08 (samples A–G). All the samples exhibit positive MR and have a cusp-shaped minimum at B = 0 when the gate voltage VG is set at zero. The corresponding Hall resistance (R yx ) curves are nearly linear and have negative slopes (Fig. 1e), indicating n-type charge carriers. The sheet electron densities extracted with ns = −1/(eRH) are in a range of 0.48–1.29 × 1013 cm−2 for the (Bi1−xMn x )2Se3 thin films with x = 0.018–0.088 (samples B–G), where RH is the (ordinary) Hall coefficient. The obtained electron densities are substantially lower than ns 2.7 × 1013 cm−2 for an undoped Bi2Se3 thin film prepared in the similar condition (sample A). The basic transport parameters of these samples are summarized in Table 1, and the corresponding band diagrams are schematically drawn in Fig. 1f. The lower Fermi level in the doped samples can be attributed to the substitution of Bi3+ ions by Mn2+ ions19.

### Two-component AH effect

Figure 2a–f shows the Hall resistance curves for a subset of (Bi1−xMn x )2Se3 thin films (x = 0–0.088), and the corresponding AH resistances (Fig. 2g–l) are obtained by subtracting the linear (ordinary) component from the total Hall resistances, i.e., $$R_{{{\rm AH}}} \simeq R_{yx} - R_{{\rm H}}B$$. For the undoped Bi2Se3 thin film (sample A), the nonlinear part of the Hall resistance is nearly zero for the entire field range. In contrast, the AH resistances are clearly visible for all the Mn-doped films (samples B, D−G). At low doping levels (e.g., x = 1.8%, sample B), the sign of the AH resistance (RAH) above the (positive) magnetization saturation field is positive and opposite to that of the ordinary Hall resistance RHB. Throughout this work, we refer to this as the positive RAH, which has the same sign as the ordinary Hall resistance of the hole-type carriers. In contrast, the samples with high Mn doping levels (x = 0.074 and 0.088) exhibit the negative AH resistance. As shown in Fig. 2, the increase in the Mn concentration drives a crossover from the positive to negative RAH. At intermediate doping levels, a kink structure begins to appear in the RAH curves. This suggests coexistence of a positive RAH component with a negative one. Such a two-component AH effect can be observed for a wide range of Mn concentrations (x ≥ 2.4%). Figure 2 further shows that these two components have different dependences on the Mn doping level: the magnitude of the negative RAH component becomes larger with increasing Mn concentration, whereas the positive component has a nonmonotonic dependence. The overall trend is that negative component becomes more pronounced relative to the positive component with increasing Mn doping level, as evidenced by the reversal of the sign of RAH in the high magnetic fields.

### Gate-voltage tuning of the AH effect

The sign reversal in RAH and the two-component AH effect can also be obtained by gate-voltage tuning. This is illustrated with sample C (x = 0.02) in Fig. 3. As the gate voltage is decreased from VG = +100 to −210 V, the sheet electron density is reduced from ns = 0.91 × 1013 to 0.35 × 1013 cm−2, based on the Hall effect measurements. The sheet resistance per square (i.e., sheet resistivity $$\rho _{xx}$$) increases from 2.7 to 6.0 kΩ in the same gate-voltage range. At high electron densities (V G  ≥ 100 V), RAH only has the positive component. When the electron density is lowered by gating, a kink structure emerges at low magnetic fields and becomes more pronounced. At the lowest electron density, the magnitude of the negative component surpasses that of the positive one. Moreover, the negative RAH component tends to saturate at a magnetic field lower than that of the positive component. Similar phenomena also take place for the ungated samples with higher Mn-doping levels (x = 0.024–0.088, Fig. 2).

The gate-voltage tuning of the two-component AH effect has also been performed for samples with higher Mn concentrations. Figure 4 shows the results of sample F (x = 0.074), in which the sheet electron density can be varied from ns = 1.4 to 0.5 × 1013 cm−2 and the sheet resistivity from 5 to 65 kΩ. At high electron densities (corresponding to $$V_{\rm G} \ge 100$$ V), the magnitude of the positive component is larger than that of the negative component. When the electron density is sufficiently low (ns ≤ 0.7 × 1013 cm−2, VG ≤ 0 V), the magnitude of negative component increases rapidly with decreasing gate voltage and becomes considerably larger than the positive component. Like the other (Bi,Mn)2Se3 samples, the negative RAH component in sample F always has a lower saturation field than the positive component, and the relative magnitude between the negative and positive components is significantly enhanced when the electron density is reduced.

### Separation of the two AH components

The conspicuous differences in the two AH components suggest that they originate from different electronic states. Given that multiple parallel conduction channels coexist in the (Bi,Mn)2Se3 samples, it is more convenient to analyze the AH effect data in the form of conductivities. In general, the sheet longitudinal conductivity and Hall conductivity are given by $$\sigma _{xx} = \mathop {\sum }\nolimits_i \sigma _{xx,i} = \rho _{xx}/(\rho _{xy}^2 + \rho _{xx}^2)$$ and $$\sigma _{xy} = \mathop {\sum }\nolimits_i \sigma _{xy,i} = \rho _{yx}/(\rho _{xy}^2 + \rho _{xx}^2)$$, where the summation goes over all conduction channels and i is the channel index. Because of strong impurity scattering effect, most of the transport discussed in this work is in the low mobility regime (i.e., $$\mu B \ll 1$$), and correspondingly $$\left| {\rho _{xy}} \right| \ll \rho _{xx}$$. The Hall conductivity can thus be simplified to σ xy ρ yx /$$\rho _{xx}^2$$ (with ρ yx ρ xy ) and the AH conductivity becomes σAHRAH$$/R_{xx}^2$$. Following this definition, σAH has the same sign as the AH resistance RAH.

As described in the Supplementary Notes 1 and 2, the AH conductivity can be separated into two components: $$\sigma _{{{\rm AH}}}(B) = \sigma _{{{\rm AH}},1}(B) + \sigma _{{{\rm AH}},2}(B)$$, where the first and second terms denote the positive and negative AH components, respectively. Figure 5 summarizes the results for sample C (x = 0.02). As shown in Fig. 5a, the sheet conductivity $${{\rm \sigma }}_{xx}$$ drops from 9.1 to 4.7 e2/h as $$n_{\rm s}$$ is tuned from 0.82 to 0.35 × 1013 cm−2. Figure 5b,c display the conductivity dependences of $$\sigma _{{{\rm AH}},1}^{\rm s}$$ and $$\sigma _{{{\rm AH}},2}^{\rm s}$$, the magnitudes of the extracted $${{\rm \sigma }}_{{{\rm AH}},1}$$ and $${{\rm \sigma }}_{{{\rm AH}},2}$$ above the saturation fields. The positive component $$\sigma _{{{\rm AH}},1}^{\rm s}$$ decreases nearly linearly with decreasing $${{\rm \sigma }}_{xx}$$, whereas the negative component $$\sigma _{{{\rm AH}},2}^{\rm s}$$ become greater until $${{\rm \sigma }}_{xx}$$ drops to 6 e2/h. The ratio between the two components, $$\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$$, increases monotonically from nearly zero to over 1.6 when ns is varied from 0.82 to 0.35 × 1013 cm−2 (Fig. 5d).

Figure 6 summarizes the results for sample F (x = 0.074). Figure 6a shows that σ xx values of this sample are substantially lower than those of sample C at the same carrier densities. Similar to sample C, $$\sigma _{{{\rm AH}},1}^{\rm s}$$ of sample F also has an approximately linear dependence on σ xx , as depicted in Fig. 6b. However, $$\sigma _{{{\rm AH}},2}^{\rm s}$$ of this sample has a drastically different dependence on σ xx : it becomes smaller as σ xx decreases (Fig. 6c). Nonetheless, the $$\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$$ ratio of sample F has a qualitatively similar dependence on σ xx to sample C, namely increasing monotonically with decreasing σ xx . As depicted in Fig. 6d, the $$\sigma _{{{\rm AH}},2}^{\rm s}/\sigma _{{{\rm AH}},1}^{\rm s}$$ value increases from about 0.5 to 1.5 as σ xx decreases from 5.6 e2/h to about 0.4 e2/h.

## Discussion

The magnetism in the bulk of Mn-doped Bi2Se3 thin films have been studied with many techniques, including superconducting quantum interference device (SQUID) magnetometry17, ferromagnetic resonance18, polarized neutron reflectivity18, and X-ray absorption fine structure22. These measurements suggest a FM phase in the bulk with a Curie temperature less than 10 K (typically about 5 K) and an in-plane easy axis of magnetization. According to refs. 19,22,23, the Mn impurities are mostly divalent and located in the substitutional Bi sites if the doping level is not too high. It has been proposed that the bulk FM order in (Bi,Mn)2Se3 arise from carrier-mediated exchange coupling of Mn2+ ions with 3d5 spin configuration18, similar to the FM order in the classic diluted magnetic semiconductor (Ga,Mn)As24.

In contrast to the overall consistent results on the bulk magnetism, the surface magnetism in (Bi,Mn)2Se3 thin films has been controversial. In an early work combining ARPES and X-ray magnetic circular dichroism (XMCD) measurements, Xu et al. suggested a surface FM order with TC above 100 K and an energy gap up to 0.1 eV in the surface states due to the magnetic exchange interaction16. The gap size is, however, much greater than theoretical values for the Mn-doped TIs25,26. The nearly one order of magnitude enhancement of the surface TC over the bulk value is also at odds with theory that only predicts modest enhancement for the surface magnetism26. Recently, Sánchez-Barriga et al. showed with the XMCD measurements that the surface TC in (Bi, Mn)2Se3 thin films is about 10 K, slightly higher than the FM ordering temperature in the bulk20. They also found that in the same films the surface energy gap as large as ~0.2 eV can persist up to room temperature, and more strikingly similar energy gaps were also observed in Bi2Se3 thin films doped with non-magnetic impurities20. Based on these findings and previous theoretical work27,28, it is concluded that the energy gap seen in (Bi,Mn)2Se3 cannot be attributed to the magnetic exchange interaction, and it rather originates from resonant scatterings between the bulk impurities and the surface states20. The ARPES measurements in ref. 20 were, however, carried out at temperatures above TC. It is therefore unclear that how the non-magnetic scatterings competes with the surface FM order in the ground states of (Bi,Mn)2Se3. It is necessary to address both experimentally and theoretically how the non-magnetic interactions introduced by the doping influences the spin structures and electron transport properties in magnetically doped TIs.

Given that the FM order has been confirmed for both the bulk and surface states in (Bi,Mn)2Se3 thin films, it is very puzzling that the AH effect has not been observed in previous transport experiments17,29. The AH data presented in this article thus fill an important void in the literature. The distinctively different characteristics of the two AH components strongly suggest they are separately contributed by two sources. On one hand, the positive component $$\sigma _{{{\rm AH}},1}$$ always gets suppressed when the gate voltage is lowered. Since the (Bi,Mn)2Se3 samples remain n-type for the whole gate-voltage range, decreasing the gate voltage (or making it more negative) always lowers the bulk electron density until the bulk carriers are fully depleted. It is noteworthy that σ xx decreases monotonically with decreasing VG. Assignment of $$\sigma _{{{\rm AH}},1}$$ to the bulk states can thus provide a straightforward explanation of its dependence on σ xx , since for all known mechanisms of the AH effect in bulk systems, reducing both σ xx and ns would lead to a drop in the AH conductivity1,2. It is also very common to observe the AH effect which has the same sign as the ordinary Hall effect in a conventional magnetic system1,2. On the other hand, for the samples with low Mn concentrations (e.g., x = 0.02), the negative component $$\sigma _{{{\rm AH}},2}$$ becomes larger with decreasing gate voltage. As will be detailed below, the sign and the chemical potential dependence of $$\sigma _{{{\rm AH}},2}$$ qualitatively agree with a previous experiment on Mn-doped Bi2(Te,Se)3 thin films, in which the AH effect is dominated by the intrinsic contribution of the surface states, at least at low carrier densities9. Assignment of $$\sigma _{{{\rm AH}},2}$$ to the surface states is also consistent with the prediction that perpendicular anisotropy is favored for the magnetization of the TIs surface states26,30,31. As described in the previous section, $$\sigma _{{{\rm AH}},2}$$ has a saturation field much lower than the positive component $$\sigma _{{{\rm AH}},1}$$, which arises from the bulk FM order with the in-plane anisotropy (Please see Supplementary Note 3 and Supplementary Figures 9 and 10 for further discussion of the magnetic order in (Bi,Mn)2Se3 thin films).

The chemical-potential dependence of AH conductivity in the TI surface states has been explained with the theories based on the massive Dirac fermion model32,33,34,35. On the simplest level, the ferromagnetically ordered magnetic impurities are treated as an exchange energy term $$\Delta = J{\mathbf{S}} \cdot {\mathbf{s}}$$ in the Hamiltonian of surface states, where $$J$$ is the exchange constant between the impurity spin $${\mathbf{S}}$$ and surface electron spin $${\mathbf{s}}$$. This term turns the gapless surface states into a gapped 2D Dirac band with $${\it{\epsilon }} =\pm \left[ {\left( {\hbar ck} \right)^2 + \Delta ^2} \right]^{\frac{1}{2}}$$, where c is the Fermi velocity and k is the Fermi wavevector. The energy gap is thus 2Δ, corresponding to an effective mass m = $$\frac{\Delta }{{{{c}}^2}}$$, which could be positive or negative, depending on the sign of exchange constant $${{\rm J}}$$. In the mean field level, the intrinsic AH conductivity is given by $$\sigma _{{{\rm AH}}}^{{{\rm int}}} = - \frac{{e^2}}{{2h}}\frac{\Delta }{{{\it{\epsilon }}_{{\rm F}}}}$$, where $$\epsilon _{{\rm F}}$$ is Fermi energy. Contributions from extrinsic sources, i.e., side jump and skew scatterings, can modify the AH conductivity substantially. According to Ado et al.35, the total AH conductivity in the weakly disorder limit is given by

$$\sigma _{{{\rm AH}}}^{{{\rm tot}}} = - \frac{{8e^2}}{h}\frac{{{\it{\epsilon }}_{{\rm F}}\Delta ^3}}{{\left( {{\it{\epsilon }}_{{\rm F}}^2 + 3\Delta ^2} \right)^2}}$$
(1)

The magnitude of $$\sigma _{{{\rm AH}}}^{{{\rm tot}}}$$ decreases monotonically from $$\frac{1}{2}\frac{{{{e}}^2}}{h}$$ toward zero as $${\it{\epsilon }}_{{\rm F}}$$ increases from $$|\Delta |$$ (corresponding to the upper gap edge) to larger values. In the limit of $${\it{\epsilon }}_{{\rm F}} \gg \Delta ,$$ Eq. (1) reduces to $$\sigma _{{{\rm AH}}}^{{{\rm tot}}} \propto \left( {\frac{\Delta }{{{\it{\epsilon }}_{{\rm F}}}}} \right)^3$$. In the (Bi,Sb)2(Te,Se)3 family of magnetic TIs, the sign of m (and hence the sign of intrinsic surface AH effect) can be tuned by varying the chemical composition. For instance, the (Bi,Sb)2Te3 thin films doped with Cr14, V36, or Mn37 exhibit positive AH resistances, whereas the Mn-doped (Bi,Sb)2(Te,Se)3 has an opposite sign9. These results suggest that the sign difference may be attributed to the details of the p–d exchange interactions between the Mn ions and the anions (Te2− or Se2−). For the samples with low Mn concentration, the gate-voltage dependence of the negative component $$\sigma _{{{\rm AH}},2}$$ is qualitatively in agreement with the above theory. As shown in Fig. 5, when $${\it{\epsilon }}_{{\rm F}}$$ in sample C (x = 0.02) is lowered by gating, $$\sigma _{{{\rm AH}},2}^{\rm s}$$ increases by an order of magnitude. Based on the Hall effect data, as well as the ARPES results reported in ref. 20, $${\it{\epsilon }}_{{\rm F}} \approx 0.3$$ eV can be estimated for the large positive gate voltages. Following Eq. (1) and using the $$\sigma _{{{\rm AH}},2}^{\rm s}$$ values in Fig. 5, one obtains a magnetic gap Δ of the order 10 meV, which is comparable to the non-magnetic energy gap Δnm observed in (Bi1.98Mn0.02)2Se3 thin film with ARPES measurements20. The fair agreement between the experiment and the massive Dirac fermion model can be attributed to $${\it{\epsilon }}_{{\rm F}} \gg \Delta _{{{\rm nm}}}$$, which makes the non-magnetic effect of impurities insignificant.

For the samples with high Mn concentration or with very low Fermi levels, the surface AH conductivity can no longer be described with the massive Dirac fermion model. As shown in Fig. 6, $$\sigma _{{{\rm AH}},2}^{\rm s}$$ in sample F (x = 0.074) becomes smaller with decreasing electron density, opposite to the chemical potential dependence given by Eq. (1). According to ref. 20, a Mn concentration of x = 0.074 would lead to a non-magnetic gap of 2Δnm ≈ 0.2 eV, which is comparable to the Fermi energy $${\it{\epsilon }}_F$$ for the entire range of gate voltages. For sample C (x = 0.02), the observed saturation of $$\sigma _{{{\rm AH}},2}^{\rm s}$$ at large negative gate voltages can also be attributed to the chemical potential being comparable to Δnm. The strong resonance scatterings between the magnetic impurities in the bulk and the surface states have been identified to the origin of the robust non-magnetic gap observed in the ARPES experiment20. The ground state spin structure is expected to deviate significantly from the standard hedgehog spin texture derived from the massive Dirac fermion model32, because of such impurity scattering effects of the bulk dopants. This can account for the suppression of the surface state AH conductivity ($$\sigma _{{{\rm AH}},2}^{\rm s}$$) shown in Figs. 5 and 6 (as well as in Supplementary Figure 8), when the $$\epsilon _{{\rm F}}$$ becomes comparable to Δnm. It is noteworthy that the non-magnetic impurity scatterings have also been invoked to explain the absence of the energy gap on the TI surfaces covered by magnetic dopants27. According to recent theoretical studies, such processes can also cause the gap filling effect27, and even the sign reversal of the AH conductivity in magnetically doped TIs38. In short, magnetic doping can lead to many intriguing quantum phenomena that requires further investigation into the microscopic connection between the surface and bulk states in TIs.

Finally, it is worth noting that the sign reversal of AH conductivity has also been observed in other magnetic systems, such as field effect structures of (Ga,Mn)As thin films39 and heavily doped TI thin films Bi1.78Cr0.22(Te1−ySe y )340. In these systems, however, the AH conductivity has only one component, either positive or negative, for a certain gate voltage and/or chemical composition. The sign reversal of the AH effect in the (Bi,Mn)2Se3 thin film is caused by the gate-voltage or doping induced change in the relative weight of the two AH components, which rather originate from different sources, namely the bulk states and surface states. The observed chemical potential dependence of the negative AH component reveals, for the first time, an important role of the non-magnetic potential scatterings of the magnetic impurities in the transport properties of the surface states in the magnetically doped TIs. It will also be very interesting to explore whether the interplay between the drastically different surface and bulk magnetizations, along with the competition from various impurity effects, could lead to novel spin structures, such as spin canting, noncolinear, or topological spin textures11,41.

## Methods

### Thin film growth and characterization

The (Bi1−xMn x )2Se3 thin films were grown on SrTiO3 (111) substrates in a molecular beam epitaxy (MBE) system with base pressure of 3 × 10−10 mbar. For the growth of high quality thin films, it is important to have atomically clean and flat substrates. The STO substrates were cleaned with solvents and deionized water, followed by annealing at 1150 °C for 4 h in high pure oxygen (99.999%) before being transferred into the MBE chamber. During the growth, the substrate temperature was kept at 200 °C with a tungsten-filament heater mounted on the back of the sample holder. High-purity Bi, Mn, and Se sources (99.999%) were used and the evaporation rates were controlled by adjusting the temperatures of effusion cells. The flux ratio of Bi and Se was set at about 1:20, and the magnetic doping level was regulated by the flux ratio between Mn and Bi fluxes. In this work, the Mn concentrations in the (Bi1−xMn x )2Se3 thin films were limited to x ≤ 8.8%. According to ref. 19, such low Mn doping levels are necessary for obtaining samples free from secondary phases. The growth process was monitored with an in situ reflection high-energy electron diffraction (RHEED), and the high quality of the epitaxial growth was confirmed by streaky RHEED patterns. The thicknesses of the (Bi,Mn)2Se3 films were chosen to be 10 nm to obtain good gate-tunability while avoiding the hybridization effect of the top and bottom surfaces. The (Bi1−xMn x )2Se3 samples were characterized by AFM, X-ray diffraction, and vibrating sample magnetometry.

### Transport measurements

After being taken out of the MBE chamber, the (Bi1−xMn x )2Se3 samples used for electron transport measurements were quickly patterned into Hall bars by hand scratching in order to minimize the surface degrading effects. Electrical contacts as well as the backgate electrodes were prepared with thermally deposited of Cr/Au bilayers. The transport measurements were carried out in a helium vapor flow cryostat with temperatures down to 1.6 K and magnetic fields up to 9 T with standard lock-in technique. The samples were typically excited by a 100 nA ac current with a frequency of ~13.7 Hz. The back-gate voltage was applied with a source-measurement unit with the leakage current limited below 2 nA.

### Analyses of the Hall effect data

The AH resistances are obtained by subtracting the ordinary Hall resistances from the raw Hall effect data. The method is described in detail in Supplementary Note 1 and illustrated in Supplementary Figures 1 and 2. Its validity is further confirmed with the temperature dependent data shown in Supplementary Figures 2 and 3. Additional AH data extracted for two lightly doped samples (Samples D and H) are depicted in Supplementary Figure 4. Two methods for separating the two AH components are described in Supplementary Note 2 and illustrated in Supplementary Figures 5 and 6. The consistency between these two methods is shown in Supplementary Figures 7 and 8.

### Data availability

The data that support the findings of this study are available from the corresponding author upon request.

## Additional information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## References

1. 1.

Nagaosa, N., Sinova, J., Onoda, S., MacDonald, A. H. & Ong, N. P. Anomalous Hall effect. Rev. Mod. Phys. 82, 1539 (2010).

2. 2.

Xiao, D., Chang, M. C. & Niu, Q. Berry phase effects on electronic properties. Rev. Mod. Phys. 82, 1959 (2010).

3. 3.

Jungwirth, T., Niu, Q. & MacDonald, A. H. Anomalous Hall effect in ferromagnetic semiconductors. Phys. Rev. Lett. 88, 2070208 (2002).

4. 4.

Haldane, F. D. M. Berry curvature on the fermi surface: anomalous Hall effect as a topological fermi-liquid property. Phys. Rev. Lett. 93, 206602 (2002).

5. 5.

Fert, A. Skew scattering in alloys with Cerium impurities. J. Phys. F: Met. Phys. 3, 2126 (1973).

6. 6.

Berger, L. Side-jump mechanism for the Hall effect of ferromagnets. Phys. Rev. B: Solid State 2, 4559 (1970).

7. 7.

Hasan, M. Z. & Kane, C. L. Colloquium: topological insulators. Rev. Mod. Phys. 82, 3045 (2010).

8. 8.

Qi, X. L. & Zhang, S. C. Topological insulators and superconductors. Rev. Mod. Phys. 83, 1057 (2011).

9. 9.

Checkelsky, J. G., Ye, J. T., Onose, Y., Iwasa, Y. & Tokura, Y. Dirac-fermion-mediated ferromagnetism in a topological insulator. Nat. Phys. 8, 729 (2012).

10. 10.

Janíček, P., Drašar, Č., Lošťák, P., Vejpravová, J. & Sechovskỳ, V. Transport, Magnetic, Optical and thermodynamic properties of Bi2−xMn x Se3 single crystals. Phys. B 403, 3553 (2008).

11. 11.

Yasuda, K. et al. Geometric Hall effects in topological insulator heterostructures. Nat. Phys. 12, 555 (2016).

12. 12.

Choi, J. et al. Mn-doped V2VI3 semiconductors: single crystal growth and magnetic properties. J. Appl. Phys. 97, 10D324 (2005).

13. 13.

Kou, X. F., Fan, Y. B., Lang, M. R., Upadhyaya, P. & Wang, K. L. Magnetic topological insulators and quantum anomalous Hall effect. Solid State Commun. 215–216, 34–53 (2015).

14. 14.

Chang, C. Z. et al. Experimental observation of the quantum anomalous Hall effect in a magnetic topological insulator. Science 340, 167 (2013).

15. 15.

Chen, Y. L. et al. Massive Dirac fermion on the surface of a magnetically doped topological insulator. Science 329, 659 (2010).

16. 16.

Xu, S. Y. et al. Hedgehog spin texture and Berry’s phase tuning in a magnetic topological insulator. Nat. Phys. 8, 616 (2012).

17. 17.

Zhang, D. M. et al. Interplay between ferromagnetism, surface states, and quantum corrections in a magnetically doped topological insulator. Phys. Rev. B 86, 205127 (2012).

18. 18.

von, H. J. et al. Ferromagnetism in Bi2Se3:Mn epitaxial layers. Phys. Rev. B 88, 075149 (2013).

19. 19.

Collins-McIntyre, L. J. et al. X-ray magnetic spectroscopy of MBE-grown Mn-doped Bi2Se3thin films. AIP Adv. 4, 127136 (2014).

20. 20.

Sánchez-Barriga, J. et al. Nonmagnetic band gap at the Dirac point of the magnetic topological insulator (Bi1 x Mn x )2Se3. Nat. Commun. 7, 10559 (2016).

21. 21.

He, L., Kou, X. F. & Wang, K. L. Review of 3D topological insulator thin-film growth by molecular beam epitaxy and potential applications. Phys. Status Solidi RRL 7, 50–63 (2013).

22. 22.

Figueroa, A. I. G. et al. Local structure and bonding of transition metal dopants in Bi2Se3 topological insulator thin films. J. Phys. Chem. C. 119, 17344 (2015).

23. 23.

Babakiray, S. et al. Structural properties of Bi2−xMn x Se3 thin films grown via molecular beam epitaxy. J. Appl. Phys. 118, 045302 (2015).

24. 24.

Dietl, T. & Ohno, H. Dilute ferromagnetic semiconductors: physics and spintronic structures. Rev. Mod. Phys. 86, 187 (2014).

25. 25.

Abanin, D. A. & Pesin, D. A. Ordering of magnetic impurities and tunable electronic properties of topological insulators. Phys. Rev. Lett. 106, 136802 (2011).

26. 26.

Rosenberg, G. & Franz, M. Surface magnetic ordering in topological insulators with bulk magnetic dopants. Phys. Rev. B 85, 195119 (2012).

27. 27.

Black-Schaffer, A. M. & Balatsky, A. V. Subsurface impurities and vacancies in a three-dimensional topological insulator. Phys. Rev. B 86, 115433 (2012).

28. 28.

Fransson, J., Black-Schaffer, A. M. & Balatsky, A. V. Engineered near-perfect backscattering on the surface of a topological insulator with nonmagnetic impurities. Phys. Rev. B 90, 241409(R) (2014).

29. 29.

Choi, Y. H. et al. Simple tuning of carrier type in topological insulator Bi2Se3 by Mn doping. Appl. Phys. Lett. 101, 152103 (2012).

30. 30.

Liu, Q., Liu, C. X., Xu, C., Qi, X. L. & Zhang, S. C. Magnetic impurities on the surface of a topological insulator. Phys. Rev. Lett. 102, 156603 (2009).

31. 31.

Efimkin, D. K. & Galitski, V. Self-consistent theory of ferromagnetism on the surface of a topological insulator. Phys. Rev. B 89, 115431 (2014).

32. 32.

Sinitsyn, N. A., MacDonald, A. H., Jungwirth, T., Dugaev, V. K. & Sinova, J. Anomalous Hall effect in a two-dimensional Dirac band: the link between the Kubo-Streda formula and the semiclassical Boltzmann equation approach. Phys. Rev. B 75, 045315 (2007).

33. 33.

Lu, H. Z., Shan, W. Y., Yao, W., Niu, Q. & Shen, S. Q. Massive Dirac fermions and spin physics in an ultrathin film of topological insulator. Phys. Rev. B 81, 115407 (2010).

34. 34.

Lu, H. Z. & Shen, S. Q. Extrinsic anomalous Hall conductivity of a topologically nontrivial conduction band. Phys. Rev. B 88, 081304(R) (2013).

35. 35.

Ado, I. A., Dmitriev, I. A., Ostrovsky, P. M. & Titov, M. Anomalous Hall effect with massive Dirac fermions. Eur. Phys. Lett. 111, 37004 (2015).

36. 36.

Chang, C. Z. et al. High-precision realization of robust quantum anomalous Hall state in a hard ferromagnetic topological insulator. Nat. Mater. 14, 473 (2015).

37. 37.

Lee, J. S. et al. Ferromagnetism and spin-dependent transport in n-type Mn-doped bismuth telluride thin films. Phys. Rev. B 89, 174425 (2014).

38. 38.

Deng, M. X. et al. Competing effects of magnetic impurities in the anomalous Hall effect on the surface of a topological insulator. Phys. Rev. B 94, 235116 (2016).

39. 39.

Chiba, D. et al. Anomalous Hall effect in field-effect structures of (Ga,Mn)As. Phys. Rev. Lett. 104, 106601 (2010).

40. 40.

Zhang, J. S. et al. Topology-driven magnetic quantum phase transition in topological insulators. Science 339, 1582 (2013).

41. 41.

Zhu, J. J., Yao, D. X., Zhang, S. C. & Chang, K. Electrically controllable surface magnetism on the surface of topological insulators. Phys. Rev. Lett. 106, 097201 (2011).

Download references

## Acknowledgements

We thank H. Z. Lu, S. Q. Shen, and P. Ostrovsky for valuable discussions. This work was supported by the National Key Research and Development Program (Project No. 2016YFA0300600), the National Science Foundation of China (Projects No. 61425015, No. 11604374, and No. 11374337), the National Basic Research Program of China (Projects No. 2015CB921102), and the Strategic Initiative Program of Chinese Academy of Sciences (Projects No. XDPB0801, No. XDB070200, and No. XDPB0602).

## Author information

### Affiliations

1. #### Beijing National Laboratory for Condensed Matter Physics, Institute of Physics, Chinese Academy of Sciences, Beijing, 100190, China

• Nan Liu
• , Jing Teng
•  & Yongqing Li
2. #### School of Physical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China

• Nan Liu
• , Jing Teng
•  & Yongqing Li
3. #### Beijing Key Laboratory for Nanomaterials and Nanodevices, Beijing, 100190, China

• Jing Teng
•  & Yongqing Li

### Contributions

Y.L. and J.T. initiated the project. L.N. and J.T. carried out the MBE growth. L.N. fabricated the devices and performed the electron transport measurements. L.N. and Y.L. analysed the data. All of the authors prepared the manuscript.

### Competing interests

The authors declare no competing interests.

### Corresponding authors

Correspondence to Jing Teng or Yongqing Li.

## About this article

### DOI

https://doi.org/10.1038/s41467-018-03684-0

## Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.