## Introduction

Magnetically doped topological insulators (MTI) form a cornerstone in the field of topological quantum materials. Particularly in Cr-doped and V-doped (Bi,Sb)2Te3, the combination of ferromagnetism and a topologically non-trivial electronic band structure led to the discovery of the quantum anomalous Hall (QAH) effect1,2,3,4,5, i.e. a dissipationless quantised edge-state transport in the absence of external magnetic fields. These materials are now being widely utilised for possible realisations of topological superconductor6 and axion insulator states7, as well as in the context of metrology8 and spintronic functionalities9. However, despite the broad interest in these dilute MTI, controversy still remains as to the microscopic origin of the ferromagnetism and to the electronic states inducing the ferromagnetic (FM) coupling.

In a pioneering work, predicting the QAH effect in transition metal (TM)-based MTI, the FM state was proposed to arise from a van Vleck mechanism, as a result of strong spin–orbit coupling (SOC) and the topologically non-trivial band ordering in these materials10. Although some experimental support for this scenario has been reported2,11,12, more recent first-principles calculations find that the strength of the exchange interactions in Cr:(Bi,Sb)2Te3 and V:(Bi,Sb)2Te3 is, in fact, largely independent of SOC, suggesting that the van Vleck mechanism, in the form proposed in ref. 10, plays no decisive role13,14. Other theoretical works predict a dependence of the magnetic coupling on the precise configuration of the impurity 3d states13,14,15,16,17,18, as it is known in the context of dilute magnetic semiconductors19,20,21,22,23,24. Experimentally, the robustness of the FM state in Cr:(Bi,Sb)2Te3 and V:(Bi,Sb)2Te3 films was indeed found to vary substantially with dopant type and host stoichiometry3,25,26,27,28,29,30,31. It was speculated that these variations arise from differences in the electronic structure of the two systems3. So far, however, a comprehensive understanding of how this behaviour is related to the local impurity electronic structure is still absent. A realistic theory of the impurity 3d states in MTI shall not only explain the differences observed in the electronic and magnetic properties of these two coined QAH insulators, but also bring important insights for the physical description of other MTI, with possible implications to other TM-based van der Waals materials.

In this work, we present a combination of key experiments and theory that establishes the fundamental link between local impurity electronic structure and magnetic coupling in MTI. By means of X-ray magnetic circular dichroism (XMCD) and resonant photoelectron spectroscopy (resPES) we systematically probe the electronic and magnetic fingerprints of the 3d states of V and Cr impurities embedded in the bulk of (BixSb1−x)2Te3 thin films, as well as the effect of Bi/Sb substitution in the host. Supported by multiplet ligand field theory (MLFT) and ab initio density functional theory (DFT) calculations, our results unveil the essential role of impurity-state-mediated exchange interactions underlying the magnetic properties of V-doped and Cr-doped (Bi,Sb)2Te3 thin films, paving the way towards a more general theory of the magnetic interactions in MTI.

## Results

### Electronic and magnetic ground state of V and Cr impurities

We start by presenting in Fig. 1a, b the X-ray absorption (XAS) and XMCD spectra from V0.1(Bi0.32Sb0.68)1.9Te3 and Cr0.1(Bi0.1Sb0.9)1.9Te3, respectively, at the V and Cr L2,3 edges, measured at temperatures above (hollow circles) and below (full circles) the Curie temperature TC (Supplementary Fig. 1). No energy shifts or changes in the branching ratio are observed at the L2,3 edges with varying temperature across TC, contradicting recent reports where an apparent energy shift was interpreted as evidence of the van Vleck mechanism11. The XMCD spectra were measured under a small applied magnetic field of 10 mT (remanent state), oriented perpendicular to the surface, at 5 K. They confirm a persistent FM state at this temperature, with a sizable magnetic moment carried by the TM 3d states for both V-doped and Cr-doped (Bi,Sb)2Te3 systems.

A state-of-the-art MLFT analysis of the V and Cr L2,3 lineshapes has been performed (Supplementary Fig. 1) and described in detail elsewhere32. We find a strong deviation from the 3+ ionic (V d2, Cr d3) ground state25,29,30, with resulting d-shell occupations of d3.13 (mS = 2.33μB per atom) for V and d4.37 (mS = 3.32μB per atom) for Cr. Correspondingly, our first-principles calculations for single V and Cr impurities embedded in a quintuple layer of Sb2Te3, on an octahedral Sb site, find d3.35 (mS = 2.51μB per atom) and d4.40 (mS = 3.72μB per atom), (Supplementary Fig. 5 and below). This implies a substantial charge transfer (CT) from the ligand p states into the 3d impurity states as a result of strong pd hybridisation (Supplementary Fig. 4).

### Fingerprint of the impurity 3d states

We next assess the contribution of the 3d states to the valence band (VB) by resPES at the V and Cr L3 edges. The data sets shown in Fig. 2a, b are obtained from the normalised difference between the on-resonant and off-resonant photoemission spectra, taken at 30 K (Supplementary Fig. 2), and represent the V and Cr 3d partial DOS. The data establish a remarkably pronounced difference in the character of the 3d DOS for V and Cr impurities. For V, the 3d states pile up predominantly in a narrow peak just below EF with the centre at EB = 170 meV. For Cr, on the other hand, the 3d states are broadly distributed over the host VB, with a maximum at EB = 1.71 eV and low spectral weight near EF. Our first-principles calculations nicely capture these main characteristics. Figure 2c shows the calculated spin-polarised 3d DOS and the corresponding integrated DOS for V (upper panel) and Cr (lower panel) impurities in Sb2Te3, which allow us to assign the experimentally observed states to majority-spin (spin-up) t2g states (Supplementary Fig. 6). The examination of the integrated 3d DOS at EF allows us to extract the predicted d-shell filling of d3.4 for V (red curve) and d4.4 for Cr (blue curve) impurities, in good agreement with our MLFT analysis. Furthermore, our calculations are able to reproduce our resPES data without self-interaction corrections, which suggests a minor role of the latter for the V and Cr occupied 3d states. While small self-interaction corrections tend to shift the 3d resonances deeper in the VB13,14,18,21, this is compensated by the natural charge doping of the host, as we discuss below. Our findings yet support previous resPES results33,34, and recent scanning tunnelling microscopy and spectroscopy studies of the electronic fingerprints of substitutional V and Cr impurities in Sb2Te318,35,36.

The presence of exchange-split 3d states at EF suscitate the emergence of localised magnetic moments and, possibly, long-range magnetic order. In order to study the effect of the impurity states on the magnetic coupling mechanism, we first consider the spin-up DOS of V and Cr impurities in Sb2Te3, calculated for different artificial EF shifts (ΔEF), i.e. different positions of EF with respect to the bulk VB and conduction band (CB) (Supplementary Fig. 3). This approach allows us to simulate the effect of n-type and p-type charge doping on the 3d states, i.e. the valence of the TM impurities37. The results for the spin-up V and Cr 3d DOS are shown in Fig. 3a, b, respectively. The grey-shaded areas correspond to the Sb2Te3 host total DOS, emphasising the position of the impurity states with respect to the bulk band gap, where the topological surface states (TSS) typically lie. We find that the position of the impurity states depends sensitively on the charge doping, shifting towards higher binding energies when going from p-type (dark-coloured curves) to n-type (light-coloured curves) doping. In particular, the narrow V 3d peak gradually moves away from EF. This trend is experimentally supported by our resPES data for (BixSb1−x)2Te3 films with different Bi concentrations x, which will be discussed later. For higher x, and thus higher n-doping3,15,16, the V 3d peak shifts to significantly higher binding energies.

### Impurity-state-mediated magnetic exchange interactions

Based on our findings regarding the local electronic and magnetic impurity configurations, we now discuss their implications for the magnetic exchange interactions. In Fig. 4a, b we plot the calculated exchange-coupling constants Jij for different separation distances between two TM ions located in one atomic plane (intra-layer coupling, upper panels) and at neighbouring atomic planes within the same quintuple layer (inter-layer coupling, lower panels), using the same ΔEF values as in Fig. 3a, b (note the corresponding colour code). For both V and Cr, the interlayer coupling is dominant, in good agreement with ref. 17. Overall, at nearest-neighbour (NN) distances the calculated Jij contributions are larger for V, while at the second and third neighbour positions the Jij values for Cr are markedly larger. This behaviour follows the spatial decay of the calculated impurity-induced features in the 3d DOS (Supplementary Fig. 7). Most importantly, we find that Jij strongly varies with charge doping. Already small shifts of EF give rise to considerable changes in Jij, whose sign and strength are linked to characteristic features in the 3d DOS at EF. For V, Jij rapidly decreases in the n-doped regime, while for Cr the behaviour is more complex and depends more strongly on the relative impurity positions. Figure 4c, d show the energy dependence of Jij(E) for NN spins for ΔEF = −300 meV, which best corresponds to our experimental resPES data. The inter-layer and intra-layer components are, respectively, depicted as dashed and solid lines, and they exhibit qualitatively similar trends. In the same plots, the corresponding spin-up and spin-down 3d DOS are shown as shaded areas. For V, Jij(E) near EF consists of a sharp peak overlapping with a broad flat ridge that spans between the onsets of the t2g and eg peaks. Jij(E) is maximum when EF is positioned inside the V t2g manifold, and rapidly decreases otherwise. For Cr, the sharp peak in Jij(E) is absent, and only the broad ridge is seen. In sharp contrast to V, we find for Cr a maximal Jij(E) when EF lies in the gap between the t2g and eg peaks, where the 3d DOS is low. The calculated Jij(E) in Fig. 4d, thus, indicates a less pronounced dependence of the magnetic coupling on the EF position for Cr. This may explain the stronger gate-voltage dependence of the magnetic properties observed in V-doped3 as compared to Cr-doped films1,12,29. These distinct features in the V and Cr 3d DOS are fully in line with our resPES measurements in Fig. 2. In the context of dilute magnetic semiconductors, similar characteristics in the exchange coupling constants have been extensively discussed19,20,23,24,38,39,40. By comparison, we may tentatively associate the sharp peak in Jij(E) for V with the double-exchange mechanism14,17,19,20,22,24,31 and the broad ridge, found for both dopant types, with the FM superexchange mechanism13,19,23,38.

### Effect of the host charge doping on the impurity 3d states

In order to verify the predicted trends of the impurity-mediated magnetic exchange interactions with the charge doping of the host, we now discuss the effect of the host stoichiometry on the local electronic and magnetic properties of V-doped (BixSb1−x)2Te3 via Bi/Sb substitution. Our resPES and XMCD data from V0.1(Bi0.32Sb0.68)1.9Te3 (dark red curve) and V0.1Bi1.9Te3 (light red curve) in Fig. 5 demonstrate the strong sensitivity of the magnetic coupling on the position of the V impurity states. The V 3d maximum in Fig. 5a is found at EB = 170 meV in the former, and at EB = 300 meV in the latter (EB = 130 meV for x = 0.24, as seen in Supplementary Fig. 2), confirming the trend predicted by our DFT calculations (Fig. 3). This may be attributed to the effect of CT from the Bi ions into the V impurities (Supplementary Fig. 4)15,16,41. Figure 5b shows a comparison of V L2,3 XMCD spectra for the same samples, measured at saturation (hollow circles) and in remanence (full circles) at 5 K. While in Sb-rich V:(Bi,Sb)2Te3 the remanent and saturated spectra almost coincide, in V0.1Bi1.9Te3 the remanent XMCD is critically lower than in saturation, demonstrating a suppression of FM interactions coinciding with the shift of the V states away from EF, as predicted in our theoretical calculations.

## Discussion

The observed weakening of the ferromagnetism upon Bi doping in V:(Bi,Sb)2Te3 is in agreement with previous works3,28,31, as well reported for single-crystalline Cr:(Bi,Sb)2Te326. It is known from recent XMCD studies18,26,29,31,32 that the Sb ions become partially polarised in the presence of substitutional V or Cr, while Bi, on the other hand, does not. Thus, our findings are consistent with a scenario where loosely localised, spin-polarised holes in Sb ions facilitate a longer ranged magnetic coupling in both systems13,19,20,21,22. The substitution of Sb by Bi, thus, gradually disables the network of spin-polarised p orbitals that contribute to stabilise a more robust FM state.

Our results yet show that the sharp V and Cr resonances are mostly localised inside the bulk band gap (see Fig. 3), indicating a considerable overlap with the TSS. While the effect of the TSS on the magnetic interactions has not been explored here, our theory suggests that the scenario pictured in the bulk may possibly be affected by the presence of the Dirac electrons at the surface, potentially leading to modified magnetic properties as compared to the bulk.

In conclusion, our systematic experimental and theoretical results highlight the central role of impurity-state-mediated exchange coupling for the magnetism in the paradigmatic QAH insulators Cr:(Bi,Sb)2Te3 and V:(Bi,Sb)2Te3. The latter cannot be explained based solely on the van Vleck mechanism, bearing its origin on the topologically non-trivial band structure10,11,12. Instead, our theoretical calculations on the basis of pd hybridisation and exchange coupling unambiguously elucidate the experimental observations. They show that the nature and strength of the magnetic exchange coupling vary with the position of EF in the 3d DOS, i.e. with the occupation of the 3d states, thereby reconciling, in a unified theory, the differences observed between V and Cr doping of (BixSb1−x)2Te3 films, as well as unveiling the role of Sb in mediating a robust ferromagnetism in QAH insulators. The presence of the Dirac states might as well have further impact on the charge and magnetic ground states of 3d impurities in the vicinity of the surface and, consequently, on the pd magnetic interactions37. Conversely, the 3d impurity states may mediate spin scattering channels for the spin-momentum-locked Dirac electrons36. This advance in the knowledge on the microscopic electronic and magnetic properties in V-doped and Cr-doped (Bi,Sb)2Te3 may eventually facilitate an improved understanding of the microscopic origin of the QAH effect in these systems. Finally, the developed theory can be further applied to other instances of MTI for tailoring the properties of these materials in regard to potential spintronic applications.

## Methods

### Sample preparation, structural, transport and magnetic characterisation

Thin films (about 9–10 nm thick) of Crz(BixSb1−x)2−zTe3 and Vz(BixSb1−x)2−zTe3 were grown by molecular beam epitaxy (MBE) on hydrogen-passivated Si(111) substrates. The growth details and characterisation, e.g. by X-ray diffraction, atomic force microscopy and Hall magnetotransport, confirming the realisation of QAH effect in the V-doped samples (for x = 0.76−0.79 and z = 0.1–0.2), are published elsewhere5,28,42,43. After growth, the films were capped by a protective Te layer (~100 nm), which was mechanically removed in UHV conditions, prior to the spectroscopic measurements, revealing a chemically clean surface (Supplementary Fig. 8). Recent results have demonstrated the effectiveness of this decapping method on Bi2Te3 layers with high pristine quality44. The stoichiometries applied in this work are comparable to those that exhibited a stable and reproducible QAH effect.

### XAS, XMCD and resPES

The XAS and XMCD data was acquired at the HECTOR endstation, located at BOREAS beamline of the ALBA storage ring (Barcelona, Spain)45. The measurements were performed in total electron yield mode, under magnetic fields of up to 6 T and temperatures down to 5 K. The resPES experiments were conducted at the ASPHERE III endstation located at beamline P04 of the PETRA III storage ring of DESY (Hamburg, Germany). The on-resonant and off-resonant VB photoemission spectra were taken at hνon = 514.8 eV and hνoff = 508 eV for V-doped samples, and at hνon = 575.6 eV and hνoff = 560 eV for Cr-doped ones, according to the respective XAS spectra in Fig. 1. The energy resolution of the resPES measurements was typically better than 67 meV. All experiments were performed in ultra-high vaccuum (UHV) at pressures below 3 × 10−10 mbar.

### DFT calculations

The Sb2Te3 and Bi2Te3 bulk crystals were simulated using the experimental bulk lattice structure (see ref. 46 for Sb2Te3 and ref. 47 for Bi2Te3). The electronic structure was calculated within the local density approximation (LDA)48 to DFT by employing the full-potential relativistic Korringa–Kohn–Rostoker Green’s function method (KKR)49,50 with exact description of the atomic cells51,52. The truncation error arising from an $${\ell }_{\max }=3$$ cutoff in the angular momentum expansion was corrected for using Lloyd’s formula53. The V and Cr defects, together with a charge-screening cluster comprising the first two shells of neighbouring atoms (consisting of about 20 surrounding scattering sites), were embedded self-consistently using the Dyson equation in the KKR method50 and have been chosen to occupy the substitutional Sb/Bi position in the quintuple layers and structural relaxations were neglected, while keeping the direction of the impurity’s magnetic moment fixed along the out-of-plane direction. The shift in the Fermi level occurring in (Bi,Sb)2Te3 was accounted for by adjusting the self-consistently computed Fermi level of the host systems in the impurity embedding step. The exchange interactions among two impurities were computed using the method of infinitesimal rotations54, which map the exchange interaction to the Heisenberg Hamiltonian $${\mathcal{H}}=-\frac{1}{2}{\sum }_{\left\langle ij\right\rangle }{J}_{ij}{\overrightarrow{s}}\!_{i}\cdot {\overrightarrow{s}}\!_{j}$$.

### MLFT calculations

Theoretical XAS and XMCD spectra for the L2,3 (2p → 3d) absorption edges of V and Cr ions were calculated by means of a configuration interaction (CI) cluster model, considering the central TM ion surrounded by six ligands (Te anions). We take into account all the 2p−3d and 3d−3d electronic Coulomb interactions, as well as the SOC on every open shell of the absorbing atom. We consider nominal 2p63dn (n = 2 for V3+and n = 3 for Cr3+) configurations and further include three more CT states $${d}^{n+1}{\underline{L}}^{1}$$, $${d}^{n+2}{\underline{L}}^{2}$$ and $${d}^{n+3}{\underline{L}}^{3}$$ ($${\underline{L}}^{1}$$ denotes a hole in the Te 5p orbitals) to account for hybridisation effects. To perform the CI calculation, the following fit parameters were introduced: scaling parameter β for the Hartree–Fock values of the Slater integrals, the CT energy Δ, the Coulomb interaction energy Udd between the 3d electrons, the hybridisation energy Veg and the octahedral crystal field parameter 10Dq. The simulations were performed using the Quanty software for quantum many-body calculations, developed by M. W. Haverkort et al.55. We assume V/Cr ions embedded in the cation sites and describe the crystal field in Oh symmetry, with C4 axes of the octahedron along the V–Te bonds. The spectral contributions from each of the split ground state terms to the absorption spectra were weighted by a Boltzmann factor. The calculated spectra were broadened by a Gaussian function to account for the instrumental broadening and by an energy-dependent Lorentzian profile for intrinsic lifetime broadening.