## Abstract

Manipulating the spins of the topological surface states represents an essential step towards exploring the exotic quantum states emerging from the time reversal symmetry breaking via magnetic doping or external magnetic fields. The latter case relies on the Zeeman effect and thereby we need to estimate the *g*-factor of the topological surface state precisely. Here, we report the direct observations of the Zeeman effect at the surfaces of Bi_{2}Se_{3} and Sb_{2}Te_{2}Se by spectroscopic-imaging scanning tunnelling microscopy. The Zeeman shift of the zero mode Landau level is identified unambiguously by appropriately excluding the extrinsic effects arising from the nonlinearity in the band dispersion of the topological surface state and the spatially varying potential. Surprisingly, the *g*-factors of the topological surface states in Bi_{2}Se_{3} and Sb_{2}Te_{2}Se are very different (+18 and −6, respectively). Such remarkable material dependence opens up a new route to control the spins of the topological surface states.

## Introduction

The helical Dirac fermions on the surfaces of topological insulators (TIs) host novel relativistic quantum phenomena in solids^{1,2}. When the time reversal symmetry (TRS) of the topological surface state (TSS) is broken, a gap opens at the Dirac point. This brings about novel topological excitations, such as the magneto-electric effect^{3,4}, the quantum anomalous Hall effect^{3,5} and the magnetic monopole effect^{6}. Magnetic doping has proven to be an effective way to break the TRS via magnetic exchange interactions^{7,8,9}, thereby enabling the experimental observation of the quantum anomalous Hall state^{10}. However, magnetic dopants may introduce charge inhomogeneity^{11} and weaken the spin–orbit coupling of TI compounds^{12}.

The Zeeman effect, the coupling of spins with a magnetic field, offers an alternative way to break the TRS of the TSS without causing any of the above issues. The opened gap Δ is fully tunable by a perpendicular magnetic field *B*, that is, Δ=*g*_{s}*μ*_{B}*B*, where *g*_{s} is the electron *g*-factor of the TSS and *μ*_{B} is the Bohr magneton^{13}. Consequently, to manipulate the TSS via the Zeeman effect, it is crucial to know its *g*_{s}.

Under *B*, Landau levels (LLs) are formed as a result of the cyclotron motion of the electrons. Inclusion of the Zeeman effect can lead to an energetic shift of the LLs as the corresponding TSS lack spin degeneracy (Fig. 1a). This is in stark contrast to the Zeeman splitting of the LLs observed in graphene^{14} and conventional two-dimensional (2D) electron systems^{15}. Such a Zeeman effect is most pronounced for the zeroth LL (LL_{0}), and decreases dramatically with increasing Landau index *n* (Supplementary Note 1).

Understanding the Zeeman shift behaviour and determining the *g*-factor of the TSS have entailed intense research investigations primarily by the means of quantum oscillation measurements^{13,16,17}. However, the reported data are still largely controversial. The inconsistency is due to the fact that the host materials are inevitably doped by defects. This prevents the lower LLs, which exhibit prominent Zeeman shifts, from reaching the Fermi level and contributing to quantum oscillations in the accessible *B* (ref. 16). Furthermore, the sign of *g*_{s} cannot be determined from the Zeeman shift of the nonzero LLs^{17}. A recent tunnelling spectroscopy study appears to be successfully probing the LL_{0} of the TSS formed at an interface with a conventional semiconductor^{18}. However, the method used depends on specific details of band-bending in such heterostructures, and therefore cannot be readily applied to other TI compounds. Spectroscopic imaging scanning tunnelling microscopy (SI-STM) can access electronic states in a wide energy range with high spatial and energy resolutions. Thus, it can be used to study any LL, regardless of the doping level of the TI compounds.

Using this technique, we have developed a methodology to unambiguously observe the Zeeman shift of LL_{0}. This allows us to determine the *g*-factors of the TSSs in Bi_{2}Se_{3} and Sb_{2}Te_{2}Se precisely, which turn out to be very different.

## Results

### Modelling the Zeeman effect of the TSS

In principle, the Zeeman shift of the LL_{0} energy (*E*_{0}) of the TSS is linear with *B*, and its slope determines *g*_{s}. In practice, however, more factors are involved, hindering its direct observation. On one hand, a finite curvature is superimposed on the linear dispersion of the TSS in actual compounds (Fig. 1b). This induces an extra *B*-linear change in *E*_{0} that is irrelevant to the Zeeman effect. On the other hand, there exists spatial potential variations in the TSS originating from the inhomogenously distributed charged defects^{19}. This introduces an extrinsic *B* dependence of LL energies as the spatial extension of the LL wave functions shrinks with increasing *B* (Fig. 1c).

To quantify the effects of nonlinear dispersions and the potential variations, we consider the following model Hamiltonian^{17}.

Here, **σ** are the Pauli matrices and **II** are the canonical momenta. The first term denotes the nonlinearity of band dispersions, with *m** being the effective mass relative to that of the free electron (*m*_{e}). The second term represents the helical Dirac component of the TSS, with *v* being the electron velocity. The third term is the Zeeman term and the last term represents the potential variation. The energy of the *n-*th LL, *E*_{n}, has been given in ref. 20, albeit without contribution from the last term of equation (1).

To include the effect of this term, we consider a 2D parabolic potential model to approximate the shape and location of the potential extremes, where *E*_{D} is the Dirac-point energy. At the potential extreme, the *B*-dependence of *E*_{0} can be calculated using a first-order approximation^{21} (Supplementary Note 1). Including the effects of nonlinear dispersions and potential variations, the *E*_{0}(*B*) is accordingly given as

Note that *m** renormalizes *g*_{s}, and the potential variation introduces an additional 1/*B* dependence term. To determine the intrinsic *g*_{s}, we have developed an analysis scheme for our SI-STM data, which corrects these extrinsic factors, *m** and *α*_{x}+*α*_{y}, and applied it to two different TI materials, Bi_{2}Se_{3} and Sb_{2}Te_{2}Se.

### SI-STM study of the Zeeman effect of the TSS

As the first step, we evaluate *m** using momentum-resolved LL spectroscopy^{22}. Figure 2a,b display the LL spectra of Bi_{2}Se_{3} and Sb_{2}Te_{2}Se (Supplementary Note 2, Supplementary Figs 1and 2), respectively, as measured at a fixed location at various *B*. In contrast to the electron-doped Bi_{2}Se_{3}, Sb_{2}Te_{2}Se is hole-doped and thus its Dirac point is in the empty state. For each material, the corresponding *E*_{n} exhibits a quasi-linear scaling relation with a scaling variable (*nB*)^{1/2} (Fig. 2c), which represents the energy–momentum dispersion of the TSS^{22}. The potential effect on *E*_{n} is more significant for the LLs with small *n* exposed to high *B* than those with large *n* exposed to low *B* (ref. 21). In this regard, the observed scaling with (*nB*)^{1/2} demonstrates that such potential effect has negligible influence on it at the measured location. For both compounds, a finite curvature is evidently seen in the dispersions indicating that *m** is finite, as also demonstrated by angle-resolved photoemission spectroscopy measurements and band calculations^{7,23}. Remarkably, both compounds show a very similar band curvature, despite their different constituent elements. We thus expect *m** to be nearly the same for both the compounds. To evaluate this, *m** is extracted from the scaling function, while ignoring the potential variations. For each compound, we have performed three measurements at different locations/samples. On the basis of these measurements, *m** is found to be 0.12±0.03 for Bi_{2}Se_{3} and 0.13±0.02 for Sb_{2}Te_{2}Se with negligible inhomogeneity.

Next, we assess the impact of potential variations on *E*_{0}(*B*). The spatial variation of *E*_{0} represents the potential landscape, although it is smeared out by the magnetic length *l*_{B} (refs 19, 21). We start with Bi_{2}Se_{3} and map out the potential landscape by performing a spectroscopic imaging of *E*_{0} at a high *B* of 11 T, where *l*_{B}∼7.7 nm (Fig. 3a,d). We focus on areas around the potential extremes (Fig. 3a,d) and fit their shapes with the 2D parabolic potential model introduced above (Supplementary Fig. 3). After positioning the tip at the fitted potential centre (Fig. 3a,d, cross point), the *B* dependence of the LL_{0} peak is measured (Fig. 3b,e). The measured *E*_{0}(*B*) is plotted in Fig. 3c,f (black symbols). Intriguingly, *E*_{0}(*B*) exhibits ∼1/*B* behaviours at both the potential minimum (*α*_{x}>0,*α*_{y}>0) and maximum (*α*_{x}<0,*α*_{y}<0), but shifts towards opposite directions. This is exactly expected from equation (2) and directly highlights the influence of potential variations on *E*_{0}(*B*).

To eliminate the potential effect, we estimate the last term of equation (2) using the fitted values of *α*_{x} and *α*_{y}. After subtracting the estimated potential effect, we obtain a nearly *B* independent *E*_{0} in the high *B* region at both the potential minimum and maximum (red symbols in Fig. 3c,f). This validates our methodology and indicates that the effect of *m** and the Zeeman effect contribute oppositely with an accidental cancellation. By further subtracting the contribution from *m**, we obtain the genuine Zeeman shift (Fig. 3c,f, blue symbols). Subsequently, *g*_{s} in Bi_{2}Se_{3} is determined to be +18±4. (The error in *g*_{s} is the propagation of uncertainty evaluated from the errors of *m** and , which are extracted above and measured at different potential extremes, respectively.) This differs significantly from its corresponding bulk value +32 as indicated by magneto-transport and nuclear magnetic resonance measurements^{24,25}.

The same methodology is applied to Sb_{2}Te_{2}Se (Fig. 4; Supplementary Figs 4 and 5). Distinct from Bi_{2}Se_{3}, the LL_{0} state measured at the potential minimum (Fig. 4b) exhibits a nonmonotonic *B* dependence (Fig. 4c) and increases with *B* at high fields. Given that the effective masses of both compounds are almost the same, *g*_{s} in Sb_{2}Te_{2}Se must differ significantly from that of Bi_{2}Se_{3}. Indeed, we evaluate *g*_{s} in Sb_{2}Te_{2}Se to be −6±2. While both systems share very similar band dispersions in their TSSs, their *g*-factors turn out to be strikingly different in both size and sign.

## Discussion

We elucidate the above results under the framework of **k**·**p** theory. For narrow gap semiconductors, the conduction bands are coupled to the spin–orbit-split valence bands through a second-order perturbative term. Such a coupling substantially enhances the orbital sector of the *g*-factor. As a result, the total *g*-factor of such electrons/holes can significantly deviate from that of the free electrons in both magnitude and sign^{26}. In the case of TIs, the strong atomic spin–orbit coupling creates a symmetry-inverted band gap. Thus, the atomic orbital characters of the wave function undergo a strong variation at and in the vicinity of the inverted band edges^{27}. Such a variation manifests itself in the orbital character of TSS around the Dirac point as well^{28}. As Sb_{2}Te_{2}Se and Bi_{2}Se_{3} comprise different elements, their wave functions must have different orbital characters. Such a difference further implies that the *g*-factors of the two compounds should be different too. Note that a considerable energy dependence of the *g*-factor is expected, since the orbital character of the wave function changes notably in these systems. Our measurement is merely around the Dirac point, which is directly relevant to the gap opening of the TSS via *B*. In this regard, the *g*-factor of the TSS may be different from that of the bulk since they are measured at different energies. Further theoretical investigations regarding those factors are needed to develop a general theory that describes the *g*-factor.

Considering the significant material dependence of *g*_{s}, we envision an interesting possibility to tailor the *g*-factor of the TSS by controlling the chemical composition of the chalcogenide TI materials in the form of solid solutions^{29,30}. This provides a new knob in manipulating the TSS for its spin-related applications.

## Methods

### Experiment description

The experiments were performed with a modified commercial UNISOKU low temperature STM at 4.4 or 1.5 K. Magnetic fields up to 12 T can be applied perpendicularly to the sample surface. Sb_{2}Te_{2}Se and Bi_{2}Se_{3} crystals grown by a modified Bridgman technique were cleaved *in situ* under ultrahigh vacuum conditions at∼77 K. After cleaving, the crystals were transferred quickly to the low-temperature STM for subsequent measurements. Two Bi_{2}Se_{3} and three Sb_{2}Te_{2}Se samples were measured. A tungsten tip was used as the STM probe, which had been cleaned and characterized with a field-ion microscope. The tunnelling spectra were obtained by lock-in detection of the tunnelling current with a modulation voltage at 617.3 Hz feeding into the sample bias. The tip was grounded to provide the reference voltage.

### Evaluation of the finite curvature of LL scaling

We evaluated *m**, which characterizes the finite curvature, using the scaling analysis of the LL energies shown in Fig. 2c. Since only the Zeeman shift of the LL_{0} is prominent, Supplementary equation (10) (Supplementary Note 1) can be approximated by . This indicates that the scaling of *E*_{n} with (*nB*)^{1/2} still applies even in the case of finite *m**. Regarding *E*_{0}, its energy at 3 T was used for the scaling analysis, because its shift is negligible at low *B*. By fitting the low-energy part of the scaling relation, the *m** value can be obtained. It must be noted that the low-energy fitting considerably deteriorated when high-energy data were also included. Since the effect of *m** on *E*_{0} is determined by the electronic states around the Dirac point, we only fitted the low-energy parts.

### Fitting the potential extremes with 2D parabolic potential model

First, two sectional lines across the potential extreme were drawn and fitted with a one-dimensional parabolic potential to estimate its shape and location. The obtained parameters were subsequently input as the initial guess of the 2D parabolic fitting. The 2D fitting provided the shape and location of the fitted potential, whose equipotential lines were superimposed on the potential map (Supplementary Figs 3a,b and 4a,b). The generated fitting error was small, demonstrating that the 2D parabolic potential model fitted the measured potential well. This conclusion was further augmented by evaluating two sectional lines extracted along the major eclipse axes of both the fitted potential and the measured *E*_{0} map (Supplementary Figs 3c–f and 4c–f). Since the 2D parabolic potential merely applies to potential extremes, it cannot be used at low *B*, where the LL_{0} state spatially expands beyond the potential extremes. (The effect of the potential variations at low *B* is discussed in Supplementary Note 3, Supplementary Figs 6 and 7). Therefore, for the fitting of Figs 3c,f and 4c,f, we did not include the data below the critical *B* value, where the potential-corrected *E*_{0} begins to deviate from the *B*-linear behaviour. The size of the LL_{0} state (2*l*_{B}) at the critical *B* (Supplementary Figs 3a,b and 4a,b, dashed circles) has a good correspondence with the fitted area, which substantiates our model.

## Additional information

**How to cite this article:** Fu, Y.-S. *et al*. Observation of Zeeman effect in topological surface state with distinct material dependence. *Nat. Commun.* 7:10829 doi: 10.1038/ncomms10829 (2016).

## References

- 1.
Hasan, M. Z. & Kane, C. L. Topological insulators.

*Rev. Mod. Phys.***82**, 3045–3067 (2010). - 2.
Qi, X. L. & Zhang, S. C. Topological insulators and superconductors.

*Rev. Mod. Phys.***83**, 105–1110 (2010). - 3.
Qi, X. L., Hughes, T. L. & Zhang, S. C. Topological field theory of time-reversal invariant insulators.

*Phys. Rev. B***78**, 195424 (2008). - 4.
Essin, A. M., Moore, J. E. & Vanderbilt, D. Magnetoelectric polarizability and axion electrodynamics in crystalline insulators.

*Phys. Rev. Lett.***102**, 146805 (2009). - 5.
Yu, R.

*et al.*Quantized anomalous Hall effect in magnetic topological insulators.*Science***329**, 61–64 (2010). - 6.
Qi, X. L., Li, R., Zang, J. & Zhang, S. C. Inducing a magnetic monopole with topological surface states.

*Science***323**, 1184–1187 (2009). - 7.
Chen, Y. L.

*et al.*Massive Dirac fermion on the surface of a magnetically doped topological insulator.*Science***329**, 659–662 (2010). - 8.
Checkelsky, J. G., Ye, J. T., Onose, Y., Iwasa, Y. & Tokura, Y. Dirac-fermion-mediated ferromagnetism in a topological insulator.

*Nat. Phys.***8**, 729–733 (2012). - 9.
Xu, S. Y.

*et al.*Hedgehog spin texture and Berry’s phase tuning in a magnetic topological insulator.*Nat. Phys.***8**, 616–622 (2012). - 10.
Chang, C. Z.

*et al.*Experimental observation of the quantum anomalous Hall effect in a magnetic topological insulator.*Science***340**, 167–170 (2013). - 11.
Zhang, D. M.

*et al.*Interplay between ferromagnetism, surface states, and quantum corrections in a magnetically doped topological insulator.*Phys. Rev. B***86**, 205127 (2012). - 12.
Zhang, J. S.

*et al.*Topology-driven magnetic quantum phase transition in topological insulators.*Science***339**, 123905 (2013). - 13.
Analytis, J. G.

*et al.*Two-dimensional surface state in the quantum limit of a topological insulator.*Nat. Phys.***6**, 960–964 (2010). - 14.
Song, Y. J.

*et al.*High-resolution tunnelling spectroscopy of a graphene quartet.*Nature***467**, 185–189 (2010). - 15.
Hashimoto

*et al.*Quantum Hall transition in real space: from localized to extended states.*Phys. Rev. Lett.***101**, 256802 (2008). - 16.
Xiong, J.

*et al.*High-field Shubnikov–de Haas oscillations in the topological insulator Bi_{2}Te_{2}Se.*Phys. Rev. B***86**, 045314 (2012). - 17.
Taskin, A. A. & Ando, Y. Berry phase of nonideal Dirac fermions in topological insulators.

*Phys. Rev. B***84**, 035301 (2011). - 18.
Yoshimi, R.

*et al.*Dirac electron states formed at the heterointerface between a topological insulator and a conventional semiconductor.*Nat. Mater.***13**, 253–257 (2014). - 19.
Fu, Y. S.

*et al.*Imaging the two-component nature of Dirac–Landau levels in the topological surface state of Bi_{2}Se_{3}.*Nat. Phys.***10**, 815–819 (2014). - 20.
Reynoso, A., Usaj, G., Sánchez, M. J. & Balseiro, C. A. Theory of edge states in systems with Rashba spin-orbit coupling.

*Phys. Rev. B***70**, 235344 (2004). - 21.
Okada, Y.

*et al.*Visualizing Landau levels of Dirac electrons in a one-dimensional potential.*Phys. Rev. Lett.***109**, 166407 (2012). - 22.
Hanaguri, T., Igarashi, K., Kawamura, M., Takagi, H. & Sasagawa, T. Momentum-resolved Landau-level spectroscopy of Dirac surface state in Bi

_{2}Se_{3}.*Phys. Rev. B***82**, 081305 (R) (2010). - 23.
Lin, H.

*et al.*An isolated Dirac cone on the surface of ternary tetradymite-like topological insulators.*New J. Phys.***13**, 095005 (2011). - 24.
Köhler, H. & Wöchner, E. The

*g*-factor of the conduction electrons in Bi_{2}Se_{3}.*Phys. Stat. Sol. (b)***67**, 675 (1975). - 25.
Mukhopadhyay, S.

*et al.*Hyperfine coupling and spin polarization in the bulk of the topological insulator Bi_{2}Se_{3}.*Phys. Rev. B***91**, 081105 (R) (2015). - 26.
Roth, L. M., Lax, B. & Zwerdling, S. Theory of optical magneto-absorption effects in semiconductors.

*Phys. Rev.***114**, 90–104 (1959). - 27.
Liu, C. X.

*et al.*Model Hamiltonian for topological insulators.*Phys. Rev. B***82**, 045122 (2010). - 28.
Cao, Y.

*et al.*Mapping the orbital wavefunction of the surface states in three-dimensional topological insulators.*Nat. Phys.***9**, 499–504 (2013). - 29.
Arakane, T.

*et al.*Tunable Dirac cone in the topological insulator Bi_{2x-}Sb_{x}Te_{3-y}Se_{y}.*Nat. Commun.***3**, 636 (2012). - 30.
Zhang, J. S.

*et al.*Band structure engineering in Bi_{2-x}Sb_{x}Te_{3}topological insulators.*Nat. Commun.***2**, 574 (2011).

## Acknowledgements

We thank Y. Tokura, X.-C. Xie, J.-H. Gao, P. Zhang, Z.-G. Wang, Y.-Y. Wang and X. Liu for helpful discussions. This work is funded by Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology of Japan, as well as the National Science Foundation of China.

## Author information

## Affiliations

### School of Physics and Wuhan National High Magnetic Field Center, Huazhong University of Science and Technology, Wuhan 430074, China

- Ying-Shuang Fu

### RIKEN Center for Emergent Matter Science, Wako, Saitama 351-0198, Japan

- Ying-Shuang Fu
- , T. Hanaguri
- , M. Kawamura
- & M. S. Bahramy

### Materials and Structures Laboratory, Tokyo Institute of Technology, Yokohama, Kanagawa 226-8503, Japan

- K. Igarashi
- & T. Sasagawa

### Department of Applied Physics, University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan

- M. S. Bahramy

## Authors

### Search for Ying-Shuang Fu in:

### Search for T. Hanaguri in:

### Search for K. Igarashi in:

### Search for M. Kawamura in:

### Search for M. S. Bahramy in:

### Search for T. Sasagawa in:

### Contributions

Y.-S.F. and T.H. performed the experiments and analysed the data. K.I. and T.S. grew single crystals of Sb_{2}Te_{2}Se and Bi_{2}Se_{3}. M.K. and M.S.B. contributed to the theoretical understanding. T.H. supervised the project. Y.-S.F. and T.H. wrote the manuscript with contributions from M.K. and M.S.B.

### Competing interests

The authors declare no competing financial interests.

## Corresponding authors

Correspondence to Ying-Shuang Fu or T. Hanaguri.

## Supplementary information

## PDF files

- 1.
### Supplementary Information

Supplementary Figures 1-7, Supplementary Notes 1-3 and Supplementary References

## Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/