Observation of Zeeman effect in topological surface state with distinct material dependence

Manipulating the spins of the topological surface states represents an essential step towards exploring the exotic quantum states emerging from the time reversal symmetry breaking via magnetic doping or external magnetic fields. The latter case relies on the Zeeman effect and thereby we need to estimate the g-factor of the topological surface state precisely. Here, we report the direct observations of the Zeeman effect at the surfaces of Bi2Se3 and Sb2Te2Se by spectroscopic-imaging scanning tunnelling microscopy. The Zeeman shift of the zero mode Landau level is identified unambiguously by appropriately excluding the extrinsic effects arising from the nonlinearity in the band dispersion of the topological surface state and the spatially varying potential. Surprisingly, the g-factors of the topological surface states in Bi2Se3 and Sb2Te2Se are very different (+18 and −6, respectively). Such remarkable material dependence opens up a new route to control the spins of the topological surface states.


Supplementary
Measuring the g-factor of Sb 2 Te 2 Se at several potential extremes. Potential maps of the Sb 2 Te 2 Se surface obtained by the spectroscopic imaging of E 0 at 12 T showing a potential minimum (V min ) and maximum (V max ) in a and a single potential minimum in b. The scale bar corresponds to 10 nm. The solid ellipses represent the equipotential lines of the fitted 2D parabolic potential. The innermost ellipse corresponds to 248 meV for V min in a, 238.5 meV for V max in a and 238meV in b. The adjacent equipotential lines have an energy interval of 0.5 meV. The dashed circles in a and b characterize the location and size of the LL 0 state at 7 T for V min of a, 5 T for V max of a, and 9 T for V min of b. (c, d, e) E 0 at different B (black symbols) measured at the fitted potential extremes (marked as crosses in a and b) and their fitting according to Eq. 2 of main text (blue curves). The error bars of E 0 are the standard of deviation generated from the LL 0 peak fitting with a Lorentz line shape. (f) Table showing Table showing the parameters of the Gaussian potential minimums for constructing the potential in a.
Pixel size of a-c: 512  512. The value for s * 2  g m is 20 during the calculation.

Supplementary Note 1 -Models of LLs in TSS in the presence of Zeeman effect
We consider two models for the TSS and study their LLs in the presence of the Zeeman effect: one is the ideal helical Dirac fermions, another one is the non-ideal helical Dirac fermions perturbed by a parabolic curvature in their energy dispersion and the potential variation.

Ideal helical Dirac fermions
The Hamiltonian for the ideal helical Dirac fermions in a perpendicular magnetic field B is given as: Here,  e Π kA is the canonical momentum, with k and A being the momentum and the vector potential, respectively; v is the electron velocity; σ are the Pauli matrices, and s g is the electron g-factor of the TSS. We assume   For ladder operators, †  a a n n n , † 11    a n n n , 1  a n n n , where n is a non-negative integer. This, in combination with Supplementary Eq. 3 and 4, We further evaluate the situation of reversing the direction of B. Applying a negative B to the TI is equivalent to probing its opposite surface in a positive B [1].
For the negative B, the Hamiltonian becomes: The ladder operators should accordingly change to () 2

Supplementary Note 2 -Topography of Sb 2 Te 2 Se surface
Sb 2 Te 2 Se has the tetradymite structure, which is identical to that of Bi 2 Se 3 . Its quintuple-layer unit consists of Te-Sb-Se-Sb-Te ( Supplementary Fig. 2, insert). The bonding forces between the quintuple layers are weak van der Waals interactions.
Therefore, the crystal cleaves easily. STM image of the cleaved surface clearly resolves the ordered atoms of the triangular lattice ( Supplementary Fig. 2). Its lattice constant is estimated to be 4.2 Å, which is consistent with the bulk value. Since cleaving occurs between the adjacent Te layers, the imaged atoms should be Te.

Supplementary Note 3 -Modeling the effect of potential extensions on E 0 at low B
In the TSS of Sb 2 Te 2 Se, the non-ideal dispersions and the Zeeman effect both make E 0 shift towards higher energy with B. In contrast, the effect of a potential minimum make E 0 shift oppositely with B. As a result, E 0 first decreases and then increases with B at a potential minimum, as is seen from Fig. 4c. However, the shifting trend of the E 0 (B) differs at different potential minimums. For instance, E 0 exhibits a monotonic B-shift at the potential minimum of Supplementary Fig. 5d. As the spatial extension of the LL 0 state expands at low B, the effect of the potential at large extensions takes place. In this section, we model this effect to understand the observed diverse shifting behavior of E 0 at different potential minimums.
We first simply use two superimposed Gaussian potentials to model the potential  Fig. 6a-c) to elucidate the different shifting behavior of E 0 (B). Their parameters are listed in Supplementary Fig. 6h. The three potentials all have a dip in the center, and decrease in energy after reaching a maximum as their sizes spatially extend, thereby forming a hump shape. The depth of the potential dip is deepest for Potential a (V a ), and shallowest for Potential c (V c ) ( Supplementary Fig. 6g).
Subsequently, the E 0 value can be calculated according to Supplementary Eq. 10 and 11 as The calculated E 0 (B) exhibit different shifting trends at different potential minimum centers ( Supplementary Fig. 6d-f). Such differences can be interpreted from the weighting of the LL wave functions at the potentials. When the potential dip is deep (Potential a), the LL 0 state is mostly weighted by the potential dip even at low B.
Therefore, its E 0 first decreases and then increases with decreasing B ( Supplementary   Fig. 6d), as is expected for a potential minimum shown in Fig. 4c. It must be noted that the normalization value of the LL 0 state at 1 T is less than 1, which means its calculated E 0 should be neglected. When the potential dip is shallow (Potential b), the LL 0 state significantly enhances its weighting at the potential hump. This makes E 0 shift monotonically with decreasing B (Supplementary Fig. 6e) in a similar manner as Supplementary Fig. 5d. The monotonic B-shifting behavior of E 0 is more evident ( Supplementary Fig. 6f) as the potential dip gets even shallower (Potential c).
On the basis of the single potential indicated above, we further construct a multi-minimum potential to reproduce the actual potential variations shown in Supplementary Fig. 5b. Supplementary Fig. 7a shows the modeled potential, which is composed of 20 Gaussian potentials with identical shapes but different coordinate centers, i.e.  Supplementary Fig. 7c). The E 0 values of the boundary regions, whose normalization values are smaller than 1, should be neglected. The calculated E 0 map ( Supplementary   Fig. 7b, red rectangle) reproduces the measurement of Supplementary Fig. 5b well.
We then calculate E 0 of the potential minimum ( Supplementary Fig. 7b, cross) at different B (Supplementary Fig. 7d). The obtained shifting trend reproduces that of Supplementary Fig. 5d as well. Therefore, our model calculations substantiate our experimental observations, demonstrating that the potential at large extensions could not only affects the amount of shifting of E 0 with B but also changes its trend.