Substitution-induced spin-splitted surface states in topological insulator (Bi1−xSbx)2Te3

We present a study on surface states of topological insulator (Bi1−xSbx)2Te3 by imaging quasiparticle interference patterns (QPI) using low temperature scanning tunneling microscope. Besides the topological Dirac state, we observed another surface state with chiral spin texture within the conduction band range. The quasiparticle scattering in this state is selectively suppressed. Combined with first-principles calculations, we attribute this state to a spin-splitted band induced by the substitution of Bi with Sb atoms. Our results demonstrate that the coexistence of topological order and alloying may open wider tunability in quantum materials.

T hree dimensional (3D) topological insulator (TIs) is a new state of quantum matter with nontrivial gapless surface states in the bulk energy gap where the directions of spin and moment are locked. The topological surface state (TSS) is originated from strong spin-orbit coupling (SOC) effect, and robust to nonmagnetic perturbation, as the topological order is protected by the particle number conservation and time-reversal (TR) symmetry [1][2][3] . Such unique electronic properties provide a new ground not only for novel quantum phenomena (such as magnetic monopole and Majorana fermions) but also for applications in spintronics and quantum computing [4][5][6][7] .
Among all 3D TIs, bismuth telluride (Bi 2 Te 3 ) has been widely investigated due to its large SOC gap. However, the position of Dirac point (DP) for pure Bi 2 Te 3 is beneath its bulk valence bands maxima, making it difficult for application in electronic devices. Furthermore, although significant efforts have been made to realize the surface transport, the progress is still hampered by large unintentional bulk contribution to the total conductance caused by Te vacancies. Recently, improved bulk-insulating property was achieved in the Bi-based ternary (Bi 12x Sb x ) 2 Te 3 (BST) [8][9][10][11] , which has a tetradymite structure similar to Bi 2 Te 3 . The existence of TSS in BST was confirmed by angle resolved photoemission spectroscopy (ARPES) and electronic transport measurements for 0 # x # 1. The energy position of DP for BST relative to the Fermi level can be tuned by adjusting the stoichiometry of the Bi and Sb, eventually leading to be an ideal TI with insulating bulk. Such great advantage has been substantially demonstrated by the recent discovery of quantum anomalous Hall Effect (QAHE) in Cr-doped BST films 12 . However, despite the application in transport studies, fundamental properties of BST have not been fully understood yet. For example, regarding the random alloying of Bi and Sb atoms in BST, the effects of structural disorder and on the electronic structure and transport properties are still unknown.
In this article, besides the conventional TSS, we found in ternary compound (Bi 12x Sb x ) 2 Te 3 films a new surface state located within the conductance band. This band is absent in binary compounds Bi 2 Te 3 . Remarkbly, the band also exhibits chiral spin texture, as revealed by quasiparticle interference (QPI) patterns. First principle calculations suggest that this band is originated from the Sb/Bi layer due to the random substitution of Bi by Sb atoms. This demonstrates that alloying of semiconductor components provides not only a method to tune the semiconducting gap or the existing surface state, but also capable of creating new exotic electronic states for potential applications.

Results
Characterization of (Bi 12x Sb x ) 2 Te 3 surface. The morphology of the atomic flat terraces on the surface of a (Bi 0.45 Sb 0.55 ) 2 Te 3 film is illustrated in Fig. 1(a). The single step height is about 1 nm, corresponding to one quintuple layer. High-resolution STM image ( Fig. 1(b)) indicates a well-ordered hexagonal close-packed structure with a lattice constant of 4.4 6 0.1 Å , which is close to that of Bi 2 Te 3 (inset in Fig. 1(b)). But the brightness of atoms is not identical, which is different from the Bi 2 Te 3 . Considering that STM image contains the contribution from local density of states (LDOS), the different brightness may be the result of electronic distortions caused by random substitution of Bi with Sb. It is similar to the surfaces of Bi 12x Sb x alloy 13 , Bi 21x Te x Se 14 and Bi 1.5 Sb 0.5 Te 1.7 Se 1.3 15 . The STS measurements were performed to investigate the electronic structures of BST. We obtained the typical dI/dV curves of (Bi 12x Sb x ) 2 Te 3 films with different Sb composition shown in Fig. 1(c). The positions of DP, corresponding to the minima of LDOS or the extension of the linearly dispersing Dirac bands to bias axis 16,17 , are about 2230 meV, 250 meV, 15 meV and 98 meV for x 5 0, 0.45, 0.62, and 0.87, respectively. Obviously, the energy of DP gradually lifts up from the valence band as x increases, as illustrated in the schematic band diagram ( Fig. 1(d)). From it, we estimate that an intrinsic topological insulator, where the DP is in the bulk energy gap and close to the Fermi level, will be achieved in sample with x 5 0.5 6 0.1, in agree with transport results in the (Bi 12x Sb x ) 2 Te 3 films 10 .
Surface states measurement from quasiparticle interference (QPI) analysis. To obtain how the surface states disperse in momentum space, we need to investigate the standing waves caused by quasiparticle interference (QPI) on surface. A series of dI/dV maps of a (Bi 0.45 Sb 0.55 ) 2 Te 3 thin film taken at a tip bias voltage (V T ) from 20.2 V to 21.5 V are shown in Fig. 2(b1-b12), which exhibit pronounced standing waves throughout the surface. The fast Fourier transformation (FFT) of the dI/dV maps, shown in Fig. 2(c1-c12), indicates the distribution of scattering wave vectors q in momentum space. We found there are six-fold symmetric spots with intensity centered along the CM direction of the first Brillouin zone at outer edge of FFT patterns. The spots are small at V T between 20.3 and 20.6 V, and get gradually broader when V T is decreasing to more negative side. Next, between 20.9 V and 21.1 V, the QPI patterns dramatically shrink and become intensive again, and then they are gradually broadened and obscured below 21.2 V and vanished until 21.5 V once more.
The six-fold symmetric spots in FFT images are related to robust scattering vectors. To better understand the evolution of scattering vectors with energy, we integrate the FFT intensity along the radical direction, and plot it as a function of distance from C point, as shown in Fig. 3(a). The peaks marked by dots correspond to the positions of the spots. It is obviously that there are two dispersive bands of scattering vectors: one at the low energy region (from 20.8 V to 20.2 V), the other at high energy region (from 21.5 V to 20.9 V). Similar features have also been observed on the BST surface for x 5 0.62 and 0.87. However, in the case of pure Bi 2 Te 3 (BST for x 5 0) only that at the low energy region (20.5 The two different bands of scattering vectors relate to quaiparticle scattering in two different states. The low energy state at 0.15-0.8 eV above E F should be attributed to the topological Dirac surface state. The hexagonally warped Dirac cone (DC) with chiral spin texture in BST 8 and the protection by time-reversal symmetry result in scattering only in the CM direction (the schematic illustration shown in Fig. 3(b) (bottom)), corresponding to six-fold symmetric spots in reciprocal space 18,19 . The similar scattering vector along CM direction has been reported on Bi 12x Sb x alloy 13 , Bi 2 Te 3 18 , and Bi 2 Se 3 16 . In constant energy contours (CEC) of DC, the wave vector k along the CK direction and the scattering vector q should obey the equation: q 5 !3 k 18 . Thus, the dispersions of the Dirac states along the CK direction can be obtained, as shown in Fig. 3(c)-(f). The linear fittings to the data are perfect, and the slopes provide measurement of the Fermi velocity V F , which are 3.58 3 10 25 , 5.34 3 0 25 , 5.23 3 10 25 , and 3.69 3 10 25 m/s, corresponding to x 5 0, 0.45, 0.62, and 0.87, respectively. In addition, the energy positions of DP for BST films are about 2248, 255, 110, and 164 meV, respectively, by estimating the interceptions of the dispersion curves with the energy axis. There is a small deviation of the energy position of DP from measurements in dI/dV curves. The discrepancy may be due to the electrostatic induction by the electric field between tip and sample 20 .
For the high energy state at 0.9-1.5 eV above EF, which is located in the bulk conduction band region, there are also six-fold symmetric spots along the CM direction in FFT images, similar as the Dirac state. This suggests backscattering suppression of quasiparticles from a new surface state with chiral spin texture. One possible origination of this state is two dimensional electron gas (2DEG) with Rashbatype splitting on TIs surface, caused by the band bending due to foreign gas/atom adsorption [21][22][23] . However, although this state may  have the same spin chiral texture with the TSS, its energy should be near the bottom of bulk conduction band, around 200-300 meV above EF. In contrast, the new state found in our experiments distributes at higher energy region (0.9 eV to 1.5 eV). Secondly, this state is absent in pure Bi 2 Te 3 films in our experiment, indicating that it should be related to the substitution of Bi by Sb. Therefore, the possibility of 2DEG as origination of this state is ruled out.
Assignment of the surface states based on the first-principles calculation. In order to elucidate the origin of the new surface state, first-principles calculations based on density functional theory (DFT) were performed to study the electronic structures of (Bi 12x Sb x ) 2 Te 3 with x 5 0 & 0.5 (Bi 2 Te 3 and BiSbTe 3 ). The calculated band structures, as well as the layer contributions projected to both surface and bulk of Bi 2 Te 3 and BiSbTe 3 , are displayed in Fig. 4(a-b), and Fig. 4(c-d), respectively. The TSS, in other word, DC is obtained at the C points for both Bi 2 Te 3 and BiSbTe3. The positions of DP in Bi2Te3 and BiSbTe3 are about 0.11 eV and 0.05 eV below the top of bulk valence bands, respectively, showing the DC is indeed significantly pulled towards the middle of bulk band gap by doping Sb, consistent to the experimental results. Furthermore, two significant states (ST1 and ST2) were observed at 0.6 , 0.8 eV and 0.9 , 1.3 eV above Fermi energy for BiSbTe3 (Fig. 4(c)), which have larger projected intensity at surface layer. On the contrary there is only one unclear state at 0.7 , 0.9 eV for Bi2Te3 ( Fig. 4(a)). The major states from C to M points are more clearly indicated by ignoring minor contribution to the bands of surface layer, as shown in Fig. 4(e).
According to the energy range of calculated surface states, the new state observed in experiment can be assigned to the ST2. In addition, there are some additional spots near the center of FFT images, shown in Fig. 2(c5-c9) with energy range of 0.5 eV-0.8 eV above EF, which can be attributed to the quasipaticles scattering in ST1. To further confirm this conclusion, we decomposed the bands of all three states to contributions from each element in the surface layer, and the results are shown in Fig. 4(f-h). We found that the DC state is a hybridized state composed by Bi-6p, Sb-5p, and Te-5p orbitals. On the other hand, the ST1 is mainly from Bi-6p and Te-5p orbitals, and the ST2 from Sb-5p orbitals. As a consequence, the ST2 can only be observed in BST sample with x ? 0, in perfect agreement with experiments. In addition, we found the intensity of spots in FFT images at high energy region for (Bi 12x Sb x ) 2 Te 3 become weaker by increasing the value of x (x 5 0.62, and 0.87), indicating ST2 state should be more dominating for the hybrid BST sample than pure Bi 2 Te 3 or Sb 2 Te 3 . We also compared the distributions of spin moments for the DC, ST1 and ST2 of BiSbTe 3 , as exhibited in Fig.  S1. It is found that the ST2 has the similar distribution of spin with the DC from the C to M points.

Discussion
Next let us discuss why the ST2 contributed by Sb atoms has a chiral spin texture. Spin degeneracy is a consequence of both time-reversal and inversion symmetry. If one of them is broken, the degeneracy can be lifted by, e.g. spin-orbit interaction. The substitution doping of Sb atoms breaks the structural inversion asymmetry in plane, leading to spin-split electronic states 24 . On the other hand, heavy atoms (Bi) are surrounded by light atoms (Sb), resulting in a strong in-plane gradient of the crystal potential in the Bi/Sb layer, which will enhance the splitting strength. The similar giant spin splitting was found in the Bi/ Ag(111) surface 25 . We noted the ST2 from K-C-M points has nearly double parabola shape with a symmetric geometry around C point ( Fig. 4(g)), similar to the Rashba-type spin-split bands. That may be the origination of the chiral spin texture in ST2 26,27 . Therefore, the spin-momentum locking can be also realized in conductance band through surface alloying effect.
The spin-momentum locking in ST2 will make the backscattering of quasiparticle in ST2 suppressed due to the protection by timereversal symmetry. Considering the similar hexagonal warping effect, the CECs of the inner and outer spin-split subbands of ST2 are shown in Fig. 3(b) (top). Therefore, the most possible scattering vector q should be along the CM direction (shown as the solid arrows), which accords with experiment very well. The value of scattering vector q H (dot line in Fig. 3(b)) extracted from the FFT images at high energy region should be the average of the two scattering vectors in two subbands. Similar analysis performed on QPI patterns on Bi/Ag(111) 2 !3 3 !3 surface 26,27 . As a consequence, the energy dispersion E(k) of the new surface state are drew in figure 3(e-h), where q H is equal to !3 k. The parabolic fitting to the dispersion give the effective mass m* about 0.56, 0.61, and 0.09 me, respectively.
The ST2 is mainly located at the first layer of BST film. When the thickness of film is only few layers even single layer, this state will be more dominated than bulk conduction states. On the other hand, this surface state is chiral, that means it may be sensitive with polarized light, which be one effective method to separate this state from bulk states. The feature of chiality in ST2 results in the prohibition the backscattering of quasiparticle due to the protection of time-reversal symmetry. So the transport properties of quasiparticle in this state will be much different with the bulk conduction states, which may be another way to separate this state from bulk states. Therefore we believe the BST films with few layers may have potential electrical or optical applications due to the existence of this chiral surface state.
In summary, we have investigated the electronic states of topological insulator BST with different x value using STM. Two robust states with chiral spin texture are found: one is Dirac surface state near the Fermi energy in samples with all x value, another state is located in conductance band in samples with x . 0 and never reported before. Combined with DFT calculations, the new spin-split states can be attributed to the substitution of Bi by Sb atoms. We think it is significant that the spin-momentum locking realize in both the topological state and the surface state stem from surface alloying in a wide energy range, which may provide more extensive applications. Methods BST sample growth and STM measurement. We prepared (Bi 12x Sb x ) 2 Te 3 thin films on Nb-doped SrTiO 3 (111) substrates with different x by molecular beam epitaxy (MBE), and studied the electronic structures using low-temperature scanning tunneling microscopy (STM) and scanning tunneling spectroscopy (STS). Our experiments were performed in a home-built combined low temperature STM-MBE system. The (Bi 12x Sb x ) 2 Te 3 films with thickness of 40 nm were prepared on Nb-doped single crystal SrTiO 3 (111) substrates. Details of sample preparation are described in Ref. 10. The STM images were acquired in the constant-current mode with the bias voltage applied to the tip. The dI/dV images were obtained by simultaneously recording the STM images using a lock-in technique (with 20 mV modulation at 963 Hz).
Electronic structure calculations. The calculation model contains 15 Å of vacuum space, as well as 7 quintuple layers of 2 3 2 BST thin film. (For BiSbTe 3 , same number of Sb and Bi atoms are randomly distributed in each quintuple layer.) To better evaluate the interlayer space, the dispersion-corrected vdW-DF 28 was employed for fully relaxation of the geometry, combined with the projector augmented wave (PAW) pseudopotential 29,30 and a plane-wave basis set of 250 eV energy cutoff. The maximum force on the each relaxed atom was converged to less than 0.01 eV/Å , and the Brillouin zone was sampled by (8 3 8 3 1) k-points for both structure optimization and electronic structure calculation. All the calculations were carried out with VASP package 31,32 .