## Abstract

Spin-split Rashba bands have been exploited to efficiently control the spin degree of freedom of moving electrons, which possesses a great potential in frontier applications of designing spintronic devices and processing spin-based information. Given an intrinsic breaking of inversion symmetry and sizeable spin–orbit interaction, two-dimensional (2D) surface alloys formed by heavy metal elements exhibit a pronounced Rashba-type spin splitting of the surface states. Here, we have revealed the essential role of atomic orbital symmetry in the hexagonally warped Rashba spin-split surface state of the \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} monatomic alloy by scanning tunneling spectroscopy (STS) and density functional theory (DFT). From dI/dU spectra and calculated band structures, three hole-like Rashba-split bands hybridized from distinct orbital symmetries have been identified in the unoccupied energy region. Because of the hexagonally deformed Fermi surface, quasi-particle interference (QPI) mappings have resolved scattering channels opened from interband transitions of *p*_{x},*p*_{y} (*m*_{j} = 1/2) band. In contrast to the *s**,p*_{z}-derived band, the hexagonal warping is predominately accompanied by substantial out-of-plane spin polarization *S*_{z} up to 24% in the dispersion of *p*_{x},*p*_{y} (*m*_{j} = 1/2) band with an in-plane orbital symmetry.

## Introduction

Hexagonal warping, i.e., a strong deformation of the Fermi surface in topological surface states due to interaction with neighboring (bulk) states, has been observed in a wide variety of topological insulators (TIs)^{1,2,3,4}. In the case of Bi_{2}Te_{3}, it has a rhombohedral crystalline structure of space group \(R3\bar m\), resulting in *C*_{3} rotation symmetry and three *M* mirror reflections on the [111] surface. According to \(k \cdot p\) theory, the warping term from cubic spin–orbit coupling together with *C*_{3} and *M* symmetries can lead to an anisotropic Fermi surface in Bi_{2}Te_{3}^{5}. Such a hexagonally warped Fermi surface gives rise to several interesting effects on the topological surface state, including out-of-plane spin polarization, an energy band gap opened at Dirac point, the emergence of Friedel oscillation on the local density of states (LDOS), and a possible spin density wave (SDW) phase^{6,7,8,9,10}.

With sizeable spin–orbit coupling and broken inversion symmetry, the spin degeneracy of the electronic band structure can be intrinsically lifted in two-dimensional (2D) systems to create a Rashba spin splitting of surface states^{11,12,13,14}. In recent years, a class of single-atomic-layer surface alloys has been successfully fabricated by means of evaporating heavy atoms onto the noble metal surfaces, for example, Pb, Bi, and Sb grown on Ag(111) and Cu(111)^{15,16,17,18,19,20,21,22,23}. The gigantic Rashba spin splitting characterized by the Rashba parameter *α*_{R} of 3.05 has been reported for the \(\sqrt 3 \times \sqrt 3 R30^0\) BiAg_{2} binary alloy^{16,24}.

In view of the \(\sqrt 3 \times \sqrt 3 R30^0\) surface reconstruction, the system presents a *C*_{3v} point group, including the *C*_{3} rotational symmetry and the *M* mirror plane in the Γ–K direction. Since significant spin–orbit interaction is inherently rooted in heavy elements, one could therefore expect the presence of a hexagonal warping effect in Rashba-type spin-split surface states with \(\sqrt 3 \times \sqrt 3 R30^0\) structure^{25}. In addition, atomic orbital symmetries from elements of Rashba binary alloys, e.g., Ag 5 *s*, Bi 6*p*_{x}, 6*p*_{y}, and 6*p*_{z} in BiAg_{2}, are responsible for the hybridization of different Rashba-split surface states. These states locate separately in occupied and unoccupied energy regions, which in principle will offer an opportunity to investigate the mutual interplay between orbital symmetry and hexagonal warping.

In this work, we study the hexagonal warping effect in unoccupied Rashba-split surface states of the \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} surface alloy by scanning tunneling spectroscopy (STS) and DFT. Differential conductance dI/dU spectra have resolved three hole-like Rashba-split bands, and their corresponding spectroscopic features locate separately at 0.23 eV (*s,p*_{z}), 1.4 eV (*p*_{x},*p*_{y} (*m*_{j} = 1/2)), and 1.6 eV (*p*_{x},*p*_{y} (*m*_{j} = 3/2)) in the unoccupied energy region, which is consistent with calculated band structures. From energy-dependent QPI mappings on Fermi surface with hexagonal deformations, we have observed standing wave patterns developed by interband spin–flip scatterings in the dispersion of the *p*_{x},*p*_{y} (*m*_{j} = 1/2) band. Due to the in-plane symmetry of the Bi 6*p*_{x},6*p*_{y} orbitals, they are effectively coupled to in-plane potential gradients on the \(\sqrt 3 \times \sqrt 3 R30^0\) surface. A substantial out-of-plane spin polarization *S*_{z} reaching 24% maximally has been revealed in the considerably warped *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba spin-split band.

## Results

### Characterization of monatomic bismuth surface alloy

According to structure models obtained from previous X-ray diffraction and STM studies, the surface alloys of Bi grown on Cu(111) have Bi coverages of 1/3 ML and 1/2 ML for \(\sqrt 3 \times \sqrt 3 R30^0\) and [2012] phases, respectively^{26,27}. This, in principle, provides a way how we can calibrate the amount of Bi in order to prepare the sample with a well-defined and extended \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} alloy. The overview STM topography of about 0.40 ML Bi on Cu(111) has been shown in Fig. 1a and the magnified image of Fig. 1b from the square area framed by black dashed lines in Fig. 1a displays one antiphase domain boundary (blue arrow) with few Bi vacancies (red arrow) in the BiCu_{2} phase. These two types of surface defects can be served as scattering centers enabling us to study the dispersion of Rashba-split bands from energy-dependent QPI measurements^{22}. From the atomic resolution image in Fig. 1c, the surface reconstruction exhibits a \(\sqrt 3 \times \sqrt 3\) periodicity with a 30° rotation toward the high symmetry axis, e.g., \([1\bar 10]\) of Cu(111), which is in agreement with the corresponding structure model of BiCu_{2} alloy shown in the inset^{26,27}. Since the Bi coverage of this sample is slightly above 1/3 ML, a few isolated patches of [2012] phase also coexist with the \(\sqrt 3 \times \sqrt 3 R30^0\) surface alloy. For example, the atomic resolution image of Bi[2012] shown in Fig. 1d has been resolved from the blue square frame in Fig. 1a, and structure model in the inset of Fig. 1d explains the zigzag chain-like feature along the \([1\bar 10]\) direction of Cu(111) surface with threefold rotational symmetry.

### Electronic structures of unoccupied Rashba spin-split bands

In order to determine the energy positions of different Rashba-split bands of Bi/Cu(111), tunneling spectra have been acquired on both \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} and Bi[2012] phases. Figure 2a displays the conductance dI/dU curve taken in modest bias range around the Fermi level, e.g., from 0.7 to −0.4 V. On BiCu_{2} (black line), an asymmetric peak at 0.23 V originates from the singularity of the LDOS at the band edge of *s*,*p*_{z}-derived Rashba band^{22}. The Rashba splitting, Rashba energy, and Rashba parameter of this band along the high symmetry direction \(\overline {{\it{{\Gamma}{\rm{M}}}}}\) \(\left( {\overline {{\it{{\Gamma}K}}} } \right)\) derived from our first-principles calculations are *k*_{0} = 0.0347(0.0359), *E*_{R} = \(\hbar ^2k_0^2/2m^ \ast\) = 0.0156(0.0169), and *α*_{R} = \(\hbar ^2k_0/2m^ \ast\) = 0.899(0.941), respectively. Note that our DFT Rashba energy *E*_{R} = 16 meV is similar to ~ 20 meV obtained from photoemission studies^{28}.

To explore Rashba-split bands hybridized from Bi 6*p*_{x},*p*_{y} orbitals of BiCu_{2}, we have measured the dI/dU curve at a larger bias range from 2.0 to −0.4 V and the results are shown in Fig. 2b. Interestingly, a pronounced peak at 1.6 V (red arrow) accompanied by a shoulder at 1.4 V (blue arrow) have been observed on BiCu_{2} (black line), but no characteristic features can be observed from dI/dU curve taken at Bi[2012] (blue line). Note that there are surface states at 2.5 eV and 2.8 eV above the Fermi level reported from previous 2PPE measurements and calculations^{29}. Since the intensity of conductance peak at 1.6 V is unusually stronger than the Rashba peak at 0.23 V (black arrow), we have double-checked the normalized dI/dU/(I/U) from the simultaneously measured *I/U* curve. The intensity of conductance peak at 1.6 V is indeed about six times stronger than that of Rashba peak at 0.23 V (see Supplementary Fig. S1 for more details), but less pronounced as compared to the resulting output directly from lock-in amplifier.

Apart from tunneling spectroscopy measurements, we also performed first-principle electronic structure calculations to reveal the origin of spectra peaks observed at different energy positions. The orbital-decomposed band structure of the BiCu_{2} surface is shown in Fig. 2c. It can be seen that two sets of Rashba-type bands cross the \({\it{\bar {\Gamma}}}\) point with band edges at about 0.2 eV and 1.4 eV, yielding the observed minor peak and the shoulder in our STS in Fig. 2a, b. In addition, the projected DOS from another Rashba band (*p*_{x},*p*_{y} (*m*_{j} = 1/2)) crossing the \({\it{\bar {\Gamma}}}\) near 1.8 eV contributes to the major peak observed in STS of Fig. 2b. For a direct comparison with STS measurement, we further integrate the DOS around the \({\it{\bar {\Gamma}}}\) point with a broadening ∆*E* as shown in Fig. 2d. Note that the overall features of the DOS remain similar after integration at different heights above the surface. Our DFT calculations fairly support the minor peak at about 0.2 eV, and the major peak at ~1.6 eV as well as the shoulder at ~1.4 eV observed in our STS measurement. Note that the exact position of these states depends sensitively on the Bi relaxations^{30}.

### QPI mappings on hexagonally warped Rashba-split surface states

To uncover the dispersion of the *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba-split band and compare with the *s*,*p*_{z}-derived Rashba band, we have carried out the QPI measurements covering both occupied and unoccupied energy regions on BiCu_{2}^{22}. According to the theoretical calculation by Mirhosseini et al., the *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba band exhibits an unconventional spin topology, the overall feature has been illustrated in Fig. 3a with intra- and interband scattering vectors (*q*_{intra} and *q*_{inter}), respectively^{31}. Note that the energy-dependent scattering vector *q*_{n}(*E*) is defined by *q*_{n}(*E*) = *k*_{f}(*E*) − *k*_{i}(*E*) in the framework of elastic scattering, which links initial and final momentum eigenstates for mapping out the dispersion relation of surface bands^{22,32,33}.

Another feature worth being mentioned is the hexagonal shape of constant energy contour (CEC) on *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba band, which has been observed in the occupied states of both BiAg_{2} and BiCu_{2} surface alloys by photoemission studies before^{24,30}. As illustrated in Fig. 3b, the CEC of Rashba band in general consists of two closed contours, and the outer one with a larger *k* value is hexagonally deformed due to strong warping effects. The out-of-plane spin polarization can be stabilized at all corner points lying on the \(\overline {{\it{{\Gamma}{\rm{M}}}}}\) direction and vanish due to mirror symmetry along \(\overline {{\it{{\Gamma}K}}}\) on \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} surface. With such kind of spin topology and multiple stationary *k* points, one could therefore expect additional scattering channels opened from the outer contour of the hexagonal CEC^{5,34,35,36}. The multiple pairs of stationary *k* points, as well as the density of states, are mainly distributed on the extremal curvature, i.e., in particular, the corners and the midpoints, of the hexagonal CEC, giving rise to the dominant intensity in the QPI pattern^{5,35}. For instance, possible interband spin–flip scatterings (solid light and dark gray arrow lines) have been depicted in Fig. 3b for the outer hexagon, and these scattering vectors as well as their symmetric counterparts, i.e., rotating every 60°, will construct two circle-like features in the reciprocal *q* space as shown in Fig. 3c (solid light and dark gray dots). On the contrary, if we simply consider scattering vectors from intraband transitions (dashed light and dark gray arrow lines in Fig. 3b), they will merely result in a single closed contour (formed by empty light and dark gray dots) in Fig. 3d as reported before^{32,33}. Note that the expected QPI pattern from the hexagonally deformed CECs in Fig. 3b needs to take both the inter- and the intraband transitions into account, i.e., overlapping Fig. 3c with the Fig. 3d, which has been graphically illustrated in Supplementary Fig. S2.

From Fig. 3e to g, representative dI/dU mappings clearly show spatial LDOS modulations in unoccupied of *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba band^{37}. Note that the standing waves become barely visible when bias voltages are above 1.4 V (see Supplementary Movie for details). The electron interference pattern results in two circle-like features in the Fourier-transformed dI/dU (FT-dI/dU) image as indicated by red and blue arrows in the middle of Fig. 3e to g. Note that the two circle-like features in FT-dI/dU images are a result of the superposition of the inter- and the intraband transitions, which indicates the presence of hexagonally deformed CECs as conceptually illustrated in the Fig. 3b–d. In order to exclude possible defect induced artifacts, we have inverted the FT-dI/dU signals from individual circle-like features to restore the dI/dU mappings separately and found that electron-standing waves are indeed developed with a long-range coherence (see Supplementary Fig. S3 for details). According to the line profiles along \(\overline {{\it{{\Gamma}{\rm{M}}}}}\) and \(\overline {{\it{{\Gamma}K}}}\) of FT-dI/dU images with an exponential background removed at the bottom of Fig. 3e–g, two scattering vectors have been extracted by employing the Gaussian fits (red/blue lines)^{15,22,32,33} (see Supplementary Fig. S4 for line profiles without removing the exponential background). Although spin-conserving scattering, i.e., *k*_{f}(*E*) and *k*_{i}(*E*) with the same direction of spin polarization, is typically considered, we are also aware of spin–flip scattering on nonmagnetic scattering centers that need to be taken into account for the case of band structures with a spin-polarization inversion^{15,22,32,33,38}. In contrast to the simple selection rule that the scatterings between exact anti-parallel spins are forbidden, spin-flip scattering process permits the transitions from a 180°. reversal of spin direction accompanied by a 90°. rotation of orbital moment. Due to the spin–orbit entanglement, the spin-flip scattering transitions between surface states with different orbital parities, i.e., (*sp*_{z}), (*p*_{x} − *ip*_{y}), and (*p*_{x} + *ip*_{y}), also become allowed as long as the total angular momentum remains conserved^{15,33} (see Supplementary Fig. S5 for the spin–orbit entanglement in the orbital-dependent spin-resolved band structures). However, we denote that either spin-conserving or spin-flip scattering along would give rise to only a single scattering vector in the *p*_{x},*p*_{y} (*m*_{j} = 1/2)-derived Rashba band, if only the intraband transitions are involved.

## Discussion

Figure 4a summarizes the dispersions, *D*_{n}, i.e., the energy-dependent scattering vectors *q*_{n}(*E*) along \(\overline {{\it{{\Gamma}{\rm{M}}}}}\) within bias voltage range from 1.5 V to −0.45 V (for *q*_{n}(*E*) along \(\overline {{\it{{\Gamma}K}}}\) see Supplementary Fig. S6). Figure 4b is the spin-decomposed band structure with in-plane spin polarization *S*_{x,y} of the BiCu_{2} surface states. For the scattering vector dispersions of *D*_{1} (black dots) and *D*_{2} (green dots), they come from intra- and interband scattering processes of *s*,*p*_{z} and *p*_{x},*p*_{y} (*m*_{j} = 1/2) Rashba bands^{22}. The black and green lines are the parabolic fits to obtain the effective masses \(m_{\overline {{\Gamma}{\rm{M}}} }^ \ast\) of (−0.34 ± 0.02)*m*_{e} and (−0.39 ± 0.01)*m*_{e}, respectively. These \(m_{\overline {{\Gamma}{\rm{M}}} }^ \ast\) values are consistent with those reported from photoemission work^{28}. Besides *D*_{1} and *D*_{2}, QPI results also reveal the band dispersions of *D*_{3} (gray dots) in the occupied energy region. We associate *D*_{3} with the scattering between the outer branch of *s*,*p*_{z} and inner one of the *p*_{x},*p*_{y} (*m*_{j} = 1/2) bands since it starts appearing at about 0.1 eV well above the projected bulk band gap of the Cu(111) substrate, whereas the strong hybridization between the outer branch of *s*,*p*_{z} band and bulk states does not play a significant role in forbidding the emergence of this scattering vector *q*_{3}^{22,23,30}.

Interestingly, two dispersions of *D*_{4} and *D*_{5} have been resolved from the *p*_{x},*p*_{y} (*m*_{j} = 1/2) band in the unoccupied energy region, and they correspond to scattering vectors *q*_{4} and *q*_{5}, respectively. In order to explain *q*_{4} and *q*_{5} in terms of interband scattering events in the hexagonally deformed CEC as illustrated in Fig. 3c, we have further calculated spin-dependent CEC with out-of-plane spin polarization *S*_{z} in Fig. 4c and the spin-decomposed band structure with *S*_{z} in Fig. 4d. According to the CEC at 0.6 eV in Fig. 4c, there is an inner ring-like circle accompanied by an outer circle with a hexagonal shape. They both are colored by red and blue for positive and negative *S*_{z} components, which are the most important characteristics that come along with the warping deformation of the *p*_{x},*p*_{y} (*m*_{j} = 1/2) band (see Supplementary Fig. S7 for details). Typically, the Rashba band consists of two concentric circles on the CEC with an in-plane chiral spin structure perpendicularly locked to the momentum vector, leading to prohibited backscattering as a consequence of time-reversal symmetry. However, this scenario is not perfectly true if there is a warping distortion on the CEC, where not only a single pair of stationary points can be stabilized but also new scattering channels can be opened^{5,8,9,10,35}. To confirm our observation, we have also performed simulations on QPI structures based on generic \(k \cdot p\) theory with a cubic warping term included (see Supplementary Note for details)^{5,35}. Given that the effective scattering potentials in T-matrix formalism are predominately developed from the corners and midpoints on the hexagonal CEC, as shown in Fig. 4e, two circle-like features (red and blue arrows) are in line with FT-dI/dU image in Fig. 3d, where the inner circle (red arrow) arises essentially from interband transitions permitted in the hexagonal CEC^{35}. On the other hand, the outer circle (blue arrow) is attributed to not only interband but also to extra input from intraband scattering channels as described in Fig. 3b, c. Note that the quantitative values in Fig. 4e are slightly smaller than FT-dI/dU in Fig. 3d, which could be a deviation of the offset *k*_{0} in DFT calculations on *k*_{||} of the high-lying, unoccupied Rashba band.

When the energy approaches the *p*_{x},*p*_{y} (*m*_{j} = 1/2) band edge, the CEC turns into a circular shape with a reduced *S*_{z} component because hexagonal warping interaction is less influential on smaller *k* values. With the plot in Fig. 4f, in contrast to the *s*,*p*_{z}-derived band, we have found substantial out-of-plane *S*_{z} up to 24% in the dispersion of *p*_{x},*p*_{y} (*m*_{j} = 1/2) band. This result indicates that, due to the in-plane symmetry of atomic orbitals, they are more effectively coupled to in-plane potential gradient from \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} surface, leading also to a significant out-of-plane *S*_{z} polarization when spin–orbit coupling is involved. For comparison, we have also calculated the *S*_{z} of BiAg_{2} and found that BiCu_{2} can have *S*_{z} about three times larger than that of BiAg_{2} because of the stronger mixing of the *p*-states caused by the larger outward protrusion of Bi atoms ~1.0 Å on BiCu_{2} surface (see Supplementary Fig. S8 for details).

In conclusion, we have investigated the hexagonal warping effect in unoccupied Rashba spin-split surface states of the \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} surface alloy. From dI/dU spectra as well as calculated band structures, three hole-like Rashba-split bands have been resolved. Their spectroscopic features locate separately at 0.23 eV (*s*,*p*_{z}), 1.4 eV (*p*_{x},*p*_{y} (*m*_{j} = 1/2)), and 1.6 eV (*p*_{x},*p*_{y} (*m*_{j} = 3/2)). Caused by hexagonally deformed Fermi surface, the energy-dependent QPI mappings have visualized standing wave patterns from interband scattering transitions of *p*_{x},*p*_{y} (*m*_{j} = 1/2) band. Due to a strong in-plane potential gradient on BiCu_{2} surface, hexagonal warping is accompanied by out-of-plane *S*_{z} spin polarization up to 24% in the dispersion of *p*_{x},*p*_{y} (*m*_{j} = 1/2) band. This strong effect is enabled by the in-plane symmetry of atomic orbitals. In light of such an orbital-selective hexagonal warping effect, our results, therefore, demonstrate an important and generic aspect of engineering relevant spin-dependent properties on low-dimensional Rashba materials by exploiting atomic orbital symmetry.

## Methods

### Experiment

The Bi/Cu(111) surfaces were prepared in an ultrahigh vacuum (UHV) chamber with the base pressure below *P* ≤ 2 × 10^{−10} mbar. The clean Cu(111) surface was first prepared by cycles of Ar^{+} ion sputtering with an ion energy of 500 eV at room temperature and subsequent annealing up to 800 K. The Bi source with the purity of 99.999% (Goodfellow) was e-beam sublimated onto Cu(111) surface at elevated temperature of 400 K at which the well-defined and extended \(\sqrt 3 \times \sqrt 3 R30^0\) BiCu_{2} binary alloy can be grown. After preparation, the sample was immediately transferred into a low-temperature scanning tunneling microscope (LT-STM) from Unisoku Co. Ltd. (operation temperature *T* ≈ 4.5 K). The topography images were obtained from the constant-current mode with the bias voltage *U* applied to the sample. For STS measurements, a small bias voltage modulation (*U*_{mod} = 20–50 mV) was added to U (frequency *v* = 3991 Hz), such that tunneling differential conductance dI/dU spectra, as well as dI/dU maps, can be acquired by detecting the first harmonic signal by means of a lock-in amplifier.

### Theory

First-principles calculations are performed using the Vienna Ab initio Simulation Package (VASP) based on the DFT^{39,40,41}. The projector-augmented-wave-type pseudopotential with the Ceperley–Alder and Perdew–Zunger (CA-PZ) type exchange-correlation functional are adopted in the local density approximation (LDA) calculations^{42,43,44,45,46}. We consider the BiCu_{2} monolayer on top of a nine-layer Cu(111) \(\sqrt 3 \times \sqrt 3\) substrate to simulate our experimental system. The ion positions of the top three layers are optimized until the residue force is smaller than 0.02 eVÅ^{−1}. After the geometrical optimization, the buckling height of Bi ion of the BiCu_{2} alloy is fixed at 1.0 Å as deduced from our experimental spectroscopy result and from ref. ^{26} where Kaminski et al. give 1.05 Å. Spin–orbit coupling is included in the self-consistent calculations with the energy cutoff of 400 eV over the 12 × 12 × 1 **k-**mesh. The DOS is integrated over the zone center near the \({\it{\bar {\Gamma}}}\) point with an additional broadening of 0.01 and 0.1 eV. The two-dimension energy contours of the Rashba bands around the \({\it{\bar {\Gamma}}}\) point are calculated over the 40 × 40 **k**-mesh and then interpolated over the 360 × 360 **k**-mesh.

## Data availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

## References

Hsieh, D. et al. A tunable topological insulator in the spin helical Dirac transport regime.

*Nature***460**, 1101 (2009).Chen, Y. L. et al. Experimental realization of a three-dimensional topological insulator, Bi

_{2}Te_{3}.*Science***325**, 178 (2009).Kuroda, K. et al. Hexagonally deformed Fermi surface of the 3D topological insulator Bi

_{2}Se_{3}.*Phys. Rev. Lett.***105**, 076802 (2010).Arakane, T. et al. Tunable Dirac cone in the topological insulator Bi

_{2-x}Sb_{x}Te_{3-y}Se_{y}.*Nat. Commun.***3**, 636 (2012).Fu, L. Hexagonal warping effects in the surface states of the topological insulator Bi

_{2}Te_{3}.*Phys. Rev. Lett.***103**, 266801 (2009).Souma, S. et al. Direct measurement of the out-of-plane spin texture in the Dirac-cone surface state of a topological insulator.

*Phys. Rev. Lett.***106**, 216803 (2011).Herdt, A. et al. Spin-polarization limit in Bi

_{2}Te_{3}Dirac cone studied by angle- and spin-resolved photoemission experiments and ab initio calculations.*Phys. Rev. B***87**, 035127 (2013).Zhang, T. et al. Experimental demonstration of topological surface states protected by time-reversal symmetry.

*Phys. Rev. Lett.***103**, 266803 (2009).Roushan, P. et al. Topological surface states protected from backscattering by chiral spin texture.

*Nature***460**, 1106–1109 (2009).Alpichshev, Z. et al. STM imaging of electronic waves on the surface of Bi

_{2}Te_{3}: topologically protected surface states and hexagonal warping effects.*Phys. Rev. Lett.***104**, 016401 (2010).Bychkov, Y. A. & Rashba, E. I. Properties of a 2D electron gas with lifted spectral degeneracy.

*JETP Lett.***39**, 78–81 (1984).Bihlmayer, G., Rader, O. & Winkler, R. Focus on the Rashba effect.

*N. J. Phys.***17**, 050202 (2015).Gambardella, P. & Miron, I. M. Current-induced spin–orbit torques.

*Philos. Trans. R. Soc. A***369**, 3175–3197 (2011).Noguchi, R. et al. Direct mapping of spin and orbital entangled wave functions under interband spin-orbit coupling of giant Rashba spin-split surface states.

*Phys. Rev. B***95**, 041111(R) (2017).El-Kareh, L. et al. A combined experimental and theoretical study of Rashba-split surface states on the \(\sqrt 3 \times \sqrt 3 s\) Pb/Ag(111)\(\sqrt 3 \times \sqrt 3 s\) surface.

*N. J. Phys.***16**, 045017 (2014).Ast, C. R. et al. Local detection of spin-orbit splitting by scanning tunneling spectroscopy.

*Phys. Rev. B***75**, 201401(R) (2007).Bian, G., Zhang, L., Liu, Y., Miller, T. & Chiang, C. T. Illuminating the surface spin texture of the giant-Rashba quantum-well system Bi/Ag(111) by circularly polarized photoemission.

*Phys. Rev. Lett.***108**, 186403 (2012).Moreschini, L. et al. Assessing the atomic contribution to the Rashba spin-orbit splitting in surface alloys: Sb/Ag(111).

*Phys. Rev. B***79**, 075424 (2009).Wissing, S. N. P., Ritter, K. T., Krüger, P., Schmidt, A. B. & Donath, M. Spin-dependent size of interband hybridization gap: The interplay of adlayer and substrate states in Pb/Cu(111).

*Phys. Rev. B***91**, 201403(R) (2015).Bentmann, H. & Reinert, F. Enhancing and reducing the Rashba-splitting at surfaces by adsorbates: Na and Xe on Bi/Cu(111).

*N. J. Phys.***15**, 115011 (2013).Bentmann, H. et al. Spin orientation and sign of the Rashba splitting in Bi/Cu(111).

*Phys. Rev. B***84**, 115426 (2011).Steinbrecher, M., Harutyunyan, H., Ast, C. R. & Wegner, D. Rashba-type spin splitting from interband scattering in quasiparticle interference maps.

*Phys. Rev. B***87**, 245436 (2013).Moreschini, L. et al. Influence of the substrate on the spin-orbit splitting in surface alloys on (111) noble-metal surfaces.

*Phys. Rev. B***80**, 035438 (2009).Ast, C. R. et al. Giant spin splitting through surface alloying.

*Phys. Rev. Lett.***98**, 186807 (2007).Meier, F., Dil, H., Lobo-Checa, J., Patthey, L. & Osterwalder, J. Quantitative vectorial spin analysis in angle-resolved photoemission: Bi/Ag(111) and Pb/Ag(111).

*Phys. Rev. B***77**, 165431 (2008).Kaminski, D., Poodt, P., Aret, E., Radenovic, N. & Vlieg, E. Surface alloys, overlayer and incommensurate structures of Bi on Cu(111).

*Surf. Sci.***575**, 233–246 (2005).Girard, Y. et al. Growth of Bi on Cu(111): alloying and dealloying transitions.

*Surf. Sci.***617**, 118–123 (2013).Ünal, A. A. et al. Polarization dependence and surface sensitivity of linear and nonlinear photoemission from Bi/Cu(111).

*Phys. Rev. B***86**, 125447 (2012).Mathias, S. et al. Quantum-well-induced giant spin-orbit splitting.

*Phys. Rev. Lett.***104**, 066802 (2010).Bentmann, H. et al. Origin and manipulation of the Rashba splitting in surface alloys.

*EPL***87**, 37003 (2009).Mirhosseini, H. et al. Unconventional spin topology in surface alloys with Rashba-type spin splitting.

*Phys. Rev. B***79**, 245428 (2009).El-Kareh, L., Sessi, P., Bathon, T. & Bode, M. Quantum interference mapping of Rashba-split Bloch states in Bi/Ag(111).

*Phys. Rev. Lett.***110**, 176803 (2013).Schirone, S. et al. Spin-flip and element-sensitive electron scattering in the BiAg

_{2}surface alloy.*Phys. Rev. Lett.***114**, 166801 (2015).Sessi, P. et al. Visualizing spin-dependent bulk scattering and breakdown of the linear dispersion relation in Bi

_{2}Te_{3}.*Phys. Rev. B***88**, 161407(R) (2013).Lee, W. C., Wu, C. J., Arovas, D. P. & Zhang, S. C. Quasiparticle interference on the surface of the topological insulator Bi

_{2}Te_{3}.*Phys. Rev. B***80**, 245439 (2009).Rakyta, P., Pályi, A. & Cserti, J. Electronic standing waves on the surface of the topological insulator Bi

_{2}Te_{3}.*Phys. Rev. B***86**, 085456 (2012).Friedel, J. Metallic alloys.

*Nuovo Cim. Suppl.***7**, 287–311 (1958).Hirayama, H., Aoki, Y. & Kato, C. Quantum interference of Rashba-type spin-split surface state electrons.

*Phys. Rev. Lett.***107**, 027204 (2011).Kresse, G. & Hafner, J. Ab initio molecular dynamics for open-shell transition metals.

*Phys. Rev. B***48**, 13115 (1993).Kresse, G. & Furthmüller, J. E. Efficiency of ab initio total energy calculations for metals and semiconductors using a plane-wave basis set.

*Computational Mater. Sci.***6**, 15 (1996).Kresse, G. & Furthmüller, J. E. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.

*Phys. Rev. B***54**, 11169 (1996).Blöchl, P. E. Projector augmented-wave method.

*Phys. Rev. B***50**, 17953 (1994).Kresse, G. & Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method.

*Phys. Rev. B***59**, 1758 (1999).Ceperley, D. M. & Alder, B. J. Ground state of the electron gas by a stochastic method.

*Phys. Rev. Lett.***45**, 566 (1980).Perdew, J. P. & Zunger, A. Self-interaction correction to density-functional approximations for many-electron systems.

*Phys. Rev. B***23**, 5048–5079 (1981).Kohn, W. & Sham, L. J. Self-consistent equations including exchange and correlation effects.

*Phys. Rev.***140**, A1133 (1965).

## Acknowledgements

D.S.L. and P.J.H. acknowledge support from the competitive research funding and the helium liquefier system from instrumentation center in National Tsing Hua University, Ministry of Science and Technology of Taiwan under Grants No. MOST-110-2731-M-007-001, MOST-108-2636-M-007-002 and MOST-107-2112-M-007-001-MY3, and center for quantum technology from the featured areas research center program within the framework of the higher education sprout project by the Ministry of Education (MOE) in Taiwan. H.T.J. also acknowledges support from NCHC, CINC-NTU, AS-iMATE-109-13, Taiwan.

## Author information

### Authors and Affiliations

### Contributions

G.Y.C. and A.H. contributed equally to this work. G.Y.C., Y.H.L., and C.J.C. carried out the STM/STS experiments and analyzed the data. A.H., D.S.L., P.Y.C., G.B., and H.T.J. performed the theoretical calculations. H.T.J. and P.J.H. coordinated and supervised the project. All authors discussed the results and contributed to the paper.

### Corresponding authors

## Ethics declarations

### Competing interests

The authors declare no competing interests.

## Additional information

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Supplementary information

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Chen, GY., Huang, A., Lin, YH. *et al.* Orbital-enhanced warping effect in *p*_{x},*p*_{y}-derived Rashba spin splitting of monatomic bismuth surface alloy.
*npj Quantum Mater.* **5**, 89 (2020). https://doi.org/10.1038/s41535-020-00293-3

Received:

Accepted:

Published:

DOI: https://doi.org/10.1038/s41535-020-00293-3