Orbital-enhanced warping effect in px,py-derived Rashba spin splitting of monatomic bismuth surface alloy

Spin-split Rashba bands have been exploited to efficiently control the spin degree of freedom of moving electrons, which possesses a great potential in frontier applications of designing spintronic devices and processing spin-based information. Given an intrinsic breaking of inversion symmetry and sizeable spin–orbit interaction, two-dimensional (2D) surface alloys formed by heavy metal elements exhibit a pronounced Rashba-type spin splitting of the surface states. Here, we have revealed the essential role of atomic orbital symmetry in the hexagonally warped Rashba spin-split surface state of the 3×3R300\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt 3 \times \sqrt 3 R30^0$$\end{document} BiCu2 monatomic alloy by scanning tunneling spectroscopy (STS) and density functional theory (DFT). From dI/dU spectra and calculated band structures, three hole-like Rashba-split bands hybridized from distinct orbital symmetries have been identified in the unoccupied energy region. Because of the hexagonally deformed Fermi surface, quasi-particle interference (QPI) mappings have resolved scattering channels opened from interband transitions of px,py (mj = 1/2) band. In contrast to the s,pz-derived band, the hexagonal warping is predominately accompanied by substantial out-of-plane spin polarization Sz up to 24% in the dispersion of px,py (mj = 1/2) band with an in-plane orbital symmetry.


INTRODUCTION
Hexagonal warping, i.e., a strong deformation of the Fermi surface in topological surface states due to interaction with neighboring (bulk) states, has been observed in a wide variety of topological insulators (TIs) [1][2][3][4] . In the case of Bi 2 Te 3 , it has a rhombohedral crystalline structure of space group R3m, resulting in C 3 rotation symmetry and three M mirror reflections on the [111] surface. According to k Á p theory, the warping term from cubic spin-orbit coupling together with C 3 and M symmetries can lead to an anisotropic Fermi surface in Bi 2 Te 3 5 . Such a hexagonally warped Fermi surface gives rise to several interesting effects on the topological surface state, including out-of-plane spin polarization, an energy band gap opened at Dirac point, the emergence of Friedel oscillation on the local density of states (LDOS), and a possible spin density wave (SDW) phase [6][7][8][9][10] .
With sizeable spin-orbit coupling and broken inversion symmetry, the spin degeneracy of the electronic band structure can be intrinsically lifted in two-dimensional (2D) systems to create a Rashba spin splitting of surface states [11][12][13][14] . In recent years, a class of single-atomic-layer surface alloys has been successfully fabricated by means of evaporating heavy atoms onto the noble metal surfaces, for example, Pb, Bi, and Sb grown on Ag(111) and Cu(111) [15][16][17][18][19][20][21][22][23] . The gigantic Rashba spin splitting characterized by the Rashba parameter α R of 3.05 has been reported for the ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiAg 2 binary alloy 16,24 . In view of the ffiffi ffi 3 p ffiffi ffi 3 p R30 0 surface reconstruction, the system presents a C 3v point group, including the C 3 rotational symmetry and the M mirror plane in the Γ-K direction. Since significant spin-orbit interaction is inherently rooted in heavy elements, one could therefore expect the presence of a hexagonal warping effect in Rashba-type spin-split surface states with ffiffi ffi 3 p ffiffi ffi 3 p R30 0 structure 25 . In addition, atomic orbital symmetries from elements of Rashba binary alloys, e.g., Ag 5 s, Bi 6p x , 6p y , and 6p z in BiAg 2 , are responsible for the hybridization of different Rashba-split surface states. These states locate separately in occupied and unoccupied energy regions, which in principle will offer an opportunity to investigate the mutual interplay between orbital symmetry and hexagonal warping.
In this work, we study the hexagonal warping effect in unoccupied Rashba-split surface states of the ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiCu 2 surface alloy by scanning tunneling spectroscopy (STS) and DFT. Differential conductance dI/dU spectra have resolved three holelike Rashba-split bands, and their corresponding spectroscopic features locate separately at 0.23 eV (s,p z ), 1.4 eV (p x ,p y (m j = 1/2)), and 1.6 eV (p x ,p y (m j = 3/2)) in the unoccupied energy region, which is consistent with calculated band structures. From energydependent QPI mappings on Fermi surface with hexagonal deformations, we have observed standing wave patterns developed by interband spin-flip scatterings in the dispersion of the p x , p y (m j = 1/2) band. Due to the in-plane symmetry of the Bi 6p x ,6p y orbitals, they are effectively coupled to in-plane potential gradients on the ffiffi ffi 3 p ffiffi ffi 3 p R30 0 surface. A substantial out-ofplane spin polarization S z reaching 24% maximally has been revealed in the considerably warped p x ,p y (m j = 1/2)-derived Rashba spin-split band.

RESULTS
Characterization of monatomic bismuth surface alloy According to structure models obtained from previous X-ray diffraction and STM studies, the surface alloys of Bi grown on Cu  Fig. 1a displays one antiphase domain boundary (blue arrow) with few Bi vacancies (red arrow) in the BiCu 2 phase. These two types of surface defects can be served as scattering centers enabling us to study the dispersion of Rashba-split bands from energydependent QPI measurements 22 . From the atomic resolution image in Fig. 1c, the surface reconstruction exhibits a ffiffi ffi 3 p ffiffi ffi 3 p periodicity with a 30°rotation toward the high symmetry axis, e.g., ½110 of Cu(111), which is in agreement with the corresponding structure model of BiCu 2 alloy shown in the inset 26,27 . Since the Bi coverage of this sample is slightly above 1/3 ML, a few isolated patches of [2012] phase also coexist with the ffiffi ffi  Figure 2a displays the conductance dI/dU curve taken in modest bias range around the Fermi level, e.g., from 0.7 to −0.4 V. On BiCu 2 (black line), an asymmetric peak at 0.23 V originates from the singularity of the LDOS at the band edge of s,p z -derived Rashba band 22 . The Rashba splitting, Rashba energy, and Rashba parameter of this band along the high symmetry direction ΓM ΓK À Á derived from our firstprinciples calculations are k 0 = 0.0347(0.0359), E R = h 2 k 2 0 =2m Ã = 0.0156(0.0169), and α R = h 2 k 0 =2m Ã = 0.899(0.941), respectively. Note that our DFT Rashba energy E R = 16 meV is similar to2 0 meV obtained from photoemission studies 28 . To explore Rashba-split bands hybridized from Bi 6p x ,p y orbitals of BiCu 2 , we have measured the dI/dU curve at a larger bias range from 2.0 to −0.4 V and the results are shown in Fig. 2b. Interestingly, a pronounced peak at 1.6 V (red arrow) accompanied by a shoulder at 1.4 V (blue arrow) have been observed on BiCu 2 (black line), but no characteristic features can be observed from dI/dU curve taken at Bi[2012] (blue line). Note that there are surface states at 2.5 eV and 2.8 eV above the Fermi level reported from previous 2PPE measurements and calculations 29 . Since the intensity of conductance peak at 1.6 V is unusually stronger than the Rashba peak at 0.23 V (black arrow), we have double-checked the normalized dI/dU/(I/U) from the simultaneously measured I/U curve. The intensity of conductance peak at 1.6 V is indeed about six times stronger than that of Rashba peak at 0.23 V (see Supplementary Fig. S1 for more details), but less pronounced as compared to the resulting output directly from lock-in amplifier. Apart from tunneling spectroscopy measurements, we also performed first-principle electronic structure calculations to reveal the origin of spectra peaks observed at different energy positions. The orbital-decomposed band structure of the BiCu 2 surface is shown in Fig. 2c. It can be seen that two sets of Rashba-type bands cross the Γ point with band edges at about 0.2 eV and 1.4 eV, yielding the observed minor peak and the shoulder in our STS in Fig. 2a, b. In addition, the projected DOS from another Rashba band (p x ,p y (m j = 1/2)) crossing the Γ near 1.8 eV contributes to the major peak observed in STS of Fig. 2b. For a direct comparison with STS measurement, we further integrate the DOS around the Γ point with a broadening ΔE as shown in Fig. 2d. Note that the overall features of the DOS remain similar after integration at different heights above the surface. Our DFT calculations fairly support the minor peak at about 0.2 eV, and the major peak at 1.6 eV as well as the shoulder at~1.4 eV observed in our STS measurement. Note that the exact position of these states depends sensitively on the Bi relaxations 30 .
QPI mappings on hexagonally warped Rashba-split surface states To uncover the dispersion of the p x ,p y (m j = 1/2)-derived Rashbasplit band and compare with the s,p z -derived Rashba band, we have carried out the QPI measurements covering both occupied and unoccupied energy regions on BiCu 2 22 . According to the theoretical calculation by Mirhosseini et al., the p x ,p y (m j = 1/2)derived Rashba band exhibits an unconventional spin topology, the overall feature has been illustrated in Fig. 3a with intra-and interband scattering vectors (q intra and q inter ), respectively 31 . Note that the energy-dependent scattering vector q n (E) is defined by q n (E) = k f (E) − k i (E) in the framework of elastic scattering, which links initial and final momentum eigenstates for mapping out the dispersion relation of surface bands 22,32,33 .
Another feature worth being mentioned is the hexagonal shape of constant energy contour (CEC) on p x ,p y (m j = 1/2)-derived Rashba band, which has been observed in the occupied states of both BiAg 2 and BiCu 2 surface alloys by photoemission studies before 24,30 . As illustrated in Fig. 3b, the CEC of Rashba band in general consists of two closed contours, and the outer one with a larger k value is hexagonally deformed due to strong warping effects. The out-of-plane spin polarization can be stabilized at all corner points lying on the ΓM direction and vanish due to mirror symmetry along ΓK on ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiCu 2 surface. With such kind of spin topology and multiple stationary k points, one could therefore expect additional scattering channels opened from the outer contour of the hexagonal CEC 5,34-36 . The multiple pairs of stationary k points, as well as the density of states, are mainly distributed on the extremal curvature, i.e., in particular, the corners and the midpoints, of the hexagonal CEC, giving rise to the dominant intensity in the QPI pattern 5,35 . For instance, possible interband spin-flip scatterings (solid light and dark gray arrow lines) have been depicted in Fig. 3b for the outer hexagon, and these scattering vectors as well as their symmetric counterparts, i.e., rotating every 60°, will construct two circle-like features in the reciprocal q space as shown in Fig. 3c (solid light and dark gray dots). On the contrary, if we simply consider scattering vectors from intraband transitions (dashed light and dark gray arrow lines in Fig.  3b), they will merely result in a single closed contour (formed by empty light and dark gray dots) in Fig. 3d as reported before 32,33 . Note that the expected QPI pattern from the hexagonally deformed CECs in Fig. 3b needs to take both the inter-and the intraband transitions into account, i.e., overlapping Fig. 3c with the Fig. 3d, which has been graphically illustrated in Supplementary Fig. S2.
From Fig. 3e to g, representative dI/dU mappings clearly show spatial LDOS modulations in unoccupied of p x ,p y (m j = 1/2)derived Rashba band 37 . Note that the standing waves become barely visible when bias voltages are above 1.4 V (see Supplementary Movie for details). The electron interference pattern results in two circle-like features in the Fourier-transformed dI/dU (FT-dI/dU) image as indicated by red and blue arrows in the middle of Fig. 3e to g. Note that the two circle-like features in FT-dI/dU images are a result of the superposition of the inter-and the intraband transitions, which indicates the presence of hexagonally deformed CECs as conceptually illustrated in the Fig. 3b-d. In order to exclude possible defect induced artifacts, we have inverted the FT-dI/dU signals from individual circle-like features to restore the dI/dU mappings separately and found that electronstanding waves are indeed developed with a long-range coherence (see Supplementary Fig. S3 for details). According to the line profiles along ΓM and ΓK of FT-dI/dU images with an exponential background removed at the bottom of Fig. 3e-g, two scattering vectors have been extracted by employing the Gaussian fits (red/blue lines) 15,22,32,33 (see Supplementary Fig. S4 for line profiles without removing the exponential background). Although spin-conserving scattering, i.e., k f (E) and k i (E) with the same direction of spin polarization, is typically considered, we are also aware of spin-flip scattering on nonmagnetic scattering centers that need to be taken into account for the case of band structures with a spin-polarization inversion 15, 22,32,33,38 . In contrast to the simple selection rule that the scatterings between exact antiparallel spins are forbidden, spin-flip scattering process permits the transitions from a 180°. reversal of spin direction accompanied by a 90°. rotation of orbital moment. Due to the spin-orbit entanglement, the spin-flip scattering transitions between surface states with different orbital parities, i.e., (sp z ), (p x − ip y ), and (p x + ip y ), also become allowed as long as the total angular momentum remains conserved 15,33 (see Supplementary Fig. S5 for the spin-orbit entanglement in the orbital-dependent spin-resolved band structures). However, we denote that either spin-conserving or spin-flip scattering along would give rise to only a single scattering vector in the p x ,p y (m j = 1/2)-derived Rashba band, if only the intraband transitions are involved. Figure 4a summarizes the dispersions, D n , i.e., the energydependent scattering vectors q n (E) along ΓM within bias voltage range from 1.5 V to −0.45 V (for q n (E) along ΓK see Supplementary  Fig. S6). Figure 4b is the spin-decomposed band structure with inplane spin polarization S x,y of the BiCu 2 surface states. For the scattering vector dispersions of D 1 (black dots) and D 2 (green dots), they come from intra-and interband scattering processes of s,p z and p x ,p y (m j = 1/2) Rashba bands 22 . The black and green lines are the parabolic fits to obtain the effective masses m Ã ΓM of (−0.34 ± 0.02)m e and (−0.39 ± 0.01)m e , respectively. These m Ã ΓM values are consistent with those reported from photoemission work 28 . Besides D 1 and D 2 , QPI results also reveal the band dispersions of D 3 (gray dots) in the occupied energy region. We associate D 3 with the scattering between the outer branch of s,p z and inner one of the p x ,p y (m j = 1/2) bands since it starts appearing at about 0.1 eV well above the projected bulk band gap of the Cu(111) substrate, whereas the strong hybridization between the outer branch of s,p z band and bulk states does not play a significant role in forbidding the emergence of this scattering vector q 3 22,23,30 . Interestingly, two dispersions of D 4 and D 5 have been resolved from the p x ,p y (m j = 1/2) band in the unoccupied energy region, Fig. 3 QPI mappings on hexagonally warped Rashba-split surface states. a p x ,p y (m j = 1/2)-derived Rashba-split band with unconventional spin topology (red and blue colors are referred to opposite spin directions), dashed and solid black arrow lines indicate intra-(q intra ) and interband (q inter ) scattering vectors. b Hexagonal CEC with allowed intra-(dashed arrow lines) and interband transitions (solid arrow lines) along ΓM (light gray) and ΓK (dark gray), respectively. c Expected QPI pattern (outlined by solid light and dark gray dots) from only the interband transitions (3 solid arrow lines in b with half magnitude), producing two closed contours (red/blue lines). d Expected QPI pattern (outlined by empty light and dark gray dots) by solely taking the intraband transitions (two dashed arrow lines in b with half magnitude) into account, resulting in a single closed contour (blue line). e-g Top: dI/dU mappings from p x ,p y (m j = 1/2)-derived band with corresponding FT-dI/dU images. The scale bars are 4 nm −1 . Bottom: the line profiles along ΓM and ΓK of FT-dI/dU images with an exponential background removed. Two energy-dependent scattering vectors (red/blue arrows) have been identified by employing the Gaussian fits (red/blue lines). (scan parameters: I = 1.0 nA, T = 4.5 K for all dI/dU mappings). and they correspond to scattering vectors q 4 and q 5 , respectively. In order to explain q 4 and q 5 in terms of interband scattering events in the hexagonally deformed CEC as illustrated in Fig. 3c, we have further calculated spin-dependent CEC with out-of-plane spin polarization S z in Fig. 4c and the spin-decomposed band structure with S z in Fig. 4d. According to the CEC at 0.6 eV in Fig.  4c, there is an inner ring-like circle accompanied by an outer circle with a hexagonal shape. They both are colored by red and blue for positive and negative S z components, which are the most important characteristics that come along with the warping deformation of the p x ,p y (m j = 1/2) band (see Supplementary Fig.  S7 for details). Typically, the Rashba band consists of two concentric circles on the CEC with an in-plane chiral spin structure perpendicularly locked to the momentum vector, leading to prohibited backscattering as a consequence of time-reversal symmetry. However, this scenario is not perfectly true if there is a warping distortion on the CEC, where not only a single pair of stationary points can be stabilized but also new scattering channels can be opened 5,[8][9][10]35 . To confirm our observation, we have also performed simulations on QPI structures based on generic k Á p theory with a cubic warping term included (see Supplementary Note for details) 5,35 . Given that the effective scattering potentials in T-matrix formalism are predominately developed from the corners and midpoints on the hexagonal CEC, as shown in Fig. 4e, two circle-like features (red and blue arrows) are in line with FT-dI/dU image in Fig. 3d, where the inner circle (red arrow) arises essentially from interband transitions permitted in the hexagonal CEC 35 . On the other hand, the outer circle (blue arrow) is attributed to not only interband but also to extra input from intraband scattering channels as described in Fig. 3b, c. Note that the quantitative values in Fig. 4e are slightly smaller than FT-dI/dU in Fig. 3d, which could be a deviation of the offset k 0 in DFT calculations on k || of the high-lying, unoccupied Rashba band.

DISCUSSION
When the energy approaches the p x ,p y (m j = 1/2) band edge, the CEC turns into a circular shape with a reduced S z component because hexagonal warping interaction is less influential on smaller k values. With the plot in Fig. 4f, in contrast to the s,p zderived band, we have found substantial out-of-plane S z up to 24% in the dispersion of p x ,p y (m j = 1/2) band. This result indicates that, due to the in-plane symmetry of atomic orbitals, they are more effectively coupled to in-plane potential gradient from ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiCu 2 surface, leading also to a significant out-ofplane S z polarization when spin-orbit coupling is involved. For comparison, we have also calculated the S z of BiAg 2 and found that BiCu 2 can have S z about three times larger than that of BiAg 2 because of the stronger mixing of the p-states caused by the larger outward protrusion of Bi atoms~1.0 Å on BiCu 2 surface (see Supplementary Fig. S8 for details).
In conclusion, we have investigated the hexagonal warping effect in unoccupied Rashba spin-split surface states of the ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiCu 2 surface alloy. From dI/dU spectra as well as calculated band structures, three hole-like Rashba-split bands have been resolved. Their spectroscopic features locate separately at 0.23 eV (s,p z ), 1.4 eV (p x ,p y (m j = 1/2)), and 1.6 eV (p x ,p y (m j = 3/ 2)). Caused by hexagonally deformed Fermi surface, the energydependent QPI mappings have visualized standing wave patterns from interband scattering transitions of p x ,p y (m j = 1/2) band. Due to a strong in-plane potential gradient on BiCu 2 surface, hexagonal warping is accompanied by out-of-plane S z spin polarization up to 24% in the dispersion of p x ,p y (m j = 1/2) band. This strong effect is enabled by the in-plane symmetry of atomic orbitals. In light of such an orbital-selective hexagonal warping effect, our results, therefore, demonstrate an important and generic aspect of engineering relevant spin-dependent properties on low-dimensional Rashba materials by exploiting atomic orbital symmetry.

METHODS Experiment
The Bi/Cu(111) surfaces were prepared in an ultrahigh vacuum (UHV) chamber with the base pressure below P ≤ 2 × 10 −10 mbar. The clean Cu (111) surface was first prepared by cycles of Ar + ion sputtering with an ion energy of 500 eV at room temperature and subsequent annealing up to 800 K. The Bi source with the purity of 99.999% (Goodfellow) was e-beam sublimated onto Cu(111) surface at elevated temperature of 400 K at which the well-defined and extended ffiffi ffi 3 p ffiffi ffi 3 p R30 0 BiCu 2 binary alloy can be grown. After preparation, the sample was immediately transferred into a low-temperature scanning tunneling microscope (LT-STM) from Unisoku Co. Ltd. (operation temperature T ≈ 4.5 K). The topography images were obtained from the constant-current mode with the bias voltage U applied to the sample. For STS measurements, a small bias voltage modulation (U mod = 20-50 mV) was added to U (frequency v = 3991 Hz), such that tunneling differential conductance dI/dU spectra, as well as dI/dU maps, can be acquired by detecting the first harmonic signal by means of a lockin amplifier.

Theory
First-principles calculations are performed using the Vienna Ab initio Simulation Package (VASP) based on the DFT [39][40][41] . The projectoraugmented-wave-type pseudopotential with the Ceperley-Alder and Perdew-Zunger (CA-PZ) type exchange-correlation functional are adopted in the local density approximation (LDA) calculations [42][43][44][45][46] . We consider the Fig. 4 Dispersion of Rashba Bands and CEC with out-of-plane S z . a Dispersion of scattering vectors q n (E) along ΓM extracted from a series of FT-dI/dU mappings from 1.5 V to −0.45 V. b Spindecomposed band structure of the BiCu 2 surface with in-plane spin polarization S x,y . The red and blue circles indicate opposite spin components from the surface states. Differe n t q n are assigned with separate colors. c Spin-dependent CEC with out-of-plane spin polarization S z induced by the hexagonal warping effect. d Spindecomposed band structure with S z . e QPI structures calculated from CEC in (c), red and blue arrows mark the two circle-like features observed in Fig. 3d. f S z + as a function of energy in the dispersion of the p x ,p y (m j = 1/2)-derived band. BiCu 2 monolayer on top of a nine-layer Cu(111) ffiffi ffi 3 p ffiffi ffi 3 p substrate to simulate our experimental system. The ion positions of the top three layers are optimized until the residue force is smaller than 0.02 eVÅ −1 . After the geometrical optimization, the buckling height of Bi ion of the BiCu 2 alloy is fixed at 1.0 Å as deduced from our experimental spectroscopy result and from ref. 26 where Kaminski et al. give 1.05 Å. Spin-orbit coupling is included in the self-consistent calculations with the energy cutoff of 400 eV over the 12 × 12 × 1 k-mesh. The DOS is integrated over the zone center near the Γ point with an additional broadening of 0.01 and 0.1 eV. The two-dimension energy contours of the Rashba bands around the Γ point are calculated over the 40 × 40 k-mesh and then interpolated over the 360 × 360 k-mesh.