## Introduction

Hexagonal warping, i.e., a strong deformation of the Fermi surface in topological surface states due to interaction with neighboring (bulk) states, has been observed in a wide variety of topological insulators (TIs)1,2,3,4. In the case of Bi2Te3, it has a rhombohedral crystalline structure of space group $$R3\bar m$$, resulting in C3 rotation symmetry and three M mirror reflections on the [111] surface. According to $$k \cdot p$$ theory, the warping term from cubic spin–orbit coupling together with C3 and M symmetries can lead to an anisotropic Fermi surface in Bi2Te35. Such a hexagonally warped Fermi surface gives rise to several interesting effects on the topological surface state, including out-of-plane spin polarization, an energy band gap opened at Dirac point, the emergence of Friedel oscillation on the local density of states (LDOS), and a possible spin density wave (SDW) phase6,7,8,9,10.

With sizeable spin–orbit coupling and broken inversion symmetry, the spin degeneracy of the electronic band structure can be intrinsically lifted in two-dimensional (2D) systems to create a Rashba spin splitting of surface states11,12,13,14. In recent years, a class of single-atomic-layer surface alloys has been successfully fabricated by means of evaporating heavy atoms onto the noble metal surfaces, for example, Pb, Bi, and Sb grown on Ag(111) and Cu(111)15,16,17,18,19,20,21,22,23. The gigantic Rashba spin splitting characterized by the Rashba parameter αR of 3.05 has been reported for the $$\sqrt 3 \times \sqrt 3 R30^0$$ BiAg2 binary alloy16,24.

In view of the $$\sqrt 3 \times \sqrt 3 R30^0$$ surface reconstruction, the system presents a C3v point group, including the C3 rotational symmetry and the M mirror plane in the Γ–K direction. Since significant spin–orbit interaction is inherently rooted in heavy elements, one could therefore expect the presence of a hexagonal warping effect in Rashba-type spin-split surface states with $$\sqrt 3 \times \sqrt 3 R30^0$$ structure25. In addition, atomic orbital symmetries from elements of Rashba binary alloys, e.g., Ag 5 s, Bi 6px, 6py, and 6pz in BiAg2, are responsible for the hybridization of different Rashba-split surface states. These states locate separately in occupied and unoccupied energy regions, which in principle will offer an opportunity to investigate the mutual interplay between orbital symmetry and hexagonal warping.

In this work, we study the hexagonal warping effect in unoccupied Rashba-split surface states of the $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 surface alloy by scanning tunneling spectroscopy (STS) and DFT. Differential conductance dI/dU spectra have resolved three hole-like Rashba-split bands, and their corresponding spectroscopic features locate separately at 0.23 eV (s,pz), 1.4 eV (px,py (mj = 1/2)), and 1.6 eV (px,py (mj = 3/2)) in the unoccupied energy region, which is consistent with calculated band structures. From energy-dependent QPI mappings on Fermi surface with hexagonal deformations, we have observed standing wave patterns developed by interband spin–flip scatterings in the dispersion of the px,py (mj = 1/2) band. Due to the in-plane symmetry of the Bi 6px,6py orbitals, they are effectively coupled to in-plane potential gradients on the $$\sqrt 3 \times \sqrt 3 R30^0$$ surface. A substantial out-of-plane spin polarization Sz reaching 24% maximally has been revealed in the considerably warped px,py (mj = 1/2)-derived Rashba spin-split band.

## Results

### Characterization of monatomic bismuth surface alloy

According to structure models obtained from previous X-ray diffraction and STM studies, the surface alloys of Bi grown on Cu(111) have Bi coverages of 1/3 ML and 1/2 ML for $$\sqrt 3 \times \sqrt 3 R30^0$$ and [2012] phases, respectively26,27. This, in principle, provides a way how we can calibrate the amount of Bi in order to prepare the sample with a well-defined and extended $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 alloy. The overview STM topography of about 0.40 ML Bi on Cu(111) has been shown in Fig. 1a and the magnified image of Fig. 1b from the square area framed by black dashed lines in Fig. 1a displays one antiphase domain boundary (blue arrow) with few Bi vacancies (red arrow) in the BiCu2 phase. These two types of surface defects can be served as scattering centers enabling us to study the dispersion of Rashba-split bands from energy-dependent QPI measurements22. From the atomic resolution image in Fig. 1c, the surface reconstruction exhibits a $$\sqrt 3 \times \sqrt 3$$ periodicity with a 30° rotation toward the high symmetry axis, e.g., $$[1\bar 10]$$ of Cu(111), which is in agreement with the corresponding structure model of BiCu2 alloy shown in the inset26,27. Since the Bi coverage of this sample is slightly above 1/3 ML, a few isolated patches of [2012] phase also coexist with the $$\sqrt 3 \times \sqrt 3 R30^0$$ surface alloy. For example, the atomic resolution image of Bi[2012] shown in Fig. 1d has been resolved from the blue square frame in Fig. 1a, and structure model in the inset of Fig. 1d explains the zigzag chain-like feature along the $$[1\bar 10]$$ direction of Cu(111) surface with threefold rotational symmetry.

### Electronic structures of unoccupied Rashba spin-split bands

In order to determine the energy positions of different Rashba-split bands of Bi/Cu(111), tunneling spectra have been acquired on both $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 and Bi[2012] phases. Figure 2a displays the conductance dI/dU curve taken in modest bias range around the Fermi level, e.g., from 0.7 to −0.4 V. On BiCu2 (black line), an asymmetric peak at 0.23 V originates from the singularity of the LDOS at the band edge of s,pz-derived Rashba band22. The Rashba splitting, Rashba energy, and Rashba parameter of this band along the high symmetry direction $$\overline {{\it{{\Gamma}{\rm{M}}}}}$$ $$\left( {\overline {{\it{{\Gamma}K}}} } \right)$$ derived from our first-principles calculations are k0 = 0.0347(0.0359), ER = $$\hbar ^2k_0^2/2m^ \ast$$ = 0.0156(0.0169), and αR = $$\hbar ^2k_0/2m^ \ast$$ = 0.899(0.941), respectively. Note that our DFT Rashba energy ER = 16 meV is similar to ~ 20 meV obtained from photoemission studies28.

To explore Rashba-split bands hybridized from Bi 6px,py orbitals of BiCu2, we have measured the dI/dU curve at a larger bias range from 2.0 to −0.4 V and the results are shown in Fig. 2b. Interestingly, a pronounced peak at 1.6 V (red arrow) accompanied by a shoulder at 1.4 V (blue arrow) have been observed on BiCu2 (black line), but no characteristic features can be observed from dI/dU curve taken at Bi[2012] (blue line). Note that there are surface states at 2.5 eV and 2.8 eV above the Fermi level reported from previous 2PPE measurements and calculations29. Since the intensity of conductance peak at 1.6 V is unusually stronger than the Rashba peak at 0.23 V (black arrow), we have double-checked the normalized dI/dU/(I/U) from the simultaneously measured I/U curve. The intensity of conductance peak at 1.6 V is indeed about six times stronger than that of Rashba peak at 0.23 V (see Supplementary Fig. S1 for more details), but less pronounced as compared to the resulting output directly from lock-in amplifier.

Apart from tunneling spectroscopy measurements, we also performed first-principle electronic structure calculations to reveal the origin of spectra peaks observed at different energy positions. The orbital-decomposed band structure of the BiCu2 surface is shown in Fig. 2c. It can be seen that two sets of Rashba-type bands cross the $${\it{\bar {\Gamma}}}$$ point with band edges at about 0.2 eV and 1.4 eV, yielding the observed minor peak and the shoulder in our STS in Fig. 2a, b. In addition, the projected DOS from another Rashba band (px,py (mj = 1/2)) crossing the $${\it{\bar {\Gamma}}}$$ near 1.8 eV contributes to the major peak observed in STS of Fig. 2b. For a direct comparison with STS measurement, we further integrate the DOS around the $${\it{\bar {\Gamma}}}$$ point with a broadening ∆E as shown in Fig. 2d. Note that the overall features of the DOS remain similar after integration at different heights above the surface. Our DFT calculations fairly support the minor peak at about 0.2 eV, and the major peak at ~1.6 eV as well as the shoulder at ~1.4 eV observed in our STS measurement. Note that the exact position of these states depends sensitively on the Bi relaxations30.

### QPI mappings on hexagonally warped Rashba-split surface states

To uncover the dispersion of the px,py (mj = 1/2)-derived Rashba-split band and compare with the s,pz-derived Rashba band, we have carried out the QPI measurements covering both occupied and unoccupied energy regions on BiCu222. According to the theoretical calculation by Mirhosseini et al., the px,py (mj = 1/2)-derived Rashba band exhibits an unconventional spin topology, the overall feature has been illustrated in Fig. 3a with intra- and interband scattering vectors (qintra and qinter), respectively31. Note that the energy-dependent scattering vector qn(E) is defined by qn(E) = kf(E) − ki(E) in the framework of elastic scattering, which links initial and final momentum eigenstates for mapping out the dispersion relation of surface bands22,32,33.

Another feature worth being mentioned is the hexagonal shape of constant energy contour (CEC) on px,py (mj = 1/2)-derived Rashba band, which has been observed in the occupied states of both BiAg2 and BiCu2 surface alloys by photoemission studies before24,30. As illustrated in Fig. 3b, the CEC of Rashba band in general consists of two closed contours, and the outer one with a larger k value is hexagonally deformed due to strong warping effects. The out-of-plane spin polarization can be stabilized at all corner points lying on the $$\overline {{\it{{\Gamma}{\rm{M}}}}}$$ direction and vanish due to mirror symmetry along $$\overline {{\it{{\Gamma}K}}}$$ on $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 surface. With such kind of spin topology and multiple stationary k points, one could therefore expect additional scattering channels opened from the outer contour of the hexagonal CEC5,34,35,36. The multiple pairs of stationary k points, as well as the density of states, are mainly distributed on the extremal curvature, i.e., in particular, the corners and the midpoints, of the hexagonal CEC, giving rise to the dominant intensity in the QPI pattern5,35. For instance, possible interband spin–flip scatterings (solid light and dark gray arrow lines) have been depicted in Fig. 3b for the outer hexagon, and these scattering vectors as well as their symmetric counterparts, i.e., rotating every 60°, will construct two circle-like features in the reciprocal q space as shown in Fig. 3c (solid light and dark gray dots). On the contrary, if we simply consider scattering vectors from intraband transitions (dashed light and dark gray arrow lines in Fig. 3b), they will merely result in a single closed contour (formed by empty light and dark gray dots) in Fig. 3d as reported before32,33. Note that the expected QPI pattern from the hexagonally deformed CECs in Fig. 3b needs to take both the inter- and the intraband transitions into account, i.e., overlapping Fig. 3c with the Fig. 3d, which has been graphically illustrated in Supplementary Fig. S2.

From Fig. 3e to g, representative dI/dU mappings clearly show spatial LDOS modulations in unoccupied of px,py (mj = 1/2)-derived Rashba band37. Note that the standing waves become barely visible when bias voltages are above 1.4 V (see Supplementary Movie for details). The electron interference pattern results in two circle-like features in the Fourier-transformed dI/dU (FT-dI/dU) image as indicated by red and blue arrows in the middle of Fig. 3e to g. Note that the two circle-like features in FT-dI/dU images are a result of the superposition of the inter- and the intraband transitions, which indicates the presence of hexagonally deformed CECs as conceptually illustrated in the Fig. 3b–d. In order to exclude possible defect induced artifacts, we have inverted the FT-dI/dU signals from individual circle-like features to restore the dI/dU mappings separately and found that electron-standing waves are indeed developed with a long-range coherence (see Supplementary Fig. S3 for details). According to the line profiles along $$\overline {{\it{{\Gamma}{\rm{M}}}}}$$ and $$\overline {{\it{{\Gamma}K}}}$$ of FT-dI/dU images with an exponential background removed at the bottom of Fig. 3e–g, two scattering vectors have been extracted by employing the Gaussian fits (red/blue lines)15,22,32,33 (see Supplementary Fig. S4 for line profiles without removing the exponential background). Although spin-conserving scattering, i.e., kf(E) and ki(E) with the same direction of spin polarization, is typically considered, we are also aware of spin–flip scattering on nonmagnetic scattering centers that need to be taken into account for the case of band structures with a spin-polarization inversion15,22,32,33,38. In contrast to the simple selection rule that the scatterings between exact anti-parallel spins are forbidden, spin-flip scattering process permits the transitions from a 180°. reversal of spin direction accompanied by a 90°. rotation of orbital moment. Due to the spin–orbit entanglement, the spin-flip scattering transitions between surface states with different orbital parities, i.e., (spz), (px − ipy), and (px + ipy), also become allowed as long as the total angular momentum remains conserved15,33 (see Supplementary Fig. S5 for the spin–orbit entanglement in the orbital-dependent spin-resolved band structures). However, we denote that either spin-conserving or spin-flip scattering along would give rise to only a single scattering vector in the px,py (mj = 1/2)-derived Rashba band, if only the intraband transitions are involved.

## Discussion

Figure 4a summarizes the dispersions, Dn, i.e., the energy-dependent scattering vectors qn(E) along $$\overline {{\it{{\Gamma}{\rm{M}}}}}$$ within bias voltage range from 1.5 V to −0.45 V (for qn(E) along $$\overline {{\it{{\Gamma}K}}}$$ see Supplementary Fig. S6). Figure 4b is the spin-decomposed band structure with in-plane spin polarization Sx,y of the BiCu2 surface states. For the scattering vector dispersions of D1 (black dots) and D2 (green dots), they come from intra- and interband scattering processes of s,pz and px,py (mj = 1/2) Rashba bands22. The black and green lines are the parabolic fits to obtain the effective masses $$m_{\overline {{\Gamma}{\rm{M}}} }^ \ast$$ of (−0.34 ± 0.02)me and (−0.39 ± 0.01)me, respectively. These $$m_{\overline {{\Gamma}{\rm{M}}} }^ \ast$$ values are consistent with those reported from photoemission work28. Besides D1 and D2, QPI results also reveal the band dispersions of D3 (gray dots) in the occupied energy region. We associate D3 with the scattering between the outer branch of s,pz and inner one of the px,py (mj = 1/2) bands since it starts appearing at about 0.1 eV well above the projected bulk band gap of the Cu(111) substrate, whereas the strong hybridization between the outer branch of s,pz band and bulk states does not play a significant role in forbidding the emergence of this scattering vector q322,23,30.

Interestingly, two dispersions of D4 and D5 have been resolved from the px,py (mj = 1/2) band in the unoccupied energy region, and they correspond to scattering vectors q4 and q5, respectively. In order to explain q4 and q5 in terms of interband scattering events in the hexagonally deformed CEC as illustrated in Fig. 3c, we have further calculated spin-dependent CEC with out-of-plane spin polarization Sz in Fig. 4c and the spin-decomposed band structure with Sz in Fig. 4d. According to the CEC at 0.6 eV in Fig. 4c, there is an inner ring-like circle accompanied by an outer circle with a hexagonal shape. They both are colored by red and blue for positive and negative Sz components, which are the most important characteristics that come along with the warping deformation of the px,py (mj = 1/2) band (see Supplementary Fig. S7 for details). Typically, the Rashba band consists of two concentric circles on the CEC with an in-plane chiral spin structure perpendicularly locked to the momentum vector, leading to prohibited backscattering as a consequence of time-reversal symmetry. However, this scenario is not perfectly true if there is a warping distortion on the CEC, where not only a single pair of stationary points can be stabilized but also new scattering channels can be opened5,8,9,10,35. To confirm our observation, we have also performed simulations on QPI structures based on generic $$k \cdot p$$ theory with a cubic warping term included (see Supplementary Note for details)5,35. Given that the effective scattering potentials in T-matrix formalism are predominately developed from the corners and midpoints on the hexagonal CEC, as shown in Fig. 4e, two circle-like features (red and blue arrows) are in line with FT-dI/dU image in Fig. 3d, where the inner circle (red arrow) arises essentially from interband transitions permitted in the hexagonal CEC35. On the other hand, the outer circle (blue arrow) is attributed to not only interband but also to extra input from intraband scattering channels as described in Fig. 3b, c. Note that the quantitative values in Fig. 4e are slightly smaller than FT-dI/dU in Fig. 3d, which could be a deviation of the offset k0 in DFT calculations on k|| of the high-lying, unoccupied Rashba band.

When the energy approaches the px,py (mj = 1/2) band edge, the CEC turns into a circular shape with a reduced Sz component because hexagonal warping interaction is less influential on smaller k values. With the plot in Fig. 4f, in contrast to the s,pz-derived band, we have found substantial out-of-plane Sz up to 24% in the dispersion of px,py (mj = 1/2) band. This result indicates that, due to the in-plane symmetry of atomic orbitals, they are more effectively coupled to in-plane potential gradient from $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 surface, leading also to a significant out-of-plane Sz polarization when spin–orbit coupling is involved. For comparison, we have also calculated the Sz of BiAg2 and found that BiCu2 can have Sz about three times larger than that of BiAg2 because of the stronger mixing of the p-states caused by the larger outward protrusion of Bi atoms ~1.0 Å on BiCu2 surface (see Supplementary Fig. S8 for details).

In conclusion, we have investigated the hexagonal warping effect in unoccupied Rashba spin-split surface states of the $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 surface alloy. From dI/dU spectra as well as calculated band structures, three hole-like Rashba-split bands have been resolved. Their spectroscopic features locate separately at 0.23 eV (s,pz), 1.4 eV (px,py (mj = 1/2)), and 1.6 eV (px,py (mj = 3/2)). Caused by hexagonally deformed Fermi surface, the energy-dependent QPI mappings have visualized standing wave patterns from interband scattering transitions of px,py (mj = 1/2) band. Due to a strong in-plane potential gradient on BiCu2 surface, hexagonal warping is accompanied by out-of-plane Sz spin polarization up to 24% in the dispersion of px,py (mj = 1/2) band. This strong effect is enabled by the in-plane symmetry of atomic orbitals. In light of such an orbital-selective hexagonal warping effect, our results, therefore, demonstrate an important and generic aspect of engineering relevant spin-dependent properties on low-dimensional Rashba materials by exploiting atomic orbital symmetry.

## Methods

### Experiment

The Bi/Cu(111) surfaces were prepared in an ultrahigh vacuum (UHV) chamber with the base pressure below P ≤ 2 × 10−10 mbar. The clean Cu(111) surface was first prepared by cycles of Ar+ ion sputtering with an ion energy of 500 eV at room temperature and subsequent annealing up to 800 K. The Bi source with the purity of 99.999% (Goodfellow) was e-beam sublimated onto Cu(111) surface at elevated temperature of 400 K at which the well-defined and extended $$\sqrt 3 \times \sqrt 3 R30^0$$ BiCu2 binary alloy can be grown. After preparation, the sample was immediately transferred into a low-temperature scanning tunneling microscope (LT-STM) from Unisoku Co. Ltd. (operation temperature T ≈ 4.5 K). The topography images were obtained from the constant-current mode with the bias voltage U applied to the sample. For STS measurements, a small bias voltage modulation (Umod = 20–50 mV) was added to U (frequency v = 3991 Hz), such that tunneling differential conductance dI/dU spectra, as well as dI/dU maps, can be acquired by detecting the first harmonic signal by means of a lock-in amplifier.

### Theory

First-principles calculations are performed using the Vienna Ab initio Simulation Package (VASP) based on the DFT39,40,41. The projector-augmented-wave-type pseudopotential with the Ceperley–Alder and Perdew–Zunger (CA-PZ) type exchange-correlation functional are adopted in the local density approximation (LDA) calculations42,43,44,45,46. We consider the BiCu2 monolayer on top of a nine-layer Cu(111) $$\sqrt 3 \times \sqrt 3$$ substrate to simulate our experimental system. The ion positions of the top three layers are optimized until the residue force is smaller than 0.02 eVÅ−1. After the geometrical optimization, the buckling height of Bi ion of the BiCu2 alloy is fixed at 1.0 Å as deduced from our experimental spectroscopy result and from ref. 26 where Kaminski et al. give 1.05 Å. Spin–orbit coupling is included in the self-consistent calculations with the energy cutoff of 400 eV over the 12 × 12 × 1 k-mesh. The DOS is integrated over the zone center near the $${\it{\bar {\Gamma}}}$$ point with an additional broadening of 0.01 and 0.1 eV. The two-dimension energy contours of the Rashba bands around the $${\it{\bar {\Gamma}}}$$ point are calculated over the 40 × 40 k-mesh and then interpolated over the 360 × 360 k-mesh.