Introduction

Recent experimental1,2,3,4,5 and theoretical6,7,8 works have demonstrated that topological surface states (TSS) support massless spin-polarised Dirac fermions due to a strong spin-orbit coupling effect. They are robust and immune to backscattering by non-magnetic defects because of protection by time-reversal symmetry9. Unlike the Dirac fermions in graphene with pseudospin texture, the real helical spin polarization TSS exhibit has been widely studied by surface-sensitive experiments such as angle-resolved photoemission spectroscopy (ARPES), scanning tunneling microscopy (STM) and scanning tunneling spectroscopy (STS) as summarized in recent review articles10,11. TSS are attractive not only in fundamental condensed matter physics for realising novel entities such as dyons12, imaging magnetic monopoles13 and Majorana fermions12,14, but also in practical applications in spintronics and error-tolerant quantum computing. Motivated by such perspective, special attention has been focused on thin films hosting TSS suitable for device applications. Therefore, three-dimensional (3D) topological insulator (TI) thin films approaching 2D limit have been explored. Theoretical calculations predict that an energy gap can be opened at the Dirac point by inter-surface coupling15. Some of these films may exhibit quantum spin Hall effect as that observed in HgTe quantum wells16,17. Experimental studies on Bi2Se318,19 and Bi2Te320 confirm the existence of a thickness-dependent bandgap. However, the detail of inter-surface coupling effect, especially the intermediate state between weak and strong coupling of surface states (SS) from two surfaces, still needs to be interrogated. From topological nontrivial to trivial phases, interesting transformation must take place progressively as the overlap of electron wave functions on two surfaces increases with reducing film thickness.

Here we investigate the inter-surface coupling issue in antimony (Sb), which has a rhombohedral crystal structure and can be considered as a stacking of (111) bilayers (BL, 1 BL = 3.75 Å). There are several reasons to study Sb(111) films. Firstly, as the “parent” of the first-generation 3D TI Bi1−xSbx21, although bulk Sb is semimetal due to its negative indirect bandgap, its band order is inverted at the L point of the Brillouin zone. Sb(111) has been confirmed to possess TSS, leading to the absence of 180° backscattering and exotic transmission through atomic steps22. Next, the strongly distorted TSS Dirac cone on Sb(111) yields various scattering channels in QPI patterns, from which TSS dispersion and spin information can be extracted. Measuring the patterns at different film thickness helps us understand how the spin texture evolves as inter-surface coupling varies. Thirdly, a thin film with a large surface-to-volume ratio can make surface effect more observable. Lastly, compared with well-studied compound TIs such as Bi2Se3, Sb thin films provide a single-element simple system for demonstrating topological properties without much influence of residual bulk carriers from self-doping states2,4,23 or spatial fluctuations of charge and potential24. Therefore, by using Fourier-transform scanning tunneling spectroscopy (FT-STS), we measure the quasi-particle interference (QPI) patterns of TSS on thick and thin Sb(111) films. Combining with density functional theory (DFT) calculations, we aim to identify the scattering features in the patterns generated by the intra- and inter-surface couplings of TSS and to examine the dependence of the couplings on film thickness and wavevector k.

Results

We grew Sb films on Si(111)-√3 × √3:Bi-β surface in an ultrahigh vacuum (UHV) chamber25,26,27,28. Shown in Figure 1a is a STM image of an Sb(111) film of thickness ~30 BL. Steps here are all 3.75 Å corresponding to 1 BL of Sb(111). The atomic resolution image in the inset exhibits the hexagonal lattice on the (111) surface. In Fig. 1b, a typical differential tunneling conductance dI/dVb spectrum, where Vb is sample bias voltage and I the tunneling current, is displayed to reveal the local density of states (LDOS) of electrons in certain energy range. Unlike Bi2Se3 and Bi2Te3, Sb(111) has a strongly distorted TSS Dirac cone, with its lower part bowing up rapidly near point of the first surface Brillouin zone (SBZ) before bending down to merge with bulk valence bands, as shown schematically in Fig. 1b inset. Several extraordinary features of Sb(111) show up starting from the Dirac point ED at E ≈ −240 meV (with the Fermi energy EF at E = 0), which have also been confirmed by previous ARPES measurements1,29. The specific dispersion of TSS on Sb(111) allows us to determine ED being at the appearance E of QPI patterns22,30.

Figure 1
figure 1

Atomic and electronic structures on Sb(111).

(a) STM image of 30 BL Sb(111) film grown on Si(111)-√3 × √3:Bi-β. Imaging conditions: Vb = −3 V, I = 50 pA. The inset shows the atomic-resolution image of Sb(111) surface (Vb = 1 V, I = 250 pA). The period is about 4.31 Å. (b) dI/dVb spectrum taken on 30-BL Sb(111) surface. The inset shows the schematic SS bands forming a Dirac point at on Sb(111). By cutting the bands with vertical planes passing and axes, the dispersions lines of SS along these high symmetry directions can be obtained.

For TSS, it is generally accepted that, without magnetic field or magnetic impurities, scattering between states must obey spin conservation. That is, the scattering intensity P from a state of wavevector k and spin S to another state of and depends on , the angle between S and , as5,30

STS can detect the amplitude modulation of LDOS due to the scattering of momentum transfer vector 5,31. After Fourier transform, the real-space modulations can be visualised as a distribution of q in the k-space, offering us information of the SS involved in scatterings and even their spin alignment at a given E. Hence, the dI/dVb maps on 30-BL Sb(111), which can be regarded as the bulk surface, were taken under different Vb. Above E ≈ −240 meV, 6-fold symmetric patterns with dominating intensity along the direction can be identified in the FT-STS power spectra. Fig. 2a shows the coexistence of atomic structure and QPI patterns, while Fig. 2b gives its Fourier transform, which also contains both the contributions from atomic lattice (the outer 6 reciprocal lattice points) and QPI pattern near at the centre. The hexagon marks the first SBZ.

Figure 2
figure 2

QPI patterns of TSS on 30-BL Sb(111).

(a) Real-space dI/dVb map in an area of 20 nm × 20 nm taken at Vb = 40 mV, I = 120 pA. (b) Fourier transform of (a), consisting of six strong peaks along . The first SBZ (yellow hexagon) and high symmetry directions are marked. (c) FT-STS pattern around with three observable scattering vectors qA, qBand qC. It was taken on a 40 nm × 40 nm area at Vb = 20 mV. (d) Schematic of CEC as well as the spin texture around , which has a central electron pocket and six hole pockets. The small grey arrows represent the spin directions. qAand qB are allowed scattering vectors, while qC is low in probability and qD totally forbidden.

The QPI patterns are our main concern. Starting at E ≈ −40 meV, a set of spots mainly along and appear. We use the pattern at E = 20 meV in Fig. 2c as an example to investigate the origins of these spots marked with scattering vectors qA, qB and qC. From the schematic constant energy contour (CEC) in Fig. 2d with the spin texture23, one can find that qA along represents the scattering between neighbouring hole pockets and qBis that between the hole edge and the electron pocket edge across . The qAscattering has a spin misalignment of 60°, giving a spin probability factor of ¾, while the spins are nearly parallel for the qB, giving a probability of 1. These two scattering channels are both observed along in Fig. 2c with the qAintensity weaker than that of qB. The qC scattering along between the next-nearest-neighbor hole pockets with = 120°, has a probability of ¼, yielding an even weaker intensity. The qD scattering between two hole pockets across , however, is totally forbidden due to antiparallel spins of two relevant states.

In Fig. 3a, a series of FT-STS patterns for E from −40 meV to 150 meV are displayed. Based on the magnitude of q plotted as a function of E in Fig. 3b, the dispersions of qA and qB are approximately linear with slopes of 1.18 eVÅ and 1.09 eVÅ, respectively. These values are close to those reported in previous STS22 and ARPES studies1. Similar dispersions have also been obtained from our QPI patterns taken near step edges, confirming that scattering only takes place between TSS with similar spin directions. Moreover, the observed QPI patterns can be compared with DFT computational data. A QPI pattern at certain E can be simulated based on the CEC and spin texture from DFT computations as well as the spin factor given by equation (1). We re-plot the experimental QPI patterns together with the simulation patterns for E = 5 meV and 80 meV in Fig. 3c–f. Comparing the corresponding patterns, one can find a few common features. Firstly, the dominating scattering vectors qB along appear as bright spots in all cases. Secondly, the spots centred at qB spread more in the direction transverse to at 80 meV than at 5 meV. Thirdly, the relative weak qC spots along become more visible at 80 meV than at 5 meV. The agreement between experimental and computational results on these key features confirms that the QPI patterns acquired on such a “thick” Sb(111) film are generated by spin-polarized TSS only on the top surface.

Figure 3
figure 3

Scattering vectors in QPI patterns on 30-BL Sb(111).

(a) FT-STS maps of the same area taken at Vb from −40 mV to 150 mV are selected to extract the dispersion relationship. Each map shows two sets of scattering vectors corresponding to qA and qB, respectively. (b) The dispersions of qA and qB extracted from (a). Here the error bar corresponds to 0.05π Å−1, which is determined by (2π/40) nm−1. (c),(d) FT-STS mapping taken at 5 mV and the corresponding simulation pattern based on DFT computational data. (e),(f) Measured and simulated FT-STS patterns at 80 mV. qBand qCare marked for comparison with Fig. 2c–d.

When Sb(111) film is thin enough, however, the TSS on the top and bottom surfaces will couple eventually. It is normally believed that a gap will open up at the Dirac point and the Dirac cone will turn into Dirac hyperbola when the coupling is significant15,18. In order to gain insight into how the thickness affects the band structure and furthermore, the coupling of SS from two surfaces, we examined a 9-BL Sb(111) film. Interestingly, according to DFT calculations, at this thickness, the coupling of states at is not strong enough to open a gap yet, but the inter-surface coupling of states elsewhere in k-space is already significant as revealed by FT-STS mapping. Shown in Fig. 4a is the QPI pattern on 9-BL Sb(111) at E = 20 meV. Compared with the 30-BL sample, a few changes are obvious in the pattern. Firstly, besides intensity mostly along in the 30 BL case, here comparable intensity along can also be seen. Secondly, instead of discrete q for 30-BL Sb, 12 lobes of continuous intensity with clear cutoff vectors labeled as qEand qF along and , respectively, are observable on 9-BL Sb, indicating new scattering channels are effective.

Figure 4
figure 4

QPI patterns reveal strongly inter-surface coupling of TSS in 9-BL Sb(111).

(a) A representative FT-STS mapping at 20 mV. Two cutoff vectors marked as qEand qF are both shown as black dashed arrows. (b) qEand qF in the calculated CEC at EF. The grey arrows illustrate the spin directions of TSS on the top surface of a thick film. Intra-surface scatterings qAand qB still exist and they correspond to the central green zone with high intensities in (a). (c) The corresponding simulated QPI pattern based on DFT calculations. (d) DFT-computation band structure of 9-BL Sb(111) along . The blue lines represent the SS band chosen for computing the spin separation. (e) Spin separation as a function of k, indicating strong k-dependent inter-surface coupling of SS. A, B and C are the intersection points of CEC at EF with k axis along . (f) The real-space distributions of SS from near to obtained from DFT computations, showing that a well-defined localization of SS around but large penetration depth for states from 0.34 Å−1 to .

Discussion

Now we discuss the origins of these new scattering channels on 9-BL Sb. Notice the scale difference between Fig. 2c and Fig. 4a. The new intensity lobes in Fig. 4a have q lengths longer than those in Fig. 2c. Specifically, the lengths of the cutoff vectors qEand qF (both shown as dash-line arrows) are 0.63 Å−1 and 0.54 Å−1, respectively, whereas qC = 0.25 Å−1 in Fig. 2c. The qE/qFlength ratio takes a unique value 1.17 in the measured energy range. For comparison, the calculated CEC at EF is shown in Fig. 4b, with superficial spin directions labeled as grey arrows assuming the upper surface separated far from the lower one. The new scattering channels are also marked in Fig. 4b, with qE between the outer edges of opposite hole pockets along and qF between the outer edges of next-nearest-neighbor hole pockets. Such assignments yield a , in agreement with the measured ratio. These scatterings, especially qE, however, seem to violate spin conservation based on the superficial spin texture.

Different from thick film case, the two surfaces of 9-BL film are separated not far from each other, so a SS is no long confined to one surface. Now, it is possible for an electron in state k on one surface to scatter into a state with the same spin on the opposite surface. In another viewpoint, for a thick Sb(111) film, there is one TSS Dirac cone on each surface. With time-reversal and inversion symmetries, a pair of SS of a particular k are degenerate in energy but with opposite spin. They do not couple noticeably with each other since their wave functions overlap little. In a thin Sb(111) film, the coupling becomes strong if wave function overlap is significant, yielding mixing states with their spin only partially polarized or even totally degenerate. This opens up new scattering channels without violating spin conservation. Therefore, quite different from the 30-BL film where the patterns mainly originate from the so-called “intra-surface scattering”, here the QPI patterns include “inter-surface scattering”. On the other hand, the intensities in inner hexagonal zone in Fig. 4a corresponding to qAand qB scatterings marked in Fig. 4b are stronger than those of qE and qF. This part overlaps closely with the QPI patterns on the 30-BL sample, demonstrating significant contribution from intra-surface scattering. This two-part pattern clearly shows the coexistence of intra- and inter-surface scattering events.

Based on DFT computation results of CEC and spin texture near EF, we simulated the QPI pattern as shown in Fig. 4c, which is in good agreement with our experimental pattern in Fig. 4a. First of all, the characteristic outer lobes of comparable intensities along and are reproduced in the simulation pattern. Secondly, the core part of the simulated pattern has stronger intensity along , which is consistent with out observation that “intra-surface scattering” is still nontrivial. It is also worth noting that the experimental patterns were taken at 4.2 K with a 5 mV peak-to-peak modulation added to Vb, which results in an energy resolution ΔE ≈ 4.5 meV32. If this energy window is considered, the dots and arcs in the simulation pattern should be smeared out to continuous features similar to those in Fig. 4a.

Besides new features in QPI patterns, following the procedure described by Bian et al.33,34, the spin separation of SS can be defined and evaluated for our 9 BL Sb(111) film. In Fig. 4d, the electronic states in the 9 BL film obtained with DFT computation are plotted along . The SS bands remain cross each other at , i.e., the Dirac point is intact. Due to quantum confinement effect, an indirect bandgap of ~0.3 eV can be observed for the bulk bands in Fig. 4d. Simple estimation based on bulk band parameters35 gives a bandgap ~0.53 eV, whereas Zhang et al. get a value of 0.25 eV in their computations36. The calculated spin separation as a function of k along for the SS bands is plotted in Fig. 4e, where the limiting values ±1 represent full spin polarization whereas 0 represents total spin degenerate. At , the states are fully spin-polarized, but the spin separation decreases quickly as k moves from towards and it vanishes at k = 0.34 Å−1. This k point is closer to than the total spin degenerate point of 20-BL Sb(111)33, indicating that a stronger inter-surface coupling reduces spin polarization of SS.

On the other hand, the variation of spin separation shown in Fig. 4e implies a non-uniform degree of inter-surface coupling, which should depend on the SS penetration depth. The calculated real-space distributions of SS in the film normal direction are plotted in Fig. 4f for six k points of the lower SS band along . It is obvious that around the SS are well localized near the surface, so the inter-surface coupling is weak. In contrast, as k approaches , the SS become more spread over the thickness, so the coupling is stronger.

To correlate the above computational results with the observed QPI patterns, we overlay the CEC at EF on the spin separation plot in Fig. 4e. The three points marked as A, B and C correspond to the intersections of EF with the SS bands. The states near point B with k ≈ 0.12 Å−1 are involved in the qB scatterings in Fig. 2c and d. These states have spin separation of about ±0.9 and they are localized near surface based on Fig. 4f, so they largely maintain the character of TSS. The states at point C with k ≈ 0.34 Å−1 are basically spin degenerate and spread over the whole thickness of 9 BL film. According to our DFT computations, the states at this k value in the 30 BL film exhibit SS character with a penetration depth λ ~ 5.7 BL (for comparison, λ ~ 1 BL at k = 0.12 Å−1). In the 9 BL film, with such a λ value, the states originated at two surfaces can couple strongly due to significant overlap in the interior, losing TSS character. These states can scatter with any other states at the CEC of the hole pockets without violating spin conservation, resulting in the outer lobes of continuous intensity terminating at qEand qF in Fig. 4a. The wave functions for the states with k ≥ 0.34 Å−1 in Fig. 4f have quite large magnitude both at the surface and in the film centre. Since the magnitude does not decay when moving from film interior to the surface, these states should be considered as surface resonance states instead of quantum well states (QWSs)37. QWSs derived from the bulk states, with their wave functions decaying significantly from film interior to the surface, are readily observable with ARPES33, but they seem to contribute weakly as diffused background intensity in FT- STS QPI patterns. We explain this in terms of different surface sensitivity in these two methods. The photoelectrons detected in ARPES come from the top ~1–3 atomic layers of the sample whereas STS detects electronic states at ~3–10 Å above the top atomic layer.

Along , the solid- and dot-line arrows in Fig. 4a denote the end points of scattering vector qB due to intra-surface coupling and qE due to inter-surface coupling, respectively. The ratio of intensities at these points, IE/IB, in FT-STS patterns is a quantitative measure of the relative strength of intra- and inter-surface coupling. In order to analyze the thickness dependence of inter-surface coupling, we took FT-STS patterns at E = 20 meV for Sb(111) films with thickness in 4–15 BL range. In Fig. 5, the FT-STS patterns for 15, 12, 6 and 5 BL samples are displayed. The measured IE/IB as a function of film thickness is plotted in Fig. 5e. It shows that IE is almost absent in 15 BL sample and it increases as film thickness reduces from 15 to 5 BL. This monotonic trend unambiguously confirms the evolutionary process of diminishing spin polarization from thick to ultrathin films.

Figure 5
figure 5

Film thickness dependence of relative strength of intra- and inter-surface couplings of TSS.

(a–d) FT-STS patterns taken on 15, 12, 6, 5 BL Sb(111), respectively. The red and green arrows mark the position of IB due to intra-surface coupling and IE due to inter-surface coupling. (e) Plot of IE/IB as a function of film thckness.

The inter-surface coupling of TSS in Bi2Se3 thin films has been studied both theoretically and experimentally15,18. There, the most remarkable effect is the opening of a bandgap at for the TSS. Comparing to Sb(111), the TSS on Bi2Se3(111) form a fairly ideal Dirac cone with rather weak warping in the bulk bandgap6,8, so they can be considered as true surface states. Sb(111) has a highly warped Dirac cone. The SS not far from penetrate quite deep into the interior and they almost behave as surface resonance states. In 9-BL Sb(111), the inter-surface coupling of these states makes them locate more near the film centre than at the surfaces. The spin splitting of SS due to the Rashba effect, which depends on the overall potential gradient V the electrons experience, is greatly reduced. To the extreme, for those states with k ≥ 0.5 Å−1 in Fig. 4f with wave functions totally symmetric about the film centre, the overall V = 0, so the spin polarisation is completely lost. Therefore, our results demonstrate that SS spin polarization diminishes not only with decreasing film thickness, but even more dramatically with k moving from to .

Although not an ideal TI, Sb(111) ultrathin films allow us to investigate the extraordinary scatterings between SS with nearly parallel spin directions, while in other TIs (e.g. Bi2Se3) possessing an ideal Dirac cone it is hard to get similar features exclusively between SS. The “distorted” Dirac cone and dramatic k-dependent penetration depth of SS in Sb thin films offer us an interesting quasi-2D system to help interpret the detailed evolution of coupling and hybridization of TSS. The dramatic change in QPI patterns with film thickness indicates that the intra- and inter-surface coupling of TSS dominates in the thick and thin film cases, respectively. The inter-surface coupling of TSS in a “thin” film shows strong k-dependence. The tuneability of relative contributions of intra- and inter-surface scatterings by changing thickness can be further explored for tailoring the surface energetic and transport properties for potential applications.

Methods

The experiments are carried out in a UHV LT-STM system. The base pressure is better than 1 × 10−10 Torr. Clean Si(111)-7 × 7 surface is prepared by degassing at 500°C overnight, a briefing annealing at 850°C and finally flashing at 1200°C for ~1 min. The Si(111)-√3 × √3:Bi-β surface serving as the substrate for Sb film growth in this work is prepared with 2-ML Bi deposition on Si(111)-7 × 7 at room temperature followed by annealing at ~450°C for 15 min. Bi and Sb deposition fluxes are generated from Ta boats loaded with high purity (99.999%) source materials. Prior to the deposition, the evaporators are degassed at appropriate temperature for a few hours in order to remove contaminations. Bi deposition flux is calibrated by measuring the coverage and thickness of Bi(110) films on Si(111). 1 ML of Bi(110) represents 9.3 × 1014 atom/cm2. Sb is deposited on Si(111)-√3 × √3:Bi-β at 100°C to form (111)-oriented thin films with thickness from 4 BL to >30 BL. All STM images are acquired at 77 K or 4.2 K. The dI/dVb spectra are acquired at 4.2 K using a lock-in amplifier with a modulation voltage at a frequency of 700 Hz and a peak-to-peak amplitude of 5 mV. The dI/dVb mapping for FT-STS is normally taken in a square area of edge length ~25–40 nm, without any atomic step and other defect in the area.

First-principles DFT-based electronic structure calculations are performed using the VASP package38 with a plane wave basis and a 5 × 5 k-point sampling of the Brillouin zone. In all calculations, generalized gradient approximation (GGA) in Perdew-Burke-Ernzerhof (PBE) format39 and spin-orbital coupling are included. The neighbouring Sb(111) film slabs are separated with a 10-Å vacuum region along the [111] direction.