Introduction

The close correspondence between symmetry operations in 3D bulk bands and the topological character of the surface states lying on the edge of a crystal has been one of the central topics of solid state physics in this decade1. Such topological surface states (TSSs) are attractive not only for basic science but also for spintronic applications due to their momentum-dependent spin–orbital polarization2,3. Since TSSs originate from the topological order of bulk states, some qualitative characteristics of TSSs, such as the metallic band dispersion across the bulk bandgap, are robust against any non-magnetic external perturbations as long as the bulk states remain unchanged. This is a strong point on the one hand because of the expected contamination-tolerant working of devices, for example. On the other hand, the role of the surface atomic structure in TSSs is regarded as rather unimportant because of the robustness. Some theoretical and experimental studies reported modification of TSSs by surface treatments, such as surface oxidization4, chemical or photochemical ageing5,6, and a small interlayer rotation forming a moiré modulation on the surface7. However, such previous works were focused on how to perturb the pristine TSS. Actually, the unit cells of the surface lattice in these works were always the same length of basis vectors as the ideally truncated bulk crystal, with the symmetry operations obtained by simple projection of bulk 3D space group. Some studies focused on topological electronic states localized at 1D edges or hinges, but they still suppose the similar simple projection from 3D to 1D systems8,9. Although some drastic electronic phenomena, such as 1D charge-density-wave formation on 2D semiconductor surfaces10, obtained by forming a surface atomic structure independent of the substrate, are already known, the role of the surface superstructures in TSSs has not been studied in detail yet, to the best of our knowledge.

Samarium hexaboride (SmB6) is a long-known Kondo insulator, in which a bulk bandgap opens at low temperature because of the Kondo effect11. It is the first material proposed as a candidate for topological Kondo insulators (TKIs), and it hosts a metallic TSS coexisting with strong electron correlation12,13 and was recently confirmed as a TKI by angle-resolved photoelectron spectroscopy (ARPES) experiments after a long debate14,15,16,17,18, as summarized in ref. 19. The (001) surface of SmB6 is also known as a valuable example of a well-defined surface superlattice among TIs. Although cleavage provides multiple surface terminations20, in situ surface preparation, typically by the cycling of Ar ion sputtering and annealing, results in an uniform surface superstructure such as (1 × 2) and (2 × 2)14,21,22, making the SmB6(001) surface a good template to study the role of the surface atomic structure in TSSs. The remaining barrier for such research is that the obtained SmB6(001) surfaces have two or more equivalent domains coexisting with nearly the same area. For example, double-domain (1 × 2) and (2 × 1) surfaces could provide apparent fourfold rotation symmetry of the TSS by overlapping of two twofold domains rotated 90° to each other, as observed on the Si(001) surface23.

In this work, we show the TSS on the uniform and semi-single-domain SmB6(001)-p(2 × 2) surface prepared in situ. One surface domain is dominantly grown by using a vicinal (001) substrate, confirmed by low-energy electron diffraction (LEED). The dispersion and orbital–angular-momentum (OAM) polarization of the SmB6(001) TSS is observed by ARPES without ambiguity from the multidomain overlap for the first time. The ARPES data show that the TSS is anisotropic and that the Fermi contour (FC) lacks the fourfold rotation symmetry that is maintained in bulk. This result emphasizes the important role of the surface atomic structure even in the TSS.

Results

Vicinal (001) surface of SmB6

As depicted in Fig. 1a, the bulk-truncated (001) surface of SmB6 has a fourfold rotation symmetry. After cycles of Ar ion sputtering and annealing (see “Methods” for details), the vicinal SmB6(001) surface exhibited the LEED patterns shown in Fig. 1b. In contrast to most of the earlier studies reporting (2 × 1) surface periodicity14,21, the (1/2 1/2) fractional spot indicated the p(2 × 2) surface structure at the primary electron energy (Ep) of 65 eV. Moreover, other fractional spots at Ep = 45 eV, (1 − 1/2) and (1/2 − 1), show different intensities from each other, although they should be identical if the surface has fourfold rotation and time-inversion symmetries. Note that different diffraction spots become bright at Ep = 45 and 65 eV simply because to sweep LEED Ep corresponds to sweep the size of the Ewald sphere in reciprocal space. The asymmetric intensities between (1 – 1/2) and (1/2 – 1) are quantitatively shown by the LEED line profiles in Fig. 1c. This shows that the obtained surface has an anisotropic atomic structure without fourfold rotation symmetry. The line profile also shows that this surface hosts the mirror planes [100] and [010], since the spots (1 ± 1/2) have nearly the same heights. Note that the height difference between (1 − 1/2) and (1/2 − 1) does not directly reflect the area ratios of surface domains; the obtained surface periodicity is p(2 × 2), and thus both (1 − 1/2) and (1/2 − 1) spots could appear even from the perfect single-domain surface.

Fig. 1: Surface preparation of the vicinal SmB6(001) surface.
figure 1

a Crystal structure of SmB6. b LEED patterns of the cleaned SmB6(001) surface taken at 100 K. The distortions in the patterns are due to the flat microchannel plate in the LEED electron optics. Two-headed arrows indicate the lines along which the profiles in (c) were taken. c LEED line profiles taken from the pattern shown in (b). d Angle-integrated photoelectron spectra taken at 13 K. 6F and 6H are the Sm2+ 4f5 final states after photoexcitation. e A surface atomic structure of SmB6(001) without any surface superstructure. f The same as (e) but with hypothetical p(2 × 2) surface reconstruction. Arrows guide the displacements of Sm and B6 near the surface.

Full size image

Figure 1d shows the B 1s core level and Sm2+ 4f valence bands obtained from angle-integrated photoelectron spectra of SmB6(001)-p(2 × 2). Different from the cleaved SmB6(001) cases17, we found no clear “surface” term which appears at different energies from the main peaks. This would occurs because the heating process during the surface preparation removes the metastable Sm and B atoms which are the origin of the “surface” terms. The B 1s peak has a tailed shape with respect to higher kinetic energies, suggesting that some boron atoms are in a different surrounding environment from the bulk case, but the difference is not as drastic as in the "boron-terminated” case of the cleaved surfaces17,20. This suggests that small displacements of the surface boron atoms are the origin of the p(2 × 2) superlattice. While some cleaved surfaces of SmB6(001) were reported to be inhomogeneous with various termination atoms17,20,24, we found no such signature from the current surface. Further discussion on this point is shown in Supplementary Note 1.

Based on these results, we propose a possible model of p(2 × 2) surface superstructure, as illustrated in Fig. 1e and f. In this model, no surface defects nor adatoms are supposed, and the p(2 × 2) superlattice is formed only by displacements of B6 and Sm atoms at the topmost two atomic layers. Such picture agrees with the less drastic change of surrounding conditions of the surface atoms, which was suggested by the core-level spectra explained above. In addition, the number of neighboring atoms of the displaced Sm is the same as those in bulk, in contrast to the reduction for the topmost B6. It rationalizes the core-level spectra showing the "surface” term only for boron peaks. Note that this model is just a possible example to satisfy what were observed by LEED and core-level spectra. Further experiments are required to determine the surface atomic structure accurately, such as dynamical LEED analysis25 or Weissenberg reflection high-energy electron diffraction26, while the p(2 × 2) surface unit cell without fourfold rotation symmetry is enough for the following discussion in this article.

TSS of the vicinal SmB6(001)-p(2 × 2) surface

Figure 2a shows the FCs around the Fermi level (EF) measured with circularly polarized photons at 35 eV. The spectra obtained by using both right- and left-handed polarizations are summed to avoid the anisotropic intensity distribution due to the circular dichroism (CD). Figure 2b shows a schematic drawing of the obtained FCs S1, S2, and S3. The oval enclosing \(\bar{{{{{{{{\rm{X}}}}}}}}}\) (S1) has a similar shape to those observed in the earlier studies15,16,17,19,21. The stark contrast to the earlier studies is that the FC around the other \(\bar{{{{{{{{\rm{X}}}}}}}}}\) point, S1’ depicted in Fig. 2b, is quite weak, although they should be identical, if the surface has a fourfold rotation symmetry. To highlight this difference, the other \(\bar{{{{{{{{\rm{X}}}}}}}}}\) point is named as \(\bar{{{{{{{{\rm{Y}}}}}}}}}\) (Fig. 2a). One possibility to explain this anisotropy is the ARPES experimental geometry. However, as illustrated in Fig. 2c, the ± ky//[010] orientations are identical to each other, even when the photoelectron incidence orientation is taken into account. Moreover, S1’ becomes as evident as S1 near the edge of the crystal surface where the vicinal miscut is expected to be different from that at the center of the polished surface, indicating that the difference between S1 and S1’ is from the surface domains (ARPES data are shown in SM). Together with the anisotropic LEED patterns (Fig. 1b, c), the faint S1’ can reasonably be assigned to the minor-area domains and to the fact that the SmB6(001)-p(2 × 2) surface lacks the fourfold rotation symmetry with the S1 FC only around one \(\bar{{{{{{{{\rm{X}}}}}}}}}\) point.

Fig. 2: Fermi contours (FCs) obtained by ARPES.
figure 2

a ARPES FCs taken with circularly polarized photons (hν = 35 eV) at 13 K. The ARPES intensities from the left- and right-handed polarizations are summed up to show all the states without any influence of circular dichroism (CD). The photon-incident plane is (\(\bar{1}\)10). The arrows indicate the positions where the E-k dispersions shown in Figs. 3, 4, and Supplementary Fig. 1 were taken. b Schematic drawing of the observed FCs together with the border of surface Brillouin zones; solid lines for bulk-truncated (1 × 1) and dashed for (2 × 2). c Experimental geometry and definition of the in-plane wavevectors kx and ky. kx and ky are always in the photon-incident and photoelectron detection planes, respectively. In this work, two cases, the incident planes of (010) and (\(\bar{1}\)10) were measured. The relationship between kx,y and the crystal orientation is shown by subindexes such as \({k}_{y//[\bar{1}10]}\) and ky//[010].

Full size image

The other FCs enclosing \(\bar{{{\Gamma }}}\), S2 and S3, are also clearly observed, in contrast to the blurred FCs in earlier works16. This is due to the uniform single-domain surface preparation over a wide area of the sample. The shape of S2 is similar to that observed on the cleaved (001) surface. While that state was claimed as the umklapp of S1 by (1 × 2) surface periodicity at first17, this assignment has not yet reached a consensus, as the following discussion made a counterargument on this assignment19. On the other hand, as shown in Fig. 2a and b, the shapes of S1 and S2 observed here are clearly different, indicating that S2 is another, independent FC.

All the three FCs observed here lacks the fourfold symmetry, as shown in Fig. 2. Such breakdown of the bulk-related symmetry in the TSS is observed for the first time among TIs, to the best of our knowledge. It should be derived from the anisotropic surrounding conditions of Sm atoms near the surface, since the bulk states around the Kondo gap is mainly derived from Sm 4f and 5d orbitals. Such condition can be naturally satisfied by assuming p(2 × 2) surface superstructure; for example, minor displacements of Sm below the topmost surfaces as depicted in Fig. 1f. On the other hand, no folding of FCs at the SBZ boundary was observed, while it is rather common case among surface states with superstructure, as discussed in Supplementary Note 2.

Figure 3 shows the band dispersions near EF. The almost localized Sm 4f band lies slightly below EF (~20 meV) and the Sm 5d bands disperses in 150–50 meV, reflecting the itinerant character. Around the crossing point between them, the bands bend, indicating the c-f hybridization induced by the Kondo effect. Above the Sm 4f band, the metallic bands continuously disperses between EF and Sm 4f in Cuts 1–3. Their Fermi wavevectors (kF) are consistent with the FCs observed in Fig. 2. Note that distinguishing S1, S2 and S3 in Fig. 3b is difficult because they overlap with each other at EF along this orientation. While the surface bands corresponding to S2 and S3 are not very clear from the 2D intensity plot in Fig. 3c, this is because of the too strong Sm 4f band and the momentum distribution curve (MDC) at EF shows distinct peaks corresponding to them. Moreover, Fig. 3d shows the dispersion of S2 and S3 close to EF without such difficulty. This is thanks to the photon energy (18 eV) for Fig. 3d, where the photoemission cross-section for Sm 4f becomes small. The kF values for S2 and S3 obtained in Fig. 3c and d show no difference depending on the photon energy, indicating that their 2D nature arises from the surface. For further confirmation, we also checked the lack of 3D dispersion for S2 and S3 by measuring a series of MDCs at EF with different photon energies (see Supplementary Fig. 2 in SM).

Fig. 3: Band dispersions of SmB6(001)-p(2 × 2).
figure 3

ARPES intensity plots with the same condition as in Fig. 2 together with the momentum distribution curves cut at EF, measured along the (a) \(\bar{{{{{{{{\rm{X}}}}}}}}}\)-\(\bar{{{{{{{{\rm{M}}}}}}}}}\) (Cut 1), (b) \(\bar{{{\Gamma }}}\)-\(\bar{{{{{{{{\rm{X}}}}}}}}}\) (Cut 2), and (c, d) \(\bar{{{\Gamma }}}\)-\(\bar{{{{{{{{\rm{M}}}}}}}}}\) (Cut 3) orientations.

Full size image

OAM polarization of the TSS

One of the most prominent characteristics of TSSs is the helical spin and OAM polarization, which is always perpendicular to the in-plane wavevector1. Figure 4 shows a CD-ARPES map corresponding to the region shown in Fig. 3. The CD of ARPES reflects the OAM polarizations projected onto the photon-incident orientation. As illustrated in Fig. 2c, CD in the current experimental geometry corresponds to OAM polarization along the in-plane and normal to ky, or out-of-plane orientation. Figure 4a shows that the oval-shaped S1 around \(\bar{{{{{{{{\rm{X}}}}}}}}}\) has helical OAM polarization consistent with the earlier CD-ARPES15 and spin-resolved ARPES16 results. Figure 4b and c show that the surface bands S2 and S3 also have OAM polarizations whose sign inverts from +ky to −ky, obeying the time-inversion symmetry. This behavior, helical OAM polarizations with time-reversal symmetry, is consistent with what is expected for TSSs1,19.

Fig. 4: CD-ARPES band dispersions.
figure 4

a–c CD-ARPES plots taken at 13 K. The measurement geometries are the same as those in Figs. 2 and 3. d Schematic drawing of the OAM polarizations corresponding to surface FCs, assuming that they are parallel to the helicities of the incident-photon polarizations: parallel (anti-parallel) to the photon-incidence vector for right- (left-) handed polarization. Fat lines indicate the cut orientations shown in (a–c). Circled digits indicate number of surface FCs in SBZ. e Surface FCs folded by (2 × 2) SBZ boundary. Arrows guide the possible singularity points (see main text for details). f Schematic drawings of TSS with folding and distortion from the anisotropic surface superstructure.

Full size image

Figure 4d illustrates the OAM polarizations of each FC, assuming that they are parallel to the helicities of the incident-photon polarizations: parallel (anti-parallel) to the photon-incidence vector for right- (left-) handed polarization. According to this, the helicities of the OAM polarizations for S1 (counterclockwise) are opposite to those for S2 and S3 (clockwise). Such behavior is not expected from the analytical calculation with the winding number27. However, note that the sign of the CD-ARPES signal is not suitable information to be directly compared with such calculation of the initial states. First, it is easily affected by the photoelectron excitation process28. Second, the calculation does not include the surface atomic structure obtained in this work. Finally, TSSs were recently revealed to often contain multiple wavefunction components with different orbital and spin polarizations even at single (E, k) point; thus, the polarization of photoelectrons is not always the direct information of the initial state29,30. Therefore, further datasets, especially spin-resolved ARPES with multiple incident-photon energies and polarizations, are required to conclude the winding number of the TSS.

Discussion

We have revealed that the TSS of SmB6(001)-p(2 × 2) is highly anisotropic. From this anisotropy, the other electronic properties, such as the electron conductivity and net spin polarization via the surface currents, are also suggested to be anisotropic on SmB6(001)-p(2 × 2), as far as they are derived from the TSS. Note that none of these properties are topologically protected. Therefore, examining the rotation asymmetry from the vicinal SmB6(001) sample with in situ surface preparation would be an interesting new pathway to distinguish the origin and dimensionality (2D or 3D) of unconventional electronic phenomena observed in SmB6, such as the thermodynamic properties31 and quantum oscillations32,33,34,35,36,37. If one could apply the surface preparation method similar to ours to vicinal SmB6(001) sample and then insert it to a magnet equipment for de Haas van Alphen oscillation measurements without breaking the vacuum environment, such experimental setup would provide conclusive evidence about the role of surface on the quantum oscillation.

Since the folding of FCs by surface superstructure changes the apparent shape of FCs drastically, one could have a question about its role on topological order. In most cases including SmB6(001)-p(2 × 2), surface superstructure causes no change the energetic order of bulk bands nor magnetic order. Therefore, there shouldn’t be any change in topological order before and after the formation of surface superstructure. However, it would be helpful to discuss how to determine the topological order with the surface superstructure, since it is often determined by counting the number of FCs to be odd or even, around surface time-reversal invariant momenta (TRIM); \(\bar{{{\Gamma }}}\), \(\bar{{{{{{{{\rm{X}}}}}}}}}\), \(\bar{{{{{{{{\rm{Y}}}}}}}}}\), and \(\bar{{{{{{{{\rm{M}}}}}}}}}\) for SmB6(001)18,19,38. For this counting, we propose to double check the number of FCs both by using (1 × 1) bulk-truncated SBZ as well as that with folding according to the surface superstructure. Circled numbers in Fig. 4d and e are examples of this method applied to SmB6(001)-p(2 × 2), showing 3 FCs, odd number consistent with the non-trivial topological order18,19. Note that S1 at the bottom side of Fig. 4d is identical to the top one because of the translational symmetry, and thus it should not be included to this FC counting. This method emphasizes the importance to prepare the single-domain surface to discuss the topological order of unknown material; for example, surface Dirac cones doubled by two nonequivalent surface terminations, as observed in ref. 39, could derive false counting of FCs.

The folding of TSS by surface superstructure could play further role on the unconventional electronic phenomena of TSS. It duplicates FCs formed by TSS, making the additional crossing points with van-Hove singularity (VHS, arrows in Fig. 4f). Such VHS formation is discussed theoretically based on a moiré modulation to expect enhanced topological superconductivity40. Moreover, this work exhibited that the surface superstructure distorts surface FCs at the same time. It suggests that engineering of surface superstructure could tailor the number and position of VHS in the reciprocal space as well as the other parameters of FCs, such as nesting vector dominating surface density-wave formation41. Actually, the observed FC on SmB6(001)-p(2 × 2) does exhibit triple degenerate points of TSSs as guided by arrows in Fig. 4e. On the other hand, the reconstruction of TSS in Kondo gap observed here is the first case, to the best of our knowledge. In most cases, surface superstructure are regarded to play a major role in larger energy scale typically in 0.1–1 eV, orders of magnitude larger than those of Kondo gap (a few tens of meV). The VHSs formed by surface superstructure possibly also enhance the electron correlation effect expected to TSS of TKI, such as the emergence of heavy surface states and non-Hermitian exceptional points42,43. From these perspectives, detailed electron behavior in smaller energy and temperature scale, in orders of few meV and Kelvins, respectively, of SmB6(001)-p(2 × 2) would be an encouraging test peace to examine the role of the surface superstructure on TKI.

Methods

Sample preparation

Single crystalline SmB6 was grown by the floating-zone method by using an image furnace with four xenon lamps21,44. The sample cut along the (001) plane was mechanically polished in air with a small angle offset toward [010] until a mirror-like shiny surface was obtained with only a few scratches when observed under an optical microscope (multiple ×10 magnification). The miscut angle is estimated to be ~ 1° from the side-view images obtained by the optical microscope. The polished sample was moved to ultra-high-vacuum (UHV) chambers and cleaned in situ by repeated cycles of Ar ion sputtering (1 keV) and annealing up to 1400 ± 50 K. The cleaned surface was transferred to the measurement chambers for LEED or ARPES without breaking the UHV.

LEED measurements

The LEED measurements were performed by using a conventional electron optics (OCI, Model BDL800IR) at beamline 7U of UVSOR-III. The sample was mounted on a homemade 6-axis goniometer maintained at 100 K. The electron incident angle was adjusted to be parallel to the surface normal ([001]) by using the rotation angles of the goniometers.

ARPES experimental setup

The ARPES measurements were performed with synchrotron radiation at the CASSIOPÉE beamline of synchrotron SOLEIL. The photon energies used in these measurements ranged from 18 to 240 eV. The polarization of incident photons is depicted in Fig. 2c, including linear polarization with the electric field lying in the incident plane (H) and circular polarizations (R and L). The photoelectron kinetic energy at EF and the overall energy resolution of the ARPES setup ( ~ 15 meV) were calibrated using the Fermi edge of the photoelectron spectra from Ta foils attached to the sample.