Incommensurate smectic phase in close proximity to the high-Tc superconductor FeSe/SrTiO3

Superconductivity is significantly enhanced in monolayer FeSe grown on SrTiO3, but not for multilayer films, in which large strength of nematicity develops. However, the link between the high-transition temperature superconductivity in monolayer and the correlation related nematicity in multilayer FeSe films is not well understood. Here, we use low-temperature scanning tunneling microscopy to study few-layer FeSe thin films grown by molecular beam epitaxy. We observe an incommensurate long-range smectic phase, which solely appears in bilayer FeSe films. The smectic order still locally exists and gradually fades away with increasing film thickness, while it suddenly vanishes in monolayer FeSe, indicative of an abrupt smectic phase transition. Surface alkali-metal doping can suppress the smectic phase and induce high-Tc superconductivity in bilayer FeSe. Our observations provide evidence that the monolayer FeSe is in close proximity to the smectic phase, and its superconductivity is likely enhanced by this electronic instability as well.


Supplementary Note 1: The 2×1 reconstruction in 1 UC FeSe
Supplementary Fig. 1a shows a topographic image of an area including 1 UC and 2 UC FeSe.
The 2×1 domain boundaries (see the white dashed lines in Supplementary Fig. 1a) are clearly resolved and they continuously cross the step edge. A zoomed-in image on one of the boundary regions (denoted by the red rectangle in Supplementary Fig. 1a) is presented in Supplementary   Fig. 1b. The 2×1 domains show 90˚ rotation on the two sides of the domain boundary, which can be clearly seen in their corresponding fast Fourier transformation images ( Supplementary   Fig. 1c, d). The 2×1 spots are denoted by the red dashed circles. Supplementary Fig. 2a shows the raw dI/dV data in Fig. 1d. The spectra in different colors correspond to the different energy ranges in Fig. 1d. The gray dashed lines highlight the peak and dip positions of the stripe ordering. It is hard to make a direct analysis for these raw data, because the oscillation amplitudes of the spectra are significant different from each other. In order to give a complementary analysis other than calculating the d 3 I/dV 3 that is shown in the main text, each spectrum is divided by its linear background, re-scaled by multiplying a factor, shifted for clarity and finally shown in Supplementary Fig. 2b (the corresponding energies and scaling factors are labeled on the right of the spectra). It shows that the period of the stripes doesn't change with energy and the stripes in 30 meV to 60 meV have a π phase shift. These 3 conclusions are consistent with those obtained from the d 3 I/dV 3 spectra in Fig. 2d. Fig. 1d. a Raw dI/dV spectra. b The normalized dI/dV spectra. The spectra are shifted for clarity. The red, green and blue colors correspond to the different energy ranges in Fig. 1d. The gray dashed lines denote the peak and dip positions in the spectra.

FeSe
Strong nematicity in FeSe thin films leads to the lift of degeneracy between the dxz and dyz bands 1, 2 . STM can also detect such band separation by the inequivalent QPI patterns along the a-and b-directions 3,4 .
In order to obtain the symmetry breaking information and determine the a-and b-directions in 3 UC FeSe, QPI measurement is carried out in the vicinity of defects. Two Fe-vacancies are introduced as scattering centers ( Supplementary Fig. 3a), in which short-range stripes are clearly revealed. The unidirectional QPI patterns above EF are shown in Supplementary Fig. 3b-k and indicated by magenta arrows. The QPI patterns are along the diagonal direction of Se-Se lattice (corresponds to the direction of Fe-Fe lattice) and break C4 symmetry in real-space.
They propagate along the direction of the short-rang stripes. The patterns move towards the scattering centers at higher energies, indicating a shorter wavelength in real-space (or a larger scattering wave vector in q space). Such an energy dispersive behavior corresponds to an electron-like band above EF. The scattering wave vector qb is extracted from each image and fitted by a parabolic curve ( Supplementary Fig. 3l), indicating that the electron-like band reaches its band bottom at around 60 meV above EF.
The QPI results also exhibit unidirectional patterns below EF (Supplementary Fig. 4a-h), 4 but they are along the perpendicular direction to those taken above EF. The QPI patterns show two ring-like features in the vicinity of defects (denoted by the yellow dashed rings in Supplementary Fig. 4b). The scattering wavelength is determined by the far end-points of the ring features (the yellow arrows in Supplementary Fig. 4a-h), which become larger with energy closer to EF, reflecting a scattering process from a hole-like band. Figure 4i shows the extracted scattering wave vector qa and the corresponding parabolic fitting result, which indicates that the hole-like band reaches its band top at around -15 meV.

Supplementary Note 4: Evidence of nematicity: Quasiparticle interference in 2 UC FeSe
The QPI measurement is also employed in 2 UC FeSe and it presents similar results to that in 3 UC FeSe. The QPI patterns at positive energies indicate a scattering process from an electron-like band and its band bottom locates above EF ( Supplementary Fig. 5). The ring-like QPI patterns at negative energies indicate the presence of a hole-like band structure and its band top locates below EF ( Supplementary Fig. 6).
The similarity of QPI results in 2 UC and 3 UC FeSe indicates the band structures in these two layers are similar, which is consistent with previous APRES data 1 . The dxz and dyz bands in 2 UC FeSe also have a considerable separation and inequivalence along the kx and ky direction. Along the kx direction, the dxz and dyz bands cross with each other and form a Dirac-cone like feature below EF (the X-shaped crosses formed by the green and blue lines in Supplementary Fig. 7b) 5 . In 50 UC FeSe, the "Dirac point" is located at -10 meV measured by ARPES 5 . The hole branch of this "Dirac-cone" is the possible origin of the scattering wave vector qa (the double headed yellow arrow in Supplementary Fig. 7b). The band top value -15 meV obtained from the fitting to the scattering vector qa is also consistent with the energy of 8 the "Dirac point". The little deviation of the value probably originates from the different strength of tensile strain in 3 UC and 50 UC thin films. Therefore, the scattering wave vector qa is attributed to the inter-band scattering process between the dxz and dyz bands along the kx direction.
In order to find the origin of qb, which is perpendicular to qa, the band structure along ky direction is investigated (Supplementary Fig. 7c). According to the QPI results, qb is related to an electron-like band with its band bottom above EF. Therefore, the only candidate is the dyz band (the green band in Supplementary Fig. 7c) and scattering wave vector qb is attributed to the intra-band scattering process of dyz band along the ky direction.
We also make a more quantitative comparison between the QPI and APRES results.
Supplementary Fig. 7d and e show the dyz and dxy bands formed Dirac-cone like band structures in 2 UC and 3 UC FeSe, which are extracted from previous ARPES data 1 . The double headed yellow arrows correspond to the scattering wave vectors qa obtained from the QPI data taken at negative energies. The double headed arrows nicely link the hole branches of the "Diraccone", confirming that the scattering wave vector qa originates from the inter-band scattering between dyz and dxy bands. The scattering wave vector qb is not able to make a quantitative comparison to the dyz band along the ky direction, because APRES measurement cannot obtain the band structure above EF.
Due to the nematic phase transition of FeSe, the Se-Se lattice is rhombic and the Fe-Fe lattice is orthorhombic. In the Fe-plane, the lattice constant along a-direction is larger than that along b-direction, but cannot be detected by STM due to the limitation of real-space resolution.  Figure 11 demonstrates the estimation of stripe area ratio at each doping level in Fig. 3.
The areas where stripe ordering is absent are marked by yellow shaded regions. Stripe area ratio is estimated as As/At, where As is the stripe area and At is the total area in each image. Smectic domain walls are calculated into stripe-free regions. Therefore, the stripe area ratio of Rb-free 2 UC FeSe is not 100%.

Supplementary Note 11: Classification of good and bad superconducting gaps
The good-superconducting group contains the dI/dV spectra which show energetically symmetric (with respect to zero energy) coherence peaks and absence of in-gap states, i.e. a fully opened U-shaped gap. Supplementary Figure 13a presents three examples for good superconducting gaps. FeSe thin films usually exhibit two pairs of energetically symmetric coherence peaks (as shown in the spectra #1 and #2 in Supplementary Fig. 13a). The spectra which only have one pair of symmetric coherence peaks are also classified in good superconducting group (#3 spectra in Supplementary Fig. 13a).
The bad superconducting group includes the spectra that show asymmetric 14 superconducting gaps or contain in-gap states. If more than two pairs of peaks/features appear in the superconducting gap, they are recognized as in-gap states, and such spectra are sorted into bad superconducting group (see the three examples in Supplementary Fig. 13b).
In some cases, the coherence peaks and in-gap states are not symmetric in energies. For example, the spectrum #1 in Supplementary Fig. 13c shows a kink feature on the gap edge at negative energy; The spectrum #2 shows an in-gap state only at positive energy; The spectrum #3 show asymmetric coherence peaks at positive and negative energies. Such kind of spectra is also sorted into bad superconducting group. Examples for bad superconducting gaps which exhibit in-gap states and asymmetric features, respectively (set point, Vs = 25 mV, It = 100 pA). The black arrows denote the coherence peaks and the red arrows denote the in-gap states/features.

Supplementary Note 12: Comparison of the stripes to band structure
Supplementary Figure 14 shows a direct comparison between the wave vector of the stripe ordering and thickness dependent band structures of FeSe thin films measured by ARPES. Supplementary Fig. 14a is the FFT result in the inset of Fig. 1c. The wave vector length of the stripes (highlighted by red circles) is q0 = 0.309 Å -1 . Supplementary Fig. 14b-f are thickness dependence of dyz and dxy bands around the M point extracted from ARPES data 1 . The red arrow in Supplementary Fig. 14b denotes the wave vector of the stripe ordering, which cannot link to any two bands in k-space. Therefore, the itinerate electron picture, which attribute the 15 origin of smectic phase to the scattering between two nesting bands, is excluded.
One may find that the dyz and dxy bands have two almost parallel branches near EF, which can give rise to a possible energy-independent inter-band scattering wave vector (the wave vector qs denoted in black double headed arrows in Supplementary Fig. 14b-f). And it might lead to the energy-independent stripe ordering. However, this scattering process is inconsistent with the stripe ordering from three aspects: (1) In 2 UC FeSe thin film ( Supplementary Fig.   14b), the stripe ordering wave vector q0 is apparently shorter than the inter-band scattering wave vector qs = 0.406 Å -1 . (2) The hole-like dyz band reaches its band top within 60 meV above EF, and such inter-band scattering above this energy could not occur anymore. However, the stripe ordering can still be observed at higher energy, for example at 150 meV ( Fig. 2b and f in the main text). (3) The long-range stripe orderings in 2 UC FeSe and the short-range stripe ordering in 30 UC FeSe have the same period. In contrast, the inter-band scattering wave vector qs is thickness-dependent and shorter in thicker films.
In short, the smectic phase in 2 UC FeSe cannot be explained by an itinerate electron picture and the electronic correlation effect needs to be considered. Thickness dependence of band structure around the M point extracted from ARPES results 1 .
The blue and green lines denote the dxy and dyz bands in multilayer FeSe. The double headed red arrow denotes the wave vector of the stripe ordering q0. The double headed black arrows denote the possible inter-band scattering wave vectors (qs) between the parallel branches of dxy and dyz bands. The scattering wave vector qs is thickness-dependent.