Introduction

The nematic phase is a state characterized by broken rotational symmetry but intact translational symmetry, which has received renewed attention due to the possibility that it may provide important insight into unconventional superconductivity1. The nematic phase exists near the superconducting region in the phase diagrams for both cuprates2 and iron-based superconductors (IBSs)3, suggesting that it may be related to superconductivity. Identification of the origin of the nematic phase is considered one of the most important goals in research on unconventional superconductivity, especially for IBSs, in which the nematic phase was first reported.

Recently, the nematic phase in FeSe has attracted attention due to its unusual properties, which are distinct from those of the nematic phase of the pnictides. The most peculiar aspect is the absence of long-range magnetic order, which always coexists with orbital order in the pnictide nematic phase4,5. Furthermore, it was reported that the resistivity anisotropy of FeSe has the opposite sign to that of iron pnictides6, which is smaller along the longer \(a\)-axis than the shorter \(b\)-axis for iron pnictides, but the opposite for FeSe. These two observations suggested that the nematic phase in FeSe may be different from that of iron pnictides. Therefore, understanding the FeSe nematic phase may provide insight into the nematic phase in IBSs. More importantly, considering the recent discovery of orbital-selective correlations and superconductivity in FeSe in scanning tunneling microscopy (STM) studies7,8, the mechanism of superconductivity could be addressed by understanding the origin of the nematic phase, which is widely believed to induce orbital selectivity.

In this regard, a number of angle-resolved photoemission spectroscopy (ARPES) experiments have been performed to investigate the electronic structure of FeSe9,10,11,12,13,14,15,16. However, interpretations varied among experiments, which caused more controversy regarding the origin of the nematic phase in FeSe. Various mechanisms were proposed to explain the origin of the nematic phase based on ARPES results, such as ferro-orbital order9, d-wave orbital order10, unidirectional nematic bond order11, and sign reversal order12,15. These controversies were likely due to the lack of comprehensive and accurate electronic structure information in early studies, where such information is needed to determine the origin of the nematic phase in FeSe.

Here, we present the results of electronic structure studies using ARPES and X-ray linear dichroism (XLD) on fully detwinned FeSe in its nematic phase. Nearly full detwinning was obtained using a piezo stack-based strain device; this is the first report of the use of such a device in ARPES and XLD experiments. With the new strain device, difficulties encountered when using the mechanical strain method, such as uncertainty in the strain direction due to accidental detwinning caused by anisotropic shrinkage of epoxy17, are overcome by controlling the strain. As a result, we could investigate band dispersion with full orbital characters in detail using ARPES, as well as the orbital occupation imbalance between \({d}_{{xz}} \) and \({d}_{{yz}}\) orbitals (by XLD). Our ARPES results show orbital-dependent band shift and hybridization, leading to only one electron pocket at the Brillouin zone (BZ) corner, which may explain why there is no long-range magnetism. Furthermore, XLD reveals an unexpected reversal of the occupation imbalance between the \({d}_{{xz}}\) and \({d}_{{yz}}\) orbital (\({n}_{{xz}}\,<\,{n}_{{yz}}\)), which explains the opposite resistivity anisotropy. These results resolve the controversies regarding the nematic phase of FeSe and may shed light on the instability driving the nematic phase in IBSs.

Results

Electronic structures of twinned and detwinned FeSe

A controversial issue in the study of the FeSe electronic structure is the Fermi surface topology at the BZ corner. A previous study claimed that the Fermi surfaces at the BZ corner consist of elliptical and large circular pockets with \({d}_{{yz}}\) and \({d}_{{xy}}\) orbital characters, respectively12. Another study asserted that each BZ corner has two elliptical pockets that are elongated along the \({k}_{{{x}}}\)-directions and \({k}_{{{y}}}\)-directions with orbital characters of \({d}_{{xy}}\) and \({d}_{{yz}}\) (\({d}_{{xz}}\)), respectively, but none was observed along the \({k}_{{{y}}}\)-direction17. Therefore, we first focus on determining the overall Fermi surface topology in the single domain. To obatain single domain, tensile strain is transmitted to the sample glued to the piezo device (Fig. 1a and Supplementary Note 1). Polarized microscopic images (Fig. 1b) reveal detwinning of the sample. In the twinned sample image, blue and red lines are seen due to differences in the reflectance of domains. When voltage is applied to the device, most of the blue lines disappear (detwin), implying that the sample is nearly detwinned.

Fig. 1: Angle-resolved photoemission spectroscopy (ARPES) measurements on twinned and detwinned FeSe.
figure 1

a Schematic representation of the piezo sample holder for strain experiments. b Optical images of twinned and detwinned FeSe single crystals, taken with a polarized microscope. Both images were acquired at a temperature below structural transition temperature (\({T}_{{{S}}}\)). c Corresponding Fermi surface maps. \({M}_{{{X}}}\) and \({M}_{{{Y}}}\) are defined by the strain direction of the piezo. Schematic Fermi surface maps in the d 2-Fe Brillouin zone (BZ) and e 1-Fe BZ.

Full size image

Figure 1c shows clear differences between Fermi surface maps from twin and single-domain samples that may provide key information for understanding this issue. For the twin domain case (left), two perpendicularly crossing elliptical pockets are observed at the BZ corner points (\({M}_{{{X}}}\) and \({M}_{{{Y}}}\)). However, Fermi surfaces from a single-domain sample (right) consist only of a single elliptical pocket at each BZ corner. If an additional pocket exists, it should be visible in Fig. 1c because the experimental geometry (i.e., light polarization and plane of the light incidence, which was chosen to be diagonal to the orthorhombic axis) allows observation of all orbitals (\({d}_{{xz}}\), \({d}_{{yz}}\), and \({d}_{{xy}}\)). Based on this experimental observation, as well as the temperature-dependent full-band dispersion (discussed below), we conclude that there is only one elliptical pocket at each BZ corner in the nematic state, as illustrated in Fig. 1d. This implies that one of the two pockets in the normal state should disappear across the nematic phase transition, i.e., the pocket at the Y point in the 1-Fe BZ scheme shown in Fig. 1e should disappear, while the pocket at the X point remains. Note that all of the ARPES data are shown in the 2-Fe BZ scheme, whereas the 1-Fe BZ scheme is used for orbital character analysis. To distinguish between the two cases, the BZ corners in the 2-Fe BZ scheme are labeled \({M}_{{{X}}}\) and \({M}_{{{Y}}}\), and the corresponding zone edges in the 1-Fe BZ scheme are labeled X (Y) (see Fig. 1d, e).

Dispersion and orbital characters of bands

To understand how and why the pocket at the Y point in 1-Fe BZ disappears, band dispersions and orbital characters should be fully identified. Figure 2a, b show the overall band dispersions along the \(\Gamma\)-\({M}_{{{X}}}\) (along \({k}_{{{x}}}\)) and \(\Gamma\)-\({M}_{{{Y}}}\) (along \({k}_{{{y}}}\)) directions from a single-domain sample in the normal and nematic states, respectively. Here, the \(x\)-direction is the direction of the tensile strain, as shown in Fig. 2c. Starting from isotropic normal-state band dispersions, anisotropic band dispersions develop in the nematic phase. Especially, the band dispersions around the BZ corner are renormalized and become those of two merged Dirac cones (possibly with small gaps at band crossing points) (Fig. 2b). The \(\Gamma\)-\({M}_{{{X}}}\) high-symmetry cut shows a large electron band and two split hole bands. Meanwhile, in the cut along the \(\Gamma\)-\({M}_{{{Y}}}\), a tiny electron band near \({E}_{{{F}}}\), and two nearby hole bands with a higher binding energy, are observed, consistent with previous reports17.

Fig. 2: Dispersions and orbital characters of bands obtained via polarization dependent angle-resolved photoemission spectroscopy (ARPES).
figure 2

a, b Three-dimensional schematic of band dispersions and high symmetry cuts along the \({M}_{{{Y}}}\) - Z - \({M}_{{{X}}}\) direction above and below structural transition temperature (\({T}_{{{S}}}\)). All data were obtained with \(s\)-polarized 56 eV light. c Schematic Fermi surfaces with orbital characters in the nematic state in the 2-Fe Brillouin zone (BZ) scheme. \({d}_{{xz}}\), \({d}_{{yz}}\), and \({d}_{{xy}}\) bands are indicated by the red, green, and blue lines, respectively. Tensile strain is applied along the \(x\)-direction. d Fermi surface map around \({M}_{{{X}}}\). e High symmetry cuts along the \({k}_{{{x}}}\)- and \({k}_{{{y}}}\)-directions near \({M}_{{{X}}}\). The cut directions are shown in d as dotted lines. The overlaid dashed lines are band dispersions with orbital characters. The orbital characters and band type (electron or hole) are indicated at the bottom of the figure. f High symmetry cut along the \({M}_{{{X}}}\) - Z - \({M}_{{{X}}}\) direction near the BZ center. g–j Similar measurements as c–f but with the sample rotated by 90 degree (light polarization along \(a\)-, i.e., in the \(x\)-direction). Fermi surface map and high symmetry cuts are shown for \({M}_{{{Y}}}\). k, l Schematic Fermi surfaces and band dispersions with orbital characters above and below \({T}_{{{S}}}\), respectively, in the 2-Fe BZ.

Full size image

Polarization-dependent ARPES was used to precisely characterize the orbital characters (Fig. 2c–j). Here, the effect of the glide-mirror symmetry arising from nonequivalent Se positions is considered in the polarization dependence analysis (see Supplementary Notes 2, 3). The visible orbital characters with band types (electron or hole) determined based on polarization dependence are indicated in Fig. 2e, i. First, the tiny electron band seen in cut 1 in Fig. 2i must have a \({d}_{{yz}}\) orbital character, as only the \({d}_{{yz}}\) electron band can be seen in this geometry. The hole bands in cut 1 located at higher binding energies have \({d}_{{xz}}\) and \({d}_{{xy}}\) characters. The large electron band seen in cut 2 in Fig. 2i must arise from either \({d}_{{xy}}\) or \({d}_{{xz}}\) orbitals. However, the fact that this band is not observed in cut 1 near the \({M}_{{{X}}}\) point (Fig. 2e) indicates that it should be of \({d}_{{xy}}\) character. Finally, the hole band near \({E}_{{{F}}}\) in cut 1 of Fig. 2e is assigned to the \({d}_{{yz}}\) orbital, as its top position is the same as the bottom position of the tiny \({d}_{{yz}}\) electron band in cut 1 of Fig. 2i. With regard to the orbital characters of hole bands around the BZ center, our polarization-dependent results are consistent with previous reports (Fig. 2f, j)12.

The orbital characters of the bands in the 2-Fe BZ are shown schematically in Fig. 2k, l. Two unique features of the FeSe electronic structure in the nematic phase can be seen. First, there is a reversal behavior in the relative energy positions of the \({d}_{{xz}}\) and \({d}_{{yz}}\) hole bands, as noted previously12. As shown in Fig. 2l, the \({d}_{{xz}}\) hole band (red, at \({M}_{{{Y}}}\)) is located below the \({d}_{{yz}}\) hole band (green, at \({M}_{{{X}}}\)) at BZ corners, while the \({d}_{{xz}}\) hole band is situated above the \({d}_{{yz}}\) band at the BZ center. Another, more important feature is the reduced number of electron bands at \({M}_{{{X}}}\) and \({M}_{{{Y}}}\) in the nematic phase. In the normal state, there are two electron bands at both \({M}_{{{X}}}\) and \({M}_{{{Y}}}\) points–the common \({d}_{{xy}}\) electron band and the \({d}_{{xz}}\) (\({d}_{{yz}}\)) electron band at \({M}_{{{X}}}\) (\({M}_{{{Y}}}\)). In the nematic phase, only one electron band crosses \({E}_{{{F}}}\) at each BZ zone corner point (Fig. 2l). Two other electron bands, \({d}_{{xz}}\) and \({d}_{{xy}}\) at \({M}_{{{X}}}\) and \({M}_{{{Y}}}\), respectively, were not observed. Therefore, it is reasonable to speculate that these two bands are pushed above \({E}_{{{F}}}\) across the nematic phase transition.

Temperature evolution of the electronic structure

The temperature evolution of the electronic structure across the nematic phase transition clearly confirms the speculation. Figure 3a shows the temperature dependence of band dispersion along the \({k}_{{{x}}}\)-direction at \({M}_{{{Y}}}\), with momentum distribution curves (MDCs) acquired 5 meV above \({E}_{{{F}}}\). In the normal state, the \({d}_{{xz}}\) electron band crosses \({E}_{{{F}}}\) and peaks in the MDC have finite momentum (indicated by red arrows). As the temperature decreases, peaks for the \({d}_{{xz}}\) electron band shift toward \({M}_{{{Y}}}\) and disappear at low temperatures, consistent with the view that the \({d}_{{xz}}\) electron band shifts upward and is finally pushed above \({E}_{{{F}}}\). In fact, a recent ARPES study on FeSe indicated the occurrence of such a band shift below about 30 K16. Meanwhile, the \({d}_{{xz}}\) (\({d}_{{yz}}\)) hole band at \({M}_{{{X}}}\) (\({M}_{{{Y}}}\)) shifts downward (upward) as the temperature decreases, as seen in the temperature-dependent data shown in Fig. 3c. The splitting at low temperature occurs at about 60 meV and persists above the nematic phase transition temperature, possibly due to the applied strain (Fig. 3d). It is noteworthy that the two hole bands do not cross \({E}_{{{F}}}\), and remain below \({E}_{{{F}}}\) after the shift.

Fig. 3: Temperature evolution of the electronic structure.
figure 3

a Temperature dependent angle-resolved photoemission spectroscopy (ARPES) data along the \({k}_{{{x}}}\)-direction near the \({M}_{{{Y}}}\) point. Momentum distribution curves (MDCs) at 5 meV above \({E}_{{{F}}}\) are overlaid on the figure. The contributions of \({d}_{{xy}}\) and \({d}_{{xz}}\) are shown as blue and red curves, respectively. Peak positions of the \({d}_{{xz}}\) band are indicated by red arrows. b Schematic illustration of the experimental geometry. c Temperature dependent energy distribution curves (EDCs) at \({M}_{{{X}}}\) (along \({k}_{{{y}}}\)-direction, cut 1) and \({M}_{{{Y}}}\) points (along \({k}_{{{x}}}\)-direction, cut 2), showing downward and upward shifts of the \({d}_{{xz}}\) and \({d}_{{yz}}\) band upon cooling, respectively. d The peak position of the \({d}_{{xz}}\) (red square) and \({d}_{{yz}}\) (green square) band as a function of temperature. e Schematic illustration of the band reconstruction at the Brillouin zone (BZ) corner across the nematic phase transition in 1-Fe BZ scheme.

Full size image

Next, we discuss how the electron pocket at Y in 1-Fe BZ shifts upward and disappears, i.e., how the number of electron bands is reduced across the nematic phase transition. We argue that the observed evolution in the electronic structure can be explained by the orbital-dependent band shift (or splitting) and hybridization. For ease of understanding, an 1-Fe BZ scheme with fewer bands is provided in Fig. 3e. The left and right halves of each panel represent the band dispersions along the \({k}_{{{x}}}\)-directions and \({k}_{{{y}}}\)-directions, respectively. Starting from symmetric bands in the normal state (left panels), the \({d}_{{yz}}\) and \({d}_{{xz}}\) bands shift upward and downward, respectively, as the temperature decreases below structural transition temperature (\({T}_{{{S}}}\)) (middle panels); we term this phenomenon nematic band shift. Then, orbital-selective hybridization comes in. For the \({d}_{{yz}}\) band, there is only weak mixing with \({d}_{{xy}}\) and thus the overall dispersion remains unchanged. On the other hand, the \({d}_{{xz}}\) band hybridizes strongly with the \({d}_{{xy}}\) band, thereby pushing the hole band down below \({E}_{{{F}}}\) while also raising the \({d}_{{xz}}\) and \({d}_{{xy}}\) electron bands above \({E}_{{{F}}}\) (right panels). This results in the disappearance of the electron pocket at Y in the 1-Fe BZ scheme.

Reversed orbital occupation imbalance

A surprising implication of the above interpretation is that the \({d}_{{xz}}\) orbital should be less occupied than \({d}_{{yz}}\), which is the opposite of the case with iron pnictide18 and is also in contrast to the prediction regarding the ferro-orbital order19. At the BZ center, the energy position of the \({d}_{{xz}}\) band is higher than that of \({d}_{{yz}}\), and thus the \({d}_{{xz}}\) state is less occupied. The disappearance of the \({d}_{{xz}}\) electron band at the BZ corner also leads to less occupied \({d}_{{xz}}\) orbitals (the \({d}_{{xz}}\) and \({d}_{{yz}}\) hole bands at the BZ corner are irrelevant, as both are fully occupied). To directly obtain information on such anomalous orbital occupancy, we performed XLD measurements on detwinned samples. XLD, as local probe for orbital-selective density of states, can directly confirm any imbalance in the orbital occupancy18,20.

 Figure 4a shows the experimental geometry with two light polarizations, parallel and perpendicular to the direction of strain. The Fe \(L\)-edge absorption spectra from detwinned FeSe, acquired with the two light polarizations at 10 K, are plotted in Fig. 4b. With the given experimental geometry, the linear dichroism (XLD) shown by the black solid line in Fig. 4b should reflect the imbalance in \({d}_{{xz}}\) and \({d}_{{yz}}\) orbital occupations. One complication is that not only the orbital occupation imbalance, but also the orthorhombic structural distortion, is known to contribute to the XLD signal20. It was reported previously that these two contributions can be separated by considering their distinct temperature dependence18. Close inspection of the temperature-dependent XLD data (704–708 eV) in Fig. 4c reveals that the XLD signal starts to appear below \({T}_{{{S}}}\), and increases monotonically as the temperature decreases down to 10 K. This monotonic increase of the XLD signal can be visualized more clearly by plotting the integrated XLD signal (defined in the caption of Fig. 4) as a function of temperature (Fig. 4d). If the XLD signal contains only the structure contribution, it should saturate at about 40 K because the contribution should follow the orthorhombicity of the crystal (see the overlaid diffraction data)21. That is, the increase in the XLD signal below 40 K is a strong indicator of an orbital contribution. An important point to note is that the orbital contribution is positive, which indicates that the \({d}_{{xz}}\) orbital is less occupied (\({n}_{{xz}}\,<\,{n}_{{yz}}\)), contrary to the simple ferro-orbital order scenario (\({n}_{{xz}}\,> \,{n}_{{yz}}\)) in which the orbital contribution in the XLD signal is negative18,20.

Fig. 4: Observation of orbital occupancy by X-ray linear dichroism (XLD).
figure 4

a Schematic illustration of the experimental geometry. The beam is incident normal to the sample surface. The \(a\)-axis and \(b\)-axis are determined by the strain direction of the piezo. b Fe \(L\)-edge absorption spectra from detwinned FeSe acquired at 10 K with E\(\parallel\)a (blue, inverted triangle) and E\(\parallel\)b (red, triangle) polarizations. The XLD (black curve) represents the difference (E\(\parallel\)a - E\(\parallel\)b). The XLD spectrum is multiplied by 10 for better visualization. c XLD spectra at various temperatures. Overlaid black solid lines are fitting results with two Gaussian functions. d Temperature dependence of the XLD area (704–708 eV; left, blue circles) and orthorhombicity (right, red square). The XLD area is calculated by integrating the absolute value of the fit curves in c. The error bars are obtained from the standard fitting error. The orthorhombicity of FeSe is derived from diffraction measurements21.

Full size image

Discussion

Our interpretation on electronic structure evolution across the nematic phase transition in FeSe was confirmed by observation of a reversed orbital occupation imbalance; the implications are discussed presently. First, the sign reversal in hole band splitting, and the reduced number of electron bands, do not support a unidirectional nematic bond order, which requires the absence of hole band splitting11. On the other hand, d-wave form splitting10 and sign reversal order12 are partially consistent with our observations when hole band splitting is taken into consideration. The inconsistency arises because the evolution of electron band dispersions, and the hybridization between the \({d}_{{xz}}\) and \({d}_{{xy}}\) bands, were not considered in these scenarios. Not only the evolution of the \({d}_{{xz}}\) and \({d}_{{yz}}\) hole bands, but also the evolution of electron bands and hybridization between the \({d}_{{xz}}\) and \({d}_{{xy}}\) bands, should be taken into consideration for a full understanding of the nematic phase.

By characterizing the electronic structure, the unique properties of the FeSe nematic phase, i.e., the absence of magnetism and opposite resistivity anisotropy in comparison to pnictides, can also be understood. The absence of magnetism can be explained for both weak and strong coupling pictures; regarding the former22,23,24, Fermi surface nesting, which is the source of the magnetism, is suppressed due to the opposite orbital characters of two pockets at the BZ center (mostly \({d}_{{xz}}\), with a small contribution from \({d}_{{yz}}\)) and corner (mostly \({d}_{{yz}}\)). Regarding strong coupling picture25,26, the absence of magnetism could be explained within the context of an orbital weight redistribution scheme. As the magnetic moment is mainly derived from the \({d}_{{xy}}\) orbital27,28, the reduced \({d}_{{xy}}\) orbital weight due to strong orbital-dependent hybridization should weaken the overall magnetic instability29. Meanwhile, the opposite resistivity anisotropy of FeSe in comparison to pnictides can be explained by reversed orbital occupation imbalance. The preferred resistivity direction can be determined on the understanding that the \({d}_{{xz}}\) and \({d}_{{yz}}\) orbitals at \({E}_{{{F}}}\) favor conduction along the \(a\)-axis and \(b\)-axis, respectively30,31. Therefore, the resistivity along the \(a\)-axis is larger than that along the \(b\)-axis in FeSe, which is the opposite of the case of pnictide (because the \({d}_{{yz}}\) orbital is more occupied than \({d}_{{xz}}\)).

Our work provides a new perspective on the nature of the nematic phase. It has been suggested that the nematic band shift and occupation imbalance between \({d}_{{xz}}\) and \({d}_{{yz}}\) orbitals are equivalent, i.e., they are a single phenomenon associated with the ferro-orbital order, arising from symmetry breaking at the atomic level. However, our observation of a reversed orbital occupation imbalance, i.e., an occupation imbalance opposite to the nematic band shift, clearly shows that they are not equivalent. Furthermore, their opposite behaviors strongly suggest that the nematic band shift is neither a manifestation of, nor is generated by, the occupation imbalance; rather, the occupation imbalance should be a byproduct of the nematic band shift, as well as the hybridization between \({d}_{{xz}}\) and \({d}_{{xy}}\) bands discussed above. In short, our results strongly indicate that the ferro-orbital order does not represent the true nature of the nematic phase in IBSs. Instead, the nematic band shift, which is commonly observed in iron pnictide systems and FeSe32,33, appears to be the key factor in the nematic phase. Several mechanisms were proposed for the nematic band shift, including bond orbital order34, Pomeranchuk instability35,36, orbital-dependent correlation effect8, and orbital-selective spin fluctuations14. Our findings will be valuable for understanding the origin of the nematic phase in IBSs.

Methods

Experiments

X-ray absorption spectroscopy (XAS) experiments were performed at the Beamline 2A of the Pohang Light Source and spectra were recorded in total electron yield (TEY) mode. All spectra were normalized by the incident photon flux intensity, as measured using a gold mesh, and calibrated with respect to the \({L}_{3}\) absorption peak of the Fe\({}_{2}\)O\({}_{3}\) alloy located in front of the analysis chamber. ARPES measurements were carried out at Beamline 4.0.3 of the Advanced Light Source. Linearly polarized light with photon energy of 56 eV was used. All crystals were cleaved in situ at a pressure better than \(9\,\times\,1{0}^{-10}\) Torr for XAS and \(4\,\times\,1{0}^{-11}\) Torr for ARPES experiments. Samples were detwinned using uniaxial strain, which was applied along the tetragonal [110] direction. The high-quality data shown in the figures were obtained from detwinned samples.