Introduction

The origin of heavy quasiparticles with f electron character is of fundamental interest due to their participation in unconventional superconductivity. In heavy fermion metals with a valence band of metallic s, p, and d electrons, the conduction electrons can turn into heavy quasiparticles by hybridizing with the localized f electrons. Hybridization of this type, involving the formation of entangled Kondo singlets that screen the f magnetic moments, has been observed in a variety of Ce-based intermetallics1,3,7,8,9,10,11,12,13,14. Microscopically, the Kondo hybridization mechanism depends on the details of the electronic wavefunctions, and it has been argued that hybridization is driven by the shape of the f orbital wavefunctions as they are determined by the crystal electric field15,16.

The single-ion properties of Sm3+ (4f5, S = 5/2, L = 5, J = 5/2) make it an analog of Ce3+ in CeCoIn5 (4f1, S = 1/2, L = 3, J = 5/2) with the same nominal J, while exhibiting similar single-ion properties, including the crystal field environment and atomic spin-orbit coupling. The case of SmCoIn5 presents with predominantly localized f electrons, as well as an additional delocalized f component6, similar to the case of CeCoIn5. However, unlike CeCoIn5, the ground state of SmCoIn5 exhibits several antiferromagnetic (AF) ordered phases, with three phase transitions observed in specific heat at TN,1 ≈ 11 K, TN,2 ≈ 8 K, and TN,3 ≈ 6 K4,5. Magnetic ordering in rare earth intermetallics is expected to be dominated by the RKKY mechanism10,17,18. In case of an additional Kondo hybridization, the RKKY interactions are reduced by the Kondo screening cloud, which typically prohibits the formation of long-range magnetic order. Therefore, the microscopic differences between CeCoIn5 and SmCoIn5, especially changes in their most dominant magnetic interactions, may hold the key to understanding how magnetic ground states can emerge from within a system exhibiting Kondo screening.

To develop a microscopic picture of the Kondo screening environment in SmCoIn5, we conducted ARPES measurements at T = 12 K and built an electronic structure model using DFT+U calculations. Our results confirm that SmCoIn5 is dominated by the Sm3+ configuration5,6 and is highly similar to other “115” materials, where the f electrons remain mostly localized and play only a small role in the overall characteristics of the band structure. Specifically, we find that the f electrons of Sm3+ sit near a binding energy of EB ≈ 6 eV, with good agreement between ARPES and our DFT+U results, showing little hybridization of the Sm3+ states. In addition, a second band of Sm states sits above this position by a value EB + U, which happens to nearly overlap the Fermi energy EF since U ≈ 6 eV. We interpret this band loosely as an “upper Hubbard band” of Sm, because it comprises a situation where an s, p, d conduction electron tunnels onto the localized Sm3+ site, leading to an instantaneous Sm2+ configuration. To show that this picture is correct, we also conducted separate DFT calculations within a strictly Sm2+ configuration, which confirm that the Sm2+ states appear close to the Fermi energy (see supplementary information), although the details of that calculation also show that the Sm2+ configuration cannot be the ground state of SmCoIn5. A partial Sm2+ character is also in agreement with previous experiments that show a slight admixture of the Sm2+ valence state at low temperatures below T = 60 K6, similar to the intermediate valence state found in a variety of other Sm- and Yb-based materials19,20,21. Therefore, we believe the origin of the intermediate valence scenario in SmCoIn5 is captured by our DFT calculations.

Looking in more detail, we first show (Figs. 1, 2 and 3) that the band structure observed by ARPES agrees very well with the DFT+U calculations, giving us a high degree of confidence in our model. The model reveals the “hot spots” in the Brillouin zone where the Sm f states hybridize with the conduction electrons near the Fermi energy EF, and we find two spots of interest which we label γZ = (0.25, 0, 0.2) and R = 5, 0, 0.5) (Figs. 1c and 4). The γZ spot sits midway along the Γ-X cut direction with nonzero kz = 0.2, which is similar to a large hot spot position previously found in ARPES, DFT, and DMFT studies of CeCoIn51,3,12. In SmCoIn5, our model shows a smaller hot spot, which we argue may be related to the different single-ion properties of Ce3+ and Sm3+.

Fig. 1: Orbital configuration and Fermi surface of SmCoIn5.
figure 1

a Tetragonal unit cell of SmCoIn5, showing the position and orientation of the In(2) px orbital (yellow) and Co dxz/dyz orbitals (light and dark blue, respectively). b Sketch of the density of states of SmCoIn5 relative to EF. c Conventional labels within the P4/mmm Brillouin zone, highlighting two positions of interest γZ (k = [0.25, 0, 0.2]) and R (k = [0.5, 0, 0.5]). d ARPES intensity at T = 12 K, parallel to the Γ-X direction, while varying photon energy hν, demonstrating the coupling of photon energy to the out-of-plane kz direction and identifying the energies corresponding to two high-symmetry planes along the kz axis. e Fermi surface of SmCoIn5 in the Γ-X-M plane (kz = 0) using circularly polarized light with photon energy hν = 602 eV, and f Z-R-A plane (kz = π) using circularly polarized light with photon energy hν = 564 eV.

Full size image
Fig. 2: ARPES EB-kx maps showing the band structure of SmCoIn5.
figure 2

a Γ-X-Γ cut for E−EF > −7 eV. The data has been partially background-subtracted using a line-derivative method in order to enhance the visibility of the bands. b X-M-X cut. c Z-R-Z cut.

Full size image
Fig. 3: Comparison between DFT+U model calculations and ARPES within 1 eV of EF.
figure 3

a DFT calculated band structure and density of states along the Γ-X-Γ cut, and b comparison with ARPES data that has been enhanced using the background subtraction method of Fig. 2. The lines that are drawn in by hand over the DFT panel are copied directly onto the ARPES panel below, signifying good agreement, and the colored labels specify the band names discussed in the text. Note that the colors of these bands are not related to the colors indicated in the density of states plots on the right-hand side of each DFT plot. c, d X-M-X cut. e, f Z-R-Z cut.

Full size image
Fig. 4: Atomically resolved DFT calculated band structure of SmCoIn5 parallel to the Γ-X-Γ high-symmetry cut.
figure 4

a–c kz = 0.2 and d–f kz = 0.5 (Z-R-Z). Panels (a, d) show spectral weight of the In(2) site 5p orbitals, (b, e) show Co 3d, and (c, f) show Sm 4f. Each sub-panel shows our calculations for SmCoIn5 (left side) along with a nonmagnetic calculation with the 4f electrons frozen in the core (right side). In the left sub-panels, the intensity of the bands is proportional to the specific atomic orbital content of each band, and is multiplied by the scale factor written in light blue at the bottom of the plot. For example, in panel (a), the band colors for In(2) 5p have been multiplied by a factor 3.5 compared to the density of Co 3d states in panel (b). For the nonmagnetic calculations in the right sub-panels, the specific orbital content is proportional to the thickness of the red lines superimposed on the bands. The differences between the magnetic and nonmagnetic calculations is evidenced by slight changes to the position of the bands due to the presence of Sm f electrons. Two “hot spots” of interest at γZ and R (highlighted in panels (a) and (e)) show a large density of Sm 4f states, as shown in panel (c) and panel (f). In the supplementary information, we additionally show similar plots with projections into the In(1) 5s, In(1) 5p, and In(2) 5s orbitals.

Full size image

The second “hot spot” for f hybridization in SmCoIn5 is the high-symmetry R position (Figs. 3 and 4), located at the Brillouin zone boundary along both kx and kz, which was also previously identified as weakly important for CeCoIn53 but more important for Sn-doped CeCoIn513, which is in the limit of higher f electron itinerancy (higher EF). Our model calculations (Figs. 3 and 4) unambiguously show that the R point contains only Sm-f and Co-d states, indicating that a direct hybridization mechanism is possible. Moreover, orbitally resolving the model (Fig. 5) shows that the R point contains only the Co 3dxz orbital (or the equivalent 3dyz orbital at the R point rotated 90 degrees about the origin). Since the d-f band is highly flat at the R point, it is likely that a large van Hove peak of Sm/Co states suddenly appears in the density of states, pinning the Fermi energy at the bottom of this band. The appearance of this singularity exactly at EF suggests the R point plays a role in the ground state of SmCoIn5, even though we do not observe it in our experiments. More importantly, due to the direct f-d hybridization within this band, our model shows generally that magnetic interactions of the Co d states can directly influence the magnetic degree of freedom of the heavy quasiparticles, whether through interactions at the R = (0.5, 0, 0.5) wavevector or elsewhere.

Fig. 5: Orbital-resolved Co 3d orbitals from a DFT calculation of SmCoIn5 with f electrons frozen into the core.
figure 5

The orbitals hybridizing with Sm f states are found to be \({d}_{{x}^{2}-{y}^{2}}\) at Z, and dxz/dyz at R. The flat ϝ band contains dxy and \({d}_{{z}^{2}}\).

Full size image

Finally, our model shows unambiguously that a globally flat band appearing at E = − 0.7 eV in SmCoIn5, as well as in LaCoIn5 and CeCoIn51,2, consists of Co d states. Since our experiments and calculations both indicate that the band is globally flat, we suggest that the flatness of this band arises from a geometric frustration mechanism in the quasi-2D Co-In(2) layer. Specifically, we point out several square lattice models, including the checkerboard and Lieb lattices, that could apply to a subgroup of the Co d orbitals (see “Discussion”). Our results show microscopically that this flat band, where electronic and magnetic correlations are expected to be enhanced, can influence the heavy quasiparticles, either through direct exchange (such as at the R point) or via superexchange across the In(2) sites. We argue then that geometric frustration and the resulting flat d band may help explain the appearance of magnetic order within a system where the f states favor a delocalized ground state and, more broadly, that it might play an important (and until now overlooked) role in understanding magnetism and superconductivity in the “115” materials.

Results and discussion

Fermi surface and band structure of SmCoIn5

Figure 1 shows the Fermi surface of SmCoIn5 in the Γ-X-M and Z-R-A planes. ARPES intensity maps were taken as a function of the in-plane momentum (kx, ky) at the Fermi energy EF using two photon energies corresponding to the Γ and Z points (Fig. 1d). Using the standard notation used for “115”s, e.g., of ref. 2, we observe three electron-like bands at the M point (α, β, and γ). The α and β sheets are visible around the M point with circular and cloverleaf shapes, respectively. The γ sheet is visible as parallel lines around the X point, as well as in some of the features close to Γ. In a DMFT study of CeCoIn5, the γ sheet structure near X was shown to exhibit an in-plane curvature that pinches the pocket to a narrow point directly on the line between Γ and X. In SmCoIn5, we observe that the γ sheet is highly parallel, with no in-plane curvature visible until the intersection of the β and γ pockets very close to the Γ point. The absence of such a pinch point suggests that the Sm f electrons do not enter the Fermi surface here, whereas they do in CeCoIn53,12. Instead, our model shows the f electrons in SmCoIn5 only enter the Fermi surface at the γZ and R positions, as we discuss below. The absence of an in-plane hot spot might imply a more three-dimensional character to the Kondo screening in SmCoIn5.

In Fig. 2, we show ARPES intensity maps (EB-kx) showing the band structure of SmCoIn5 along high-symmetry cuts between EF and binding energy EB = 7 eV. The flat band near EB = −5.6 eV is from Sm3+, and its flatness reflects the predominantly localized nature of Sm3+, wherein the f electrons cannot hybridize strongly. The band structure near EF is accurately captured by our DFT calculations in minute detail, as shown in Fig. 3. In Fig. 3a, c, e, we zoom into the band structure near EF along the same high-symmetry cuts adopted in Fig. 2. Around the M point, shown in Fig. 3c, d, electronic-like α, β, and γ sheets are highly consistent with the band structure of LaCoIn5 and CeCoIn51,2. At the Brillouin zone boundary at Z, shown in Fig. 3e, f, the γ sheet creates an open pocket at EF near Z, while the lower δ sheet is closed at both Γ and Z and exhibits an inversion from concave down to concave up between Γ and Z. In a previous ARPES study of CeCoIn512, it was reported that depending on the surface termination, some bulk features could appear in different layers of the experimental data. In our data, we observe such an extra band in the Γ-X-Γ cut (Fig. 3b, about 0.2 eV above the δ band and forming an open pocket at Γ), which closely resembles the γ sheet that is found by our bulk DFT calculations only along the Z-R-Z cut (Fig. 3e, f). Since this is precisely the same termination state proposed for a Ce-In(1) surface in CeCoIn5 (cf. Fig. 2a of ref. 12), we attribute this feature in our data to a 2D termination state containing the Sm-In(1) crystallographic layer.

In all cuts shown in Figs. 2 and 3, we observe the presence of a very strong and flat band near E−EF = − 0.7 eV, which we label with the archaic Greek letter ϝ (digamma). A similar flat band was also observed in LaCoIn5 and CeCoIn51,2, but it was not clearly determined if this band arises from an ionization state of the localized rare earth ion, or if it is part of the p and d metallic bands. From our DFT calculations on SmCoIn5, we can confirm that the origin of this flat band is from Co 3d orbitals, as can be seen in the density of states in Fig. 3a, c, e, shown on the right side of each sub-panel. By comparing the experimental ARPES data to the calculations in Fig. 3b, d, f, we note several unusual properties of the ϝ band. Firstly, its strong intensity (on par with the Sm flat band at E−EF = −5.6 eV), overall flatness throughout the Brillouin zone, and lack of any observable hybridization points (anti-crossings) with any other metallic band, suggests that it represents a significant and relatively isolated subsystem of Co d states. Second, we note the possibility that the band is even flatter than the model calculations predict, which would indicate the presence of electronic correlations in the d band beyond the DFT+U level of approximation. The flat band may arise from geometric frustration in a sublattice of Co orbitals, as we show in the following sections, and therefore give rise to additional magnetic interactions. Moreover, since the Co d-shell is unfilled and therefore has a magnetic degree of freedom, the ϝ band can additionally couple magnetically to the entangled Kondo states, which are magnetic singlets. In general, the presence of the ϝ band shows that strong correlations in the d states can directly influence the magnetic properties of the heavy quasiparticles near EF.

Hybridization points between f states and metallic bands

To understand the details of the hybridization between Sm f electrons and the other bands, we projected our DFT calculations into the basis of different atomic shells and studied the “hot spots” near the Fermi energy that contain Sm states. In Fig. 4, we show projections along two high symmetry cuts parallel to the Γ-X-Γ direction, for kz = 0.2 (a–c) and kz = 0.5 (d–f), both with ky = 0. In panel (a), we use a yellow box to highlight the γZ hot spot position k = (0.25, 0, 0.2), which contains significant Sm weight (panel (c)). We find that the In (2) 5p orbitals dominate the shallow band extending from γZ up to kx = 0.5 (the X point), while the γZ position itself contains a mixture of In (1) and In (2) states, with the In (2) 5p states providing the largest contribution (contributions from the other orbitals In[1] 5s, In[1] 5p, and In[2] 5s are shown in the supplementary information). The existence of Sm states near the X point below EF represents another difference between CeCoIn5 and SmCoIn5.

From both the model calculations and experimental data, the γZ = (0.25, 0, 0.2) hot spot in SmCoIn5 is smaller and more isolated compared to CeCoIn5. Specifically, we find no evidence that the f states at γZ are distributed over the larger Z-centered Fermi surface pocket at EF (see Supplementary Fig. 4g; cf. ref. 3), and we find no significant f contributions in the middle of the Z-R cut (Fig. 4f; cf. refs. 37). Instead, our DFT+U model shows that the states near the Z point are dominated by the conduction electrons, and we find no experimental evidence of any deviations from the model along the Z-R-Z cut that would show the presence of f states. However, we have retained the notation γZ in this work to refer to the hybridization hot spot (0.25, 0, 0.2), and to avoid confusion with the Fermi surface γ sheet. The observation of a small γZ hot spot in SmCoIn5 could be related to different hybridization conditions between the relatively anisotropic Ce3+ ground state wavefunctions15,16, compared to the highly isotropic wavefunctions of Sm3+ 6. The lower anisotropy would also explain why a DFT+U approach results in a very realistic model for SmCoIn5 without the need to take into fully account for the effects of the small crystal field splitting of the lowest f manifold, which is beyond a mean-field approach. Our model shows that the γZ position hybridizes slightly with the 5s and 5p orbitals of both In sites, but is dominated by Sm3+ spectral weight. Since the In(2) site was also shown to have a strong effect on the ground state superconductivity in Ce-based “115” compounds7,16, we show the In(2) character in Fig. 4 while noting that there is an admixture of other orbitals at γZ. Within the In(2) atom, we find the most important contribution is from the 5px orbital (or with the equivalent 5py orbital at the position rotated 90 degrees about the origin), as is expected from the shape of the 5px orbitals.

Turning to the hot spot at the R point k = (0.5, 0, 0.5), in Fig. 4d–f we show the presence of a shallow band which appears pinned to the Fermi level and consists of Co d and Sm f states. This R-centered hot spot does not appear to be smoothly connected to the γZ hot spot, but is instead a separate feature with a distinct set of associated quasiparticles. In Supplementary Fig. 4, we show constant-energy slices plotting kz against kx, which show that two minima can be clearly distinguished from one another up to energies at least 20 meV above EF. Furthermore, the absence of In(2) states in this band (Fig. 4d) shows that the Sm and Co states exhibit magnetic coupling through a direct exchange mechanism, which is relevant not only for this locally flat band at R, but also more generally shows that direct exchange can be achieved between the rare earth and transition metal electrons in the “115” materials. Since the d+f band at R is very shallow, spanning nearly ~ 0.2 reciprocal lattice units before deviating significantly from E = EF, the R point allows a flat band of d states to directly hybridize with the heavy f quasiparticles. Moreover, the local flatness of this band suggests that the van Hove singularity in the density of states is responsible for pinning the Fermi energy to the bottom of this band at R. Therefore, although Sm f states are already present at γZ at energies below EF, we find that the hybridized flat band at R is probably touching the Fermi energy.

To further study how hot spots develop at the γZ and R hot spots, we conducted separate DFT calculations with the f electrons frozen into the core, shown on the right side of each sub-panel in Fig. 4. Qualitatively, the two calculations are highly consistent, although one important difference is at the R point, where the Co d band does not dip down low enough to reach EF in the nonmagnetic case, showing that the interaction between Sm and Co states is the driving mechanism behind creating an interaction between Sm and Co at R. Since the nonmagnetic calculations do not require the use of a magnetic supercell, we also use them to show the proper projection of the In and Co bands onto the orbitals within the crystallographic unit cell illustrated in Fig. 1a. In the case of the γZ position (see Supplementary Information), we find that the In(2) 5px orbital dominates the shallow band between kx = 0.3 and 0.5 where the Smf states occur, while exactly at γZ it is still the most important of the s, p, d orbitals but by a smaller margin. Hence, quasiparticles at γZ exhibit largely Sm-Sm interactions, and could support a Fermi surface nesting instability between neighboring γZ points—e.g., with wavevector q = (0.5, 0, 0.4)—driven by the Sm electrons. On the other hand, at the R point, shown in Fig. 5, we find that the dxz orbital completely dominates the d+f band at EF. This result shows that the hybridization between Sm and Co at R is orbitally selective, such that the interactions affecting the heavy quasiparticles there are related to the symmetry of the Co dxz/dyz orbitals. In the magnetically ordered ground state of SmCoIn54,5, zone folding and reconstruction of the electronic bands may further increase the weight of the R point below EF. Therefore, from a magnetic point of view, the orbital-selective magnetic exchange pathways in SmCoIn5 involving the dxz/dyz orbitals may be an important mechanism competing with Sm-Sm interactions, with consequences for the magnetically ordered ground state of the Sm sublattice.

Localization in the Co layer of “115” materials

In our DFT+U calculations, the density of states of the Co 3d electrons appears predominantly between EF and about EB = 3 eV, and exhibits a strong flat band (ϝ) near E = − 0.7 eV. Flat bands in general are hosts of electronic localization (e.g., refs. 22,23), and they have been previously shown to have a strong effect on magnetic ground states in transition metal systems24,25,26,27,28,29. Moveover, since our ϝ band is globally flat throughout the entire Brillouin zone, we are inclined to search for explanations in terms of a robust symmetry-protected mechanism. To this end, flat bands arising from geometric frustration have been identified in a variety of square or square-like lattices30,31,32,33,34,35,36,37, many of which can also host unconventional superconductivity, posing a deeper connection between the ϝ band and the phenomenology of the 115s.

If the ϝ band is globally flat because of geometric frustration, analysis of the 115 crystal structure can elucidate how this occurs. In the 115 structure, Co atoms occupy a square lattice. Therefore, one possibility is that the band structure of the 115s hosts a very specific hierarchy of couplings involving the square lattice, such as the checkerboard lattice proposed in30. On the other hand, as we show in Fig. 1a, the Co site is also coordinated by four In(2) sites just above the plane of Co, and four below. This Co-In(2) sublattice bears a resemblance to the Lieb lattice, a bipartite square lattice where the central site of the square is coordinated by four neighbors of another species. Phenomenologically, studies of the Hubbard model on the Lieb lattice have focused on its relationship with both magnetism32,38,39 and superconductivity31,33, including as a model for orbital selective superconductivity in the cuprates35. Since both checkerboard and Lieb lattice models also contain the ingredients for topologically nontrivial band crossings30,37,40, the ϝ band may be important to the 115s in yet another way, since such a crossing was previously proposed in a study of CeCoIn541. In general, the possibility of a specific square lattice model or a Lieb lattice model shows that SmCoIn5 can easily contain the ingredients for geometric frustration in the square-like sublattice of Co d states.

The appearance of geometric frustration does not necessarily imply that the ground state of the frustrated sublattice is the ground state of SmCoIn5. Since the ϝ band in the 115s is not situated at the Fermi energy, its quasiparticles are not thermally activated. However, since the Co d-shell is partially unfilled, the magnetic interactions within the ϝ band can nevertheless directly influence the itinerant quasiparticles. Also, as we have shown in SmCoIn5, direct hybridization can occur between heavy quasiparticles with specific Co d orbitals (in our model, with dxz/dyz at the R-point), proving that the effects of the Co flat band can be felt directly by the partly delocalized f states. Therefore, in CeCoIn5, where the delocalized f states participate in a superfluid condensate, interactions arising from the ϝ band of d states might also influence the superconducting pairing mechanism.

f-Multiplet-driven hybridization

In a DMFT study of CeCoIn5, different multiplets of the Ce f electrons were shown to selectively hybridize with the β band between X and M, and with the γ band between Γ and X3. With the much richer multiplet structure of Sm6,42, the hybridization in Sm is more complex, and may contain charge-like degrees of freedom in addition to the typical magnetic Kondo hybridization6. In addition to the spin-orbit coupling, which splits the Sm f levels into multiplets with different J, the multi-f-electron configuration in Sm also contains Coulomb repulsion between f electrons, a situation not present in the single-f-electron case of Ce. This additional term couples multiplets of different J, leading to a vastly larger subspace of f multiplet levels compared to the case of Ce, which is limited to just 14 f states with definite splittings set by the single-ion properties. Therefore, with Sm it is possible for even a tiny amount of curvature in a nominally flat band to encompass many multiplet states of the Sm ion. For example, within 0.3 eV of EF, the ground state of Ce3+ contains the 14 multiplet levels of different mJ, whereas Sm3+ exhibits nearly 200 different levels that are mixtures of different J and mJ6. Significantly, the various multiplets may have different Kondo coupling strengths that exhibit different hybridization strengths with the itinerant valence band electrons. This dramatically increases the likelihood of a Kondo-like interaction between at least one of the multiplet states with the valence band, at an energy close enough to EF to exhibit a Kondo resonance. The large subspace of f levels in a multi-f-electron system may therefore present a natural explanation for why many Sm-based materials exhibit high effective masses19, and is of particular interest in cases such as SmCoIn5 where the Sm/Co d+f band is already locally flat at the R point. It would therefore be interesting to conduct further photoemission studies on SmCoIn5 or doped SmCoIn5 with high resolution, which could possibly show if f multiplet wavefunctions other than the ground state are directly involved in the Kondo coupling, a complexity less likely to be encountered in Ce-based materials.

Magnetic exchange

Finally, we address the magnetic exchange in the Sm-Sm, Sm-Co, and Co-Co subsystems. In the tetragonal lattice structure of SmCoIn5, the Sm and Co atoms both form two-dimensional square lattices parallel to the basal ab plane, as shown in Fig. 1a, b. The Sm and Co layers alternate along the c axis with a direct stacking structure, such that a Co atom sits directly between two Sm atoms. The nearest neighboring Sm atoms in the ab plane are separated by the a lattice parameter of 4.5 Å and contain an In(1) ion in the same plane, while the In(2) ions sit between the Sm and Co layers. If the Sm and Co ions have a direct magnetic exchange pathway between them, it could also involve the In(2) sites, which could create a magnetic superexchange-like interaction. An interaction involving In(2) is plausible because the wavefunction shapes of the Co dxz/dyz orbitals lie closest to the In(2) sites, as shown in Fig. 1a, b. Depending on the details of hybridization with the In(2) px, py, or pz orbitals, the magnetic superexchange mechanism could support multiple pathways with different ferromagnetic (FM) or antiferromagnetic (AF) exchange constants, in addition to the in-plane exchange pathway coupling Sm to Sm across the In(2) px orbital. Therefore, the In(2) site may provide another degree of freedom in tuning the magnetic exchange pathway between Sm and Co, and through it, the proportion of heavy quasiparticles entering the Fermi surface at the γZ point versus those entering at R. Since it was previously shown that hybridization between the rare earth and In(2) site is an important factor in controlling superconductivity in the CeCoIn5-CeRhIn5-CeIrIn5 series15,16, the complexity of the interactions that we observe in SmCoIn5 may hold a key to understanding the origin of the ground state. Specifically, the existence of competing magnetic exchange pathways involving the rare earth (R) sites, specifically, R-In(2)-R (in plane), R-Co-R (out of plane), and R-In(2)-Co-In(2)-R, may have consequences for how magnetic and/or superconducting ground states form in the 115s.

In addition to the RKKY mechanism, the role of direct magnetic exchange and magnetic superexchange can be generally important for d and f electrons in rare earth intermetallics. For instance, in (La,Tb)Mn2Si2, it was proposed that an R-Mn-R interlayer superexchange mechanism is responsible for stabilizing AF order, while the Si atoms could act as mediators for superexchange between neighboring planes of Mn atoms along the c axis43,44. In SmCoIn5, hybridization of the Sm f and Co d orbitals, possibly involving superexchange across the In(2) sites, suggests that these exchange mechanisms have more than one way to affect the magnetic Hamiltonian, particularly along the c-axis. Such magnetic exchange pathways would therefore compete with the RKKY mechanism, and could play a destabilizing role in the magnetic ground state, particularly for interactions along the c axis. Competition between these magnetic interactions may therefore help explain the complicated magnetic phase diagram of SmCoIn55. Therefore, our observation of flat d bands interacting with states near the Fermi energy suggests a mechanism for FM and AF interactions to compete in SmCoIn5 beyond the RKKY model, and also suggests that because of this competition, fine tuning by strong correlation effects among the d states may have an outsize effect on the ultimate ground state of the material and the ordered structure of Sm local moments.

Summary

In summary, close agreement between ARPES measurements and DFT+U calculations shows that the band structure of SmCoIn5 exhibits coupling of the localized Sm f states at the Fermi level via hybridization with specific In 5p and Co 3d metallic states. Our model shows that heavy quasiparticles containing Sm f character can hybridize at multiple positions in the Brillouin zone, including a position supporting Sm-In(2)[5px] hybridization that is similar to previous results on CeCoIn5, and a second position at the zone boundary supporting Sm-Co[3dxz] hybridization (simultaneously with the equivalent orbitals 5py and 3dyz, respectively, along the other in-plane direction in the crystal). Meanwhile, the Co d orbitals exhibit a globally flat band (ϝ), which has a likely origin in geometric frustration.

The magnetic interactions arising from coupling of the itinerant quasiparticles to the Co layer, whether through direct hybridization (e.g., at R), through mean field interactions (because of a partly unfilled Co d shell), or through superexchange interactions involving the In(2) sites, all point to the importance of the Co d states in determining the ground state. Since the d shell is common to all “115” materials, electronic localization may play an overlooked role in the ground states found in this family, including unconventional superconductivity. In SmCoIn5, the delocalized nature of Sm states at the hot spots shows how magnetic interactions can go beyond the typical RKKY mechanism, and may explain why the Sm f states in SmCoIn5 exhibit both a delocalized component and localized magnetic order.

Methods

Sample growth

SmCoIn5 single crystals were grown using an indium flux method that we described previously6. Crystals were separated from the flux by centrifugation and later etched in hydrochloric acid. The SmCoIn5 samples exhibited a plate-like habit corresponding to the basal plane of the crystal.

ARPES experiments

Angle-resolved photoemission spectroscopy (ARPES) experiments were carried out at the soft-x-ray endstation45 of the ADRESS beamline46 at the Swiss Light Source, PSI, Switzerland. Single crystals were mounted using the natural basal plane on copper plates using high-strength H20E silver epoxy. Posts made of small stainless steel screws were attached onto the top surfaces of the samples with Torr-Seal epoxy. The samples were cleaved in ultrahigh vacuum of about 10−10 mbar. We used circularly polarized light with a beam spot of about 60 × 100 μm on the sample. We determined the coupling of beam energy to the out-of-plane kz direction by scanning the beam energy over hν = 480 to 700 eV, shown in Fig. 1d. We identified hν = 602 eV as the energy where the in-plane iso-EB maps correspond to the Γ-X-M plane, and hν = 564 eV as the energy corresponding to the Z-R-A plane. The use of soft-x-ray photon energies results in large photoelectron mean free paths, and the latter translates, by the Heisenberg relation, to sharp definition of the out-of-plane momentum kz47,48. Maps as a function of two-dimensional in-plane momentum (kx, ky) were then collected at these energies to observe the Fermi surface and band structure, where we found broad similarity to published results of CeCoIn5 that use DFT1,12 and dynamical mean-field theory3.

DFT calculations

All the density functional theory (DFT) calculations were carried out using the Quantum ESPRESSO package49,50,51. Exchange and correlation effects were modeled using the PBEsol functional52, augmented by a Hubbard U term to better describe the physics of the localized Sm 4f electrons. The scalar-relativistic pseudopotentials are taken from the SSSP library53. To sample the Brillouin zone, we used a 7 × 7 × 7 Monkhorst-Pack grid, a 10 × 10 × 10 grid was used for the calculation of the projected density of states. The wavefunctions (charge densities and potentials) were expanded using a kinetic energy cutoff of 70 Ry (560 Ry). We used the experimental lattice constants and internal atomic positions54,55 throughout this work. The values of U used in this work (U = 6.18 eV and U = 6.26 eV for Sm3+ and Sm2+ calculations, respectively) were computed fully ab initio by using the linear response approach56 as implemented in refs. 57,58. In order to “nudge” the self-consistent-field calculations to converge to the Sm3+ solution, we prepared the systems in initial configurations that have 5 electrons in the Sm f shell. Some of these initial configurations eventually converge to a self-consistent solution that exhibit the desired 3+ oxidation state for the Sm atoms. At the level of DFT+U this solution is always metastable, i.e., its energy is slightly higher then that of the Sm2+ (6 electrons in the Sm f shell) solution. However, it is the one that best reproduces the ARPES data. Also, we did not use any +U corrections for the Co 3d states as this deteriorates the agreement with the ARPES data. Finally, in the magnetic calculations including the f electrons, the unfolding of the bands in the 7-atom Brillouin zone was performed using the unfold-x code (https://bitbucket.org/bonfus/unfold-x/src/master/)59.