Introduction

The kagome-net, a pattern of corner-sharing triangular plaquettes, has been a fundamental model platform for exotic states of matter, including quantum spin liquids and topological band structures1,2,3. Recently, the transition metal-based kagome metals4,5,6,7,8,9,10,11,12,13 are emerging as a new class of topological quantum materials to explore the interplay between frustrated lattice geometry, nontrivial band topology, symmetry-breaking order, and many-body interaction. A kagome lattice tight-binding model generically features a Dirac crossing and a flat-band, which are the fundamental sources of nontrivial topology and strong correlation. Such topological fermionic structures arising from the correlated 3d electrons in the kagome lattice have been widely reported in several quantum materials4,5,6,7,8,9,10,11,12,13, including Mn3Sn, Fe3Sn2, Co3Sn2S2, TbMn6Sn6, FeSn, and CoSn. In parallel, the band dispersion of bosonic excitations on a kagome lattice also features Dirac crossings and flat-bands, as demonstrated in photonic crystals14,15. A question naturally arising when studying kagome lattice electrons is the possibility of a nontrivial many-body interplay between the bosonic kagome lattice phonons and the fermionic quasiparticles.

Such fermion–boson interactions often manifest as a perturbation of the bare band structures at very low-energy scales. However, most kagome lattice materials exhibit complicated multi-bands near the Fermi energy4,5,6,7,8,9,10,11,12, severely challenging the clear identification of the many-body effect by spectroscopic methods. Among all known kagome lattice materials, the paramagnetic CoSn is recently highlighted as an outstanding kagome topological metal with much cleaner bands and simpler Fermi surface13, making it an ideal platform to search for the geometrical frustrated fermion–boson interaction. Here we report the discovery of fermion–boson many-body interplay in kagome lattice of CoSn, which arises from the coupling of the phonon flat-band with the kagome electrons, utilizing the low-temperature (T = 4.2 K), high energy-resolution (ΔE < 0.3 meV), atomic layer-resolved scanning tunneling microscopy.

Results

Atomic-scale visualization of kagome lattice

CoSn has a hexagonal structure (space group P6/mmm) with lattice constants13,16a = 5.3 Å and c = 4.4 Å. It consists of a Co3Sn kagome layer and an Sn2 honeycomb layer (Fig. 1a) with alternating stacking. The side-plane atomically resolved map of the crystal measured by transmission electron microscopy (Fig. 1b) directly demonstrates this atomic stacking sequence along the c axis. Upon cryogenic cleaving, the surface yields either the Co3Sn- or Sn2-terminated atomic layer. The lower panel in Fig. 1c shows a highly rare topographic image of the cleaving surface that contains both terminations. It consists of a Co3Sn surface and islands of Sn2 layer sitting on top. The simultaneously obtained differential conductance map directly reveals their difference in the electronic structure shown in the upper panel of Fig. 1c, where the Co3Sn surface has a higher density of states at the bias voltage of 100 mV. From this map, we also find that the Co3Sn surface has detectable impurities induced quasi-particle interferences, which is the basis for our further space-momentum investigation. Scanning the Sn2 (Fig. 1d) and Co3Sn (Fig. 1e) surfaces with higher magnification, we directly reveal their honeycomb and kagome lattice symmetry, respectively. Remarkably, our topographic image was able to resolve the fine corner-sharing triangle structure of the Co kagome lattice and the Sn atom in the kagome center (Fig. 1e). Such ultra-high atomic resolution has not been achieved in the previous scanning tunneling studies of kagome lattice materials.

Bosonic mode coupling from kagome electrons

With the lattice structure of CoSn visualized at the atomic scale, we now study their electronic structure by measuring the differential conductance as shown in Fig. 2a. According to the first-principle calculations and photoemission measurement, the Fermi surface is dominated by a fairly simple electron-like band13. Strikingly, we find pronounced low-energy modulations for the spectra taken on the Co3Sn layer, while this feature is absent on the Sn2 layer as shown in Fig. 2b. The observed peak-dip-hump modulation can indicate the strong bosonic mode coupling with the coherent state at the Fermi level, as similar spectroscopic features have been found in many strong coupling superconductors, including lead17, cuprates18, and iron-pnictides19. Such a bosonic mode often arises from phonons or spin resonances. As this material is nonmagnetic and has no detectable magnetic field dependence of tunneling spectra up to 8T (Fig. 2c), the bosonic mode is more likely to arise from the coupling to phonons. Moreover, the electronic coupling to the bosonic mode can be described within the Eliashberg theory20,21 with α2F(ω) where α is the coupling matrix element and F(ω) is the bosonic density of states, and this can be studied in the tunneling experiments. α2F(ω) is intimately related to the second differentiation of the tunneling spectra17,18,19,20,21,22. Analyzing the measured tunneling spectra (Fig. 2d, e), we find a Gaussian-like state in the second derivative centered at EM = 15 meV with a full width at half maximum of 9 meV (Fig. 2e), and identify it as a candidate signal related to the Eliashberg function (Fig. 2e, blue curves). The other peak features at lower energies of the second differential spectra can be expected from a coherent state at the Fermi energy as shown by the simulation curve in Fig. 2d, e.

Many-body kink in the kagome dispersion

To gain deeper insight into the bosonic mode coupling on the kagome lattice, we perform systematic spectroscopic imaging on a large Co3Sn area with only a few Sn2 adatoms (Fig. 3a). By taking the Fourier transform of the differential conductance map (Fig. 3b), we obtain the quasi-particle interference (QPI) data. The QPI data at 0 meV (Fig. 3c) shows a clear ring-like signal, consistent with the intraband scattering of the dominant electron-like Fermi surface13 centered at Γ. Thus, the low-energy QPI dispersion reflects the behavior of the electron-like band crossing the Fermi surface (Q = 2k). Analyzing the QPI dispersion along Γ-M direction in Fig. 3d, we observe a pronounced double kink feature, different from its bare band dispersion calculated by the first-principles (dashed line). The spectroscopic kink feature has been identified as a fingerprint of the bosonic mode coupling23,24,25,26,27,28,29 and indicates a giant mode coupling strength. The energy of the QPI kink is around ±15 meV, well consistent with the mode energy EM in the second differential conductance signal. The coupling strength can be estimated from Fermi velocity renormalization λ = vf0/vf − 1 = 1.8 with s.d. error bar of 0.3, where vf0 and vf are the Fermi velocities of the bare QPI band and renormalized QPI band, respectively. We also explored the QPI on the Sn2 honeycomb lattice but did not find any clear kink. Hence the unique kagome lattice resolving capability combined with low temperature and high energy-resolution of our advanced spectroscopic technique can be the key for the kink discovery in this material.

Discussion

The pronounced kink signal from the kagome lattice allows us to analyze the electron many-body self-energy Σ(ω) and further quantify the Eliashberg function α2F(ω). Σ(ω) is intimately related to α2F(ω), the Fermi–Dirac distribution f(E) and the Bose–Einstein distribution n(ω) (see Eq. (4) in “Methods” section). The real part of the self-energy Re(Σ) is related to the energy difference between the observed kink dispersion and the bare QPI dispersion, while the imaginary part of the self-energy Im(Σ) is related to the electron band broadening, which is inversely proportional to the QPI intensity. Re(Σ) and Im(Σ) are tied to one another through the Kramers–Kronig relation, and we can more accurately acquire Re(Σ) from the QPI data in Fig. 3e. We take the shape of α2F(ω) in reference to the second differential conductance (Fig. 2), and tune the coupling strength $$\lambda = {\int} {\alpha ^2} F\left( \omega \right){\mathrm{/}}\omega \,{\mathrm{d}}\omega$$ and calculate the real part of the self-energy Re(Σ). We find that with λ = 1.9 with s.d. error bar of 0.3, the calculated Re(Σ) can account for that determined by the experiment (Fig. 3e), which agrees with the estimated λ from Fermi velocity renormalization. We further simulate the QPI signal with this α2F(ω) under the Green function formalism in Fig. 3f, which also shows reasonable consistency with the experimental data both in dispersion and intensity evolution, providing key support to our many-body analysis of the kagome fermion–boson interaction.

Having characterized the many-body fermion–boson interaction in the kagome lattice, we perform first-principles calculations of the phonon band to understand the nature of the bosonic mode. Firstly, the calculated phonon density of states exhibits a pronounced peak at 15 meV (Fig. 4a), coincide with the mode energy in experiments. Secondly, this phonon mode mainly arises from the Co3Sn kagome layer, consistent with the experiments. Thirdly, the calculation provides momentum space insight into the origin of this peak, in that it arises from a flat-band in phonon momentum space as shown in Fig. 4b. Lastly, through the atomic displacement resolved calculation, we identify that the flat-band phonon is mainly associated with the Co kagome lattice vibrations confined to the line connecting the centers of two neighboring triangles (Fig. 4b, inset).

In light of the first-principle calculation, we build a kagome lattice vibration model to elucidate the striking physics. The essential momentum features of the flat-band can be well captured by this model (Fig. 4c), with the flat-band toughing a parabolic band bottom. This model is highly analytical and offers a heuristic understanding of the non-propagating nature of the kagome flat-band phonon mode. We find that the collective lattice displacement shown in the inset of Fig. 4c, a deformation from a hexagonal ring (inner six atoms) rotating clockwise or anti-clockwise, would not exert any net force on the outer atoms. Hence, such geometrically frustrated vibrations can be localized forming the phonon flat-band. It is also clear now that this phonon flat-band is the lattice analog of kagome electron flat-band8,13, whose quadratic band touching feature distinguishes them from the isolated flat-bands in heavy-fermion systems30 and the Dirac cone touching flat-bands in Moire lattices31. The coupling to the kagome phonon flat-band is not predicted by existing research papers but is highly anticipated to explain the giant fermion–boson interaction observed here. Our findings suggest that the flat phonon dispersion can be probed by future momentum-resolved phonon-sensitive scattering experiments including inelastic X-ray scattering and neutron scattering.

The correspondence between the kagome lattice, tunneling conductance, magnetic field response, double kink feature, self-energy analysis, first-principles, and lattice vibration model provides strong evidence and conceptual framework for the fermion–boson interaction in a geometrically frustrated topological quantum material. The nontrivial kagome band structures have been widely observed in both fermionic and bosonic systems15,32, but their many-body interactions were rarely experimentally observed previously. We expect the latter to be quite general in many topological quantum materials with flat-bands. Such fermion–boson interactions can be the driving force for future discovery of incipient density waves and superconductivity in kagome lattice materials through pressure tuning or chemical engineering. Although our research addresses the coupling of the dispersive electrons and the flat-band phonon, it would be interesting to explore in the future the intriguing possibility of coupling of the kagome flat electron band and flat phonon band when they are tuned to the similar energies.

Methods

Sample preparation

High-quality single crystals of CoSn were synthesized by the Sn flux method. The starting elements of Co (99.99%), Sn (99.99%) were put into an alumina crucible, with a molar ratio of Co:Sn = 3:20. The mixture was sealed in a quartz ampoule under a partial argon atmosphere and heated up to 1173 K, then cooled down to 873 K with 2 K/h. The CoSn single crystals were separated from the Sn flux by using a centrifuge.

Scanning tunneling microscopy characterization

Single crystals with size up to 2 mm × 2 mm × 1 mm were cleaved mechanically in situ at 77 K in ultra-high vacuum conditions, and then immediately inserted into the STM head, already at He4 base temperature (4.2 K). The magnetic field was applied with zero-field cooling. Tunneling conductance spectra were obtained with an Ir/Pt tip using standard lock-in amplifier techniques with a root mean square oscillation voltage of 0.3 meV and a lock-in frequency of 977 Hz. Topographic images were taken with tunneling junction set up: V = −100 mV, I = 2–0.05 nA. The conductance maps are taken with tunneling junction set up: V = −50 mV, I = 0.5 nA.

Transmission electron microscopy characterization

Thin lamellae were prepared by focused ion beam cutting using a ThermoFisher Helios NanoLab G3 UC DualBeam. All samples for experiments were polished by 2 kV Ga ion beam to minimize the surface damage caused by the high energy ion beam. Transmission electron microscopy imaging, atomic resolution high-angle annular dark-field scanning transmission electron microscopy imaging and atomic-level energy-dispersive X-ray spectroscopy mapping were performed on a double Cs corrected ThermoFisher Titan Cubed Themis 300 scanning/transmission electron microscope equipped with an extreme field emission gun (X-FEG) source operated at 300 kV and super-X energy-dispersive spectrometry system.

Many-body theory for bosonic mode coupling

Due to the scattering by bosonic modes, the electrons do not have a definite energy but rather a finite lifetime and a broadened spectral function. In many-body theory, this phenomenon is characterized by the electron self-energy Σ(ω), which can be regarded as a correction to the free-electron Green function. The electron Green function is

$$G\left( {k,\omega } \right) = \frac{1}{{\omega - {\it{\epsilon }}_k^0 - \Sigma \left( \omega \right)}},$$
(1)

where $${\it{\epsilon }}_{{k}}^0$$ is the electron bare energy dispersion based on first-principle calculation. And the electron spectral function is

$$A\left( {k,\omega } \right) = \frac{1}{\pi }\left| {{\mathrm{Im}}G\left( {k,\omega } \right)} \right|.$$
(2)

The QPI dispersion is further described by

$$Q\left( {{{q}},\omega } \right) = - \frac{1}{\pi }\mathop {\sum}\limits_k {{\mathrm{Im}}} G\left( {q + k,\omega } \right){\mathrm{G}}\left( {k,\omega } \right).$$
(3)

With the Eliashberg function, the electron self-energy can be described by

$$\Sigma \left( \omega \right) = {\int} {\mathrm{d}} {{E}}{\int} {\mathrm{d}} \omega^ \prime \alpha ^2F\left( {\omega^ \prime } \right)\left[ {\frac{{1 - f\left( E \right) + n\left( {\omega^ \prime } \right)}}{{\omega - \omega^ \prime - E}} + \frac{{f\left( {{E}} \right) + n\left( {\omega^ \prime } \right)}}{{\omega + \omega^ \prime - E}}} \right],$$
(4)

where α2F(ω) is Eliashberg coupling function describing the electron-bosonic mode interaction, f(E) is the Fermi–Dirac distribution, and n(ω) is Bose–Einstein distribution at temperature T. Since in most cases, the bosonic mode energy is far less than the Fermi energy εF, we can make the approximation that the initial and final energies of scattered electrons are close to Fermi energy. In this way, we obtain an Eliashberg coupling function solely determined by the bosonic energy distribution. An effective coupling constant can be defined as

$$\lambda = 2{\int} {\frac{{\alpha ^2F\left( \omega \right)}}{\omega }} {\mathrm{d}}\omega.$$
(5)

In our experiment, we found a kink in the electron dispersion at EM = ±15 meV, which implies a dominant bosonic mode with ωE = 15 meV. In the calculation, we use the Einstein model and take α2F(ω) as a gaussian peak centered at ωE. The calculated Re Σ(k, ω) is given in Fig. 3e, with comparison to experimental data. With the electron Green function, we can simulate the QPI dispersion Q(q, ω) by Eq. (3) convoluted with the experimental resolution, which is given in Fig. 3f.

First-principles calculation

We perform electronic structure calculations within the framework of the density functional theory using norm-conserving pseudopotentials33 as implemented in the Quantum Espresso simulation package34. The exchange–correlation effects are treated within the local density approximation with the Perdew–Zunger parametrization35. The electronic calculations used a plane-wave energy cutoff of 80 Ry and a 12 × 12 × 12 Γ-centered k-mesh to sample the Brillouin zone. Total energies were converged to 10−7 Ry in combination with Methfessel–Paxton type broadening of 0.01 Ry. The phonon calculation is done by using a 2 × 2 × 2 q-mesh grid centered at Γ for the sampling of phonon momenta. Starting with the experimental structure, we have fully optimized both the ionic positions and lattice parameters until the Hellmann–Feynman force on each atom is <10−3 Ry/a.u. (10−4 Ry) and zero-stress tensors are obtained. We find that the flat-band phonon is mainly associated with the vibrations of Co atoms in Co3Sn lattice confined to the line connecting the centers of two neighboring triangles as shown by the dotted line in Fig. 5a. Figure 5b shows the atom displacement resolved phonon band structure.

Kagome lattice vibration model

We consider a kagome lattice vibration model, assuming the motion of the atoms is highly anisotropic and confined to the dotted line in Fig. 5a. The analysis reproduces the kagome band structure with a flat band quadratically touching another band.

We choose the vectors in real space connecting nearest neighbor kagome atoms as

$${\boldsymbol{a}}_1 = \left( {1{\mathrm{/}}2,\sqrt 3 {\mathrm{/}}2} \right)\quad {\boldsymbol{a}}_2 = \left( {1,0} \right)\quad {\boldsymbol{a}}_3 = {\boldsymbol{a}}_1 - {\boldsymbol{a}}_2.$$
(6)

The three atoms α = 1, 2, 3 in each kagome unit cell can move along the directions

$${\boldsymbol{U}}_1 = \left( {\sqrt 3 {\mathrm{/}}2,1{\mathrm{/}}2} \right)\quad {\boldsymbol{U}}_2 = \left( { - \sqrt 3 {\mathrm{/}}2,1{\mathrm{/}}2} \right)\quad {\boldsymbol{U}}_3 = \left( {0, - 1} \right).$$
(7)

All atoms are then coupled with the same spring constant β, except for the sign. The atoms in one unit cell have the potential energy

$$E_{\mathrm{intra}} = \frac{\beta }{2}\mathop {\sum}\limits_{\boldsymbol{R}} {\left[ {\left( {{\boldsymbol{U}}_{1,{\boldsymbol{R}}} - {\boldsymbol{U}}_{2,{\boldsymbol{R}}}} \right)^2 + \left( {{\boldsymbol{U}}_{1,{\boldsymbol{R}}} - {\boldsymbol{U}}_{3,{\boldsymbol{R}}}} \right)^2 + \left( {{\boldsymbol{U}}_{2,{\boldsymbol{R}}} + {\boldsymbol{U}}_{3,{\boldsymbol{R}}}} \right)^2} \right]}.$$
(8)

In contrast, the inter-unit cell coupling reads

$$E_{\mathrm{inter}} = \, \frac{\beta }{4}\mathop {\sum}\limits_{\boldsymbol{R}} {\left[ {\left( {{\boldsymbol{U}}_{1,{\boldsymbol{R}}} - {\boldsymbol{U}}_{2,{\boldsymbol{R}} - 2{\boldsymbol{a}}_2}} \right)^2 + \left( {{\boldsymbol{U}}_{1,{\boldsymbol{R}}} - {\boldsymbol{U}}_{3,{\boldsymbol{R}} - 2{\boldsymbol{a}}_1}} \right)^2 + \left( {{\boldsymbol{U}}_{2,{\boldsymbol{R}}} + {\boldsymbol{U}}_{3,{\boldsymbol{R}} - 2{\boldsymbol{a}}_3}} \right)^2} \right.} \\ \,\,\,\, + \left. { \left( {{\boldsymbol{U}}_{2,{\boldsymbol{R}}} - {\boldsymbol{U}}_{1,{\boldsymbol{R}} + 2{\boldsymbol{a}}_2}} \right)^2 + \left( {{\boldsymbol{U}}_{3,{\boldsymbol{R}}} - {\boldsymbol{U}}_{1,{\boldsymbol{R}} + 2{\boldsymbol{a}}_1}} \right)^2 + \left( {{\boldsymbol{U}}_{3,{\boldsymbol{R}}} + {\boldsymbol{U}}_{2,{\boldsymbol{R}} + 2{\boldsymbol{a}}_3}} \right)^2} \right].$$
(9)

After Fourier transformation, the potential energy reads

$$E_{\mathrm{pot}} = \mathop {\sum}\limits_k {\left( {{\boldsymbol{U}}_{1,{\boldsymbol{k}}},{\boldsymbol{U}}_{2,{\boldsymbol{k}}},{\boldsymbol{U}}_{3,{\boldsymbol{k}}}} \right)} v_k\left( {\begin{array}{*{20}{c}} {{\boldsymbol{U}}_{1, - {\boldsymbol{k}}}} \\ {{\boldsymbol{U}}_{2, - {\boldsymbol{k}}}} \\ {{\boldsymbol{U}}_{3, - {\boldsymbol{k}}}} \end{array}} \right),$$
(10)

with

$$v_k = \frac{\beta }{2}\left( {\begin{array}{*{20}{c}} 4 & { - 1 - e^{2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_2}} & { - 1 - e^{2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_1}} \\ { - 1 - e^{ - 2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_2}} & 4 & {1 + e^{2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_3}} \\ { - 1 - e^{2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_1}} & {1 + e^{ - 2i{\boldsymbol{k}} \cdot {\boldsymbol{a}}_3}} & 4 \end{array}} \right).$$
(11)

The spectrum of vk is given by

$$\omega _k^2 = \frac{\beta }{m}\left\{ {1,\quad \frac{5}{2} \pm \frac{1}{2}\sqrt {3 + 2{\mathrm{cos}}\left( {{\boldsymbol{k}}_1} \right) + 2{\mathrm{cos}}\left( {{\boldsymbol{k}}_1 - {\boldsymbol{k}}_2} \right) + 2{\mathrm{cos}}\left( {{\boldsymbol{k}}_2} \right)} } \right\},$$
(12)

where k1, k2 stand for the inner product of k vector with a1, a2. We can observe the characteristic flat-band and two quadratic bands.