Surface coupling in Bi₂Se₃ ultrathin films by screened Coulomb interaction

Abstract

Single-particle band theory has been very successful in describing the band structure of topological insulators. However, with decreasing thickness of topological insulator thin films, single-particle band theory is insufficient to explain their band structures and transport properties due to the existence of top and bottom surface-state coupling. Here, we reconstruct this coupling with an equivalently screened Coulomb interaction in Bi₂Se₃ ultrathin films. The thickness-dependent position of the Dirac point and the magnitude of the mass gap are discussed in terms of the Hartree approximation and the self-consistent gap equation. We find that for thicknesses below 6 quintuple layers, the magnitude of the mass gap is in good agreement with the experimental results. Our work provides a more accurate means of describing and predicting the behaviour of quasi-particles in ultrathin topological insulator films and stacked topological systems.

Introduction

Topological insulators (TIs) are known for their surface states protected by time-reversal symmetry¹. Owing to the strong spin-orbit coupling (SOC) and non-zero ${{\mathbb{Z}}}_{2}$ number¹, the Kramers’ pairs of these topological surface states exhibit quasi-particle behaviors with different dispersion relations, such as Dirac fermions, type-I, and type-II Weyl fermions, as well as corresponding spin-momentum locking². Many properties induced by the surface states have been investigated extensively such as quantum spin Hall (QSH) effect³, quantum anomalous Hall (QAH) effect with magnetic doping⁴, and nonlocal magnetization dynamics in ferromagnet/TI heterojunction⁵. These topological surface states are generally housed within a few nanometers of the surfaces of samples, while the bulk samples are usually not ideal insulators owing to the existence of vacancies during the sample preparation. Ultrathin TI films with a large surface-to-volume ratio are ideal systems to amplify the dominance of surface states. Meanwhile, with decreasing thickness of ultrathin films, the coupling of top and bottom surface states increases. As a typical example of a ${{\mathbb{Z}}}_{2}$ topological insulator, the topological surface state of Bi₂Se₃ with a Dirac fermion dispersion relation ${E}_{\pm }\left({{{{{\bf{k}}}}}}\right)=\mu \pm \sqrt{{\left(\hslash {v}_{{{{{{\rm{F}}}}}}}\right)}^{2}{k}^{2}+{m}^{2}}$ was predicted by the single-particle band theory with the help of the tight binding approximation and the model Hamiltonian, where μ is the position of the Dirac point, ${v}_{{{\rm{F}}}}$ is the Fermi velocity, $\hslash$ is the reduced Plank constant and m is the mass of the Dirac fermion^6,7,8. This dispersion relation was experimentally observed⁹. For the Bi₂Se₃ samples with thickness above 6 quintuple layers (QLs), the dispersion of surface state is gapless, i.e., $m=0$, while the coupling between top and bottom surface states can open an energy gap for the samples with thickness of below 6 QL, i.e. $m\,\ne \,0$. It should be noted that, within the framework of single-particle band theory, the surface states are treated as a two-level system described by the massive Dirac Hamiltonian ${H}_{{{{{{\rm{D}}}}}}}=\hslash {v}_{{{{{{\rm{F}}}}}}}\left({{{{{\boldsymbol{\sigma }}}}}}{{{\times}}}{{{{{\bf{k}}}}}}\right)\cdot \hat{{{{{{\bf{z}}}}}}}+m{\sigma }_{z}+\mu$. The Chern number of each energy band in such systems, which is proportional to Hall conductivity by the TKNN formula¹⁰, depends on the sign of mass gap m^1,11.

In previous works on Bi₂Se₃ ultrathin films^11,12, the single-particle band theory predicated that with decreasing thickness, a non-zero mass gap m induced by the surface states coupling appears and evolves from a negative to positive value around 2.5 QL. This change in the sign of m will induce a topological phase transformation, and a non-monotonic evolution of the energy gap. However, this non-monotonic change in energy gap was not observed by the angle-resolved photoemission spectroscopy (ARPES) measurements, and the observed energy gaps are much larger than the theoretical ones¹³. To further investigate this non-monotonic change, In-doped Bi₂Se₃ films are studied and it was found that the topological phase transformation does not appear below 6 QL, which confirms the trivial topological nature of the system¹⁴. Moreover, on the basis of single-particle band theory, a non-zero m will cause a spin polarization ${\sigma }_{z}$ in the z-direction near the Γ point¹¹, while this spin polarization was not observed in the spin-resolved ARPES measurement¹⁵. The nano-scaled multi-tip transport experiments also confirm that the 1D ballistic edge state of a 2D topological QSH insulator is absent in TI ultrathin films¹⁶, implying that the surface states are topologically trivial. Meanwhile, both transport experiments^17,18 and numerical calculation¹⁹ suggest that TIs in the 2D limit are intrinsically quantum many-body systems. It seems that the single-particle band theory is not sufficient to describe the band structure and transport properties in ultrathin Bi₂Se₃ TI films.

In this work, we investigate the coupling between the top and bottom surface states in the form of a screened Coulomb potential. Both the band shifting and the mass gap are analyzed in terms of the Hartree self-energy and the self-consistent gap equation, respectively. The band shifting and the mass gap of Bi₂Se₃ films with different thicknesses are extracted from ARPES spectra and fit the theoretical results well. Within the framework of our theory, the energy gap may originate from the breaking of chiral symmetry instead of the time-inversion symmetry for one band of a Kramers’ pair, which is similar to the case in graphene with EEI^20,21. Therefore, the energy gaps in the ultrathin Bi₂Se₃ films with thickness of below 6 QL are always topologically trivial and the z-component of spin polarization shall not appear, which is consistent with reported experimental results. Compared with the single-particle band theory, our work provides a more accurate means to describe and predict the behavior of quasi-particles, especially in TI ultrathin films and/or stacked systems.

Results

The Hamiltonian of massless Dirac fermions is $H=\hslash {v}_{{{{{{\rm{F}}}}}}}{{{{{\boldsymbol{\alpha }}}}}}{{\cdot }}{{{{{\bf{k}}}}}}-{\mu }_{0}$, where ${\alpha }_{i}={\gamma }_{i}{\gamma }_{0}$ (i = 1 and 2) are the base vectors of the spin space and ${\gamma }_{\mu }$ are the gamma matrices. Similar to the EEI in graphene²¹, the whole Lagrangian of the top $\left(\alpha=+ 1\right)$ and bottom $\left(\alpha=-1\right)$ surface states ${\psi }_{\alpha }$ in TIs, as shown in Fig. 1a, is $L={L}_{0}+{L}_{{{{{{\rm{int}}}}}}}$, where

$${L}_{0}={{\hslash }}{v}_{{{{{{\rm{F}}}}}}}\mathop{\sum }\limits_{\alpha }\int {{{{{{\rm{d}}}}}}}^{2}{{{{{\bf{r}}}}}}\, {\bar{\psi }}_{\alpha }\left(t,{{{{{\bf{r}}}}}}\right)i{{{{{\boldsymbol{\gamma }}}}}}\cdot \nabla {\bar{\psi }}_{\alpha }\left(t,{{{{{\bf{r}}}}}}\right),$$

(1)

$${L}_{{{{{{\rm{int}}}}}}}=\int {{{{{{\rm{d}}}}}}}^{2}{{{{{\bf{r}}}}}}{{{{{{\rm{d}}}}}}}^{2}{{{{{{\bf{r}}}}}}}^{{\prime} }\mathop{\sum }\limits_{\alpha,{\alpha }^{{\prime} }}{\bar{\psi }}_{\alpha }\left(t,{{{{{\bf{r}}}}}}\right){\gamma }_{0}{\psi }_{\alpha }\left(t,{{{{{\bf{r}}}}}}\right){V}_{\alpha,{\alpha }^{{\prime} }}\left(t,{{{{{\bf{r}}}}}}{{{{{\rm{;}}}}}}\,{t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\rm{;}}}}}}\,d\right){\bar{\psi }}_{{\alpha }^{{\prime} }}\left({t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }\right){\gamma }_{0}{\psi }_{{\alpha }^{{\prime} }}\left({t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }\right),$$

(2)

${{{{{\bf{r}}}}}}\,{{{{{\rm{and}}}}}}\,{{{{{{\bf{r}}}}}}}^{{\prime} }$ are the in-plane 2-dimensional position vectors, d is the distance between the top and bottom surfaces, i.e., the thickness of samples. ${V}_{\alpha,{\alpha }^{{\prime} }}\left(t,{{{{{\bf{r}}}}}};{t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }{;d}\right)={\delta }_{\alpha,-{\alpha }^{{\prime} }}V\left(t,{{{{{\bf{r}}}}}};{t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }{;d}\right)$ describes the coupling between these two surface states. Since the quasi-particles are essentially electrons, the dominant interaction is the Coulomb interaction. The Fermi velocity, ${v}_{{{{{{\rm{F}}}}}}}$, of Dirac fermions in Dirac material is about $\sim {10}^{5}\,{{{{{\rm{m}}}}}}/{{{{{\rm{s}}}}}}$¹³, which is much slower than the speed of light. The screened Coulomb interaction between top and bottom fermions is

$$V\left(t,{{{{{\bf{r}}}}}}{{{{{\rm{;}}}}}}{t}^{{\prime} },{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\rm{;}}}}}}d\right)=\delta \left(t-{t}^{{\prime} }\right)V\left({{{{{\bf{|r}}}}}}-{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\boldsymbol{|}}}}}}{{{{{\rm{;}}}}}}d\right) \\=\delta \left(t-{t}^{{\prime} }\right)\frac{{e}^{2}}{4\pi {\varepsilon }_{0}}\frac{{{\exp }}\left[-{\lambda }^{-1}\sqrt{{{{{{{\bf{|r}}}}}}-{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\boldsymbol{|}}}}}}}^{2}+{d}^{2}}\right]}{\sqrt{{{{{{{\bf{|r}}}}}}-{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\boldsymbol{|}}}}}}}^{2}+{d}^{2}}},$$

(3)

where $e$ is the electron charge, ${\varepsilon }_{0}$ is vacuum permittivity. $\lambda$ is the screening length, $\sqrt{{{{{{{\bf{|r}}}}}}-{{{{{{\bf{r}}}}}}}^{{\prime} }{{{{{\boldsymbol{|}}}}}}}^{2}+{d}^{2}}$ is the distance between the top and bottom electron with position vector ${{{{{\bf{r}}}}}}$ and ${{{{{{\bf{r}}}}}}}^{{\prime} }$, and $\delta \left(t-{t}^{{\prime} }\right)$ is the delta function. Since it is difficult to calculate the two-dimensional Fourier transformation of $V\left(r,d\right)$ with respect to ${{{{{\bf{r}}}}}}{{{{{\boldsymbol{,}}}}}}$ we adopt an approximate expression that reads

$$V\left(q,d\right){{{{{\rm{:=}}}}}}{{{{{{\mathscr{F}}}}}}}_{{{{{{\bf{r}}}}}}}\left[V\left(r,d\right)\right]\,\approx \,\frac{{e}^{2}}{4\pi {\varepsilon }_{0}}\frac{{{\exp }}\left[-d\sqrt{{q}^{2}+{\lambda }^{-2}}\right]}{\sqrt{{q}^{2}+{\lambda }^{-2}}}$$

(4)

Fig. 1: The form of interaction and the parameters extracted from ARPES.

a Feynman diagram of the self-energy used in the Dyson equation. The wavy lines and arrows represent boson lines and fermion lines, respectively. The polarization function of Dirac fermions is made of the fermion lines on the opposite surface. b ARPES spectra of 27 QL Bi₂Se₃ sample. The black dots in b are the peak positions of the surfaces state fitted by MDCs with the Lorentzian function, and the red dotted line is the linear fitting results of two sets of black dots. ${E}_{{{{{{\rm{B}}}}}}}$ is the binding energy and ${k}_{{||}}$ is the in-plane wave vector.

To analyze the error induced by this approximation, we numerically calculated the ${{{{{{\mathscr{F}}}}}}}_{{{{{{\rm{r}}}}}}}\left[V\left(r,d\right)\right]$ as ${V}_{{{{{{\rm{num}}}}}}}\left(q,d\right)$ and compare it with the expression $V\left(q,d\right)$. The relative error $\left|\frac{{V}_{{{{{{\rm{num}}}}}}}-V}{{V}_{{{{{{\rm{num}}}}}}}}\right|$ is less than ${10}^{-5}$ (see Supplementary Note 1). The free propagator of ${L}_{0}$ is ${S}_{0}\left({k}_{0},{{{{{\bf{k}}}}}}\right)={\left({\gamma }_{0}{k}_{0}-\hslash {v}_{{{{{{\rm{F}}}}}}}{{{{{\boldsymbol{\gamma }}}}}}{{\cdot }}{{{{{\bf{k}}}}}}\right)}^{-1}$. After turning Wick rotation and considering the Fermi surface shifting owing to vacancy doping present in the fabricated samples, the complete propagator modified by ${L}_{{{{{{\rm{int}}}}}}}$ is $S\left({k}_{0},{{{{{\bf{k}}}}}}\right)={\left[{\gamma }_{0}\left(i{k}_{n}-\mu \right){A}_{1}\left(k\right)-\hslash {v}_{{{{{{\rm{F}}}}}}}{{{{{\boldsymbol{\gamma }}}}}}{{\cdot }}{{{{{\bf{k}}}}}}{A}_{2}\left(k\right)-m\left(k\right)\right]}^{-1}$, where $m\left(k\right)$ is the renormalization mass gap. ${A}_{{{{{\mathrm{1,2}}}}}}$ was denoted as the wave-function renormalization function²¹. $i{k}_{n}=i\left(2n+1\right)\pi /\beta$ is the Matsubara frequency, $\beta=1/{k}_{{{\rm{B}}}}T$ and ${k}_{{{\rm{B}}}}$ is the Boltzmann constant. It should be noted that the mass term caused by interaction does not relate to the spin polarization, which naturally makes the none of spin in the z-direction hold. This result is based on the fact that the Coulomb interaction, rather than the time-inversion symmetry, breaks the chiral symmetry²². In this work, we focus on the calculation of the first two quantities μ and m. So far, only three parameters are unknown, the Fermi velocity ${v}_{{{\rm{F}}}}$ in ${L}_{0}$, the Fermi energy ${\mu }_{0}$, and the screening length $\lambda$ in ${L}_{{{{{{\rm{int}}}}}}}$. The former two parameters can be extracted experimentally by ARPES spectra in a thick enough sample. The last one describes the surface state decay length, which is approximately equal to half of the critical thickness when the energy gap opens²³, can be calculated by the density functional theory numerically.

Because of the ${\delta }_{\alpha,-{\alpha }^{{\prime} }}$ term in ${L}_{{{{{{\rm{int}}}}}}}$, it is easy to check that only the Hartree term survives in all one-loop self-energy, which is

$${{\Sigma }_{H}}_{\alpha }=V\left(q=0,d\right){\rho }_{-\alpha },$$

(5)

where $V\left(q=0,d\right)=\frac{{e}^{2}}{4\pi {\varepsilon }_{0}}\lambda {{\exp }}\left[-d/\lambda \right]$ is the DC part of $V\left(q,d\right)$ and ${\rho }_{-\alpha }$ is the areal density of fermions on the opposite surface. With the Dyson equation ${S}^{-1}={S}_{0}^{-1}+{\Sigma }_{H}$, ${\Sigma }_{H}$ makes the whole energy band shift since it is a real number. The second order of ${L}_{{{{{{\rm{int}}}}}}}$ gives the self-consisted gap equation for $m\left(i{k}_{n},{{{{{\bf{k}}}}}}\right)$. Since we are only concerned with the mass gap at the $\Gamma$ point, following by the widely used approximation to solve interaction-induced mass gap problems in the Dirac system, we set ${A}_{{{{{\mathrm{1,2}}}}}}=1$^21,22,24,25 and treat $m\left(k\right)$ as a constant^22,25,26. This treatment allows us to take the trace of m, and consequently obtain the self-consistent gap equation as follows

$$m(i{k}_{n},{{{{{\bf{k}}}}}})=\frac{1}{{N}_{\gamma }}{{{{{\rm{Tr}}}}}}\frac{1}{\beta }\mathop{\sum}\limits_{m}\int \frac{{{{{{{\rm{d}}}}}}}^{2}{{{{{\bf{q}}}}}}}{{(2\pi )}^{2}}{V}^{2}(q,d){\gamma }_{0}{S}_{\sigma }(i{k}_{n}-i{q}_{m},{{{{{\bf{k}}}}}}-{{{{{\bf{q}}}}}}){\Pi }_{-\sigma }(i{q}_{m},{{{{{\bf{q}}}}}}){\gamma }_{0},$$

(6)

where ${N}_{\gamma }$ is the rank of chosen gamma matrices, ${\Pi }_{-\sigma }\left(i{q}_{m},{{{{{\bf{q}}}}}}\right)$ is the polarization function of the Dirac fermion in the $-\sigma$ surface defined as

$${\Pi }_{-\sigma }(i{q}_{m},{{{{{\bf{q}}}}}})=-\frac{2}{\beta }\mathop{\sum}\limits_{l}\int \frac{{{{{{{\rm{d}}}}}}}^{2}{{{{{\bf{p}}}}}}}{{(2\pi )}^{2}}{\gamma }_{0}{S}_{-\sigma }(i{p}_{l},{{{{{\bf{p}}}}}}){\gamma }_{0}{S}_{-\sigma }(i{p}_{l}+i{q}_{m},{{{{{\bf{p}}}}}}+{{{{{\bf{q}}}}}}).$$

(7)

The related self-energy is shown as blue lines in Fig. 1a. The polarization function of Dirac fermions has been well investigated in graphene systems²⁴. We calculate the polarization function as follows (see Supplementary Note 2).

$${\Pi }_{\sigma }\left(i{q}_{m},{{{{{\bf{q}}}}}}\right)\,\approx \,\frac{1}{8\pi {{{\hslash }}}^{2}{v}_{{{{{{\rm{F}}}}}}}^{2}}{\int }_{0}^{1}{{{{{\rm{d}}}}}}x\left[2{\varpi }_{\beta }\left(B\right)-\frac{C}{B}\frac{\partial }{\partial B}{\varpi }_{\beta }\left(B\right)\right],$$

(8)

where $B={\left[x\,\left(1-x\right)\,{q}_{m}^{2}+ x\,\left(1-x\right)\,{{{\hbar }}}^{2}{v}_{{{{{{\rm{F}}}}}}}^{2}{q}^{2}+{m}^{2}\right]}^{1/2}$, $C=-i{q}_{m}{\mu }_{0}\left(1-2x\right)+ 2{m}^{2}+m {{\hslash }}{v}_{{{{{{\rm{F}}}}}}}{{{{{\boldsymbol{\gamma }}}}}} \cdot {{{{{\bf{q}}}}}}+m\left[\left(1-2x\right)i{q}_{m}-2{\mu }_{0}\right]{\gamma }_{0}+\left[2x\left(1-x\right)i{q}_{m}+\left(1-2x\right){\mu }_{0}\right] {{\hslash }}{v}_{{{{{{\rm{F}}}}}}}{{{{{\boldsymbol{\gamma }}}}}} \cdot {{{{{\bf{q}}}}}}{\gamma }_{0}$, and

$${\varpi }_{\beta }\left(B\right)=\frac{1}{\beta }{{{{{\rm{ln}}}}}}\left[2{{\cosh }}\beta B+2{{\cosh }}\beta {\mu }_{0}\right]\approx \left\{\begin{array}{cc}B & {B}^{{\prime} } \, > \,{\mu }_{0}\\ -B & {B}^{{\prime} } \, < -{\mu }_{0}\\ {\mu }_{0} & -{\mu }_{0}\, < \,{B}^{{\prime} } < {\mu }_{0}\end{array}\right.$$

(9)

The approximation holds when $T\to 0$. With decreasing $q$, ${V}^{2}\left(d,q\right)$ decays at the rate of a Lorentzian function with a width of ${\lambda }^{-1}$. For Bi₂Se₃, the $k$-range of linear Dirac particles is about $0.1\,{{{\text{\AA }}}}^{-1}$¹⁷, which is much larger than ${\lambda }^{-1}\approx 0.035\,{{{\text{\AA }}}}^{-1}$. It is therefore reasonable to use the approximation $\int {{{{{{\rm{d}}}}}}}^{2}{{{{{\bf{q}}}}}}{V}^{2}\left(d,q\right)g\left(q\right)\approx \frac{{e}^{4}}{64{\pi }^{2}{\varepsilon }_{0}^{2}}\left[\frac{1}{\pi }g\left(q=0\right)+{\lambda }^{-1}\frac{\partial }{\partial q}{|}_{q=0}g\left(q\right)\right]{e}^{-2{\lambda }^{-1}d}$ (see Supplementary Note 1) to simplify Eq. (6), where $g\left(q\right)$ is a function that does not fluctuate too much, and the definition of the $\delta$ function in polar coordinates is used. Finally, one can obtain the thickness-dependent mass gap (See the Supplementary Note 2 regarding the detailed error analysis) as:

$$m\,\approx \,\frac{{e}^{4}\mu }{128{\pi }^{3}{\varepsilon }_{0}^{2}{{{\hslash }}}^{2}{v}_{{{\rm{F}}}}^{2}}{{\exp }}\left[-\frac{2d}{\lambda }\right],$$

(10)

where $\mu={\mu }_{0}+{\Sigma }_{H}$ described by Eq. (5). In this expression the sign of the mass gap does not change when thickness decreases, and consequently it is consistent with the absence of gap reclosing probed from ARPES.

We choose Bi₂Se₃ ultrathin films to investigate the coupling between top and bottom surface states. Eight Bi₂Se₃ samples with different thickness of 1–7 QL and 27 QL are grown on Si(111) substrates by molecular beam epitaxy (MBE). The method of sample growth and the measurements of ARPES were described in previous work²⁷. Since the interaction term in Eq. (2) decays exponentially with the thickness at the rate of the surface state decay length, $\lambda \approx 3$ QL, it can be ignored for the sample of 27 QL Bi₂Se₃, and consequently the dispersion relation of Eq. (1) with ${\mu }_{0}$ is ${E}_{\pm }\left(k\right)={\mu }_{0}\pm \hslash {v}_{{{{{{\rm{F}}}}}}}k$. We extract the parameters ${v}_{{{{{{\rm{F}}}}}}}$ in Eq. (1) and the position of the Dirac point ${\mu }_{0}$ from the sample of 27 QL as shown in Fig. 1b using the following steps. First, we fit the momentum distribution curves (MDCs) with a Lorentzian function (see Supplementary Note 3), and the fitted peak positions are marked as black dots in Fig. 1b. Then we linearly fit these two sets of peak positions as shown by red dotted lines in Fig. 1b and obtain ${\mu }_{0}\approx -\left(0.299\pm 0.004\right)\,{{{{{\rm{eV}}}}}}$ and ${v}_{{{{{{\rm{F}}}}}}}\approx \left(2.81\pm 0.08\right)\times {10}^{5}\,{{{{{\rm{m}}}}}}/{{{{{\rm{s}}}}}}$. It is noteworthy that there is a difference between ${k}_{{{{{{\rm{F}}}}}}}$ and ${\mu }_{0}/\hslash {v}_{{{{{{\rm{F}}}}}}}$ owing to the quadratic term of $k$. The ${k}_{{{\rm{F}}}}$ fitted by MDC at the Fermi level is about $0.9\,{{{\text{\AA }}}}^{-1}$, and the areal density of fermions in a single surface is ${\rho }_{\sigma }\approx 1.3\times {10}^{13}\,{{{{{\rm{c}}}}}}{{{{{{\rm{m}}}}}}}^{-2}$. These values are in good agreement with those of the previous work¹⁷. For the other samples, we used the dispersion relation of the generalized Green function ${E}_{\pm }\left({{{{{\bf{k}}}}}}\right)=\mu \pm \sqrt{{\hslash }^{2}{v}_{{{{{{\rm{F}}}}}}}^{2}{k}^{2}+{m}^{2}}$ due to the existence of energy gap ${E}_{{{{{{\rm{gap}}}}}}}=2m$. By fitting the peak positions we can check whether or not there is an energy gap in the energy distribution curve (EDC) at the $\Gamma$ point, and further determine the magnitude of $m$. The remaining steps are the same as those for fitting the sample of 27 QL. More details on peak fitting are present in the Supplementary Note 3. The fitted peak positions and dispersion relation are shown in Fig. 2a–f. The fitted thickness-dependent $\mu$ and $m$ are shown as the scatter plots in Figs. 3a, b, respectively. It should be noticed that, although the Van der Waals materials like Bi₂Se₃ grow layer by layer, the film prepared by MBE is still difficult to be atomically flat in the entire area covered by the light spot with the scale of 10² microns in our ARPES measurements^13,28. The ARPES results should be a superposition of the results between two integer layers. Therefore, to confirm whether the gap will reclose or not at ~2.5 QL predicted by the previous theory¹¹, we further extracted the energy gap of 1.5 and 2.5 QL Bi₂Se₃ to improve the measurement accuracy of this thickness-depended evolution near the critical point (more details can be checked in Supplementary Note 3). As shown in Fig. 2b, the non-monotonic evolution of the energy gap around the thickness of 2–3 QL is not found.

Fig. 2: ARPES spectra of Bi₂Se₃/Si(111) samples with different thicknesses.

a–f are the ARPES spectra of the Bi₂Se₃ sample of 7, 6, 5, 4, 3, and 2 QL, respectively. The black dots are the peak positions fitted by MDCs and EDCs, while the red dot lines are the fitting results of black dots with massive or massless Dirac dispersion. ${E}_{{{{{{\rm{B}}}}}}}$ is the binding energy and ${k}_{{||}}$ is the in-plane wave vector.

Fig. 3: The results of theoretical validation.

a, b are the experimental and theoretical results of the Dirac position μ and the magnitude of the gap $2m$, respectively. The blue dots in b are the results of Y. Zhang et al. ¹³. The error bars are the root of the sum of the squared peak position fitting error and the squared energy resolution of our ARPES.

We can clearly see that the whole energy band shifts to the deep binding energy with decreasing thickness. In previous works, this phenomenon was attributed to the band bending effect^13,23. However, the band bending effect at the interface between Bi₂Se₃ and vacuum was found to be <0.1 eV for the samples with thickness below 7 QL²⁹, and it should be much smaller at the interface between Bi₂Se₃ and silicon. As shown in Fig. 3a, for the samples with a thickness <6 QL, the band shift is larger than 0.1 eV, therefore cannot be explained by the band bending effect. Meanwhile, as shown in Fig. 3a, the self-energy of the Hartree term, which is described by Eq. (5) with ${\rho }_{\sigma }$ and $\lambda$ discussed above, fits the experimental results quite well. The curve described by Eq. (10) is plotted in Fig. 3b. We find that the curve fits both our experimental results and previous work as the cyan circles in Fig. 3b for comparison¹³. Furthermore, we can also explain the reason for none of energy gap reclosing ~2–3 QL. It should be mentioned that with the thickness decreasing, the critical thickness of the phase transition from massless to massive Dirac fermion cannot be determined by Eq. (10). Other methods for analyzing critical behavior, such as the bifurcation theory or parameter embedding method²¹, are needed to find this critical thickness. In this work, we only consider the magnitude to the energy gap. The energy gap calculated by Eq. (10) at the phase transition thickness is <10 meV, which is smaller than the energy resolution of ARPES or the band broadening in most cases³⁰. When the thickness further decreases to 1 QL, the mass gap described by Eq. (10) is about $0.84$ eV, which is even larger than the energy gap of the bulk state⁶. In this region, the lower branch of surface state first drops into the bulk states because of the band shifting. As shown in Fig. 4a, the Dirac cone is no longer complete, and consequently, the calculated mass gap by our theory cannot be measured from ARPES for 1 QL.

Fig. 4: ARPES spectra of the Bi₂Se₃(1 QL)/X/Si(111) system.

a is the ARPES spectrum of Bi₂Se₃(1 QL)/Si(111). X in b and c is Bi₂Te₃(2 QL) (Sample S1) and Bi₂Te₃(1 QL)/Bi₂Se₃(1 QL) (Sample S2), respectively.

Only one parameter $\lambda$ is used to describe the coupling between the top and bottom surface by Eq. (2) and Eq. (3), and all thickness-dependent results are based on the form of ${\lambda }^{-1}d$ in the exponential function. This method makes the analysis of coupling between surface states intuitive and concise, especially in multilayered systems. For example, we further prepare and measure the ARPES spectra of Bi₂Se₃(1 QL)/Bi₂Te₃(2 QL) (noted as Sample S1), and Bi₂Se₃(1 QL)/Bi₂Te₃(1 QL)/Bi₂Se₃(1 QL) (noted as Sample S2) as shown in Figs. 4b and 4c. The screening length of Bi₂Te₃, ${\lambda }_{{{{{{\rm{BT}}}}}}}\approx 1\,{{{{{\rm{nm}}}}}}$, is less than that of Bi₂Se₃, ${\lambda }_{{{{{{\rm{BS}}}}}}}\,\approx \,2.8\,{{{{{\rm{nm}}}}}}$²³. The effective exponential factor ${\left[{\lambda }^{-1}d\right]}_{{{{{{\rm{eff}}}}}}} \sim {\sum }_{i}{\lambda }_{i}^{-1}{d}_{i}$ in Sample S1 is about 2.33, which is equivalent to the value found in Bi₂Se₃ film of about $7$ QL, while the effective thickness of Sample S2 is ~5 QL Bi₂Se₃. One can find the difference between these two samples and the 1 QL of Bi₂Se₃ as shown in Fig. 4a.

We reconstructed the surface coupling by the screened Coulomb interaction between the top and bottom surface states. The evolution of the energy gap with thickness fits well with the self-cosistent mass gap equation. Interestingly, we find that the Dirac fermion coupling between the top and bottom surface state is quite similar to that between the K and K’ points in graphene, in which chiral symmetry breaking with a mass gap is also observed³¹. Considering that the energy gap predicted by the single-particle theory¹¹ is less than one-third of the experimental one, the EEI dominants in ultrathin films and the influence of the size effect should be treated as a perturbation. Meanwhile, if there are systems where QSH can still be realized even after taking Coulomb repulsion into account, the interaction-induced gaps in such systems should be smaller than the size-effect-induced ones. For example, it may occur in thicker TI films or systems with small $\mu$. On the one hand, we find a better method for fitting the relationship between the mass gap and thickness in Bi₂Se₃, but this gap is topologically trivial. On the other hand, the ARPES results of samples S1 and S2 hint at being able to treat the surface coupling as a screened Coulomb potential, which makes the analysis of ultrathin films much easier. Our method also makes it possible to analyze the interactions between the high-energy-like particles in condensed matter physics.

Discussion

Instead of calculating the specific form of surface states by the single-particle energy band theory, we investigate the coupling between two quasi-particle systems using Green function with the screened Coulomb interaction. A set of Bi₂Se₃ films with different thicknesses are analyzed. The theoretical value of band shifting and the mass gap fit our experimental results well. Our method offers valuable insight into the nature of energy gaps in ultrathin Bi₂Se₃ films, which may be induced by the breaking of chiral symmetry instead of the time-reversal symmetry. This method also allows us to analyze the Green function in ultrathin TI films much more easily and explains the problems of the sign of the mass gap and the spin polarization near the $\Gamma$ point in previous work perfectly.

Methods

Sample preparation and characterization

High-quality Bi₂Se₃ films were grown on Si(111) substrates by in-situ ultrahigh vacuum (base pressure: $1.0\times {10}^{-10}$ mbar) MBE. Si substrates were firstly annealed to 1000 °C to get Si(111)-7$\times$7 reconstructed surface. Then, in order to remove the surface dangling bonds, one monolayer of Bi was deposited on the Si(111)-7$\times$7 reconstructed surface and annealed at 300 °C to get the Si(111)-Bi-$\beta$-$\sqrt{3}\times \sqrt{3}$ phase as the wafer layer. High-purity Bi and Se with a ratio of 1:14 were co-evaporated from standard Knudsen cells onto Si(111)-Bi-$\beta$-$\sqrt{3}\times \sqrt{3}$ with the substrate temperature of 220 °C. The Si(111)-7 × 7 reconstructed surface, Si(111)-Bi-$\beta$-$\sqrt{3}\times \sqrt{3}$ surface, and the surface of Bi₂Se₃ are characterized by low energy electron diffraction pattern. The crystal structure was confirmed by ex-situ X-ray diffraction. The growth rate determined by ex-situ small angle X-ray reflection is ~0.59 QL/min.

ARPES measurements

To perform ARPES measurements, a Se capping layer with a thickness of $5\,{{{{{\rm{nm}}}}}}$ was deposited to protect the surface from contamination before transferring to another MBE-ARPES system. To probe the spectral function of Bi₂Se₃, the samples were heated up to 280 °C and kept for 5 h to remove the Se capping layer and then transferred to the ARPES chamber. The spectral function of the top surface is detected by ARPES with He-$\alpha$I light (21.2 eV) and a Scienta DA30 apparatus as the electron energy analyzer under $T=5\,{{{{{\rm{K}}}}}}$ with a base pressure ~$3.0\times {10}^{-11}$ mbar. The pass energy is set to 5 meV and the energy resolution is better than 20 meV, which is confirmed bythe polycrystalline gold standard sample during the ARPES measuring.