Introduction

Massless Dirac particles preserve both parity and time-reversal symmetry (TRS). However, an infinitesimal mass will inevitability break these symmetries through the radiative correction1,2,3, resulting in an Abelian or nonAbelian Chern–Simons theory of the external gauge field. This effect is called parity anomaly. Quasiparticles with linear dispersion in condensed matter physics have recently drawn broad attention4,5,6,7,8,9,10,11,12,13, and they provide a quintessential platform to test these effects. Indeed, the gapless Dirac fermions of topological insulator (TI) surface states have been predicted to exhibit the parity anomaly and a number of exotic phenomena when different types of mass terms are considered, including the p + ip type topological superconductor induced by the superconductor (SC) proximity effect14,15, and the quantum anomalous Hall effect in the TI surface with a Zeeman field16,17,18,19.

In this work, we expand the realm of two-dimensional (2D) topological states by devising a study on the particle–-hole instability of Dirac fermions. We show that the formation and condensation of excitons can lead to a novel topological excitonic insulator that exhibits the parity anomaly. Furthermore, in sharp contrast to conventional semiconductors, the 2D helical Dirac liquid with spin-momentum locking (e.g., the TI surface), has an inherent tendency towards the formation of spin-triplet excitons. This remarkable property manifests itself most clearly by the interband optical absorption processes, as shown by Fig. 1.

For conventional semiconductors with absence of the spin–orbit coupling, the direct interband excitation generates a conduction electron and a valence hole with the same momentum k and the opposite spins (Fig. 1a), resulting in a spin-singlet exciton after they form into a pair. The situation is, however, different in TI surfaces, as shown by Fig. 1b. Due to the spin-momentum locking of TI surface, the direct interband excitation is accompanied by the spin-flip of the electron; the resultant exciton consists of an electron and a hole with the same spin polarization, and therefore a spin-triplet state. This unique optical property of TI surface, as will be shown below, can be used for generation and stabilization of a unique p + ip excitonic insulator. The natural topological p-wave excitonic insulator has p + ip symmetry inherent to their order parameters. We firstly explore the salient feature of the p + ip excitonic state by studying a minimal theoretical model. In contrary to the conventional excitonic insulators20 described by a s-wave Bardeen–Cooper–Schrieffe (BCS)-like theory21, the p + ip state mathematically resembles the spinless p + ip SC22 in the chiral basis. However, unlike the p + ip SC, it does not break the global U(1) gauge symmetry but only violates the TRS. To characterize such exotic states, we propose a chiral Chern number Cc and unveil the underlying parity anomaly by deriving an effective Chern–Simons gauge theory associated with topological defects. This is further responsible for (i) fractionalized excitations with charge e/2 at vortices and (ii) a spontaneous in-plane magnetization of ground state. Then, we investigate the excitonic instability of a strongly correlated TI surface model where we show that a p-wave-type interaction component emerges intrinsically, which stabilizes the p + ip excitonic phase in a significant parameter region of the phase diagram. Moreover, we also consider the effects of interband optical processes and we demonstrate that the electron–photon coupling can effectively drive the system into the proposed p + ip excitonic insulator as a result of the spin-flip mechanism demonstrated in Fig. 1b.

Fig. 1
figure 1

Schematic plot of the direct interband optical absorption process. a The s-wave excitons are formed in the conventional semiconductors. b The helical spin texture of topological insulator (TI) surface state favors p-wave rather than s-wave excitons

Full size image

Results

Model

We start from a spontaneous symmetry breaking of 2D Dirac fermions with spin-momentum locking and a momentum-dependent mass term, \(H = H_{{\mathrm{Dirac}}} + \hat M({\bf{k}})\), where

$$H_{{\mathrm{Dirac}}} = \mathop {\sum}\limits_{\bf{k}} {\kern 1pt} {\mathrm{\Phi }}_{\bf{k}}^\dagger v_{\mathrm{F}}{\boldsymbol{\sigma }} \cdot {\bf{k}}{\kern 1pt} {\mathrm{\Phi }}_{\bf{k}}.$$
(1)

k is the 2D wave vector, Φk = [ψk,↑, ψk,↓]T is the fermionic spinor and \(\psi _{{\bf{k}},\sigma }^\dagger\) (ψk,σ) denotes the creation (annihilation) operator of electrons with spin σ. We firstly focus on a concrete mass term \(\hat M({\bf{k}})\) = \(\mathop {\sum}\nolimits_{\bf{k}} {\kern 1pt} {\mathrm{\Phi }}_{\bf{k}}^\dagger \left( {{\mathrm{\Delta }}_{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \sigma ^z + \Delta _{\mathrm{c}}{\kern 1pt} {\mathrm{sin}}^2\theta \sigma ^x - \Delta _{\mathrm{c}}{\mathrm{sin}}{\kern 1pt} \theta {\kern 1pt} {\mathrm{cos}}{\kern 1pt} \theta \sigma ^y} \right){\mathrm{\Phi }}_{\bf{k}}\), while under which conditions \(\hat M({\bf{k}})\) can be spontaneously generated will be investigated below. θ is the polar angle of k, and Δc is the overall gap scale. It is straightforward to verify that \(\hat M({\bf{k}})\) breaks TRS, \(\hat T\hat M({\bf{k}})\hat T^{ - 1} \ne \hat M( - {\bf{k}})\), where \(\hat T = {\mathrm{i}}\sigma ^yK\) being the time-reversal operator, with K the complex conjugation operator. In contrast with the TI surface with a Zeeman field, where the diagonal term mσz breaks TRS23, here the diagonal term Δc cos θσz in \(\hat M({\bf{k}})\) respects the TRS. The TRS is however violated by the two off-diagonal terms in \(\hat M({\bf{k}})\).

One can unveil the physical meaning of \(\hat M({\bf{k}})\) by making a unitary transformation onto the chiral basis with HDirac being diagonalized: \(C_{\bf{k}} = \hat R({\bf{k}}){\mathrm{\Phi }}_{\bf{k}}\), where \(\hat R({\bf{k}})\) is a 2 by 2 rotation matrix with \(\hat R_{11} = - \hat R_{21} = e^{{\mathrm{i}}\theta }{\mathrm{/}}\sqrt 2\), \(\hat R_{12} = \hat R_{22} = 1{\mathrm{/}}\sqrt 2\), and Ck = [ck,+, ck,−]T is the spinor composed of the conduction and valence band fermions. In the chiral basis, H becomes

$$H = \mathop {\sum}\limits_{\bf{k}} \left( {\begin{array}{*{20}{c}} {c_{{\bf{k}}, + }} \\ {c_{{\bf{k}}, - }} \end{array}} \right)^\dagger \left( {\begin{array}{*{20}{c}} {v_{\mathrm{F}}k} & { - {\mathrm{\Delta }}_{\mathrm{c}}e^{ - {\mathrm{i}}\theta }} \\ { - {\mathrm{\Delta }}_{\mathrm{c}}e^{i\theta }} & { - v_{\mathrm{F}}k} \end{array}} \right)\left( {\begin{array}{*{20}{c}} {c_{{\bf{k}}, + }} \\ {c_{{\bf{k}}, - }} \end{array}} \right).$$
(2)

The off-diagonal Δce−iθ describes the process where an electron in the valence band is scattered up to the conduction band, leaving a particle–hole excitation on top of the many-body ground state. For Δc ≠ 0, the particle–hole pairs condense and gap out the Dirac cone, generating an excitonic insulator. Another interesting feature of the off-diagonal term is the factor e±iθ = (kx ± iky)/k24, similar to the spinless p + ip-wave SC25, but we note that the order parameter resides in the particle–hole rather than the particle–particle basis. The model Hamiltonian can be analyzed similarly as the BCS superconductor theory. The ground state wave function |Ψ〉 is obtained after a Bogoliubov transformation as,

$$\left| {\mathrm{\Psi }} \right\rangle = \mathop {\prod}\limits_{\bf{k}} {\kern 1pt} u_{\bf{k}}e^{\mathop {\sum}\limits_{\mathbf{k}} g_{\bf{k}}c_{{\bf{k}}, + }^\dagger c_{{\bf{k}}, - }}\left| 0 \right\rangle ,$$
(3)

with gk = vk/uk, uk = Δc(kx − iky)/kξk, vk = vFk/ξk, where \(\xi _{\bf{k}} = \sqrt {{\mathrm{\Delta }}_{\mathrm{c}}^2 + v_{\mathrm{F}}^2k^2}\) is the energy spectrum of quasiparticles. |Ψ〉 clearly describes a coherent state of electron–hole pairs with the distribution factor gk of the p + ip symmetry.

Topological classification and chiral Chern number

To examine the topological classification of the p + ip excitonic insulator, we write Eq. (2) into the compact form as, \({\cal H} = \mathop {\sum}\nolimits_{i = 1}^3 {\kern 1pt} d({\bf{k}})\tau ^i\), with τi being the Pauli matrices defined in the chiral basis. For the continuum model, where k → ∞ is envisioned as a single point, the three-vector d(k) = (−Δc cosθ, −Δc sinθ, vFk) defines a mapping from the k-space (S2) to the d-vector space. Since the z-component vFk ≥ 0, the normalized d-vector \(\hat d({\bf{k}}) = {\bf{d}}({\bf{k}}){\mathrm{/}}\left| {{\bf{d}}({\bf{k}})} \right|\) resides in a hemisphere surface. For \(\hat d({\bf{k}})\) covering the hemisphere n times, the winding number becomes n/2. This winding number is a topological invariant defined in the chiral basis Ck = [ck,+, ck,−]T, which we term as the chiral Chern number Cc.

Recalling that for the TI surface with a Zeeman field, a conventional Chern number C is defined in the same way16,23 but in the spin basis, Φk = [ψk,↑,ψk,↓]T. The difference between C and Cc becomes clear from the perspective of Berry phase as following. In the original spin basis, following a closed loop in momentum space, the accumulation of Berry phase is \(\phi = - {\mathrm{i}}{\kern 1pt} {\oint} {\kern 1pt} d{\bf{k}} \cdot \left\langle {u_{\bf{k}}} \right|\nabla _{\bf{k}}\left| {u_{\bf{k}}} \right\rangle\), where \(\left| {u_{\bf{k}}} \right\rangle\) is the eigenstate of the valence band in the spin basis. For a Bloch state defined in the chiral basis, a similar geometric phase can be defined as \(\phi ^{\mathrm{c}} = - {\mathrm{i}}{\kern 1pt} {\oint} {\kern 1pt} d{\bf{k}} \cdot \left\langle {u_{\bf{k}}^{\mathrm{c}}} \right|\nabla _{\bf{k}}\left| {u_{\bf{k}}^{\mathrm{c}}} \right\rangle\), where \(\left| {u_{\bf{k}}^{\mathrm{c}}} \right\rangle\) is the eigenstate of the valence band in the chiral basis. The Bloch states in the above two formalisms are related to each other by the unitary transformation \(\left| {u_{\bf{k}}^{\mathrm{c}}} \right\rangle = \hat R({\bf{k}})\left| {u_{\bf{k}}} \right\rangle\). It then becomes obvious that the two Berry phases satisfy ϕc − ϕ = \(- {\mathrm{i}}{\kern 1pt} {\oint} {\kern 1pt} d{\bf{k}} \cdot \left\langle {u_{\bf{k}}} \right|\hat R^\dagger ({\bf{k}})\nabla _{\mathbf{k}}\hat R({\bf{k}})\left| {u_{\bf{k}}} \right\rangle\), leading to Cc − C = \(- ({\mathrm{i/}}2\pi ){\kern 1pt} {\oint} {\kern 1pt} d{\bf{k}} \cdot \left\langle {u_{\bf{k}}} \right|\hat R^\dagger ({\bf{k}})\nabla _{\bf{k}}\hat R({\bf{k}})\left| {u_{\bf{k}}} \right\rangle\) = 1/2, after inserting the \(\hat R({\bf{k}})\) matrix we used to diagonalize HDirac.

To explicitly demonstrate the topological invariants, we plot the normalized vector \(\hat d({\bf{k}})\) with traversing the whole k-space. Figure 2a, b shows the mapping of the \(\hat d({\bf{k}})\) vector in the spin and chiral basis, respectively. It is found that \(\hat d({\bf{k}})\) in latter basis completely covers the hemisphere while \(\hat d({\bf{k}})\) in the former does not, leading to Cc = 1/2 and C = 0. Therefore, two topological invariants are needed to completely characterize the p + ip excitonic state, namely, (C, Cc) = (0, 1/2). The zero Chern number C = 0 means that the p + ip excitonic insulator is topologically distinct from the TI surface with a Zeeman field that has C = 1/216. This is clear from the Landau quantization of the two states. For the TI surface with a Zeeman field, the Landau level (LL) for |n| ≥ 1 reads as \(E_n = \pm \sqrt {2ev_{\mathrm{F}}^2B\left| n \right| + m^2}\), where m is the Zeeman gap. In addition, there is an unconventional zeroth LL E0 = sgn(m)m which is located exactly at the gap edge26,27,28. This zeroth LL is a manifestation of the nontrivial Zeeman mass as a result of C = 1/227,28. For the p + ip excitonic insulator phase, however, one cannot find the unconventional zeroth LL located at the gap edge, consistent with C = 0.

Fig. 2
figure 2

The mapping of \(\hat d({\bf{k}})\). a The mapping in the original spin basis. b The mapping in the chiral basis. The difference between the two situations shows that the two topological invariants are needed to completely characterize the p + ip excitonic insulator, i.e., (C, Cc) = (0, 1/2)

Full size image

Chern–Simons theory and fractionalization

The nonzero chiral Chern number Cc = 1/2 indicates that the p + ip excitonic insulator must be topologically distinct from normal insulators. Recalling the p + ip SCs where Majorana fermions are encoded in vortices22, we shift our attention to topological defects in the p + ip excitonic state. To this end, we can solve the Bogoliubov de Gennes (BdG)-type equations looking for zero-energy solutions. In real space, the equations can be derived as (see Methods for details):

$$e^{{\mathrm{i}}\alpha }\left( { - {\mathrm{i}}\partial _r + r^{ - 1}\partial _\alpha } \right)u({\bf{r}}) - \frac{1}{{v_{\mathrm{F}}}}{\mathrm{\Delta }}_{\mathrm{c}}({\bf{r}})v({\bf{r}}) = 0,$$
(4)
$$e^{ - {\mathrm{i}}\alpha }\left( { - {\mathrm{i}}\partial _r - r^{ - 1}\partial _\alpha } \right)v({\bf{r}}) + \frac{1}{{v_{\mathrm{F}}}}\Delta _{\mathrm{c}}^ \ast ({\bf{r}})u({\bf{r}}) = 0,$$
(5)

where α is the polar angle in real space and u, v are the two components of the spinor wave function. Similar equations have been studied in refs. 22,29,30. However, note that the physics in the present model is different from previous works. Read and Green22 derive the equations for SCs, where charge conservation is broken and Majorana fermions are the emergent excitations. For our system, the U(1) gauge symmetry is preserved, therefore no Majorana modes are allowed. Hou et al.29 investigate graphene with a chirality mixing texture which induces the mass term, and Seradjeh et al.30 discuss the exciton condensation between two opposite TI surfaces. In contrary, here we focus on the excitonic order in a Dirac liquid state and the mass term is generated by the condensation of p + ip excitons. To solve the equations in the presence of a topological defect31, we assume Δc(r) = Δc(r)eiα. A unique normalized zero-energy bound state32,33 can be obtained as, u = Af(r)ei(α+π/2)/2 and v = u*, where A is the normalization constant and the function f(r) = exp[−|Δc|r/|vF|] decays exponentially.

We note that the single particle Hamiltonian h in Eq. (2) satisfies (iτ2)h(iτ2) = h. This allows one to define a time-reversal-type operator Θ = iτ2K, with which it is clear that for any eigenstate ψE with energy E, ΘψE is also a eigenstate with energy −E. Besides, the total number of fermion states are conserved, namely, \({\int}_{ - \infty }^\infty {\kern 1pt} dE\delta \rho (E) = 0\), where δρ(E) is the change of density of states. Applying this formula to the case where a topological defect is in presence, and then taking into account the fact that ρ(E) is a symmetric function with respect to E = 0, one comes to the conclusion that both the conduction and valence band lose half a state. This state is compensated by the zero mode that is bound to the topological defect. Consequently, at half filling, the electron charge associated to the defect is either −e/2 or e/2, depending on whether the zero mode is occupied or empty.

We have shown that the p + ip excitonic insulator is characterized by Cc = 1/2 and displays fractional charge e/2 at vortices. However, it is still unclear how the two observations are related to each other. This can be answered by the parity anomaly34 of the p + ip excitonic insulator and the effective gauge theory associated with the topological defects. First, we couple an external U(1) gauge field Aμ (with μ = 0, 1, 2) to the p + ip excitonic state35, and we introduce the Dirac matrices defined as γ0 = τ3, γ1,2 = iτ1,2. In order to extract the effective theory for the vortices, we include the vortex degrees of freedom χ in the order parameter. Second, we perform a singular gauge transformation, c± → e±iχ/2c±36, which results in the following Lagrangian (see Methods for details),

$${\cal L} = \overline {\cal C} _{\bf{k}}\left( {{\mathrm{i}}\tilde \gamma ^\mu \partial _\mu + \tilde \gamma ^\mu a_\mu + m} \right){\cal C}_{\bf{k}},$$
(6)

where \({\cal C}_{\bf{k}}\) \(\left( {\bar {\cal C}_{\bf{k}}} \right)\) is the Grassmann field. \(\tilde \gamma ^\mu\) are defined as \(\tilde \gamma ^0 = \gamma ^0\), \(\tilde \gamma ^1 = \gamma ^2\), \(\tilde \gamma ^2 = - \gamma ^1\). In Eq. (6), the vortex degrees of freedom have been absorbed into an effective gauge field aμ after the singular transformation. For a static defect with ∂0χ = 0, \(a_\mu = A_\mu + \frac{1}{2}\partial _\mu \chi\). Finally, after one integrates out the Grassmann field, a Chern–Simons term with respect to the effective gauge field aμ can be obtained in the second order expansion:

$${\cal L}_{{\mathrm{CS}}} = \frac{I}{{2\pi }}\epsilon ^{\mu \nu \rho }a_\mu \partial _\nu a_\rho ,$$
(7)

where the coefficient I is found to be 1/2 for the proposed p + ip excitonic insulator.

We note that for the TI surface with a Zeeman field23, a similar Chern–Simons Lagrangian arises as a result of the parity anomaly, where the coefficient in front of the Chern–Simons term is exactly the Chern number C. Here, the coefficient I = 1/2 is identified to be the chiral Chern number Cc defined above. The similarity between the two gauge field theories is due to the underlying parity anomaly; however, for the p + ip excitonic insulator, it is the effective gauge field aμ rather than the external field Aμ that obeys the Chern–Simons form. Further inserting \(a_\mu = A_\mu + \frac{1}{2}\partial _\mu \chi\), the full Lagrangian \({\cal L}_{{\mathrm{CS}}} = \frac{I}{{2\pi }}\epsilon ^{\mu \nu \rho }A_\mu \partial _\nu A_\rho - A_\mu j^\mu + {\cal L}_{\mathrm{v}}\) consists of a Lagrangian describing the vortex \({\cal L}_{\mathrm{v}}\) and a current operator associated with the vortex,

$$j^\mu = \frac{I}{{2\pi }}\epsilon ^{\mu \nu \rho }\partial _\nu \partial _\rho \chi .$$
(8)

For a static vortex (with vortex number n = 1) that centered at r0, we have \(\epsilon ^{ij}\partial _i\partial _j\chi = 2\pi \delta \left( {{\bf{r}} - {\bf{r}}_0} \right)\), inserting which into Eq. (8), the electron charge of the defect is then obtained as \(Q = {\int} {\kern 1pt} d^2xj^0 = I \equiv C_{\mathrm{c}} = 1{\mathrm{/}}2\). Therefore, we have proved that the chiral Chern number Cc equals to the fractional charge (in unit of e) of a topological defect, justifying Cc to be the essential topological invariant characterizing the p + ip excitonic insulator.

Spontaneous in-plane magnetization

The order parameter −Δce−iθ in Eq. (2) breaks TRS. This suggests that the ground state carries a magnetization associated with it. Before proceeding, we note that the order parameter −Δce−iθ still enjoys a redundant global phase. In the following, we take this into account and write the off-diagonal term in Eq. (2) as −Δce−iθ′, where θ′ = θ − β with β being a constant angle independent of k. The global phase β is analogous to U(1) phase of a superconductor.

To clearly show the feature of magnetization, we calculate the spin expectation values 〈σx〉, 〈σy〉, 〈σz〉 from the single particle eigenstates and study the spin texture of the p + ip excitonic ground state |Ψ〉 (Eq. (3)). We obtain

$$\begin{array}{l}\left\langle {\sigma ^x} \right\rangle = - \frac{{\left( {\Delta _{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \prime + \xi _{\bf{k}}} \right)\left( {v_{\mathrm{F}}k{\kern 1pt} {\mathrm{cos}}\theta \prime + \Delta _{\mathrm{c}}{\kern 1pt} {\mathrm{sin}}^2\theta \prime } \right)}}{{v_{\mathrm{F}}^2k^2 + {\mathrm{\Delta }}_c^2 + {\mathrm{\Delta }}_{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \prime \xi _{\bf{k}}}},\\ \left\langle {\sigma ^y} \right\rangle = - \frac{{\left( {\Delta _{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \prime + \xi _{\bf{k}}} \right)\left( {v_{\mathrm{F}}k{\kern 1pt} {\mathrm{sin}}\theta \prime - \Delta _{\mathrm{c}}{\kern 1pt} {\mathrm{sin}}\theta \prime {\mathrm{cos}}{\kern 1pt} \theta \prime } \right)}}{{v_{\mathrm{F}}^2k^2 + {\mathrm{\Delta }}_c^2 + {\mathrm{\Delta }}_{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \prime \xi _{\bf{k}}}},\\ \left\langle {\sigma ^z} \right\rangle = - \frac{{{\mathrm{\Delta }}_{\mathrm{c}}^{\mathrm{2}}{\mathrm{sin}}^{\mathrm{2}}\theta \prime - {\mathrm{\Delta }}_c^2{\kern 1pt} {\mathrm{cos}}^2\theta \prime - {\mathrm{\Delta }}_c^2 - 2{\mathrm{\Delta }}_{\mathrm{c}}{\mathrm{cos}}{\kern 1pt} \theta \prime \xi _{\bf{k}}}}{{v_{\mathrm{F}}^2k^2 + {\mathrm{\Delta }}_c^2 + {\mathrm{\Delta }}_{\mathrm{c}}{\kern 1pt} {\mathrm{cos}}\theta \prime \xi _{\bf{k}}}}.\end{array}$$

As has been discussed before, 〈σz〉 does not break TRS. The integral of 〈σz〉 over the whole k-space vanishes. The in-plane spin texture (〈σx〉, 〈σy〉) is however nonzero, and we plot it in Fig. 3 for different values of β, i.e., β = 0, β = π/2, β = π, β = 3π/2. Interestingly, it is found that the in-plane spin configuration is tilted from the standard helical spin texture of the original TI surface, and the spins at two TRS-related momentum points, k and −k, are no longer always opposite to each other, violating the TRS. Though being tilted, the chiral nature of electron’s spin is preserved as we can see that the direction of the vector (〈σx〉, 〈σy〉) still undergoes a change of 2π when traversing a closed path around k = 0 point. Furthermore, we calculate the total in-plane magnetization M by integrating (〈σx〉, 〈σy〉) over the momentum space. It is found that M always points towards the opposite direction of the global phase β, as shown by the thick red arrows in Fig. 3. We also note that the emergent magnetization of ground state can lead to anisotropic in-plane transport of excitons which serves as an evident experimental signature of the p + ip excitonic insulator. Furthermore, since β determines the direction of magnetization of ground state, one expects to control β by applying a weak in-plane magnetic field during the process of TRS symmetry breaking.

Fig. 3
figure 3

The in-plane spin texture of the p + ip excitonic insulator. a–d The results for different global phase of the exciton order parameter, β = 0, β = π/2, β = π, β = 3π/2, respectively. The total net magnetization M is shown by the arrow for each case

Full size image

The excitonic phases on interacting TI surfaces

After careful characterization of the topological properties of the p + ip excitonic insulator, we now study its possible realization on interacting TI surface states. Although various instabilities of interacting topological insulators and topological superconductors have been extensively studied37,38,39,40,41,42,43,44,45,46,47,48, we focus on the excitonic instability of the surface states and propose to realize the minimal model of the topological excitonic insulators via spontaneous symmetry breaking that enjoys a fractional chiral Chern number Cc = 1/2, which is different from earlier proposals based on semiconductor heterostructure with an integer Thouless–Kohmoto–Nightingale–Nijs (TKNN) number49.

The effective model Eq. (1) describing the surface state is usually modified by a higher order wrapping effect, reducing the rotational symmetry of the Dirac cone down to a discrete C3 subgroup. Our further studies on the wrapping effect shows that the exciton gap remains very robust and therefore one can neglect the wrapping effect for the following analysis. We firstly consider the particle–hole symmetric case, while the generalization to finite chemical potential μ is very straightforward and will be discussed below.

First we start with demonstrating that a p-wave-type interaction that spontaneously generates the p + ip-wave exciton insulator naturally arises from the Hubbard model, which reads in k-space as:

$$H_{\mathrm{I}} = \frac{V}{2}\mathop {\sum}\limits_{{\bf{k}},{\bf{k}}\prime ,{\bf{q}}} \mathop {\sum}\limits_\sigma \psi _{{\bf{k}} + {\bf{q}},\sigma }^\dagger \psi _{{\bf{k}}\prime - {\bf{q}},\overline \sigma }^\dagger \psi _{{\bf{k}}\prime ,\overline \sigma }\psi _{{\bf{k}},\sigma }.$$
(9)

To facilitate the study of excitonic instability, we transform HI onto the chiral basis. Then, considering only interband forward scattering channel, the low-energy effective vectorial interaction component can be derived as (see Methods for details),

$$\begin{array}{*{20}{l}} H \hfill & = \hfill & { = \mathop {\sum}\limits_{{\bf{k}},n} {\kern 1pt} nv_{\mathrm{F}}kc_{{\bf{k}},n}^\dagger c_{{\bf{k}},n}} \hfill \\ {} \hfill & {} \hfill & { - \frac{V}{4}\mathop {\sum}\limits_{{\bf{k}},{\bf{k}}\prime ,n} \left[ {\hat kc_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}} \right] \cdot \left[ {\hat k\prime c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}} \right],} \hfill \end{array}$$
(10)

where we assume a momentum cutoff Λ implicit in the sum of k. n is the band index. The interaction term in the second line, \(H_{{\mathrm{eff}}}^{{\mathrm{(1)}}} = - \frac{V}{4}\mathop {\sum}\nolimits_{{\bf{k}},{\bf{k}}\prime ,n} \left[ {\hat kc_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}} \right] \cdot \left[ {\hat k\prime c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}} \right]\), is of p-wave type, and it arises from the spin-momentum locking of the surface state. After introducing a complex bosonic vector field Δ and performing the Hubbard–Stratonovich transformation to decouple \(H_{{\mathrm{eff}}}^{{\mathrm{(1)}}}\), we arrive at a mean-field theory with the order parameter given by \({\bf{\Delta }} = \frac{1}{2}V\mathop {\sum}\nolimits_{\bf{k}} \left\langle {\hat kc_{{\bf{k}}, - }^\dagger c_{{\bf{k}}, + }} \right\rangle\). When Δ ≠ 0, the condensation of excitons gaps out the Dirac point, resulting in the excitonic insulator phase. Similar to 3He superfluid theory25, symmetry analysis implies two possible configurations of the order parameter that are the extrema of the free energy: (1) (chiral) p + ip excitonic insulator with \({\bf{\Delta }} = \frac{1}{{\sqrt 2 }}\Delta _{\mathrm{c}}\left( {\widehat {\bf{e}}_x - i\widehat {\bf{e}}_y} \right)\) and (2) the polar excitonic insulator with \({\bf{\Delta }} = {\mathrm{\Delta }}_{\mathrm{p}}\widehat {\bf{e}}\), where \(\widehat {\bf{e}}\) is a unit vector in the xy plane. The polar excitonic phase is topologically trivial with zero chiral Chern number.

By self-consistently solving the mean-field gap equation, we obtain the condensation energy and the order parameter (Δc, Δp), which are plot in Fig. 4a, b, respectively. Figure 4b shows the emergence of a threshold Vc, beyond which the system develops the excitonic instabilities with |Δc/p| ≠ 0. Figure 4a indicates that even though both the p + ip excitonic insulator and polar excitonic insulator are stable compared to the gapless Dirac cone for V > Vc, the p + ip excitonic insulator is more energetically favored due to the lower condensation energy. Moreover, we also performed addition mean-field calculations by treating the p + ip and polar excitonic instability on equal footing. It is found that the p + ip channel always suppresses the polar order down to zero. Therefore, even though the ground state energies of the two states are close, the mass term of the polar exciton insulator is negligible in the mean-field theory. Hence, the resultant mean-field Hamiltonian exactly matches with Eq. (2), and the proposed mass term \(\hat M({\bf{k}})\) is generated by spontaneous TRS breaking as a result of the p-wave interaction \(H_{{\mathrm{eff}}}^{(1)}\). Moreover, we note that for particle–hole asymmetric case with a finite chemical potential μ, the critical Vc increases with increasing μ. Although the exciton insulator is always stable for strong enough interaction V > Vc, it is more experimentally favorable to tune μ close to the Dirac point50,51.

Fig. 4
figure 4

The results from mean-field calculations. a Condensation energy versus interaction V for p + ip excitonic insulator and polar excitonic insulator. b Calculated mean-field excitonic order parameter for p + ip excitonic insulator (Δc) and polar excitonic insulator (Δp) versus interaction V. \(\left| {{\mathrm{\Delta }}_{{\mathrm{c/p}}}} \right|^2{\mathrm{/}}v_{\mathrm{F}}^2{\mathrm{\Lambda }}^2 \ll 1\) is assumed. c The phase diagram based on unbiased mean-field with considering both the s- and p + ip-wave interaction channel. The dash-dotted red line denotes the Hubbard interaction case with Vs = Vp

Full size image

In the above calculations, we temporarily focused on the p-wave interaction \(H_{{\mathrm{eff}}}^{{\mathrm{(1)}}}\) without considering the s-wave component. Nevertheless, the results clearly show that the TI surface state indeed has a strong tendency towards the p + ip excitonic instability via spontaneous TRS breaking. We now take into account the interaction in the s-wave channel, which can also be derived from the local Hubbard interaction as (see Methods for details), \(H_{{\mathrm{eff}}}^{{\mathrm{(2)}}} = - \frac{V}{4}\mathop {\sum}\nolimits_{{\bf{k}},{\bf{k}}\prime ,n} c_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}\). In order to be more general, we assume in the following two independent couplings Vp and Vs for the p-wave and s-wave interaction respectively, and treat both terms on equal footing. An s-wave bosonic scalar field \({\mathrm{\Delta }}_{\mathrm{s}} = V_{\mathrm{s}}\mathop {\sum}\nolimits_{\bf{k}} \left\langle {c_{{\bf{k}} - }^\dagger c_{{\bf{k}} + }} \right\rangle\) is introduced to decouple \(H_{{\mathrm{eff}}}^{{\mathrm{(2)}}}\) in addition to the p + ip excitonic order parameter Δc. The coupled self-consistent equations with respect to Δs and Δc are obtained in the standard approach which leads to the calculated phase diagram shown by Fig. 4c. It is found that both the s-wave and p + ip-wave excitonic insulator can be spontaneously generated from the Dirac liquid phase with strong enough interactions. The former and latter are stabilized for large Vs and Vp respectively. We note that although the p + ip excitonic phase, which is absent in conventional semiconductors, becomes stable in a large parameter region of the phase diagram, it does not occur as the ground state for the discussed local Hubbard model, which lies in the parameter line Vs = Vp that only enters into the s-wave excitonic condensation.

The stabilization of the p + ip excitonic insulator

Now we study the stabilization of the p + ip excitonic insulator on TI surfaces by going beyond the natural Dirac semimetals and taking into account external perturbations. Recalling the interband optical absorption process shown by Fig. 1b, one expects that additional photons should prefer the p-wave pairing due to the spin-flip mechanism that gives rise to spin-triplet excitons. Since excitons are usually generated by absorption of photons in a material, we consider a linearly polarized perpendicular incident laser with frequency \(\bar \omega\) and light intensity I. The corresponding microscopic theory can be constructed by quantization of the vector potential A(r), i.e.,

$${\bf{A}}({\bf{r}}) = \mathop {\sum}\limits_{{\bf{q}}\lambda } \sqrt {\frac{{2\pi }}{{\omega _{\bf{q}}}}} {\bf{e}}_\lambda \left( {a_{{\bf{q}},\lambda } + a_{ - {\bf{q}},\lambda }^\dagger } \right)e^{ - i{\bf{q}} \cdot {\bf{r}}},$$
(11)

where aq,λ \(\left( {a_{{\bf{q}},\lambda }^\dagger } \right)\) is the annihilation (creation) operator of photons, eλ (λ = 1, 2) are the polarization vectors, ωq = c|q| is the light frequency, with c being the light velocity and q the wave vector of photons.

The coupling of the external field to the surface electrons is described within the formalism of the minimal coupling, which leads to the electron–photon interaction,

$$\begin{array}{*{20}{l}} {H_{{\mathrm{el - ph}}}} \hfill & = \hfill & { - ev_{\mathrm{F}}\mathop {\sum}\limits_{{\bf{kq}}\lambda \sigma \sigma \prime } \sqrt {\frac{{2\pi }}{{\omega _{\bf{q}}}}} \left( {{\boldsymbol{\sigma }}_{\sigma \sigma \prime } \cdot {\bf{e}}_\lambda } \right)\left( {\psi _{{\bf{k}},\sigma }^\dagger a_{{\bf{q}},\lambda }\psi _{{\bf{k}} - {\bf{q}},\sigma \prime }} \right.} \hfill \\ {} \hfill & {} \hfill & {\left. { + \psi _{{\bf{k}},\sigma }^\dagger a_{ - {\bf{q}},\lambda }^\dagger \psi _{{\bf{k}} - {\bf{q}},\sigma \prime }} \right),} \hfill \end{array}$$
(12)

where the term (σσσ′ · eλ) arises due to the helical spin texture and the spin-momentum locking of TI surface states. For linearly polarized perpendicular incident photons, \({\bf{e}}_1 = \widehat {\bf{e}}_x\) and \({\bf{e}}_2 = \widehat {\bf{e}}_y\). Thus, the spin-flip matrices σx and σy take place in Hel-ph. This verifies the spin-flip mechanism illustrated by Fig. 1b in a microscopic perspective. Moreover, since \(c \gg v_{\mathrm{F}}\), the conservation of energy and momentum requires \(\left| {\bf{k}} \right| \gg \left| q \right|\) in Hel-ph, and therefore ensures the direct interband excitation with k − q ≃ k. Furthermore, the photon fields are Gaussian with the Hamiltonian \(H_{{\mathrm{ph}}} = \mathop {\sum}\nolimits_{{\bf{q}}\lambda } {\kern 1pt} \omega _{\bf{q}}\left( {a_{{\bf{q}},\lambda }^\dagger a_{{\bf{q}},\lambda } + 1{\mathrm{/}}2} \right)\), one can therefore exactly integrate them out. This procedure leads to the effective action Sint describing the renormalization from photons:

$$\begin{array}{*{20}{l}} {S_{{\mathrm{int}}}} \hfill & = \hfill & { - {\int} {\kern 1pt} dt\mathop {\sum}\limits_{{\bf{kk}}\prime {\bf{q}}} \mathop {\sum}\limits_{\alpha \beta \rho \gamma \lambda } \frac{{2\pi {\mathrm{e}}^2v_{\mathrm{F}}^2}}{{\omega _{\bf{q}}}}\left( {{\boldsymbol{\sigma }}_{\alpha \beta } \cdot {\bf{e}}_\lambda } \right)\left( {{\boldsymbol{\sigma }}_{\rho \gamma } \cdot {\bf{e}}_\lambda } \right)} \hfill \\ {} \hfill & {} \hfill & { \times \frac{1}{{{\mathrm{i}}\partial _t - \omega _{\bf{q}}}}\psi _{{\bf{k}},\alpha }^\dagger \psi _{{\bf{k}},\beta }\psi _{{\bf{k}}\prime ,\rho }^\dagger \psi _{{\bf{k}}\prime ,\gamma }.} \hfill \end{array}$$
(13)

Where \(\mathop {\sum}\nolimits_\lambda \left( {{\boldsymbol{\sigma }}_{\alpha \beta } \cdot {\bf{e}}_\lambda } \right)\left( {{\boldsymbol{\sigma }}_{\rho \gamma } \cdot {\bf{e}}_\lambda } \right) = \sigma _{\alpha \beta }^x\sigma _{\rho \gamma }^x + \sigma _{\alpha \beta }^y\sigma _{\rho \gamma }^y\), inserting which it is found that Sint consists of an equal-time renormalization term that produces the effective interaction between electrons,

$$H_{{\mathrm{int}}} = - V_{{\mathrm{eff}}}\mathop {\sum}\limits_{{\bf{kk}}\prime \sigma } {\kern 1pt} \psi _{{\bf{k}},\sigma }^\dagger \psi _{{\bf{k}},\overline \sigma }\psi _{{\bf{k}}\prime ,\overline \sigma }^\dagger \psi _{{\bf{k}}\prime ,\sigma },$$
(14)

where \(V_{{\mathrm{eff}}} = 4\pi e^2v_{\mathrm{F}}^2n_{{\mathrm{ph}}}{\mathrm{/}}\bar \omega ^2\) and nph is the number density of photons that interact with the electrons, which is proportional to the light intensity I. We have utilized the fact that all the wave vectors q satisfy \({\bf{q}} = q\widehat {\bf{e}}_z = \bar \omega \widehat {\bf{e}}_z{\mathrm{/}}c\) for photons with the frequency \(\bar \omega\). The first two operators in Hint, \(\psi _{{\bf{k}},\sigma }^\dagger \psi _{{\bf{k}},\overline \sigma }\), are generated by the optical interband absorption process with spin-flip depicted by Fig. 1b. Similarly, the last two operators, \(\psi _{{\bf{k}}\prime ,\bar \sigma }^\dagger \psi _{{\bf{k}}\prime ,\sigma }\), are originated from the interband relaxation process with spin-flip. After a unitary transformation to the chiral basis, Eq. (14) is exactly cast into:

$$H_{{\mathrm{int}}} = - \frac{{V_{{\mathrm{eff}}}}}{2}\mathop {\sum}\limits_{{\bf{k}},{\bf{k}}\prime ,n} \left[ {\hat kc_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}} \right] \cdot \left[ {\hat k\prime c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}} \right].$$
(15)

Interestingly, Hint enjoys the p-wave factor \(\hat k \cdot \hat k\prime\), exactly the same as the p-wave interaction component reduced from the Hubbard model, \(H_{{\mathrm{eff}}}^{{\mathrm{(1)}}}\). The interband optical processes automatically select the p-wave channel due to the spin-flip scatterings (Fig. 1b). More importantly, Hint has a strength Veff tunable by external field parameters, \(\bar \omega\) and I. According to the mean-field study above and the calculated phase diagram in Fig. 4, it is known that the tunable external field can renormalize the Dirac cone state and drive it into the proposed p + ip-wave excitonic insulator. The resultant state displays the predicted fractional charges associated with vortex excitations, the spontaneous in-plane magnetization and thus the anisotropic transport of excitons on the TI surface.

Discussion

Excitonic insulators require strong interactions since the topological surface states become unstable only when the interaction is as large as the ultraviolet cutoff (band gap) as shown in Fig. 4b. Therefore, we argue that topological Kondo insulators (TKIs) are good candidates for excitonic instability on TI surface. For instance, in the case of SmB6, which is the archetypal TKI, the renormalized Coulomb interaction (U ~ 1 eV) is much larger than the direct band gap (ΛvF ~ 20 meV). For these parameters, our mean-field theory predicts an excitonic gap of order Δs/c ≃ 2–10 meV. Indeed, magnetism and hysteresis in magnetotransport have been observed in SmB652. Whether these experiments indeed observe an excitonic insulator require future work. We would like to note that our theory is not related with the recent works on excitonic contributions to quantum oscillations53 or the intervalley excitonic instabilities54 in SmB6.

In addition to the short-range interaction, we also considered a long-range Coulomb interaction on the TI surface and make generalization to finite temperature55. It is found that the critical interaction Vc for BCS exciton condensation is significantly reduced for long-range interaction. At finite temperature, taking into account the thermal fluctuation, the strict long-range order is absent, the Kosterliz–Thouless transition temperature TKT can be calculated by evaluating the phase stiffness of the exciton order based on the mean-field calculations56. Experimentally, the driven TI surface offers a rare platform for realization of new optical devices and facilitates the study of possible collective instabilities of Dirac electrons. Experiments measuring bulk metallic TI materials have obtained the lifetime of the excited states to be around a few picoseconds57,58,59, which is significantly larger than than that in graphene. Moreover, a remarkably long lifetime exceeding 4 μs has been observed by experiment for the surface states in bulk insulating Bi2Te2Se60. These indicate promising observation of a transient or quasi-equilibrium excitonic insulator59,61. Besides, the surface has a reduced dielectric constant which can be tuned even smaller by doping or gating62, and this possibly makes the long-range interaction more effective.

To summarize, we proposed a new type of topological p + ip excitonic insulator that achieves the parity anomaly and is characterized by the nontrivial chiral Chern number. We provided a concrete example and discussed its possible realization in strongly correlated TI surface. Interestingly enough, similar to the TIs due to band inversion, the predicted topological behaviors, i.e., the charge fractionalization and the emergent magnetization of ground state, are robust and immune to continuous deformation of the Hamiltonian, as long as the exciton gap remains unclosed. This suggests that realizations of topological p-wave excitonic insulators are likely in many other physical systems. For example, the spectrum under study can be continuously deformed from the Dirac cone to a quadratic band crossing point (QBCP)63. This indicates that the instability is also possible and may be more feasible in interacting 2D electron gas with spin-momentum locking and QBCP. In this sense, the present results are general and may provide new physical settings for the realization of quantum anomalies in condensed matter system, and may stimulate different directions to search for more exotic topological state of matters. We also note that a laser-induced non-equilibrium version of our proposal has been experimentally observed recently in Bi2Se355, where a millimeter-long transport distance of excitons up to 40 K is reported, supporting an excitonic condensation state formed on the TI surface. Our study on the p + ip excitonic insulator stabilized by the optical pumping further predicts an anisotropic transport of excitons, which we believe would be of great interest for future experimental investigations.

Methods

The zero-energy BdG-type equations

From Eq. (2) in the main text, the zero-energy eigen-equations can be written down as:

$$\begin{array}{l}v_{\mathrm{F}}ku_{\bf{k}} - {\mathrm{\Delta }}({\bf{r}})e^{ - i\theta }v_{\bf{k}} = 0,\\ - e^{{\mathrm{i}}\theta }{\mathrm{\Delta }}^ \ast ({\bf{r}})u_{\mathbf{k}} - v_{\mathrm{F}}kv_{\bf{k}} = 0.\end{array}$$

Here, we assume a real-space dependence of the order parameter Δ(r). With a vortex,we have Δ(r) = Δc(r)eiα, where α is the polar angle of r in real space. To simplify the eigen-equations, one can multiply a factor eiθ from the right in the first equation, and a factor e−iθ from the left in the second equation. Then, a Fourier transformation yields,

$$\begin{array}{l}\left( { - {\mathrm{i}}\partial _x + \partial _y} \right)u_{\bf{r}} - \frac{1}{{v_{\mathrm{F}}}}{\mathrm{\Delta }}({\bf{r}})v_{\bf{r}} = 0,\\ \frac{1}{{v_{\mathrm{F}}}}{\mathrm{\Delta }}^ \ast ({\bf{r}})u_{\bf{r}} + \left( { - i\partial _x - \partial _y} \right)v_{\bf{r}} = 0.\end{array}$$

After transformation to the polar coordinate, we arrive at the zero-energy BdG-like equations in real space.

$$\begin{array}{l}e^{{\mathrm{i}}\alpha }\left( { - {\mathrm{i}}\partial _r + r^{ - 1}\partial _\alpha } \right)u({\bf{r}}) - \frac{1}{{v_{\mathrm{F}}}}{\mathrm{\Delta }}_{\mathrm{c}}({\bf{r}})v({\bf{r}}) = 0,\\ e^{ - {\mathrm{i}}\alpha }\left( { - {\mathrm{i}}\partial _r - r^{ - 1}\partial _\alpha } \right)v({\bf{r}}) + \frac{1}{{v_{\mathrm{F}}}}{\mathrm{\Delta }}_{\mathrm{c}}^ \ast ({\bf{r}})u({\bf{r}}) = 0.\end{array}$$

Effective gauge field theory

In the second quantized form, the effective Hamiltonian of the p + ip excitonic insulator reads as

$$H = \mathop {\sum}\limits_{\bf{k}} {\kern 1pt} C_{\bf{k}}^\dagger \left[ {v_{\mathrm{F}}k\tau ^z - {\mathrm{\Delta }}_{\mathrm{c}}e^{ - {\mathrm{i}}\theta }\tau ^ + - {\mathrm{\Delta }}_{\mathrm{c}}^ \ast e^{{\mathrm{i}}\theta }\tau ^ - } \right]C_{\bf{k}}.$$

Formally, one can make transformation \(C_{\bf{k}} \to {\cal C}_{\bf{k}} = C_{\bf{k}}{\mathrm{/}}\sqrt k\), while keeping the total Hamiltonian unchanged. The action in the functional representation is then cast into:

$$S = {\int} {\kern 1pt} d\tau d{\bf{k}}\overline {\cal C} _{\bf{k}}\left[ {{\mathrm{i}}\partial _0 + \tau ^zv_{\mathrm{F}}k^2 + \Delta _{\mathrm{c}}k\prime \gamma ^ - + {\mathrm{\Delta }}_c^ \ast k^{\prime \ast }\tau ^ + } \right]{\cal C}_{\bf{k}},$$

where we have defined k′ = kx − iky and introduced the Dirac matrices in 2D, γ0 = τ3, γ1 = iτ1, γ2 = iτ2, and γ± = (γ2 ± iγ1)/2. The above action formally describes a 2D Dirac fermion with a mass vFk2. Since the sign of mass determines the topology of 2D Dirac systems, and vFk2 ≥ 0 for all k, one can continuously deform the mass into a constant m while keeping the topological property unchanged. With a U(1) gauge transformation of the Grassmann field, the gauge invariance requires the minimal coupling \(\tilde k\prime = k_x - A_x - {\mathrm{i}}\left( {k_y - A_y} \right)\). With further including the vortex degrees of freedom in the order parameter, the Lagrangian can be written as,

$${\cal L} = \overline {\cal C} _{\bf{k}}\left( {\gamma ^0\widetilde {{\mathrm{i}}\partial _0} + \gamma ^ + e^{{\mathrm{i}}\chi }\tilde k\prime + \gamma ^ - \tilde k\prime e^{ - {\mathrm{i}}\chi } + m} \right){\cal C}_{\bf{k}},$$

where χ is the vortex phase which goes from 0 to 2π when one completes a closed loop in real space. Without losing generality, we set |Δc| = 1. The behavior of vortex degrees of freedom can be extracted from the above Lagrangian after integrating out the Grassmann fields. To the second order expansion, the Chern–Simons action in Eq. (7) is obtained.

Mean-field analysis of the Hubbard interaction

Let us denote the four k vectors in the Hubbard interaction Eq. (9) as k1 = k, k2 = k′, k3 = k′ − q and k4 = k + q, respectively. For zero temperature, only those degrees of freedom near μ become dominant. Due to conservation of momentum in 2D, only the forward scattering is least irrelevant for repulsive interactions in the low-energy window in the renormalization group sense, which consists of two scattering channels: k1 = k3 (k2 = k4) and k1 = k4 (k2 = k3). Then we make a unitary transformation into the chiral basis. For the scattering channel k1 = k3 (k2 = k4), after expansion, there are 16 different combinations in total. The requirement of conservation of energy greatly simplifies the expression, leaving us six terms that read as: \(( {\hat k\prime \cdot \hat k} )c_{k\prime , + }^\dagger c_{k, + }^\dagger c_{k\prime , + }c_{k, + }\), \(- ( {\hat k\prime \cdot \hat k} )c_{k\prime , + }^\dagger c_{k, - }^\dagger c_{k\prime , + }c_{k, - }\), \(( {\hat k\prime \cdot \hat k} )c_{k\prime , + }^\dagger c_{k, - }^\dagger c_{k\prime , - }c_{k, + }\), \(( {\hat k\prime \cdot \hat k} )c_{k\prime , - }^\dagger c_{k, + }^\dagger c_{k\prime , + }c_{k, - }\), \(- ( {\hat k\prime \cdot \hat k} )c_{k\prime , - }^\dagger c_{k, + }^\dagger c_{k\prime , - }c_{k, + }\), \(( {\hat k\prime \cdot \hat k} )c_{k\prime , - }^\dagger c_{k, - }^\dagger c_{k\prime , - }c_{k, - }\). Only those scattering terms that involve two bands can open a gap in the Dirac state, while the scatterings within one band only renormalize the Fermi velocity but do not contribute to any instabilities. Moreover, the translation invariant mean-field treatment to the second and fifth term contributes to a self-energy that only renormalizes the chemical potential. With these consideration, the forward scattering channel k1 = k3 (k2 = k4) produces the following reduced interaction,

$$\begin{array}{*{20}{l}} {H_{eff}^{(1)} = - \frac{V}{4}\mathop {\sum}\limits_{{\bf{k}},{\bf{k}}\prime ,n} \left( {\hat k \cdot \hat k\prime } \right)c_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}.} \hfill \end{array}$$
(16)

For the scattering channel k1 = k4 (k2 = k3), following the same approach above, one obtains the following reduced interaction,

$$\begin{array}{*{20}{l}} {H_{eff}^{(2)} = - \frac{V}{4}\mathop {\sum}\limits_{{\bf{k}},{\bf{k}}\prime ,n} c_{{\bf{k}},n}^\dagger c_{{\bf{k}}, - n}c_{{\bf{k}}\prime , - n}^\dagger c_{{\bf{k}}\prime ,n}.} \hfill \end{array}$$
(17)

Both \(H_{{\mathrm{eff}}}^{{\mathrm{(1)}}}\) and \(H_{{\mathrm{eff}}}^{{\mathrm{(2)}}}\), p-wave and s-wave type respectively, are reduced from the original Hubbard interaction, which are the least irrelevant channels that can possibly gap out the Dirac cone.