Introduction

The recently discovered topological insulators (TIs) are classified accordingly to the symmetries of its crystal lattice and/or time-reversal symmetry, which yields its topologically protected edge or surface states1,2. Particularly, the class of topological crystalline insulators (TCIs) have Dirac-like bands protected by space group symmetries3,4. The first material predicted to be a TCI was SnTe5, which was promptly verified experimentally6, and followed by other IV-VI compounds7,8,9,10. The monolayers of IV-VI materials have been predicted to be two-dimensional (2D) TCIs11,12. Both the bulk and monolayers of SnTe are classified by their mirror Chern number |n M | = 2, yielding an even number of Dirac cones in the Brillouin zone. In between these two limits, the stacking of monolayers show an intriguing nonmonotonic evolution of the band gap for stackings oriented along the [001]12,13,14,15, [111]16,17,18,19,20 and [110]21,22 directions. Particularly, odd and even stackings of [001] SnTe belong to distinct space groups: the first is symmorphic, while the latter is nonsymmorphic; i.e. some symmetry operations require translations by a fraction of the unit cell. It is known that nonsymmorphic groups yield an extra degeneracy with respect to its symmorphic counterpart23,24. Moreover, from the nonsymmorphic space groups arise newly proposed exotic topological properties25,26,27,28,29,30,31,32.

In this paper the distinct topological phases of odd and even stackings of SnTe monolayers are explained by the contrast between their symmorphic and nonsymmorphic crystal symmetries. First, we analyze first principles [density functional theory (DFT)33,34,35,36, see Methods] band structures as a function of the number N of stacked layers. Interestingly, we find that the symmorphic stacking presents two topological phase transitions, while no phase transition is seen for the nonsymmorphic stackings. To understand this disparity, we develop low energy models for the surface states of both symmorphic (odd N) and nonsymmorphic (even N) stackings. We show that, due to finite-size effects, topological surface states located at opposite surfaces hybridize differently in the symmorphic and nonsymmorphic cases. Particularly, the nonsymmorphism of the even stackings enforces an extra degeneracy, thus avoiding a band inversion. Ultimately, this protects the system from a topological phase transition. Our results provide a clear picture of finite size effects, and its respective space group, on the hybridization of surface states. Although we focus here on the [001] stacking of SnTe monolayers, equivalent considerations shall hold for other directions, and as well for any material that may present distinct space groups for different stacking heights.

Results and Discussions

DFT analysis

Bulk SnTe has a rock-salt lattice and shows a small band gap at the L point. There, the conduction and valence bands are inverted due to the spin-orbit (SO) interaction, as shown in Fig. 1(a,b), yielding its 3D TCI regime characterized by the mirror Chern number |n M | = 2. Around the L point, the effective Hamiltonian37 obeys the \(P\bar{{\rm{3}}}1{\rm{m}}\,({\rm{or}}\,{{\rm{D}}}_{3{\rm{d}}}^{1})\) symmorphic space group. In the opposite limit, a single monolayer of SnTe is a 2D TCI4,5,24,37. In this case, without SO the bands are already inverted, but it is metallic around the \(\overline{{\rm{X}}}\) point of the 2D Brillouin zone, Fig. 1(c). The SO coupling opens the gap shown in Fig. 1(d), yielding the topological regime. The band structure around \(\overline{{\rm{X}}}\) obeys the symmorphic space group Pmmm \(({\rm{or}}\,{{\rm{D}}}_{2{\rm{h}}}^{1})\)24. The spinful double group of both bulk and monolayer admits doubly degenerate irreducible representations (IRREPs) with odd and even inversion characters38, which can be characterized by the orbital projections \({{\rm{\Phi }}}_{-}=[{\phi }_{P}^{Sn}+{\phi }_{S}^{Te}]\) and \({{\rm{\Phi }}}_{+}=[{\phi }_{P}^{Te}+{\phi }_{S}^{Sn}]\), respectively39. These projections label the color code in Fig. 1(a–d) and illustrates the band inversions.

Figure 1
figure 1

Band inversions for (a,b) bulk and (c,d) monolayer SnTe. Blue and red dots label are projections of \({{\rm{\Phi }}}_{+}=[{\phi }_{P}^{Te}+{\phi }_{S}^{Sn}]\) and \({{\rm{\Phi }}}_{-}=[{\phi }_{P}^{Sn}+{\phi }_{S}^{Te}]\), respectively. (e) 3D bulk Brillouin zone and its projection into the (001) surface. (f) Symmorphic (odd N) and nonsymmorphic (even N) stackings of SnTe monolayers. The blue arrows illustrate the mirror operation along z for the symmorphic (M z ) and nonsymmorphic (\(\{{M}_{z},\overrightarrow{\ell }\}\)) cases.

Full size image

Finite Stackings

At the intermediate nanosize regime, composed by a finite number N of SnTe monolayers stacked along [001], Fig. 1(f), finite-size effects arise distinctively for odd or even N. In both cases, the stacking projects a pair of distinct L points into \(\overline{{\rm{X}}}\), Fig. 1(e). For large N, the distinction between odd or even number of layers is small. Each surface shows a pair of Dirac bands, which are highlighted by the colors in Fig. 2(a,b). However, as N is reduced, topological states localized at opposite surfaces hybridize differently for the odd/even N (symmorphic/nonsymmorphic). The hybridization becomes relevant for stackings \(\lesssim 4\lambda \), where \(\lambda \sim \hslash {v}_{F}/{E}_{g}\) is the penetration length of the surface states. From Fig. 1(b) we estimate the bulk gap E g  ≈ 0.11 eV and Fermi velocity ℏv F  ≈ 2 eVÅ, which gives λ ≈ 18 Å or N ≈ 6 layers. Therefore, the hybridization becomes relevant for \(N\lesssim 24\), which matches the gap evolution in Fig. 3(a,b).

Figure 2
figure 2

Band structure for SnTe stackings with N layers and k around the \(\overline{{\rm{X}}}\) TRIM along the \(\overline{{\rm{\Gamma }}}\) − \(\overline{{\rm{X}}}\) and \(\overline{{\rm{X}}}\) − \(\overline{{\rm{M}}}\) directions [see Fig. 1(e)]. (A) The top row show the results for odd N, which corresponds to the symmorphic lattice symmetry, while (B) in the bottom row we show the even N nonsymmorphic case. The blue bands are colored to guide the eye throughout the hybridization of the surface states as N is reduced. The red arrows on the N = 27 and N = 28 panels illustrate the extra degeneracy imposed by the nonsymmorphic symmetry.

Full size image
Figure 3
figure 3

(a) Total energy band gap ΔE and (b) the local gap ΔE X at \(\overline{{\rm{X}}}\) [see inset of panel (a)] as a function of the number of layers N. (c) and (d) Inversion eigenvalues of the highest occupied (solid circles) and lowest unoccupied (empty circles) states from the DFT data for the symmorphic and nonsymmorphic stackings, respectively. In (c) the mirror Chern number n M is indicated by the shade colors. For the symmorphic case two phase transitions are apparent, while for the nonsymmorphic stacking the fourfold degeneracy at \(\overline{{\rm{X}}}\) avoids any phase transitions.

Full size image

Next we first present these gaps and the inversion eigenvalues obtained from the DFT simulations for the symmorphic and nonsymmorphic cases for various N. Later on we introduce a group theory model for the surface states, which allow us to understand how the states from opposite surfaces hybridize as N is reduced. We find that the nonsymmorphic symmetry of even N stackings imposes an extra degeneracy at \(\overline{{\rm{X}}}\), which protects the band crossings for all even N; e.g. see arrows in Fig. 2(b) for N = 28. In contrast, for odd N these bands hybridize as N is reduced; e.g. see arrows in Fig. 2(a) for N = 27. Within the model we will see that these distinct behaviors explain the phase transitions seen for the symmorphic case, and its absence for the nonsymmorphic stacking.

Symmorphic stacking

The ab-initio band structure for odd stacking ranging from N = 5 to 39 layers is shown in Fig. 2(a). For large N, the Dirac crossings that characterizes the TCI regime are noticeable (highlighted in blue). As N is reduced, the hybridization between the top and bottom surface states take place and the bands become massive. To analyze this process, we plot the total gap ΔE and the local gap ΔE X at \(\overline{{\rm{X}}}\) in Fig. 3(a,b). For the symmorphic case (solid green circles) two phase transitions are striking. That is, ΔE X changes sign from N = 1 to 3, and from N = 11 to 13, which is characterized by the inversion eigenvalue in Fig. 3(c). For N = 1 the monolayer is a 2D TCI, in agreement with previous results4,5,24,37.

To understand the transition from N = 11 to 13, the density of states for the surface bands around the Dirac crossing off the \(\overline{{\rm{X}}}\) point are projected along the [001] layers, as shown in Fig. 4. Interestingly, in all cases (symmorphic and nonsymmorphic for all N) it shows that the densities are localized at the top and bottom surfaces, within the length λ (≈6 layers) as discussed previously. Each surface state belongs to an even or odd inversion IRREP of the Pmmm \(({\rm{or}}\,{{\rm{D}}}_{2{\rm{h}}}^{1})\) space group38. These surface states do not change their character when passing from N <11 to> 13. In fact, the sole effect of reducing N is to increase the overlap and hybridization between top and bottom states. Therefore, for any \(N\lesssim 6\) the staking maintains its 3D TCI character defined by its bulk invariants. Consequently, the band inversions observed for odd stacking shown in Fig. 3(c) must be of a lower order, i.e. a 2D topological phase transition. An illustrative model40 for the edge states is shown in the Supplementary Information. This 2D topological phase transition, can be better characterized by the mirror Chern number computed using the surface state bands. Using the effective model described below, we calculate this mirror Chern number n M as a function of the gap sign (ΔE X ). We find |n M | = 2 for ΔE X  < 0 and n M  = 0 for ΔE X  > 0. This is shown by the color code in Fig. 3(c). For the monolayer case (N = 1) it is known that |n M | = 212,24.

Figure 4
figure 4

Projected density of sates (PDOS), at each layer along the [001] direction, for the surface states for different N stackings. States localized at the bottom (top) layer are drawn with dashed (dot-dashed) lines. Along the lines, each circle corresponds to a layer, which are normalized between the bottom and top layers. In all cases, the PDOS show that the densities are localized within \(\lesssim 6\) layers from the surfaces.

Full size image

Nonsymmorphic stacking

For nonsymmorphic stackings we have a different picture. Both ΔE and ΔE X show smooth monotonic and asymptotic behaviors in Fig. 3(a,b) (open pink circles). There is no phase transition. The band structures in Fig. 2(b) show clear hybridized Dirac bands for N > 6. Notice that the crossings at \(\overline{{\rm{X}}}\) remain closed for all even N, in contrast with the hybridization seen in Fig. 2(a) for odd N (c.f. arrows for N = 27 and 28). For large N the nonsymmorphic stacking is indistinguishable from the symmorphic one, since finite size effects become negligible as the surface states are spatially split [c.f. N = 39 in Fig. 2(a), and N = 40 in Fig. 2(b)], yielding gapless Dirac crossings. As we will see later on, the lack of phase transitions in this case is due to the degeneracies imposed by the nonsymmorphic symmetry. Its space group is the Pmmn \(({\rm{or}}\,{{\rm{D}}}_{2{\rm{h}}}^{13})\), and due to the Bloch phase at \(\overline{{\rm{X}}}\), the only allowed double group IRREPs are degenerate due to time-reversal symmetry24,38. This yields a fourfold degeneracy with pairs of states with even and odd inversion eigenvalues. This is illustrated in Fig. 3(d). Moreover, from the PDOS shown in Fig. 4 we observe that the 3D topological character is always present for nonsymmorphic stacking, independent on the number of layers. Since there is no phase transition, its topological nature is governed by the precursor bulk 3D TCI with mirror Chern number |n M | = 2.

Effective model

Next, we develop low energy models for symmorphic and nonsymmorphic stackings, which will help us to understand the phase transitions presented above. Indeed we find that only two transitions should be expected for the symmorphic case, in agreement with ab-initio results.

Let us start by defining a basis as the surface states from a semi-infinite stacking limit. Then, we allow it to hybridize with the solutions from the opposite surface due to finite symmorphic or nonsymmorphic stackings. In all cases we consider the wave-functions around \(\overline{{\rm{X}}}\) to be composed by s, p x and p z atomic orbitals of Sn and Te, as obtained from the DFT first principles calculations.

Semi-infinite stacking

Around \(\overline{{\rm{X}}}\), the semi-infinite stacking obeys the symmorphic Pmm2 (or C2v) space group. Starting with the spinless case, we find by inspection that the surface states belong to the A1 and B1 IRREPs of C2v. Therefore we label the spinful eigenkets for our basis as {A1; ↑, A1; ↓, B1; ↑, B1; ↓}, where {↑, ↓} refers to the spin along z. The C2v group is composed by a rotation C2(z) = iτ3⊗σ z around the z axis, and mirrors M x  = −iτ3 ⊗ σ x and M y  = −iτ0 ⊗ σ y that reflect x → − x, and y → − y, respectively. Here the τ i (i = 0, 1, 2, 3) are su(2) matrices acting on the A_1/B_1 orbitals and σ j (j = 0, x, y, z) are the su(2) spin matrices. The time-reversal symmetry operator is \({\mathscr{T}}=i{\tau }_{0}\otimes {\sigma }_{y}{\mathscr{K}}\), where \({\mathscr{K}}\) is the complex conjugation operator. Up to linear order in k x and k y the effective Hamiltonian that commutes with these symmetries is

$${H}_{\infty }=(\begin{array}{cccc}{e}_{0}+{{\rm{\Delta }}}_{0} & i{\boldsymbol{\alpha }}\cdot {{\bf{k}}}_{-} & i{\boldsymbol{\gamma }}\cdot {{\bf{k}}}_{-} & {{\rm{\Delta }}}_{1}\\ -i{\boldsymbol{\alpha }}\cdot {{\bf{k}}}_{+} & {e}_{0}+{{\rm{\Delta }}}_{0} & -{{\rm{\Delta }}}_{1} & i{\boldsymbol{\gamma }}\cdot {{\bf{k}}}_{+}\\ -i{\boldsymbol{\gamma }}\cdot {{\bf{k}}}_{+} & -{{\rm{\Delta }}}_{1} & {e}_{0}-{{\rm{\Delta }}}_{0} & i{\boldsymbol{\beta }}\cdot {{\bf{k}}}_{-}\\ {{\rm{\Delta }}}_{1} & -i{\boldsymbol{\gamma }}\cdot {{\bf{k}}}_{-} & -i{\boldsymbol{\beta }}\cdot {{\bf{k}}}_{+} & {e}_{0}-{{\rm{\Delta }}}_{0}\end{array}),$$
(1)

where k± = (k x , ±ik y ). The coefficients α = (α x , α y ) and β = (β x , β y ) are in-plane (σ x and σ y ) SOCs that define the anisotropic Fermi velocity near k = 0 for the Dirac cones at energies \(E={e}_{0}\pm \sqrt{{{\rm{\Delta }}}_{0}^{2}+{{\rm{\Delta }}}_{1}^{2}}\), while γ = (γ x , γ y ) couples orbitals A1 to B1 with the same spin at finite k.

We fit this Hamiltonian to the DFT data for a large stacking (N = 59 or 60), which gives the parameters: e0 = 0.0065 eV, Δ0 = 0.043 eV, Δ1 ≈ 0, α x  = 1.6 eV, α y  = 3.3 eV, β x  = 1.3 eV, β y  = 2.0 eV, γ x  = 0.4, γ y  = 1.5 eV. The resulting band structure is shown in Fig. 5. For small k near \(\overline{{\rm{X}}}\) the agreement is patent, while for large k parabolic terms would be necessary to improve it. Particularly, along the \(\overline{{\rm{X}}}-\overline{{\rm{M}}}\) direction one branch enters the conduction band quickly and can be disregarded, as we do not account for its coupling to the bulk in the effective model. More importantly, the spin projections (color coded) match well the DFT results. Along \(\overline{{\rm{\Gamma }}}-\overline{{\rm{X}}}\), both the mirror M y and the spin σ y are preserved, allowing us to label the states with its eigenvalues M y  = ±i, which match the σ y projections since [M y ,σ y ] = 0. Along \(\overline{{\rm{X}}}-\overline{{\rm{M}}}\) the mirror M x is preserved, while [H∞, σ x ] ≠ 0, thus we label the states with the eigenvalues of M x  = ±i, while the spin σ x projections are mixed (color code) at the anti-crossing induced by the γ y term.

Figure 5
figure 5

(a) Effective model for a semi-infinite stacking fitted to (b) the DFT data for a N = 60 monolayer stacking. Along the \(\overline{{\rm{\Gamma }}}-\overline{{\rm{X}}}\) direction, both the mirror M y  = ±i (solid/dashed) and spin σ y  = {↑ y , ↓ y } (green/yellow) along y are equivalent. On the other hand, along \(\overline{{\rm{X}}}-\overline{{\rm{M}}}\) only the mirror M x  = ±i (solid/dashed) is a good quantum number, while the spin σ x  = {↑ x , ↓ x } projection is labeled by the color gradient (from green to yellow). The agreement with the DFT data from panel (b) is clear.

Full size image

Finite stackings and phase transitions

The space groups of the symmorphic (\({{\rm{D}}}_{2{\rm{h}}}^{1}\)) and nonsymmorphic (\({{\rm{D}}}_{2{\rm{h}}}^{13}\)) stackings share the C2v as a common subgroup. Therefore, to model the finite stackings, we consider that the top (t) and bottom (b) surface states are approximately given by the semi-infinite solutions above. That is, our basis is now labeled by \(\{{A}_{1}^{t};\uparrow ,{A}_{1}^{t};\downarrow ,{B}_{1}^{t};\uparrow ,{B}_{1}^{t};\downarrow ,{A}_{1}^{b};\uparrow ,{A}_{1}^{b};\downarrow ,{B}_{1}^{b};\uparrow ,{B}_{1}^{b};\downarrow \}\). Introducing s j (j = 0, 1, 2, 3) as su(2) matrices acting on the top/bottom subspace, the C2v operations are now C2(z) = is0 ⊗ τ3 ⊗ σ z , M x  = −is0 ⊗ τ3 ⊗ σ x and M y  = −is0 ⊗ τ0 ⊗ σ y . For the odd symmorphic stacking, the \({{\rm{D}}}_{2{\rm{h}}}^{1}\) also contains the operations that connect the top and bottom surfaces: inversion I, mirror M z , and rotations C2(x) and C2(y). On the other hand, for an even number of monolayers, the \({{\rm{D}}}_{2{\rm{h}}}^{13}\) space group is given by the C2v operations, plus the nonsymmorphic counterparts of I, M z , C2(x) and C2(y). Using Seitz notation, these are \(\{I,\overrightarrow{\ell }\}\), \(\{{M}_{z},\overrightarrow{\ell }\}\), \(\{{C}_{2}(x),\overrightarrow{\ell }\}\), \(\{{C}_{2}(y),\overrightarrow{\ell }\}\), where \(\overrightarrow{\ell }\) is a nonprimitive translation by half a unit cell [see Fig. 1(f)]. Since M z  = I ⋅ C2(z), C2(x) = I ⋅ M x , and C2(y) = I ⋅ M y (and equivalent expressions for the nonsymmorphic operations), it is sufficient to define how the inversion operator acts on our basis in both cases. For the odd stacking, the inversion transforms \({A}_{1}^{t}\rightleftharpoons {A}_{1}^{b}\) and \({B}_{1}^{t}\rightleftharpoons {B}_{1}^{b}\), thus I = s1 ⊗ τ0 ⊗ σ0. In contrast, due to the nonprimitive translation, the \(\{I,\overrightarrow{\ell }\}\) transforms \({A}_{1}^{t}\rightleftharpoons {B}_{1}^{b}\) and \({A}_{1}^{b}\rightleftharpoons {B}_{1}^{t}\), which yields \(\{I,\overrightarrow{\ell }\}={s}_{1}\otimes {\tau }_{1}\otimes {\sigma }_{0}\).

From the symmetries above, we construct the effective Hamiltonians up to linear order in k. Its full form is shown in the Supplementary Information. However, to understand the topological phase transitions seen in the DFT results, it is sufficient to analyze the \(\overline{{\rm{\Gamma }}}-\overline{{\rm{X}}}\) direction (i.e. k y  = 0). Along this direction M y is a good quantum number, therefore we can block diagonalize the Hamiltonian into M y  = ±i subspaces. For M y  = + i, the dominant terms are

$${H}_{+i}^{(S)}\approx (\begin{array}{cccc}{{\rm{\Delta }}}_{0}-{\alpha }_{x}{k}_{x} & {\delta }_{0} & 0 & i{\delta }_{2}\\ {\delta }_{0} & {{\rm{\Delta }}}_{0}+{\alpha }_{x}{k}_{x} & i{\delta }_{2} & 0\\ 0 & -i{\delta }_{2} & -{{\rm{\Delta }}}_{0}-{\beta }_{x}{k}_{x} & {\delta }_{1}\\ -i{\delta }_{2} & 0 & {\delta }_{1} & -{{\rm{\Delta }}}_{0}+{\beta }_{x}{k}_{x}\end{array}),$$
(2)
$${H}_{+i}^{(NS)}\approx (\begin{array}{cccc}{{\rm{\Delta }}}_{0}-{\alpha }_{x}{k}_{x} & 0 & 0 & {\delta }_{3}\\ 0 & {{\rm{\Delta }}}_{0}+{\alpha }_{x}{k}_{x} & {\delta }_{3} & 0\\ 0 & {\delta }_{3} & -{{\rm{\Delta }}}_{0}-{\beta }_{x}{k}_{x} & 0\\ {\delta }_{3} & 0 & 0 & -{{\rm{\Delta }}}_{0}+{\beta }_{x}{k}_{x}\end{array}),$$
(3)

for the symmorphic [\({H}_{+i}^{(S)}\)] and nonsymmorphic [\({H}_{+i}^{(NS)}\)] cases. The effective Hamiltonians for M y  = −i are obtained simply replacing α x → − α x and β x → − β x . In the semi-infinite limit, α x , β x and Δ0 are the same as in H∞, but deviations should be expected for a small N. The new terms δ j (j = 0, 1, 2, 3) represent the hybridization between top and bottom surface states, which couple the linear dispersions due to the finite size of the system. As the number of layers N is reduced, these hybridizations induce anticrossings δ j , which increase as indicated by the arrows in Fig. 6(a,b). Other terms allowed by symmetry only contribute to the fine tuning of the spectrum (see Supplementary Information). Three-dimensional visualizations of spectrums E(k x , k y ) are shown in Fig. 6(c) using the parameters from the semi-infinite model above, and δ0 = δ1 = δ2 = 5 meV for the symmorphic stacking, and δ3 = 5 meV for the nonsymmorphic case. A direct comparison show that the only qualitative difference between the dispersions are the hybridization at \(\overline{{\rm{X}}}\). For δ1 ≈ δ0, the gaps at \(\overline{{\rm{X}}}\) [see inset of Fig. 3(a)] are

$${\rm{\Delta }}{E}_{X}^{(S)}\approx -\,2\sqrt{{{\rm{\Delta }}}_{0}^{2}+{\delta }_{2}^{2}}+2{\delta }_{0},$$
(4)
$${\rm{\Delta }}{E}_{X}^{(NS)}=-\,2\sqrt{{{\rm{\Delta }}}_{0}^{2}+{\delta }_{3}^{2}},$$
(5)

for the symmorphic and nonsymmorphic cases, respectively.

Figure 6
figure 6

Illustration of the surface states dispersions along \(\overline{{\rm{\Gamma }}}-\overline{{\rm{X}}}\) (i.e. k y  = 0) for the (a) symmorphic and (b) nonsymmorphic stackings within the M y  =  +i subspace. Due to the finite stacking, semi-infinite solutions from the top (purple, dashed) and bottom (orange, solid) layers hybridize, yielding the anticrossings δ(0|1|2|3). The arrows indicate how each δ j opens as one reduces the number of monolayers. These are degenerate with the M y  = −i subspace. (c) Energy dispersion as a function of k around \(\overline{{\rm{X}}}\) extracted from the model for the symmorphic and nonsymmorphic stackings, respectively.

Full size image

For the nonsymmorphic stacking, Eq. (5) tell us that the gap at \(\overline{{\rm{X}}}\) is always \({\rm{\Delta }}{E}_{X}^{(NS)} < 0\) for any intensity of the finite-size hybridization δ3. Consequently, there cannot be an inversion of its surface state bands. Hence, there is no phase transition for the even stacking. This is a direct consequence of the fourfold degeneracy imposed by the nonsymmorphic symmetry, which does not allow the band crossing at \(\overline{{\rm{X}}}\) to hybridize in Fig. 6(b). On the other hand, for the symmorphic stacking, the gap at \(\overline{{\rm{X}}}\), given by Eq. (4), changes sign with increasing δ0 ≈ δ1, with the crossing point at \({\delta }_{0}\approx {\delta }_{1}=\sqrt{{{\rm{\Delta }}}_{0}^{2}+{\delta }_{2}^{2}}\), which corresponds to the N = 11 transition in Fig. 3(b). In both cases, we diagonalize the inversion operator within a degenerate subspace. The resulting eigenvalues match the DFT data in Fig. 3(c,d). Moreover, this hybridization picture agrees well with the DFT band structures from Fig. 2, and the gap evolutions shown in Fig. 3(a,b) for N ≥ 3.

The transition from N = 3 to 1 is more subtle. Indeed for a single monolayer (N = 1) the definition of top and bottom surfaces become meaningless. In order to understand this transition, first let us recall that the penetration length of the surface states throughout the bulk is \(\lambda \sim 18\) Å or \(N\sim 6\) layers. Therefore, for \(N\lesssim 6\) we can safely assume that states from opposite surfaces hybridize, but does not reach all the way to the opposite surface, as seen in Fig. 4. This validates the model described above and the hybridization picture from Fig. 6. However, for \(N\sim 6\) the surface states lengths are larger than the stacking itself, which now goes beyond our hybridization model above. Instead, we find that as N approaches the monolayer limit, a structural phase transition occurs. This can be characterized by the lattice parameters a N . For the monolayer we obtain a1 = 6.19 Å, while for N = 2 it jumps to a2 = 6.31 Å, and for N = 3, a3 = 6.33 Å. For larger N it smoothly evolves to the bulk value a∞ = 6.40 Å. This sharp transition from N = 3 to N = 1 enhances the atomic orbital hybridizations and atomic spin-orbit couplings, which dictates this phase transition.

Conclusions

In conclusion, we have shown that the band structure of the symmorphic and nonsymmorphic stackings evolve differently as a function of N. This illustrates how the specific space group of a finite size sample is essential to understand its properties within the nanometer scale. Particularly, we show that the nonsymmorphic symmetry imposes an extra protected degeneracy that avoids an hybridization at the \(\overline{{\rm{X}}}\) point, thus yielding a smooth evolution of the band structure with the number of layers. Consequently, there is no phase transition in this case, and its topological nature is governed by the precursor bulk 3D TCI regime, even for samples consisted of only a few layers. In contrast, the hybridization of the symmorphic stacking yields two 2D phase transitions. One is due to a band inversion induced by the hybridization, and the other occurs as the system approaches the monolayer limit. Our space-group-dependent hybridization picture provides a clear understanding of these two cases and matches well the DFT data. Moreover, this analysis can be easily extended to other finite-sized materials that may present a variety of distinct space groups.

Methods

Density functional theory

The first principles calculations are performed using the density functional theory (DFT) within the generalized gradient approximation (GGA) for the exchange and correlation functional35. Fully relativistic j-dependent pseudopotential within the projector augmented wave method36 has been used in the non-colinear spin-DFT formalism self-consistently. We use the Vienna Ab initio Simulation Package (VASP)33,34, with plane wave basis set with a cut-off energy of 400 eV. The Brillouin zone is sampled by using a number of k-points such that the total energy converges within the meV scale. All stackings have been fully relaxed. The optimized force criteria for convergence was less than 0.01 eV/Å. A lattice parameter of 6.19 Å was obtained for the monolayer, and 6.31 Å for the bilayer. For larger stackings, the lattice parameter varies smoothly as a function of the number of layers, and converges to the bulk value (6.40 Å) for \(\lesssim 20\) layers.

Group theory analysis

The effective models were built using the method of invariants23,24,41. In each case we consider the little group \({\mathscr{G}}\) that keeps the TRIM of interest invariant. To define the relevant irreducible representations (IRREPs) for our basis, we consider the orbitals that contribute to band structure around the Fermi level, as obtained in the DFT simulation. Matrix representations for each IRREP can be found by inspection. The spin degree of freedom is then included via a direct product with its su(2) operators to form the double group38. The character tables for each double group are shown in the Supplementary Information.

To obtain the effective Hamiltonians we consider the reducible representation composed by the direct sum of the IRREPs that define the chosen basis. For a generic Taylor expansion of H(k) as

$$H(k)=\sum _{n,m}{H}_{n,m}{k}_{x}^{n}{k}_{y}^{m},$$
(6)

we impose that H(k) commutes with all symmetry elements S i of the group \({S}_{{i}}=\{{S}_{0},{S}_{1},{S}_{2},\ldots \}\). This constrains the matrices Hn,m, thus yielding which of its matrix elements can be finite.

For the symmorphic lattices, it is sufficient to analyze its point group \({\mathscr{P}}={\mathscr{G}}/ {\mathcal B} \), which is the factor group of the full space group \({\mathscr{G}}\) with respect to the invariant subgroup of Bloch translations \( {\mathcal B} \). In contrast, for the nonsymmorphic groups the factor group \( {\mathcal F} ={\mathscr{G}}/ {\mathcal B} \) is not a point group. Instead, it contains elements of the form \(\{{S}_{i}|{\ell }_{i}\}\) (Seitz notation), where S i is a point group operation and \({\ell }_{i}\) is a fractional translation of the unit cell. Its action on a coordinate r, its multiplication rule, and its matrix representations are38

$$\{{S}_{i}|{{\boldsymbol{\ell }}}_{i}\}{\boldsymbol{r}}={S}_{i}{\boldsymbol{r}}+{\ell }_{i},$$
(7)
$$\{{S}_{i}|{{\boldsymbol{\ell }}}_{i}\}\{{S}_{j}|{{\boldsymbol{\ell }}}_{j}\}=\{{S}_{i}\cdot {S}_{j}|{S}_{i}{{\boldsymbol{\ell }}}_{j}+{{\boldsymbol{\ell }}}_{j}\},$$
(8)
$${D}^{{\rm{\Gamma }}}(\{{S}_{j}|{\ell }_{j}\})={D}^{{\rm{\Gamma }}}({S}_{j}){e}^{i{\boldsymbol{k}}\cdot {\ell }_{j}},$$
(9)

where Γ refers to a given IRREP. Equation (9) shows that the matrix representation is the product of the corresponding point group matrix representation and a Bloch factor for the fractional translation \({\ell }_{j}\) for the point k of the Brillouin zone.