Unconventional topological phase transition in non-symmorphic material KHgX (X = As, Sb, Bi)

Traditionally topological phase transition describes an evolution from topological trivial to topological nontrivial state. Originated from the non-symmorphic crystalline symmetry, we propose in this work an unconventional topological phase transition scheme between two topological nontrivial insulating states mediated by a Dirac gapless state, differing from the traditional topological phase transition. The KHgX (X = As, Sb, Bi) family is the first experimentally realized topological non-symmorphic crystalline insulator (TNCI), where the topological surface states are characterized by the Mobius-twisted connectivity. Based on first-principles calculations, we present a topological insulator–metal transition from TNCI into a Dirac semimetal (DSM) via applying an external pressure on KHgX. We find an unusual mirror Chern number Cm = −3 for the DSM phase of KHgX in the non-symmorphic crystal structure, which is topologically distinct from the traditional DSM such as Na3Bi and Cd3As2. Furthermore, we predict a new TNCI phase in KHgX via symmetry breaking. The topological surface states in this new TNCI phase display zigzag connectivity, different from the unstressed one. Our results offer a comprehensive study for understanding how the topological surface states evolve from a quantum phase transition in non-symmorphic system.


INTRODUCTION
Understanding the fundamental physics in different phases of matter is one of the most important goals of condensed matter physics. A powerful route to understand an unrevealed phase is investigating the nature of a phase transition. [1][2][3][4][5][6][7] Topological phases and topological phase transitions are new phenomena distinct from Landau's phase transition 8 and have attracted worldwide research interests because of their unusual fundamental physical properties as well as potential applications in next-generation electronics. [9][10][11][12][13] Consequentially, searching for new type of topological phases in real materials and studying its phase transition become a research frontier in condensed matter physics and materials science.
In the past decade, many materials have been demonstrated to be a time-reversal symmetry-protected Z 2 topological insulator in both three-dimensional (3D) and two-dimensional systems. 8,14 Scientists also realize that additional crystalline symmetries play an important role in the topological classifications. Hence, various topological crystalline phases have been proposed and discovered, including the topological crystalline insulators (TCIs) [15][16][17][18][19][20][21][22] and the topological semimetals (TSMs), [5][6][7][23][24][25][26][27][28][29][30][31][32][33] which correspond to fully gapped and gapless bulk states, respectively. Recent works further raise a new class of TCI states in the time-reversal-invariant systems with spin-orbit coupling, which are protected by the nonsymmorphic glide reflection symmetries. [34][35][36][37][38][39] It was shown that, for a 3D insulating system under both glide reflection and timereversal, a Z 4 invariant can be defined such that four topologically distinct phases can be identified under the symmetries, 36,37 called the topological non-symmorphic crystalline insulator (TNCI). 40 The topological surface states of TNCI exhibit exotic Mobius-twisted connectivity because the glide eigenvalue does not coincide with the 2π periodicity of the Brillouin zone (BZ). 36 The KHgX (X = As, Sb, Bi) family is recently predicted and confirmed as a TNCI both theoretically 38,39 and experimentally. 41 However, the topological phase transition of this insulating phase remains still elusive. In this work, we investigate the topological phase transition in KHgX under external pressure and breaking symmetry via first-principles calculations based on density functional theory (DFT). Our calculations show a topological insulator-metal transition that KHgX transfers from TNCI into an unusual topological Dirac semimetal (DSM) phase under external stress within the non-symmorphic symmetry. More importantly, we find the mirror Chern number of this Dirac state is C m = −3 on the k z = 0 plane, which has never been found in previous Type-I and Type-II DSMs. 5,23,42,43 Furthermore, our calculation predicts that KHgX transfers into a new type of TNCI state distinct from the unstressed one when breaking the rotational symmetry. Consequentially, KHgX provides an unrevealed surface connectivity beyond the Mobius-twisted one. These intriguing topological phases and corresponding phase transitions may lead to brand new signatures in various experiments, including photoemission and scanning tunneling, as well as transport measurements, paving a way for experimental studies on new type of topological phases. structure of KHgX in the space group P6 3 /mmc (no. 194). The K atom acts as the electron donor while Hg and X form the honeycomb layer with weak interlayer interactions. 45 The alternatively stacking HgX honeycomb layers along the c axis indicates that the space group contains non-symmorphic symmetries. Taking K ion as the origin of the Cartesian coordinates, the space group can be generated by three symmetry operators: (1) The glide reflectionM y : ðx; y; zÞ7 !ðx; Ày; z þ c=2Þ, i.e., a reflection by the y = 0 plane followed by a half translation parallel to c; (2) The six-fold screw rotationĈ 6 : ðx; y; zÞ7 !ðx cos θ À y sin θ; x sin θ þ y cos θ; z þ c=2Þ where θ ¼ 2π=6, i.e., a six-fold rotation along caxis followed by a half translation; (3) The inversion through the originP : ðx; y; zÞ7 !ðÀx; Ày; ÀzÞ. HereM y andĈ 6 are nonsymmorphic symmetries involving a fractional translation that cannot be removed by different choices of the origin. All the crystal symmetries of the Hamiltonian can be written in terms of the above generators together with the translation operator tðx; y; zÞ of the Bravais lattice. The Hamiltonian also exhibits the time-reversal symmetryT since KHgX is non-magnetic.
Symmetries play important roles in this work such that: (a) The Z 4 invariant is defined underM y andT; (b) The mirror Chern number is defined underM z , whereM z ¼tð0; 0; 2cÞPðĈ 6 Þ 3 is the reflection (not glide) by the z ¼ c=4 plane, namely, the HgX layer; (c) The rotational symmetry (Ĉ 6 orĈ 3 ) together withP andT protect the Dirac node (DN) under stress, whereĈ 3 1 tð0; 0; ÀcÞðĈ 6 Þ 2 is the three-fold rotation (not screw). In the following, given a symmetry operatorĥ, a path or plane in the bulk or surface BZ is called invariant underĥ if all the k on this path or plane satisfyhk ¼ k þ G whereh is the 3 × 3 matrix acting on the Cartesian coordinate and G is a vector of the reciprocal lattice. On suchĥ invariant path or plane, common eigenstates of the HamiltonianĤðkÞ and the symmetry operatorĥ can be found. Figure 1c, d show, respectively, the calculated band structures of stress-free KHgAs and KHgBi based on the optimized lattice constants (Table 1) with minimum and maximum spin-orbit coupling strength among the KHgX family. A common feature of the KHgX band structures is the two occupied Hg-s bands as can be seen along Γ−A. In fact, these Hg-s bands show inverted energy order compared to that of KZnP, 38,45,46 where the latter is The bulk BZ, (010)-surface BZ (gray plane), and (100)-surface BZ (blue plane). Here (010) or (100) refers to the surface normal vector in terms of the Cartesian coordinate. In the bulk BZ, the k z = 0 plane is colored by red. c, d The calculated band structure of KHgAs and KHgBi, respectively, with the optimized lattice constant. The red dots indicate the contribution from the Hg-s orbitals adiabatically connected to the atomic limit. Owing to such nontrivial band inversion, it was predicted that the unstressed KHgX exhibits gapless surface states characterized by the hourglass flow connectivity 38,39 (also called the Mobius-twisted connectivity 36,37 ). Previous works 38,39 have demonstrated the non-trivial topology of KHgX from Wilson loop spectra. Here we instead unveil the topological properties of KHgX from another viewpoint through the Z 4 invariant and mirror Chern number as discussed below.
Owing to the symmetries ofM y andT, the Z 4 invariant χ ∈ {0, 1, 2, 3} is well defined for the orthorhombic BZ of KHgX. Following the conventional approach, we carry out the Wilson loop calculations for KHgBi in different phases as discussed in section II of the Supplementary Information (SI-2). The one-to-one relation between the Wilson loop spectrum and the topological invariant χ is clearly seen. Here we also provide a new gauge-independent formula of this invariant, Eq. S14 in SI-3. We will adapt this equation in this work because it is more suitable for numerical calculations. On the other side, the mirror Chern number C m ¼ ðC þi À C Ài Þ=2 2 Z is well defined on the twoM z invariant planes k z = 0 and k z = π/c, where C ±i is the Chern number of the subspace such that in which F ± i ðkÞ is the Berry curvature of the occupiedM z eigenstates (with eigenvalues ±i). However the mirror Chern number on k z ¼ π=c plane must vanish since the subspace spanned byM z eigenstates at k z ¼ π=c is symmetric underPT, making the Berry curvature zero. 38,47 In the following, "mirror Chern number" or "C m " refers to the one defined on k z = 0 plane.
The relation between C m and χ will also be addressed in the next section (Eq. 5). By using the effective tight-binding model described in the "Method" section, we numerically calculate these topological invariants and conclude that χ = 2 and C m ¼ À2 for stress-free KHgX. For the bulk terminations that preserveM y , χ = 2 ensures that the gapless surface states display the Mobius-twisted connectivity, 36,37 drawing the same conclusion as ref. 38 . We have noticed that ref. 38 The sign ambiguity may arise since the integral (Eq. 1) depends on the surface orientation, and here we set dajj þ k z when performing the integration.
KHgX under stress Interestingly, KHgX transforms into a DSM after gap closing. We schematically illustrate this phase transition in Fig. 2a-c for KHgX around Γ. Figure 2a shows the unstressed case or the so-called TNCI phase with χ = 2 and C m = −2. By applying suitable stress while preserving the symmetries of the space group, Fig. 2b demonstrates the critical point of the phase transition, where the gap between the conduction (CB) and valence (VB) band vanishes. By further enhancing the stress so that the CB (VB) goes downward (upward) with increasing band overlaps, in general any band crossing points can be gapped out due to band hybridizations. 48 However, it is not the case for Γ → A, along which the hybridization is forbidden. The path Γ → A is invariant under C 6 , where the CB (the red band in Fig. 2a-c belonging to the representation Δ 9 ) and VB (the blue band belonging to Δ 7 ) have theĈ 6 eigenvalues given as CB : fe iπ=2 ; e Àiπ=2 g; VB : fe iπ=6 ; e Àiπ=6 g: Here the phase factor exp(−ik z c/2) arising from the half translation is omitted, and these eigenvalues are degenerate in pairs due toP andT. Having differentĈ 6 eigenvalues, the CB and VB do not hybridize along Γ → A owing to the orthogonal theorem. 49 Therefore, a four-fold degenerate band crossing point (i.e., the DN) is formed along Γ → A once the CB has energy lower than VB, realizing the DSM phase (Fig. 2c). Note that the CB and VB also have different eigenvalues under the three-fold rotationĈ 3 , given by the square of Eq. (2). We remark that the DN of the stressed KHgX is protected byĈ 6 (orĈ 3 ),P, andT, in the sense that the node cannot be gapped out by symmetry-preserving perturbations. In Fig. 2e, f, we show the calculated band structure of KHgAs and KHgBi under the uniaxial stress along c (the in-plane stress is relaxed to zero). We find that the DSM phase is realized when the uniaxial stress exceeds Δc/c 0 = 0.085 (2.23 GPa) for KHgAs; Δc/c 0 = 0.140 (2.86 GPa) for KHgSb; Δc/c 0 = 0.115 (1.56 GPa) for KHgBi, where Δc refers to the lattice distortion parallel to c while c 0 refers to the ideal lattice constant ( Table 1).
The Z 4 invariant is not well defined for the DSM phase of KHgX since there is a gapless point, the DN, along Γ → A, However, we can still consider the mirror Chern number. During the phase transition process, the two inverted Hg-s bands at Γ and A are unaffected by the stress since they are fully occupied deeply below the Fermi level (Figs 1c, d, and 2e, f). As the DN is formed at the Fermi level along Γ → A, a band inversion is also created at Γ. In addition to the aforementioned two inverted Hg-s bands below the Fermi level, there are triple times of band inversion at Γ for the DSM phase, implying an interesting mirror Chern number C m = −3. To confirm this conjecture, we express the Chern number C ±i (Eq. 1) of the subspace as 50,51 where η ± i n ðΓÞ is theĈ 6 eigenvalue of the occupiedM z eigenstate at Γ (withM z eigenvalue ±i), while θ ± i n ðKÞ and ζ ± i n ðMÞ is of theĈ 3 and C 2 ¼ tð0; 0; ÀcÞðĈ 6 Þ 3 eigenvalues at K and M, respectively. As the DN is formed, the highest occupied state at Γ changes from Γ À 7 to Γ þ 9 (Fig. 2a-c). TheĈ 6 eigenvalues of the doubly degenerate states Γ À 7 and Γ þ 9 were given in Eq. (2). BecauseM z ¼tð0; 0; 2cÞPðĈ 6 Þ 3 , both the Γ À 7 and Γ þ 9 states withĈ 6 eigenvalues of, respectively, exp (−iπ/6) and exp(−iπ/2) have commonM z eigenvalues of +i. As the highest occupied state changes from Γ À 7 to Γ þ 9 , we have ΔC þi ¼ À1 according to Eq. (3), which immediately implies ΔC Ài ¼ þ1 underT. Knowing that C m = −2 before the band inversion, this results in C m = −3 for the DSM phase.
Owing to the non-trivial mirror Chern number obtained here, the DSM phase of KHgX exhibits protected surface states distinct from other DSMs, such as Na 3 Bi 5 and Cd 3 As 2 . 23 To demonstrate, we consider the (010)-surface of the stressed KHgBi with Δc/c 0 = 0.115, where (010) refers to the surface normal vector in terms of the Cartesian coordinates. The surface BZ and the surface band structure are shown in Figs 1b and 3a, respectively. The (010)surface preserves the symmetryM z , under which C m is defined. Accordingly, along ÀX ! Γ ! X of the surface BZ, to which the bulk k z = 0 plane projects, there should exist three pairs of counter-propagating surface states with oppositeM z eigenvalues due to C m = −3. 16,17 These non-trivial surface states can be clearly observed in Fig. 3a (see the schematic illustration in the lower inset). Owing to C m < 0, the surface states withM z eigenvalues +i propagate toward the −x direction. 16 On the other hand, along Γ ! Z, the bulk-projected DN appears, which is emphasized by the white dashed circle in Fig. 3a. Moreover, there is a surface band connecting to this bulk-projected DN from Z. When viewed from the Fermi surface as presented in Fig. 3b, the connected surface bands become a pair of Fermi arcs. More precisely, in Fig.  3b, there is a pair of Fermi arcs (labeled by I and II) connecting the two bulk-projected DNs (yellow dots) located along ÀZ ! Γ ! Z. Such Fermi arc surface states have been widely observed in many DSMs 5,23,43,52 although there is no topological origin. [52][53][54] Nonetheless, for the DSM phase of KHgX, the Fermi surface is guaranteed to intersect the high-symmetry line ÀX ! Γ ! X by (at least) six times due to C m = −3.
Generally speaking, a DSM phase can be regarded as a critical point on the topological phase diagram, which transforms into various topological distinct phases via symmetry breaking. 5,24,43 Starting from the DSM phase of KHgX, an interesting phase can be realized by breaking the rotation (Ĉ 6 andĈ 3 ) while keeping all the other symmetries. As we now show, the resulting phase is a TNCI with χ = 3 and C m = −3. Rotational symmetry breaking has the effect of gapping out the DN along Γ → A (Fig. 2d), while the mirror Chern number C m , as a topological invariant, is left intact. Therefore, we still have C m = −3 after rotational symmetry breaking, the same as the DSM phase. On the other hand, the Z 4 invariant is now well defined since the DN has been gapped out and which can be inferred through 36 where ν kz¼0 is the Z 2 invariant 55,56 defined on the k z = 0 plane. We have ν kz¼0 ¼ 1 since C m = −3 is odd. 16 Equation 4 determines χ ∈ {0, 1, 2, 3} up to modulo 2, implying that χ = 1 or 3 after rotational symmetry breaking. A general relation between the mirror Chern number and the Z 4 invariant is also suggested by Eq. (4), i.e., where n 2 Z. To determine the actual χ value (1 or 3) after rotational symmetry breaking, we resorted to the DFT calculation, where we confirmed χ = 3 based on the proposed formula (Eq. S14). However, it should be emphasized that χ is a modulo 4 invariant; therefore, there is an ambiguity between χ = 1 and 3, which depends on the orientation of the surface and line integrals in Eq. S14 (or S10) for the definition of the invariant. This ambiguity also arises when the domain of the BZ (Eq. S1) is shifted by k z 7 !k z þ n 2π=c where n is some odd integer. 57 Nonetheless, once the orientation or domain of the BZ is fixed, different χ values still describe distinct topological phases. 36,57 Here we note that the band gap of KHgX, which is sensitive to the lattice constant, 46 can be manipulated by stress. We have systematically investigated the phase transition parameters such as the spin-orbit coupling strength as well as the interlayer spacing by replacing variant species to realize the topological  Fig. S1, the stress-induced band gap modulation can be achieved through replacing the big X anion by the small P anion or replacing the large K + cation by the smaller Na + or Li + cations without applying stress. We have found three candidates KHgP, NaHgAs, and LiHgSb, which are DSMs with the mirror Chen number of −3 at their equilibrium volumes (SI-1).

Evolution of the surface band topology
We have shown that by varying the stress and breaking the rotational symmetry, KHgX undergoes the topological phase transition: TNCI (χ = 2, C m = −2) → DSM (C m = −3) → TNCI (χ = 3, C m = −3), in which χ is ill-defined for the DSM phase. Each phase exhibits unique gapless surface states resulting from the nontriviality of χ or C m . Here we demonstrate how the surface band topology of KHgX evolves during the topological phase transition. Since χ is defined underM y andT while C m is defined underM z , we will focus on the (100)-surface (Fig. 1a) that preserves bothM y andM z . The (100) is the only surface that keepsM y ; meanwhile due to the rotational symmetry, there are also another two glide reflections given byĈ 3My ðĈ 3 Þ À1 and ðĈ 3 Þ 2M y ðĈ 3 Þ À2 , under which the Z 4 invariant can be defined. Therefore, the following discussion also applies to the set of symmetrical equivalent surfaces f2110g (Miller-Bravais index).
Below we consider KHgBi under stress, in which the phase transition from TNCI (χ = 2, C m = −2) to DSM (C m = −3) occurs around Δc/c 0 = 0.115. Figure 4a shows the TNCI surface band structure with Δc/c 0 = 0.100 less than the critical value. Consistent with the χ = 2 topology, 36,37 there exist gapless surface states along the closed pathỸ !T !Z !Γ !Ỹ where the Mobiustwisted connectivity can be observed along the twoM y invariant pathsỸ !T andZ !Γ. In addition, along theM z invariant path ÀỸ !Γ !Ỹ, there exist two pairs of counter-propagating surface states with oppositeM z eigenvalues, consistent with C m = −2.
Here the surface states withM z eigenvalues +i propagate toward +y since C m < 0. If we consider the ð100Þ-surface instead, then these "+i surface states" will propagate toward the −y direction.
With the stress up to Δc/c 0 = 0.115, KHgBi turns into the DSM phase with C m = −3. Figure 4b shows the surface band structure along the pathZ !Γ !Ỹ, which is different from the previous phase (Fig. 4a). Along theM y invariant pathZ !Γ, the occurrence of the bulk-projected DN breaks the Mobius-twisted connectivity and makes the Z 4 invariant undefined for the DSM phase. While along theM z invariant pathΓ !Ỹ, the two surface bands labeled as I and II no longer degenerate at Γ but connect to the bulk states. As such, we now have three pairs of counter-propagating surface states along ÀỸ !Γ !Ỹ instead of two pairs, consistent with the change from C m = −2 to C m = −3.
Finally, upon breaking the rotational symmetry, KHgBi undergoes the phase transition from DSM (C m = −3) to another TNCI phase (χ = 3, C m = −3) as shown in Fig. 4d. The resulting surface band structure is nearly identical to the DSM phase (Fig. 4b, c) except the bulk-projected DN is gapped out. In Fig. 4d, a reference line drawn inside the gap will intersect the surface band(s) by odd number of time(s). This implies the surface band topology along theM y invariant pathZ !Γ is identical to the quantum spin Hall (QSH) effect, 8 consistent with χ = 3 topology. 36,37 In addition, there also exist three pairs of counter-propagating surface states along ÀỸ ! Γ !Ỹ since C m = −3 (see Fig. 4g), the same as the DSM phase. Figure 4e-g schematically demonstrate the phase transition in KHgX. Starting from the TNCI phase with χ = 2 and C m = −2 in Fig.  4e, the surface states alongỸ !T andZ !Γ display the Mobiustwisted connectivity with two pairs of counter-propagating surface states along ÀỸ !Γ !Ỹ with oppositeM z eigenvalues. Figure 4f shows the DSM phase with C m = −3 over the critical stress, where the Mobius-twisted connectivity alongZ !Γ is broken by the bulk-projected DN with an extra pair of counterpropagating surface states along ÀỸ !Γ !Ỹ. Another TNCI phase with χ = 3 and C m = −3 given by breaking the rotational symmetry is depicted in Fig. 4g, where the bulk-projected DN is gapped out, and now the surface bands alongZ !Γ connect in a zigzag way equivalent to the QSH effect.
In summary, the KHgX (X = As, Sb, Bi) family is the first experimentally realized TNCI, where the topological surface states are characterized by the Mobius-twisted connectivity. We diversify the topological phase diagram of KHgX by introducing two new phases based on first-principles calculations. By applying stress, KHgX undergoes a topological insulator-metal transition from the TNCI phase with C m = −2 into the DSM phase with a non-trivial mirror Chern number C m = −3 in the non-symmorphic crystal structure. Via symmetry breaking, the DSM phase transforms into another new TNCI phase with C m = −3 hosting the QSH effect. The change in the connectivity of the surface bands provides a direct justification of the topological phase transition, and manipulating the band gap is the key to realize these predicted new topological phases. The red color represents the bulk projected bands. The upper inset is a zoom-in of the calculated surface band structure along ÀZ ! Γ ! X with ±i indicating theM z eigenvalues of the surface states. The lower inset is a schematic illustration of the surface band structure along ÀX ! Γ ! X with blue and red lines representing, respectively, positive and negative eigenvalues ofM z . b Constant energy cut at the Fermi energy (i.e., the Fermi surface). The two yellow dots indicate the position of the bulk-projected Dirac nodes, which are connected by a pair of Fermi arcs labeled by I and II

METHOD
The bulk electronic structure calculations were performed using the VASP package 58 based on the DFT. The generalized gradient approximation of Perdew-Burke-Ernzerhof 59 was used for the exchange correlation functional. The cutoff energy of 800 eV and the BZ sampling grid of 12 × 12 × 6 were used in this work. The topological invariants were calculated by using the effective tight-binding Hamiltonian constructed by the WANNIER90 code 60 on the basis of the DFT wavefunctions. The initial projections of the Wannier orbitals were selected as Hg-s and X-p (X = As, Sb, Bi). The surface states were obtained from the surface Green function of the semi-infinite system based on the iterative method. 61

DATA AVAILABILITY
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request. Green and brown lines indicate the surface states with differentM y eigenvalues. The label ±i indicate theM z eigenvalues of the surface states depicted in blue and red, respectively. Dotted, dashed, and solid lines represent the first, second, and third pair of the surface states, respectively. The other surface state of each pair locates along ÀY ! Γ and is not shown here. a, e The topological non-symmorphic crystalline insulator (TNCI) phase with χ = 2 and C m = −2 with Δc/c 0 = 0.100. The Mobius-twisted connectivity can be observed alongỸ !T andZ !Γ, shaping like an hourglass (The shape spoils alongZ !Γ; however, the same topology remains.). b, f The Dirac semimetal (DSM) phase with C m = −3 under the stress Δc/c 0 = 0.115. The Mobius-twisted connectivity alongZ !Γ is spoiled owing to the occurrence of the bulkprojected Dirac node (DN). A magnified view for the DN is also provided in c. d, g Another TNCI phase with χ = 3 and C m = −3, transformed from the DSM phase by breaking the rotational symmetry. The quantum spin Hall effect can be observed alongZ !Γ