Main

Whether a band insulator is topological or not is determined by the parity of the valence-band wave function, which is described by the Z2 topological invariant. Strong spin–orbit coupling can lead to an inversion of the character of valence- and conduction-band wave functions, resulting in an odd Z2 invariant that characterizes the topological insulator4,5. All known topological insulators6,7,8,9,10,11,12,13,14 are based on this band-inversion mechanism4,5,15,16,17,18, but the successive evolution of the electronic state across the quantum phase transition (QPT) from trivial to topological has not been well studied in 3D topological insulators owing to the lack of suitable materials. TlBi(S1−xSex)2 is therefore the first system where one can investigate the 3D topological QPT (ref. 19). The advantage of this system is that it always maintains the same crystal structure (Fig. 1a), irrespective of the S/Se ratio. Low-energy, ultrahigh-resolution angle-resolved photoemission spectroscopy (ARPES), which has recently become available, is particularly suited to trace such a QPT in great detail.

Figure 1: Comparison of the valence-band structure between TlBiSe2 and TlBiS2.
figure 1

a, Crystal structure of TlBi(S1−xSex)2. b, Valence-band ARPES intensity along the direction of the surface Brillouin zone for TlBiSe2 and TlBiS2, plotted as a function of wave vector and binding energy, measured using the He Iα resonance line (h ν=21.218 eV) at T=30 K. c, Direct comparison of the valence-band dispersions between TlBiSe2 and TlBiS2. The energy positions of the bands are determined by tracing the peak position of the second derivatives of the ARPES spectra. Several branches of dispersive bands in both compounds are found at EB higher than 1 eV; these are attributed to the hybridized states of Tl 6p, Bi 6p and Se 4p (S 3p) orbitals16,17,18. Clearly, the bands in TlBiS2 are shifted towards higher EB with respect to those in TlBiSe2, with the energy shift being more enhanced in deeper-lying bands, indicating the relative expansion of the VB width in TlBiS2. d, Comparison of the near- EFARPES intensity around the point between TlBiSe2 and TlBiS2, measured using the Xe I resonance line (h ν=8.437 eV). The absence of the surface band in TlBiS2 was also confirmed by varying the photon energy and light polarization of the synchrotron radiation.

The bulk band structures of the two end members, TlBiSe2 and TlBiS2, are shown in Fig. 1b, where one can see several common features, such as the prominent hole-like band at the binding energy EB of 0.5–1 eV and a weaker intensity at the Fermi level (EF), both being centred at the point (Brillouin-zone centre). These features correspond to the top of the bulk valence band (VB) and the bottom of the bulk conduction band (CB), respectively, demonstrating that both TlBiSe2 and TlBiS2 samples are originally insulators with a bandgap of 0.3–0.4 eV, but electron carriers are doped in the naturally-grown crystals10,11,12. Besides the wider VB width in TlBiS2, which is naturally expected from its smaller lattice constant, the VB structures in the two systems are very similar (Fig. 1c).

A critical difference in the electronic states of the two compounds is recognized by looking at the band dispersion in the vicinity of EF around the point (Fig. 1d). An ‘X’-shaped surface band that traverses the bulk bandgap is clearly recognized in TlBiSe2 (ref. 10), whereas such a surface state is completely absent in TlBiS2. This indicates a topologically distinct nature between the two, despite the similar overall bulk band structure. One can thus conclude that the band parity is inverted in TlBiSe2, whereas it is not in TlBiS2.

What would happen if we mixed the topologically non-trivial TlBiSe2 and trivial TlBiS2 phases? One natural consequence of such an alloying would be that the bulk bandgap closes owing to the switching of the parity of the VB wave function at a certain Se content xc, marking a topological QPT across which the massless Dirac-cone surface band appears (vanishes) once the system enters into the topological (non-topological) phase. The surface band in the topological phase would retain the Kramers degeneracy at the Dirac point and retain the massless character, as long as the alloying disorder does not break the TRS. It turns out that the electronic-structure evolution in TlBi(S1−xSex)2 indeed presents the topological QPT, but it bears a feature that is totally unexpected.

Figure 2a shows the near- EF ARPES intensity around the zone centre for a series of x values, including TlBiSe2 (x=1.0) and TlBiS2 (x=0.0). One can immediately see that the surface state is seen for x≥0.6, whereas it is absent for x≤0.4 (see also Supplementary Information), which points to the topological QPT occurring at xc≈0.5. In fact, the bulk bandgap estimated from our data approaches zero on both sides of the QPT (see Fig. 3d), suggesting that a band inversion takes place across the QPT, in accordance with the natural expectation and also with a recent ARPES study independently done on TlBi(S1−xSex)2 (ref. 19).

Figure 2: Mass acquisition of surface Dirac fermions in TlBi(S1−xSex)2.
figure 2

a, Near- EFARPES intensity around the point as a function of wave vector and binding energy in TlBi(S1−xSex)2 for a series of Se concentrations, measured using the Xe I line with the same experimental geometry. White dotted lines for x≤0.4 represent the top of the VB and the bottom of the CB, thus highlighting the bandgap; owing to the kz dispersion of the bulk bands, the true gap is likely to be smaller than these estimates, as is reflected in the error bars in Fig. 3d. b, Energy distribution curves (EDCs) of the data shown in a. c, Second-derivative plots of the ARPES intensity for x=1.0, 0.9, 0.8 and 0.6.

Figure 3: Massive Dirac fermions and the electronic phase diagram across the topological quantum phase transition.
figure 3

a, Numerical fittings of the ARPES-determined surface band dispersions (open circles) using the theoretical band dispersion which incorporates the finite mass term20. Note that the Dirac energy for x=0.9 is shifted downward by 0.015 eV for clarity. b, Schematic 3D image of the surface Dirac cone and its energy-gap evolution at low sulphur concentrations. c, ARPES spectra around the VB and CB edges for x=1.0 and 0.6, which are used for the bandgap determination for x>0.5, as was done in ref. 10; the photon energies of the synchrotron radiation were chosen to optimally probe the VB top and the CB bottom respectively10. d, Electronic phase diagram of TlBi(S1−xSex)2 showing the surface Dirac gap (green symbols) and the bulk bandgap (red symbols), together with a schematic picture of the band evolution (top) summarizing the present ARPES experiment. Error bars for the Dirac gap energy correspond to the experimental uncertainty in determining the energy position of the band dispersion at the point in a, whereas those for the bulk bandgap originate from the experimental uncertainty in determining the energy difference between the top of the valence band and the bottom of the conduction band in Figs 2 and 3c.

The unexpected physics manifests itself at the Dirac point. The bright intensity peak at 0.4 eV at x=1.0 is no longer visible at x=0.9and is markedly suppressed at x=0.6, suggesting that the Kramers degeneracy is lifted on S substitution whereas the surface state is still present. In fact, a closer look at the energy-distribution curves (EDCs) in Fig. 2b shows that the originally X-shaped surface band at x=1.0, where the EDC at the point is well fitted by a slightly asymmetric Lorentzian, splits into lower and upper branches at x=0.9, with a finite energy gap at the point. Further substitution of S results in the reduction and the broadening of the intensity of the surface band (see EDCs for x=0.6), but the energy position of the surface band can be still traced, as illustrated in the second-derivative intensity plots in Fig. 2c. The surface-state nature of this band was confirmed by the stationary nature of its energy position with respect to the photon energy (Supplementary Information), so this band evidently represents massive Dirac fermions on the surface. As a result of the gapped nature, this phase cannot be called topological in the strict sense, but the massive Dirac fermions are obviously of topological origin, suggesting that the bulk bands are kept inverted. On the other hand, the photon-energy dependence of the ARPES spectra for x=0.4 signifies the absence of the surface state, in contrast to the clear signature of it for x=0.6 (Supplementary Information). The disappearance of the surface state and the very narrow bulk gap at x=0.4 (which can be inferred from Supplementary Fig. S2) point to the topological QPT being located between x=0.6 and 0.4. Interestingly, the surface bandgap, called here the Dirac gap, grows with decreasing x (less than 0.1 eV at x=0.9and 0.8, and greater than 0.1 eV at x=0.6), indicating that the S content is closely related to the magnitude of the Dirac gap. We also found that the magnitude of the Dirac gap does not diminish with increasing temperature (Supplementary Information), which argues against a magnetic-order origin of the gap. We note that in a recent independent work19 the Dirac cone was reported to remain gapless for x>0.5, in contrast to the gapped surface states for 0.6≤x≤0.9 observed here. This discrepancy may be due to the difference in the energy resolution (15 meV in ref. 19, as opposed to 2–4 meV in the present experiment).

One may wonder if the bulk bandgap really closes at xc≈0.5. If the bandgap never closes, the samples for both x=1.0 and 0.0 should be in the same topological phase. Apparently, this is inconsistent with our ARPES data in Fig. 1d. According to the fundamental principle of the topological band theory, the QPT must always be accompanied by a bandgap closing. Hence, based on our data, the bandgap closing must be happening either at 0.4<x<0.6 or at 0.9<x<1.0 (at which the Dirac gap starts to open). Taking into account the gradual reduction of the bandgap size on approaching x=0.5, it is most sensible to conclude that the inevitable closing of the gap takes place at x=0.5.

One may also question if the observed Dirac gap might be an artefact of an inhomogeneous S distribution in the sample. To address this question, we employed electron-probe microanalysis (EPMA) on the surface and found that our crystals are exceedingly homogeneous (Supplementary Information). The persistently narrow X-ray diffraction peaks, together with a systematic change of the lattice constants, further corroborate this conclusion (Supplementary Information).

To quantify the magnitude of the Dirac gap, we use the theoretical surface-band dispersion to account for the finite mass term20 (which was originally proposed to explain the Bi2Se3 ultrathin-film data21) and numerically simulate the experimentally obtained surface band dispersion near the point; although the origin of the mass term in TlBi(S1−xSex)2 is not clear at the moment. As shown in Fig. 3a, the simulated curves reasonably well reproduce the experimental data, and the obtained Dirac gaps for x=0.9, 0.8 and 0.6 are 50±10, 70±10 and 130±20 meV, respectively. The evolution of the massive Dirac cone is schematically illustrated in 3D images of the band dispersions in Fig. 3b. We have confirmed that the obtained sizes of the Dirac gap are highly reproducible by measuring more than five samples for each composition and also by varying the incident photon energy. Taking into account that all the elements contained in TlBi(S1−xSex)2 are nonmagnetic and also that the sample shows no obvious magnetic order (Supplementary Information), our result is a strong indication that the substitution of Se with S in TlBi(S1−xSex)2 leads to an unconventional mass acquisition of the surface Dirac fermions without explicitly breaking the TRS.

Based on the present ARPES results, one may draw the electronic phase diagram of TlBi(S1−xSex)2, as shown in Fig. 3d. The massless Dirac topological phase is achieved only near x=1.0. Once a small amount of S is substituted for Se, the Dirac gap opens, growing almost linearly as a function of the S content, 1- x. Such a massive Dirac phase is present until the topological QPT occurs at xc≈0.5, where the bulk gap closes and the band parity is interchanged.

The mass acquisition of the Dirac fermions indicates that the Kramers degeneracy is lifted, which means that the TRS must be broken on the surface. Given that there is no explicit TRS breaking, the only possibility is that a spontaneous symmetry breaking takes place on the S substitution, which is reminiscent of the Higgs mechanism in particle physics. Therefore, TlBi(S1−xSex)2 may serve as a model system to bridge condensed-matter physics and particle physics. The exact mechanism of the mass acquisition is not clear at the moment, but an interesting possibility is that it originates from some exotic many-body effects that can lead to an electronic order, although a simple mechanism like the spin-density wave does not seem to be relevant (Supplementary Information). When the top and bottom surface states coherently couple and hybridize, a Dirac gap can open21, but the sufficiently large thickness (>10 μm) of our samples precludes this origin. Another possibility is that critical fluctuations associated with the QPT are responsible for the mass acquisition, but it is too early to speculate along this line. From the application point of view, the Dirac gap can be much larger than that of the magnetically doped topological insulator Bi2Se3 (ref. 1) and is tunable by means of the S/Se ratio, making the TlBi(S1−xSex)2 system a prime candidate for device applications that require a gapped surface state.

Methods

High-quality single crystals of TlBi(S1−xSex)2 were grown by a modified Bridgman method (see Supplementary Information for details). X-ray diffraction measurements indicated the monotonic shrinkage of a and c axis lengths on substitution of S for Se, without any apparent change in the relative atomic position with respect to the unit cell. ARPES measurements were performed at Tohoku University using VG-SCIENTA SES2002 and MBS-A1 spectrometers with high-flux He and Xe discharge lamps and a toroidal/spherical grating monochromator. The He Iα (h ν=21.218 eV) line and one of the Xe I (h ν=8.437 eV) lines22 were used to excite photoelectrons. Samples were cleaved in situ along the (111) crystal plane in an ultrahigh vacuum of 5×10−11 torr. The energy resolutions for the measurement of the VB and near- EF regions were set at 15 and 2–4 meV, respectively. The angular resolution was 0.2°, corresponding to a k resolution of 0.007 and 0.004 Å−1 for the He Iα and Xe I photons, respectively. The Fermi level of the samples was referenced to that of a gold film evaporated onto the sample holder. A shiny mirror-like surface was obtained after cleaving samples, confirming its high quality.