Introduction

The recently discovered topological insulator (TI) is a novel state of quantum matter and is characterized by time-reversal-symmetric gapless topological surface states, which appear within the bulk energy gap1,2,3. The key ingredient for creating the TI phase is spin-orbit coupling (SOC) and the strong SOC due to the heavy elements can inverse the band characters of the valence band and conduction band4,5. By now, TI has been at the core of a very active research area due to their exotic physical properties and potential technological applications. The bulk-insulating TI with tunable surface state is necessary and important for applications in spintronics and quantum computation. Recently, a class of strong TIs (Bi2Se3, Bi2Te3 and Sb2Te3) has been predicted5 and demonstrated6,7,8 and has attracted much interest due to their relatively large bulk band gaps and a single Dirac cone at the Γ point of the Brillouin zone9,10,11,12,13,14,15. However, there are some challenges remain that if binary TIs Bi2Se3, Bi2Te3 and Sb2Te3are to become functional components of electronic devices. Chief among them is the unacceptably high bulk conductivity introduced by the strong defect doping (e.g., anion vacancy and antisite defects)16. In addition, for Bi2Te3, the Dirac point of the (111) surface lies below the Fermi level and is buried in the bulk valence bands, which is a potential source for scattering between the surface and bulk states5,12. Substantial efforts involving nanostructuring17, chemical doping16,18,19 and electrical gating20,21,22,23 have been try to modulate their electronic structure, but limited progress has been made to solve the problems.

Recently, it has been shown that solid-solution, which has a similar structure to its parent compounds, is a powerful approach for engineering the electronic structure of TIs24,25,26,27,28,29. A topological phase transition from a normal to a topological insulator is observed in TlBi(S1−xSex)2 by changing the S/Se ratios24,25. While for (Bi1−xSbx)2Te3, the topological surface states maintain over the entire Bi/Sb ratios x with the reducing of the bulk carrier density and realization of the truly insulating bulk26,27. This paves a new route to achieving novel devices based on topological insulators. However, the mechanism of the robustness of topological property and the factors that affect the decrease of bulk carrier density in (Bi1−xSbx)2Te3remain unclear. In addition, so far, the unique band inversion in (Bi1−xSbx)2Te3 has not been observed experimentally26,27. Therefore, in the present work, to address exactly these questions and show the band inversion, we explore the electronic characters of (Bi1−xSbx)2Te3 via the first-principles electronic structure calculations. We identify that both the SOC strength and lattice parameters can be effectively tuned by effectively varying the fraction of Bi and Sb. The effects of the SOC strength and lattice parameters on the band inversion, topologic surface states and formation energies of native defects are investigated. The results provide a way to tune the properties of topological insulators and are helpful to design novel topologically insulating devices.

Results

Crystal structure of (Bi1−xSbx)2Te3

The advantage of this system is that Bi2Te3 and Sb2Te3 share the same crystal structure26,27 with the rhombohedral setting for the primitive cell and hexagonal setting for the conventional cell. Hence, for the bulk investigation, both a 40-atom 2×2×2 supercell with rhombohedral setting and a 15-atom conventional cell with hexagonal setting ( Fig. 1a )30 are used. We determine the lattice constants of the solid-solution (Bi1−xSbx)2Te3 as follows. The most stable configuration of (Bi1−xSbx)2Te3 at a given Sb concentration x, from 0 to 1 by a step of 1/6 for 15-atom case and a step of 0.125 for 40-atom case, is searched through an exhaustive search (see Supplemental Information) at each level of x and determined through the cohesive energies calculations:

Where A (A = Bi, Sb and Te) represents different atoms of the solid solution, NA the number of A atoms, NF.U. the number of formula units and the energies of isolated A atoms. For the 15-atom and 40-atom cases, NF.U. = 3 and NF.U. = 8, respectively. The total energies, lattice parameters and geometry structures of determined most stable configurations are given in Supplementary Information. The cohesive energies Ecoh for both 15-atom and 40-atom cases, as shown in Figs. 2a and b, show a nearly linear variation versus Sb concentration x, which unveils that there is no driving force for phase segregation. Figs. 2c and d show the lattice parameters of the most stable configurations as a function of Sb concentration x for 15-atom and 40-atom cases, respectively. The lattice parameters, for both cases, vary linearly and agree well with the Vegard’s Law31, , which is ascribed to the small lattice mismatch of parent compounds Bi2Te3 and Sb2Te3, demonstrating that the (Bi1−xSbx)2Te3 are ideal to form with arbitrary x.

Figure 1
figure 1

Crystal structure.

(a) The hexagonal conventional unit cell of (Bi1−xSbx)2Te3 and (b) side view of the quintuple layer structure of perfect bulk, (c) Te1 vacancy and (d) TeBi antisite. Numbers in panel (a) are the possibility sites of native point defects.

Full size image
Figure 2
figure 2

Cohesive energies, lattice parameters and projection of the lowest-conduction band.

Cohesive energies per formula unit of the most stable configurations of (a) 15-atom and (b) 40-atom (Bi1−xSbx)2Te3 at each x and the lattice parameters of (Bi1−xSbx)2Te3 with (c) 15-atom and (d) 40-atom plotted as a function of the Sb content x. The solid lines represent the lattice parameters calculated from Vegard’s law. (e) and (f) Projection of the lowest-conduction band at Γ point on Te and Sb (Bi) pz orbital as functions of the SOC strength (λ0 is the actual SOC strength) with x = 1 and the Sb content x, respectively.

Full size image

Electronic structures of (Bi1−xSbx)2Te3

The topological insulators Bi2Te3 and Sb2Te3, as distinct from normal insulators, are band insulators with band invention driven by strong SOC5. To get a better understanding of the inversion, the projections of the lowest-conduction band of (Bi0Sb1)2Te3 at Γ point on Sb and Te pz orbital versus the relative SOC strength λ/λ0 are plotted in Fig. 2e . For week relative SOC strength, λ/λ0< 0.68, it is mainly coming from the Sb pz orbital. The SOC forces a particular relation between the spin and orbital angular momentum, which leads to a level repulsion that pushing down the Sb pz orbital and pushing up the Te pz orbital, resulting in the two orbitals reverse when the relative SOC is strong enough (λ/λ0 = 0.68), namely, band inversion occurs and the system twists into the topological insulator phase through a topological quantum phase transition. For the solid-solution (Bi1−xSbx)2Te3, Fig. 2f shows the projections of the lowest-conduction band at Γ point as a function of the Sb concentration x. The lowest-conduction band at Γ point almost comes from the pz orbital of Te atoms over the entire range of x from 0 to 1, indicating that the SOC of (Bi1−xSbx)2Te3 is strong enough to inverse the band and the topological insulator phase remains for (Bi1−xSbx)2Te3 with , which agrees with the experimentally observation that the topological surface states are shown to exit with 26,27.

The calculated band structures of (Bi1−xSbx)2Te3 on the (111) surface with five QLs27 are shown in Fig. 3 . For both end members (Bi1Sb0)2Te3 and (Bi0Sb1)2Te3, as shown in Figs. 3a and b, the topological surface states form a single Dirac cone at the Γ point, which is consistent with the previous reported results5. However, the shapes of their Dirac cones are significantly different. Fig. 4a shows the three-dimensional upper Dirac cone around the Γ point of the (Bi1Sb0)2Te3 (111) surfaces. Similar to the previous experiment7 and theory5,32,33, the ideal circular constant-energy contours, centered at the Γ point, appear below a critical energy, but the Dirac cone is significantly warped above this critical energy. For the lower Dirac cone of (Bi1Sb0)2Te3, as shown in Figs. 3a , 4a, and 4c, it is concave and the Dirac point is buried in the bulk valence band, making it cannot be accessed by transport experiment and opening the electron scattering channel from surface states to bulk continuum states, while (Bi0Sb1)2Te3 shows an ideal ‘x’-shaped Dirac point with the Fermi level passes through the Dirac point ( Figs. 3b , 4b, and 4d). To address the mechanism of the defining properties of the striking topological surface states vary systematically with the Bi/Sb ratios, which leads to a series of new TIs26,27, we perform calculations for the band structures of five QLs (Bi1Sb0)2Te3 with rescaled SOC strength shown in Figs. 3c . It can be seen that with decreasing λ/λ0(equivalent to x increasing), the Dirac fermion velocity increases accompanied the slope of the upper Dirac cone becomes steeper. Meanwhile, as shown in Fig. 3d , the Dirac point moves upward, resulting in the distance between the Dirac point and the Fermi level decreasing. These follow the experimental results very well27. With the increase of Sb concentration x, the SOC strength reduces due to that of Sb is weaker than Bi by a factor of 3. Therefore, the weaker SOC strength is the key ingredient for the change brought about by the increasing weight of the Sb when the change in lattice parameters can be neglected (inset of Fig. 3a ). Consequently, it can be easily understand why Bi2Se3 and Sb2Te3 possess the ideal Dirac cone while Bi2Te3 not.

Figure 3
figure 3

Surface states.

The band structures calculated for the 5QLs (Bi1−xSbx)2Te3 with (a) x = 0 and (b) x = 1. (c) and (d) The evolution of the Dirac cone with different SOC strength λ/λ0 (λ0 is the actual SOC strength) for (Bi1Sb0)2Te3. The highest occupied and the lowest unoccupied bands are plotted to clearly show the evolutions. The bottom of the conduction band at Γ point is explicitly aligned to 0 eV in panel (c). The inset in panel (a) shows the zoomed-in view of band for (Bi1Sb0)2Te3 with the lattice parameters of (Bi0Sb1)2Te3. The Fermi level is indicated by the dashed line at 0 eV.

Full size image
Figure 4
figure 4

Comparison of the Dirac cone around the Γ point for (Bi1−xSbx)2Te3 with x = 0 and x = 1.

(a) and (b) the upper Dirac cone for x = 0 and x = 1 and (c) and (d) the lower Dirac cone for x = 0 and x = 1, respectively. The constant-energy contour plots are given at the bottom.

Full size image

Suppression of bulk defects in (Bi1−xSbx)2Te3

The pure Bi2Te3 family is expected to be undoped insulator5, however, the currently available samples always show conductive bulk states owing to defect-induced excess carries6,7,16 and the unacceptably high bulk conductivity is the most challenging problem that prevent the TIs to become functional components of electronic devices. The following results show the solid-solution (Bi1−xSbx)2Te3 can effectively eliminate the unwanted defects and reduce the bulk carrier density. To show this, it is necessary to look at the formation energy, ie., the energy cost of creating the defect and that of a charge neutral defect is given by34

where E(defect) and E(bulk) are the total energies of the system with and without defects. Δni is the change in the number of atom i (i = Bi, Sb, and Te) and μi is the chemical potential of the corresponding atom. The formation of a point defect depends on the growth or annealing conditions, which can be expressed by the chemical potentials, such as for Bi and Te, in thermodynamic equilibrium, the Te-rich condition corresponds to the chemical potential of Te (μTe) set to that of bulk Te, whereas the Te-poor condition corresponds to μTe is taken as, with and μBi equal to the total energy of per formula bulk Bi2Te3 and bulk Bi, respectively.

The most prominent defects in Bi2Te3 family are vacancies and antisite defects on the Bi(Sb), Te1(Se1) and Te2(Se2) sublattices. For (Bi1Sb0)2Te3, the natural samples are n-type semiconductors, mainly owing to the Te vacancies (VTe1 and VTe2) ( Fig. 1c ), or the TeBi antisite defects, ie., by replacement of Bi atoms by Te atoms ( Fig. 1d )7,27,35, while for (Bi0Sb1)2Te3, the natural samples are p-type semiconductors due to the hole type carriers induced by SbTe antisite defect30. Firstly, different sites of defects are considered. Table 1 summarizes the formation energies of Te vacancies and the TeBi antisite defects in 15-atom (Bi1Sb0)2Te3. Similar to the previous studies30,35, the formation energies for all the defects (VTe1, VTe2 and TeBi) are found to be strongly dependent on the sites of defects and the formation energies of vacancies on the Te1 sites (VTe1) are lower than that for vacancies on the Te2 sites (VTe2). The lower vacancy formation energy of VTe1 may be result from the weak van der Waals interaction between quintuple layers5. VTe1 has the higher formation energies at the Te-rich condition than that at the Te-poor condition, namely, the Te-rich limit is needed to reduce the electron-type bulk carries and realize a more insulating bulk. However, for the TeBi formation energies, the situation is drastically different. The Te-rich condition leads to the lower formation energy, meaning more electron-type bulk carries induced by TeBi. Calculations of the formation energies of VTe1, VTe2 and TeBiusing the 40-atom (Bi1Sb0)2Te3 show the similar results ( Table 1 ). Therefore, controlling the growth or annealing conditions is not an effective way to reduce the bulk carrier density.

Table 1 The formation energies Ef (eV) for different sites of vacancies (VTe1and VTe2) and antisite defects (TeBi) in 15-atom and 40-atom (Bi1Sb0)2Te3 at the Te-rich and Te-poor conditions including spin-orbit coupling
Full size table

For the solid-solution (Bi1−xSbx)2Te3, the most stable configuration at a given Sb concentration x is used. Fig. 5a presents the formation energies of the lowest-energy VTe1 and TeBi with x = 0 and x = 1/6 and that of SbTe with x = 1 and x = 1/6 as a function of Te chemical potential. Obviously, when x increases from x = 0 to 1/6, both the formation energies of VTe1 and TeBi become larger for the entire range of Te chemical potentials, indicating that the formation of VTe1 and TeBi is much more difficult for x = 1/6 than that for x = 0, namely the bulk carrier density is reduced effectively. With increasing x, the SOC strength of (Bi1−xSbx)2Te3becomes smaller due to the weaker SOC of Sb than that of Bi. To see the effects of SOC strength on defects formation, formation energies of VTe1 and TeBi under rescaled SOC strength are calculated and the relative formation energies are plotted in Figs. 5c and d, respectively. It is quite remarkable that the relative formation energies of VTe1 and TeBiincrease monotonically with decreasing SOC strength, which implies that the formation of VTe1 and TeBibecomes more difficult by the increasing Sb concentration x. This is in good agreement with the formation energies shown in Fig. 5a . In addition, it should be note that in a recent work35 the formation energies of VTe1 and TeBiwere calculated with neglect of SOC and the values of both end conditions are larger than that of our results, which should compare to the tendency of Figs. 5c and d. However, for the case of SbTe as shown in Fig. 5a , the situation is rather different. The antisite defect SbTe in x = 1 system is much more stable than that in x = 1/6 system though the x = 1 system shows weaker SOC. Thus an available together factor exists. As shown in Figs. 2c and d, the lattice parameters a and c are effectively tuned with the x = 1 system behaves the smallest ones. To investigate their effect, the isotropic strain is applied and strain dependence of relative formation energy of SbTe in (Bi0Sb1)2Te3 is plotted in Fig. 5e . It is found that the relative formation energy increases monotonically with increasing isotropic strain from 0 to 5%. Hence, the decreasing of the formation energy of SbTe is mainly determined by the effect of lattice parameters changes. In addition, we also study the native point defect (VSe) in solid-solution (Bi1−xSbx)2Se3 and the results show that the introducing of Sb is an effect way to reduce the bulk carrier density ( Fig. 5d )36. In a word, the solid-solution is a novel promising pathway to eliminating the bulk states that depends on the changes of both the SOC and the lattice parameters.

Figure 5
figure 5

Formation energies and relative formation energies.

(a) Formation energies of the lowest-energy vacancy (VTe1) and antisite (TeBiand SbTe) defects in (Bi1−xSbx)2Te3 under various Te chemical potential (μTe) and (b) that of Vse1 in (Bi1−xSbx)2Se3 under various μSe. (c) and (d) Relative formation energies of VTe1 and TeBi in (Bi1Sb0)2Te3 as a function of the SOC strength. The formation energy for the actual SOC strength (λ/λ0 = 1) is set to zero. (e) Strain dependence of relative formation energy of SbTe in (Bi0Sb1)2Te3. c0 refers to the lattice constants of the unstrained (Bi0Sb1)2Te3 and Δc to the difference of that between strained and unstrained systems.

Full size image

Discussion

Systematic first-principles calculations are carried out to investigated the band structure engineering in (Bi1−xSbx)2Te3. The effects of artificially rescaled SOC and lattice parameters are considered due to their changes caused by the increasing of Sb concentration. Our results can be summarized into three main points:

  1. 1

    The SOC of (Bi1−xSbx)2Te3 is strong enough to invert the highest valence band and lowest conduction band over the entire range of x, indicating the robustness of bulk Z2 topology.

  2. 2

    For the topological surface states, with decreasing SOC, i.e., increasing x, Dirac point moves upward to the bulk energy gap and Dirac fermion velocity vD increases, resulting in the ideal Dirac point and a series of novel ternary topological insulators.

  3. 3

    The (Bi1−xSbx)2Te3 ternary compounds have much more insulting bulk than the parents due to the formation of the native point defects is eliminated effectively that depends on both lattice changes and reduced SOC.

The highly insulating bulk and tunable Dirac cone have been achieved in (Bi1−xSbx)2Te3, which is consistent with experimentally observation and is significant for the development of dissipationless devices based on the TIs.

Methods

All of the calculations, including geometry relaxation and electronic structure calculations, are performed using the projector augmented wave method with a plane-wave basis set as implemented in the Vienna ab initio simulation package (VASP) code37. The plane wave cutoff energy is 400 eV and the exchange correlation functional is treated by Perdew–Burke–Ernzerhof from generalized gradient approximation (GGA)38. For the Brillouin-zone sampling, a Γ-centered 7×7×5 Monkhorst-Pack k mesh is used for 15-atom conventional cell, a 5×5×5 k mesh for the 40-atom rhombohedral supercell and a 15×15×1 k mesh for the five QLs. Both a 40-atom supercell and a 15-atom conventional cell are used for the bulk investigation30 and a slab of five quintuple layers (QLs) with the vacuum layer of 20 Å to avoid interactions between adjacent slabs for the (111) surface27. The lattice parameters and all the atomic positions are fully relaxed until the residual forces on each atom are smaller than 0.001 eV/Å. SOC is invoked in the calculation on a fully self-consistent basis.