Abstract
As the thickness of a threedimensional (3D) topological insulator (TI) becomes comparable to the penetration depth of surface states, quantum tunneling between surfaces turns their gapless Dirac electronic structure into a gapped spectrum. Whether the surface hybridization gap can host topological edge states is still an open question. Herein, we provide transport evidence of 2D topological states in the quantum tunneling regime of a bulk insulating 3D TI BiSbTeSe_{2}. Different from its trivial insulating phase, this 2D topological state exhibits a finite longitudinal conductance at ~2e^{2}/h when the Fermi level is aligned within the surface gap, indicating an emergent quantum spin Hall (QSH) state. The transition from the QSH to quantum Hall (QH) state in a transverse magnetic field further supports the existence of this distinguished 2D topological phase. In addition, we demonstrate a second route to realize the 2D topological state via surface gapclosing and topological phase transition mechanism mediated by a transverse electric field. The experimental realization of the 2D topological phase in a 3D TI enriches its phase diagram and marks an important step toward functionalized topological quantum devices.
Similar content being viewed by others
Introduction
Despite the wellstudied topological surface states in 3D TIs^{1}, their peculiar gapped surface states in the presence of broken symmetry have still been a subject of intense studies for exotic topological quantum states. Even in the absence of symmetry breaking, the strong coupling between the topological surface states in a 3D TI can be a medium to realize interesting states, e.g. the quantum spin Hall (QSH) effect^{2,3,4}. Such 2D TIs host a surface gap with inverted bands initiated via hybridization of the surface states of the parent 3D TIs. From the theoretical perspective, the gapped surface spectrum exhibits an oscillatory behavior alternating between topologically trivial and nontrivial 2D states as a function of layer thickness^{2,3,4}. Although the hybridization gap has been systematically probed in prototypical 3D TIs Bi_{2}Se_{3} by angleresolved photoemission spectroscopy (ARPES)^{5,6}, and Sb_{2}Te_{3} by scanning tunneling microscopy^{7}, the anticipated 2D TI phases and their QSH states await to be confirmed electrically in device measurements. Previous transport measurements in the hybridization regime were restricted by disorder and inhomogeneity in TI thin films preventing the observation of the surface hybridization gap^{8,9}. Also, the crystal quality of the TI films limited the mobility which constrained the mean free path to microscopic length scales^{10}. From the technical point of view, electrical measurements can resolve the prevailing small energy gap scales beyond the resolution limit of ARPES^{5,6}.
Band inversion through a gapclosing and reopening mechanism is a hallmark of a topological phase transition (TPT)^{1}. In addition to the layerdependent topological phases varying with the thickness of 3D TIs, a TPT can be controllably induced through band distortions by inplane or outofplane magnetic field^{11,12}, strain, or pressure^{13,14}, and electric field^{7,15,16,17,18}. The external inplane magnetic field can induce gapclosing by oppositely shifting the two surface bands^{11}. However, a strong magnetic field is required to fully close the surface gap which is impractical for most applications. Another mechanism with which to transform the band topology is by distorting the crystal lattices, such as strain or pressureinduced TPTs in ZrTe_{5}^{13} and BiTeI^{14}, respectively. Finally, the most preferred route is a TPT driven by perpendicular electric fields, which can realize a functional allelectrical topological switch between normal and inverted gap states by controlling the electric potential between top and bottom surfaces^{16,17}.
In this work, we study the magnetoelectrical transport properties of the ultrathin BiSbTeSe_{2} 3D TIs in the intersurface hybridization regime. We identify the normal insulator and 2D topological insulator phases as indicated by their distinct responses to temperature and magnetic field. We further demonstrate that these trivial and topological insulating phases can be reversibly tuned by a perpendicular electric field through a gapclosing mechanism. The highly tunable topological phases enrich the phase diagram of 3D TIs in the 2D thin limit.
Results
Topological phase diagram
We first perform the firstprinciples calculations for the intersurface hybridized Bi_{0.7}Sb_{1.3}Te_{1.05}Se_{1.95} (hBSTS) 3D TI. The hybridization between the top and bottom surface states of the BSTS creates a tunneling gap. The gap sizes extracted from the calculated band structures for BSTS thickness ≤10 quintuple layers (QL), where 1QL≈ 1 nm^{19}, are shown in Supplementary Figs. 1, 2. The parity of the hybridization gap is evaluated in Supplementary Table 1. A negative parity of the gap occurs when the hole subband is at a higher energy level than the electron subband, and thus the total parity reads negative, analogous to the description of the negative gap due to the bulk band inversion in 3D TIs. Similar to the binary Bi_{2}Se_{3} and Bi_{2}Te_{3} 3D TI compounds^{2,3,4}, the parity of BSTS hybridization gap exhibits an oscillatory pattern, indicating switching between the band and QSH insulators via thickness modulation as shown at the zerofield limit in Fig. 1a.
The hBSTS exhibits unique electromagnetic responses. In a perpendicular magnetic field, the surface states’ hybridization essentially lifts the degeneracy of the N = 0 surface Landau level (LL) and causes a splitting into an electron and holelike N = 0 sublevels^{20}. The hybridization gap can thus be defined as the energy difference between the two sublevels, denoted by E_{0}. The analytical formula of E_{0} reveals a linear relation with magnetic field, expressed as \({E}_{0}={\widetilde{C}}_{0}+\frac{{eB}}{\hslash }{\widetilde{C}}_{2}\frac{{\mu }_{B}{gB}}{2}\)^{21}, where \({\mu }_{B}\) and g are the Bohr magneton and effective gfactor; \({\widetilde{C}}_{0}\) and \({\widetilde{C}}_{2}\) are the parameters related to the constant and quadratic terms of the topological surface states’ Hamiltonian, and the third term describes the Zeeman energy splitting of surface bands (Fig. 1b). Meanwhile, in the presence of an external perpendicular electric field, the surface states’ Hamiltonian is again modified by the electric field as \({H}_{U}=e{{{{{{\bf{E}}}}}}}_{z}\cdot {{{{{\bf{z}}}}}}\)^{15}. The external electric field can induce a Rashbatype splitting in the surface band^{15,16} due to the structure inversion asymmetry in the gapped surface states, resulting in a shrinkage of the hybridization gap (Fig. 1b).
Combining the responses of the hybridization gap to magnetic and electric fields, we construct phase diagrams for hBSTS with trivial and nontrivial topological phases, based on analytical formulations, as illustrated in Fig. 1a. The figures investigate and compare the switching of topological phases in thickness range with distinct topology. For hBSTS with an inverted surface gap, both perpendicular magnetic and electric fields are accountable for the termination of the QSH gap by breaking the timereversal and inversion symmetries, respectively. The critical switching fields, denoted as B_{c} and E_{c}, draw the maximum magnetic and electric fields for the QSH state at which boundaries the surface gap vanishes. In contrast, the normal surface hybridization gap reveals noteworthy phase switching only in electric field, while the outofplane magnetic field results in monotonic widening of the surface gap^{21}. The closure of trivial gap and reopening of inverted gap as controlled by electric field is another route to realize a QSH state.
Hybridization gap
Experimentally we have employed three different methods to evaluate energy gaps in the hBSTS. The first method is thermal activation energy by fitting the temperature dependent conductance. Figure 2a shows the fourterminal resistance (R_{xx}) as a function of back gate voltage (V_{bg}) measured at different temperatures for a 9 nm hBSTS. The amplification of R_{xx} peak with the decrease in temperature implies an insulating state developed at charge neutrality point (CNP) due to the intersurface hybridization. The inset of Fig. 2a displays the G_{xx} (=1/ρ_{xx}) data taken at the CNP on an Arrhenius plot, where two thermally activated conductance slopes corresponding to the bulk (>100 K) and surface activation (<100 K) gaps are observed. Fitting of the second slope to the activation relation \({{{G}}}_{{{xx}}}={{{G}}}_{{{xx}}}^{0}\exp ({{{E}}}_{{{{{{\rm{A}}}}}}}/2{k}_{{{B}}}{{T}})\)^{22} yields a surface activation energy (E_{A}) of ~3 meV. The second method is to probe the nonlinear current–voltage characteristics by measuring the differential conductance (dI/dV). Figure 2b (inset) displays a dI/dV map as functions of bias voltage (V_{b}) and V_{bg} for the 9 nm hBSTS. As the V_{bg} sweeps across the CNP, the dI/dV reaches an overall minimum, which elucidates the nature of the surface gap. The diamondshaped feature arises from charge transport across the surface states when the chemical potential is aligned to or detuned from the hybridization gap at the CNP^{23}. A tunneling gap of ~16 meV is determined from the dI/dV versus V_{b} plot (Fig. 2b) across the center minimum. The third approach is to determine the chemical potential relation, μ(n), by integrating the reciprocal quantum capacitance, 1/C_{Q}, with respect to charge density (n)^{24}. The capacitance dip at the CNP (n ≈ 0/cm^{2}) (Fig. 2c) indicates a minimum density of state (DoS) corresponding to the insulating state. The μ(n) plotted in Fig. 2c reveals a step feature, where the step height of ~21 meV gives the hybridization gap for the 9 nm hBSTS.
The hybridization gap sizes as a function of hBSTS flake thickness extracted from the three methods are summarized in Fig. 2d. The details of the layerdependent hybridization gap analyses are presented in Supplementary Fig. 3–6. We note that the thermal activation is often smeared by disorder potential fluctuation, resulting in a much smaller activation energy gap than the actual surface gap especially when gap size is comparable to the disorder^{25}. Despite varying values, these three methods reveal the same trend of nearly exponential decay with the increase of hBSTS thickness. The hybridization gap can thus be approximated by the relation as \({\triangle }_{h}\propto {{{{{{\rm{e}}}}}}}^{\lambda d}\)^{26}, where the characteristic length (λ) is fitted to ~0.44±0.02 – 0.66±0.01 nm^{−1} for the three approaches. This exponential dependence confirms the singleparticle gap nature by excluding the predicted many body effects, such as topological excitonic states in the overlapped thickness range, as they are less sensitive to the change in film thickness^{27}. Furthermore, our DFT calculations (Supplementary Fig. 1) based on the molecular structure of BSTS, as depicted in Fig. 2d, are in good agreement with our experimental trend, further verifying its singleparticle origin. We note that our measured singleparticle tunneling gap of hBSTS crosses over to the 3D limit at a larger thickness than the previously measured crossover thickness in Bi_{2}Se_{3}^{5}. Such discrepancy can be explained by the resolution limit of ARPES in resolving gap size of the order of meV energy scale. A more detailed triggering of the hybridization gap in Bi_{2}Se_{3} using a tightbinding model^{28} resolves the exponentially decay of the hybridization gap at thickness beyond 6QL. SubmeV hybridization gaps were also detected in 12–17 nm Bi_{2}Se_{3} using phasecoherent transport^{9}.
Normal and inverted gaps
We have noted two inverted surface gap regimes at hBSTS thickness of 9 QL (<10 meV) and 5–6 QL (30–40 meV), based on our DFT calculations of the surface gap topology (Supplementary Table 1). The transport properties in the first inverted regime are studied. Different from the monotonically increasing R_{xx} for the 9 nm hBSTS, the 10 nm sample shows R_{xx} maximum saturating at ~12 kΩ (~h/2e^{2}) at temperature below 50 K. This weak temperature response in the 10 nm hBSTS at low temperature suggests a finite conductance state existing/developing in the hybridization gap^{29}. The full data set of R_{xx} at all temperatures for the 9 nm and 10 nm hBSTS as presented in color maps of Fig. 3a(i) and (ii), respectively, further elaborate their distinct temperature dependence behaviors. Figure 3b compares the gatedependence of R_{xx} at temperature of 1.5 K for the 9 nm and 10 nm hBSTS. The R_{xx} reaches a maximum value of ~500 kΩ (~h/0.05e^{2}) as the chemical potential is tuned into the surface gap, indicating a normal insulating state for the 9 nm hBSTS. Contrary to the strong resistive signal, the 10 nm hBSTS exhibits a finite resistance of ~h/2e^{2} within the hybridization gap regime as indicated by its dI/dV color map inserted in the figure. This again supports the inverted nature of the surface gap for the 10 nm hBSTS. We further show that the twoterminal and nonlocal resistances measured in different configurations agree with the expected values for helical edge states derived from the Landauer–Buttiker formalism^{30} (Supplementary Fig. 7 and Supplementary Table 3). Similar observations are reproduced in an additional 10 nm hBSTS device (Supplementary Fig. 8). However, we note that similar transport features were not observed in the second inverted surface gap regime (5–6 QL) as identified by our DFT calculations. One possible reason is the significant reduction of the mobility below 7 nm, thus shortening the mean free path and obscuring the transport signature for the 2D topological phase. A more complex device configuration as implemented in monolayer WTe_{2}^{10} will be needed to clarify the QSH phase in the second inverted regime.
To quantitatively evaluate the bulk surface and edge transport, we construct a simple conductance model by considering the total conductance as a parallel sum of the surface and edge conductance, \({{G}}_{{xx}}{=}{{G}}_{{{{{\rm{surf}}}}}}{+}{{G}}_{{{{{\rm{edge}}}}}}{=}\frac{1}{{{{{{{\rm{\rho }}}}}}}_{{{{{\rm{surf}}}}}}}{+}\frac{1}{{{{{{{\rm{\rho }}}}}}}_{{{{{\rm{edge}}}}}}}\). The surface resistivity (ρ_{surf}) exhibits a thermally activated behavior with temperature as ~exp(αT), whereas the edge resistivity (ρ_{edge}) will saturate in the quantum limit of ~h/2e^{2} at sufficiently low temperature. Since the edge channel is absent in the normal gap, the total resistivity for 9 nm hBSTS can be simplified to \({{{{{{\rm{\rho }}}}}}}_{{xx}}\left(9\,{{{{{{\rm{nm}}}}}}}\right){=}{{{{{{\rm{\rho }}}}}}}_{{{{{\rm{surf}}}}}}\). Thus, the total resistivity for 10 nm hBSTS can be approximated to the relation as \(\frac{1}{{{{{{{\rm{\rho }}}}}}}_{{xx}}\left(10\,{{{{\rm{nm}}}}}\right)}\approx \frac{1}{{{{{{{\rm{\rho }}}}}}}_{{xx}}\left(9\,{{{{\rm{nm}}}}}\right)}{+}\frac{1}{{{{{{{\rm{\rho }}}}}}}_{{{{{\rm{edge}}}}}}}\). To verify this relation, we plot the ρ_{edge} as a function of temperature as inserted in Fig. 3a. Figure 3a shows excellent quantization of the edge conductance at a value of h/2e^{2} in accordance with the quantum spin Hall state. This analysis also shows that the quantization of ρ_{edge} persists up to 20 K, indicating a QSH gap of ~2 meV for the 10 nm hBSTS. We note that this QSH gap is smaller than the surface hybridization gap (~8 meV) as estimated from the differential conductance measurement. Given that the device size is larger than the inelastic mean free path, backscattering between the helical edge states^{31} can occur at higher temperature, which can cause the deviation of ρ_{edge} from quantization.
Magneticfield response
The normal and inverted surface gaps of hBSTS present distinct behaviors in perpendicular magnetic field. We first discuss the normal gap feature. Line profiles in Fig. 3c plot the longitudinal conductivity (σ_{xx}) and Hall conductivity (σ_{xy}) as a function of V_{bg} measured at a magnetic field of 18 T for the 9 nm hBSTS. The two N = 0 LL bands developed in the σ_{xx} plot are assigned to the electron and hole band edges of the hybridization gap as illustrated by the schematic of the surface band diagram in the inset of Fig. 3c. This splitting of N = 0 LLs is a key signature of the Landau quantization of intersurface hybridization^{20,21}. The established ν = 0 plateau in the σ_{xy} plot within the N = 0 LL, together with the development of ν = −1 and +1 QH plateaus symmetrically about zeroth plateau, further supports the LLs of the hybridized surface states. Color map of σ_{xx} in Fig. 3c illustrates the development of the N = 0 LLs as a function of magnetic field. Similar behaviors were also observed for the 8 nm hBSTS (Supplementary Fig. 9). The N = 0 LL energy spacing (E_{0}) derived from the μ(n) relation shows a linear magnetic field dependence (Supplementary Fig. 10), in agreement with the analytical model^{21}. The fitting of E_{0} to magnetic field yields \({\widetilde{{{{{{\boldsymbol{C}}}}}}}}_{{{{{{\boldsymbol{0}}}}}}}\) ~ 0.016 eV and \({\widetilde{{{{{{\boldsymbol{C}}}}}}}}_{{{{{{\boldsymbol{2}}}}}}}\) ~ 80 eV Å^{2} for the 9 nm hBSTS, which is comparable to the fitting parameters of the surface states’ Hamiltonian^{32}.
The gatedependent σ_{xx} and σ_{xy} plots for the inverted gap hBSTS (10 nm) measured at the magnetic field of 18 T (Fig. 3d) show a similar type of N = 0 LLs splitting. The relatively narrow zeroth LL plateau width in charge density compared to the 9 nm hBSTS is consistent with its smaller size of hybridization gap as previously discussed in the gap analyses. Interestingly, the development of the N = 0 LLs with magnetic field traced by the dashed lines along the σ_{xx} minimum in Fig. 3d reveals an intriguing feature. The two N = 0 sublevels develop oppositely as revealed by the σ_{xx} color map at low magnetic field, signifying the inversion of electron and hole sublevels at zero field. As the magnetic field increases, these two sublevels eventually cross and develop into normal QH states with the two sublevels interchanged. This is equivalent to an invertedtonormal surface gap evolution with magnetic field to transform from its topologically nontrivial into a trivial gap state as illustrated in surface band diagram in Fig. 3d. This magnetic field mediated TPT is similar to the band crossing for the QSH effect observed in HgTe/CdTe quantum well 2D TIs^{33}. The sublevels’ crossing feature forming symmetrically about the opposite magnetic field further confirms this magnetic fielddriven topological transition.
Figure 3c, d(i) compare G_{xx} curves as a function of magnetic field for the 9 and 10 nm hBSTS, respectively, with their chemical potentials tuned into surface gaps. The levels crossing point at B_{c} of ~9(−9) T is clearly resolved in the G_{xx} curves of the 10 nm hBSTS (Fig. 3d). This B_{c} appears to be consistent with its surface gap size (~8 meV) obtained from gap analyses and the opening rate of ~1 meV/T revealed in 9 and 8 nm hBSTS (Supplementary Fig. 10). A similar feature is captured in a parallel magnetic field (Supplementary Figs. 11, 12) except no QH state evolves. Comparing with quantum wells, the QSH state in hBSTS persists strongly in magnetic field. The G_{xx} falls off more slowly with the magnetic field as inferred by the width of G_{xx} peak centered at zero magnetic field of ~2 T (~28 mT for HgTe/CdTe quantum wells^{31}). Also, the slope of dG_{xx}/dB in linear region (B = 0–2 T) of ~0.2–0.3 e^{2}/h/T is nearly two orders of magnitude smaller than the quantum wells^{31,34}. The dG_{xx}/dB slope reflects directly the disorder strength (W) of the QSH state^{35}. In our case, the estimated W of a few meV indicates a moderate to weak disorder (W ≤ ∆_{h}) regime with insignificant bulk scattering, thus providing a cleaner platform to study the onedimensional edge channel.
Electricfield response
An external electric field can also modulate the surface bands to induce TPTs. The effect of a transverse electric field can be triggered by controlling voltages on top and bottom gates through the hBSTS as illustrated in Fig. 1c. The diagonal resistance maximum along the transverse electric field observed in the dualgate mapping (Supplementary Fig. 15) presents a key feature resulting from the strong coupling between surface states. To better elucidate the electric field response, we converted the dualgate voltages into the displacement field (D) versus total charge density (n) relation^{36}, as presented in descending order of thickness in Fig. 4a–d. Magnitudes of the corresponding hybridization gaps at zero D are revealed by their dI/dV maps inserted in the figures. Line profile of ρ_{xx} versus D (Fig. 4a) for the inverted gap hBSTS (10 nm) shows a gradual decrease from its quantized h/2e^{2} value at large D, inferred as a transition into a semimetallic phase. Whereas for the normal gap hBSTS (9 nm), the ρ_{xx} at the CNP is suppressed by more than one order of magnitude at D ~100 mV/nm and tends to saturate to a value of order h/e^{2} at large D, suggesting a significant gap reduction by D (Fig. 4b). This ρ_{xx} change is highly symmetric with respect to opposite polarities of D. The suppression of ρ_{xx} in D becomes less pronounced in thinner hBSTS (Fig. 4c, d), further signifying modulation of hybridization gap with D. For comparison, we plot the change in resistivity ∆ρ_{xx}/ρ_{xx}(0) for the variable thickness hBSTS in the normal gap regime in Fig. 4e. Consistently, the ∆ρ_{xx}/ρ_{xx}(0)–D response scales down monotonically with decreasing thickness of hBSTS.
To further investigate the ∆_{h}–D relation, we again deploy our three gap estimation methods to probe the gap size for the 9 nm hBSTS versus D, as summarized in Fig. 4f. The gap analyses are examined in detail in Supplementary Fig. 16–19. Consistent with the ρ_{xx}–D trend, a significant reduction in hybridization gap with D is indicated by all three methods. The same trend is observed in opposite polarity of D, suggesting that the gap reduction is due to broken structure inversion symmetry between top and bottom surface states by transverse electric field^{15,16}. The appreciable reduction of the surface gap is attributed to the electric field that is effectively applied through the vdW layers. The gap size reaches a minimum saturation value of <1 meV, indicating a critical displacement field, D_{C} at ~250 mV/nm. Similar analyses are implemented for the 8 and 6 nm hBSTS, while for simplicity only gap calculated from C_{Q} are included in Fig. 4f. Their D_{C} can be extrapolated from the curves as ~400 and ~750 mV/nm for the 8 and 6 nm hBSTS, respectively. Meanwhile, an effective gapclosing strength of ~1.2 eÅ can be estimated from the ∆_{h}/D ratio for the 9 nm hBSTS, which is comparable to DFT calculated value ∆_{h}/E of ~2–3 eÅ (Supplementary Figs. 13, 14).
A direct consequence of the surface gap termination is the emergence of topological edge state due to the change in surface band topology. To analyze this effect, we again apply the parallel conductance addition model and evaluate ρ_{edge} for the 9 nm hBSTS at applied D values. Assuming an edge channel evolves with D, ρ_{xx}(D) will follow the relation: \(\frac{1}{{{{\rho }}}_{{xx}}\left({D}\right)}{=}\frac{1}{{{{\rho }}}_{{xx}}\left(0\right)}{+}\frac{1}{{{{\rho }}}_{{{{{\rm{edge}}}}}}}\). From this relation, ρ_{edge}(D) is obtained at different temperatures and plotted in Fig. 4g showing suppression of thermal activation with the increase in D. This analysis shows that, at large D (250 mV/nm), ρ_{edge} behavior resembles the temperature curve of the inverted gap hBSTS (inset of Fig. 3a) and approaches the h/2e^{2} value at low temperature (<40 K), indicating an emerging helical edge state. This gapclosing mechanism mediated by transverse electric field is illustrated by the evolution of surface spectra (insets of Fig. 4g), which offers an alternative route toward hightemperature topological edge channels.
In summary, we realized distinct topological phases by mapping the evolution of the surface gap of BSTS 3D TI with thickness, temperature, and transverse magnetic and electric fields in the quantum tunneling regime. The trivial and topological phases in hBSTS were identified by their diverging and finite (~h/2e^{2}) resistances, respectively, in response to temperature and gate voltages. The helical edge state reveals more than one order of magnitude smaller disorder strength compared to HgTe/CdTe quantum wells. By studying the development of zeroth LL subbands in the perpendicular magnetic field, we observed an invertedtonormal surface gap crossing accompanied by a transition from QSH to QH edge states. We further realized surface gap closing with electric field, together with an emerging edge conductance for the normal insulating gap hBSTS, implying a transition between trivial and topological phases mediated by the electric field. These compelling signatures of TPTs and the reversible switching mechanisms hold promise for TIbased topological fieldeffect transistors (Supplementary Table 4).
Methods
DFT calculations
Our calculations were performed within the framework of density functional theory as implemented in the Vienna ab initio simulation (VASP)^{37}. The projector augmented wave potentials were adopted with the generalized gradient approximations of Perdew–Burke–Ernzerhof exchangecorrelation functional, and the cutoff energy was set to 520 eV. The relaxation is performed until all forces on the free ions converge to 0.01 eV/Å and the MonkhorstPack kpoint meshes of 10 × 10 × 1 were used, which have been tested to be well convergence. The vacuum space is at least 20 Å, which is large enough to avoid the interaction between periodical images. The different BSTS systems were treated with virtual crystal approximation. The van der Waals interaction is described by DFTD3 method. In addition, the spinorbitcoupling was included in the calculations of the electronic structure.
Device fabrication
Variable thicknesses of hBSTS crystal flakes were exfoliated from the bulk crystal^{38} and then transferred using a micromanipulator transfer stage into the heterostructures of Gr/hBN sandwiched layers. We fabricated the hBSTS devices into the Hall bar configuration with Cr/Au (2 nm/60 nm) as the contact electrodes. The top and bottom Gr/hBN layers serve as the gateelectrode/dielectric layers for applying of perpendicular electric field to the hBSTS (Fig. 1c). The thickness of the BSTS flakes was measured by a Bruker Dimension Icon atomic force microscopy. The device dimensions were obtained from the images taken by a highresolution optical microscope. hBSTS devices with thickness ranging from 10 nm down to 1 nm were fabricated and studied. The device specifications are presented in Supplementary Table 2.
Measurements
Lowtemperature electrical transport measurements were performed in a helium4 variable temperature insert with a base temperature of 1.6 K and magnetic field up to 9 T. Four probe resistances of the devices were measured using a Stanford Research SR830 lockin amplifiers operated at a frequency of 17.777 Hz. The devices were typically sourced with a constant AC excitation current of 10–20 nA. Two Keithley 2400 source meters were utilized to source DC gate voltages separately to the top and bottom Gr gate electrodes. Variable temperature transport measurements were controlled by a Lakeshore temperature controller. The differential conductance was measured using a Stanford Research SR830 lockin amplifier coupled to a model 1211 current preamplifier. The devices were sourced with an AC excitation voltage of 100 μV. The DC bias voltage in a range from 40–1200 mV was swept across the sourcedrain electrodes. Magnetotransport measurements at high magnetic field were carried out in a helium3 variable temperature insert at a base temperature of 0.3 Kelvin and magnetic field up to 18 tesla based at the National High Magnetic Field laboratory. Two synchronized Stanford Research SR830 lockin amplifiers were used to measure the longitudinal and Hall resistances concurrently on the BSTS devices. Capacitance was measured in a capacitance bridge configuration^{24} connected between the BSTS device and the parallel gold strip as a reference capacitor. Two synchronized (at a frequency of ~50–70 kHz) and nearly equalamplitude AC excitation voltages (range of 15–40 mV) were applied separately to the top and bottom Gr gates, whose relative magnitude was chosen to match the ratio of geometric capacitances of top and bottom surfaces. A third AC excitation voltage was applied to the reference capacitor with the amplitude set to null the measured signal. The reference capacitors were calibrated to be ~300–400 fF using a standard capacitor (Johanson Technology R14S, 1 pF). The capacitance data were acquired by monitoring the offbalance current at the balance point as the DC gate voltages were changed.
Data availability
The data that support the findings of this study are available within the article and its supplementary information files. Source data are provided with this paper.
References
Xu, S.Y. et al. Topological phase transition and texture inversion in a tunable topological insulator. Science 332, 560–564 (2011).
Liu, C.X. et al. Oscillatory crossover from twodimensional to threedimensional topological insulators. Phys. Rev. B 81, 041307 (2010).
Lu, H.Z., Shan, W.Y., Yao, W., Niu, Q. & Shen, S.Q. Massive Dirac fermions and spin physics in an ultrathin film of topological insulator. Phys. Rev. B 81, 115407 (2010).
Linder, J., Yokoyama, T. & Sudbø, A. Anomalous finite size effects on surface states in the topological insulator Bi_{2}Se_{3}. Phys. Rev. B 80, 205401 (2009).
Zhang, Y. et al. Crossover of the threedimensional topological insulator Bi_{2}Se_{3} to the twodimensional limit. Nat. Phys. 6, 584–588 (2010).
Sakamoto, Y., Hirahara, T., Miyazaki, H., Kimura, S.I. & Hasegawa, S. Spectroscopic evidence of a topological quantum phase transition in ultrathin Bi_{2}Se_{3} films. Phys. Rev. B 81, 165432 (2010).
Zhang, T., Ha, J., Levy, N., Kuk, Y. & Stroscio, J. Electricfield tuning of the surface band structure of topological insulator Sb_{2}Te_{3} thin films. Phys. Rev. Lett. 111, 056803 (2013).
Nandi, D. et al. Signatures of longrangecorrelated disorder in the magnetotransport of ultrathin topological insulators. Phys. Rev. B 98, 214203 (2018).
Kim, D., Syers, P., Butch, N. P., Paglione, J. & Fuhrer, M. S. Coherent topological transport on the surface of Bi_{2}Se_{3}. Nat. Commun. 4, 2040 (2013).
Wu, S. et al. Observation of the quantum spin Hall effect up to 100 kelvin in a monolayer crystal. Science 359, 76–79 (2018).
Xu, Y., Jiang, G., Miotkowski, I., Biswas, R. R. & Chen, Y. P. Tuning insulatorsemimetal transitions in 3D topological insulator thin films by intersurface hybridization and inplane magnetic fields. Phys. Rev. Lett. 123, 207701 (2019).
Zhang, S.B., Lu, H.Z. & Shen, S.Q. Edge states and integer quantum Hall effect in topological insulator thin films. Sci. Rep. 5, 13277 (2015).
Mutch, J. et al. Evidence for a straintuned topological phase transition in ZrTe5. Sci. Adv. 5, eaav9771.
Xi, X. et al. Signatures of a pressureinduced topological quantum phase transition in BiTeI. Phys. Rev. Lett. 111, 155701 (2013).
Liu, G., Zhou, G. & Chen, Y.H. Modulation of external electric field on surface states of topological insulator Bi_{2}Se_{3} thin films. Appl. Phys. Lett. 101, 223109 (2012).
Kim, M., Kim Choong, H., Kim, H.S. & Ihm, J. Topological quantum phase transitions driven by external electric fields in Sb_{2}Te_{3} thin films. Proc. Natl Acad. Sci. 109, 671–674 (2012).
Collins, J. L. et al. Electricfieldtuned topological phase transition in ultrathin Na_{3}Bi. Nature 564, 390–394 (2018).
Qian, X., Liu, J., Fu, L. & Li, J. Quantum spin Hall effect in twodimensional transition metal dichalcogenides. Science 346, 1344–1347 (2014).
Ko, W. et al. Atomic and electronic structure of an alloyed topological insulator, Bi_{1.5}Sb_{0.5}Te_{1.7}Se_{1.3}. Sci. Rep. 3, 2656 (2013).
Zyuzin, A. A. & Burkov, A. A. Thin topological insulator film in a perpendicular magnetic field. Phys. Rev. B 83, 195413 (2011).
Pertsova, A., Canali, C. M. & MacDonald, A. H. Thin films of a threedimensional topological insulator in a strong magnetic field: microscopic study. Phys. Rev. B 91, 075430 (2015).
Cho, S., Butch, N. P., Paglione, J. & Fuhrer, M. S. Insulating behavior in ultrathin bismuth selenide field effect transistors. Nano Lett. 11, 1925–1927 (2011).
Han, M. Y., Özyilmaz, B., Zhang, Y. & Kim, P. Energy bandgap engineering of graphene nanoribbons. Phys. Rev. Lett. 98, 206805 (2007).
Chong, S. K., Tsuchikawa, R., Harmer, J., Sparks, T. D. & Deshpande, V. V. Landau levels of topologicallyprotected surface states probed by dualgated quantum capacitance. ACS Nano 14, 1158–1165 (2020).
Rosen, I. T. et al. Absence of strong localization at low conductivity in the topological surface state of lowdisorder Sb_{2}Te_{3}. Phys. Rev. B 99, 201101 (2019).
Shan, W.Y., Lu, H.Z. & Shen, S.Q. Effective continuous model for surface states and thin films of threedimensional topological insulators. N. J. Phys. 12, 043048 (2010).
Tilahun, D., Lee, B., Hankiewicz, E. M. & MacDonald, A. H. Quantum Hall superfluids in topological insulator thin films. Phys. Rev. Lett. 107, 246401 (2011).
Pertsova, A. & Canali, C. M. Probing the wavefunction of the surface states in Bi_{2}Se_{3} topological insulator: a realistic tightbinding approach. N. J. Phys. 16, 063022 (2014).
Du, L., Knez, I., Sullivan, G. & Du, R.R. Robust helical edge transport in gated InAs/GaSb bilayers. Phys. Rev. Lett. 114, 096802 (2015).
Roth, A. et al. Nonlocal transport in the quantum spin Hall state. Science 325, 294–297 (2009).
König, M. et al. Quantum spin Hall insulator state in HgTe quantum wells. Science 318, 766–770 (2007).
Liu, C.X. et al. Model Hamiltonian for topological insulators. Phys. Rev. B. 82, 045122 (2010).
Bernevig, B. A., Hughes Taylor, L. & Zhang, S.C. Quantum spin Hall effect and topological phase transition in HgTe quantum wells. Science 314, 1757–1761 (2006).
Gusev, G. M., Olshanetsky, E. B., Kvon, Z. D., Mikhailov, N. N. & Dvoretsky, S. A. Linear magnetoresistance in HgTe quantum wells. Phys. Rev. B 87, 081311 (2013).
Maciejko, J., Qi, X.L. & Zhang, S.C. Magnetoconductance of the quantum spin Hall state. Phys. Rev. B. 82, 155310 (2010).
Chong, S. K., Han, K. B., Sparks, T. D. & Deshpande, V. V. Tunable coupling between surface states of a threedimensional topological insulator in the quantum Hall regime. Phys. Rev. Lett. 123, 036804 (2019).
Kresse, G. & Hafner, J. Ab initio molecular dynamics for openshell transition metals. Phys. Rev. B 48, 13115–13118 (1993).
Han, K.B. et al. Enhancement in surface mobility and quantum transport of Bi_{2−x}Sb_{x}Te_{3−y}Se_{y} topological insulator by controlling the crystal growth conditions. Sci. Rep. 8, 17290 (2018).
Acknowledgements
This material is based upon work supported by the National Science Foundation, the Quantum Leap Big Idea under Grant No. 1936383. A portion of this work was performed at the National High Magnetic Field Laboratory, which is supported by National Science Foundation Cooperative Agreement No. DMR1644779 and the State of Florida. The authors acknowledge Brian Skinner for helpful comments.
Author information
Authors and Affiliations
Contributions
S.K.C. and V.V.D. designed, conducted the experiments, and prepared the manuscript. K.W. and T.T. provided single crystal hexagonal boron nitride. T.D.S. provided a single crystal BiSbTeSe_{2} threedimensional topological insulator. L.L. and F.L. performed theoretical calculations to support the experimental data. All authors contributed to the discussion of the results and approved the final version of the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Source data
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Chong, S.K., Liu, L., Watanabe, K. et al. Emergent helical edge states in a hybridized threedimensional topological insulator. Nat Commun 13, 6386 (2022). https://doi.org/10.1038/s41467022336439
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467022336439
This article is cited by

Anomalous Landau quantization in intrinsic magnetic topological insulators
Nature Communications (2023)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.