Introduction

Higher-order nonlinear optical processes in solids are sensitive to the crystal lattice symmetry through the nonlinear susceptibilities χ(n)(n = 2, 3, …)1. For instance, surface second-harmonic generation in solids has been widely used as a tool to detect symmetry breaking on the surface2, which leads to non-zero χ(2), while χ(2) remains zero in the centrosymmetric bulk region. In addition, cascaded lower-order nonlinear processes frequently dominate over direct higher-order nonlinear processes, e.g. the cascaded second-order susceptibility leads to an effectively third order susceptibility χ′(3)(=χ(2) · χ(2))3, which is in some cases greater than the direct susceptibility χ(3). Thus, higher-order nonlinear optical processes on solid surfaces are complex phenomena and are often useful for exploring symmetry and have been applied to the detection of topological surface states (SSs)4,5.

Topological insulators (TIs) are new quantum phase of matter having an electronic band gap in the bulk and robust SSs on the surface6,7,8. Bi2Se3 is a well-known TI while Bi2Te3 and Sb2Te3 are also considered to be model TIs although Sb2Te3 has been seldom investigated experimentally due to its strong p-type behavior stemming from point defects8,9. In the prototypical TI, Bi2Se3, the crossover from a 2D insulator to a 3D TI occurs at a thickness of six quintuple layers (QL), while for Bi2Te3 and Sb2Te3 this crossover is believed to occur at 3 QL and 4 QL, respectively9,10,11. Under an external electric field, moreover, it may be possible to switch a 3D-TI to a 2D insulator or vice versa, depending on the sample thickness12. Characterization of the crossover from a 2D insulator to 3D-TI is generally difficult using optical methods where a near infrared laser pulse is applied, mainly due to the large optical penetration depth, which can result in the surface state contribution being masked by the much stronger bulk contribution. Therefore, SHG has been applied to selectively detect only the surface contribution13. Here we demonstrate that nonlinear magnetic excitation and the subsequent probing of the cascaded second-order susceptibility can explicitly detect the presence of a surface Dirac cone and revisit the 2D insulator to 3D-TI crossover in X2Te3 (X = Bi, Sb) thin films, whose thickness are less than 10 nm.

Our idea is described in Fig. 1. As is well established, in TI the quantum spin Hall (QSH) effect occurs due to the presence of inherently strong spin-orbit coupling (SOC) (Fig. 1a). The QSH phase is defined as a time-reversal-symmetric system, which is gapped in the bulk while remaining gapless at the edge (surface), stemming from the system’s topology14. Here we consider a TI system under irradiation by an ultrashort laser pulse (Fig. 1b). In general, transient magnetization \(\overrightarrow{M}\) can be induced via the inverse Faraday effect (IFE) by irradiation with circularly polarized light pulses1,15, which can be measured through the rotation of the linearly polarized light transmitted through a magnetic medium15,

$${\theta }_{F}=\frac{\chi }{n}\overrightarrow{M}\cdot \overrightarrow{k},$$
(1)

and

$$\overrightarrow{M}=\frac{\chi }{16\pi }\overrightarrow{E}(\omega )\times {\overrightarrow{E}}^{\ast }(\omega ),$$
(2)

where χ is the magneto-optical susceptibility, n is the refractive index of the medium, \(\overrightarrow{k}\) is the wavevector, and \(\overrightarrow{E}\)(ω) is the electric field of the incident light with frequency ω. The first relation, Eq. (1), corresponds to detection via the Faraday rotation and the second, Eq. (2), characterizes the pumping action, which can be regarded as optical rectification1. In the case of pump-probe methods, \({\theta }_{F}=(\chi /n)\,(\chi \mathrm{/16}\pi )\,|\overrightarrow{E}(\omega {)|}^{2}\). This confirms the cascading nature of the magneto-optical susceptibility, χ ⋅ χ. Each χ is second-order, so χ ⋅ χ is χ(2) ⋅ χ(2), which corresponds to χ(3) (ref.1). This means that the third-order nonlinear effect is the combination of the pump and probe actions16. In the case of TI materials, we can measure the transient magnetization \(\overrightarrow{M}\) through the optical Kerr effect (OKE)1,17, which is a rotation of the linearly polarized light reflected from the sample. In our experiment, we used extremely thin samples of TIs. Under this ultimate condition, it is possible to detect dominant contributions from the surface, as explained below. In the bulk region, where χ(3) ≠ 0 and χ(2) = 0, we observed only the usual IFE via direct OKE, while at the surface, χ(3) ≠ 0 and χ(2) ≠ 04,18, leading to cascading OKE, whose amplitude can be written as χ′(3) = χ(2) · χ(2), can even dominate (Fig. 1b).

Figure 1
figure 1

Schematic of the quantum spin Hall effect and the pump-probe experiment. (a) In TIs, the quantum spin Hall effect occurs due to inherent strong spin-orbit coupling (SOC), which leads to edge (surface) states with two spins (spin-up and spin-down) propagating in the opposite directions. (b) A schematic diagram of the experiment. The TI has bulk and surface domains indicated by the shaded gray and green colors respectively. VB and CB denote the valence and conduction bands, respectively. In the bulk domain, χ(3) ≠ 0 and χ(2) = 0, whereas χ(2) ≠ 0 appears at the surface due to the breaking of inversion symmetry at the surface. The Kerr rotational signal (Δθ k ) from the TI surface states is dominated by cascading second-order nonlinear effects (χ′(3) = χ(2) · χ(2)), which have a stronger response than a direct higher (χ(3)) order nonlinear process.

Full size image

Results and Discussion

Figure 2 presents the time evolution of the Kerr-rotation (Δθ k ) signals for two Sb2Te3 samples (2 and 4 QL thick) as a function of the time delay between the pump and probe pulses under optical excitation by circularly polarized pump pulses. The Δθ k signal changes sign upon the reversal of the helicity of the pump pulses from left to right circular polarization which are denoted by and , respectively as shown in Fig. 2a,b. An instantaneous change in the Δθ k signal near zero delay is observed due to the photoexcitation of spin-polarized electrons to unoccupied surface states (for 4 QL) or to bulk excited states (for 2 QL), known as the IFE15,17,19.

Figure 2
figure 2

Time-evolution of the Kerr-rotation. (a) The Δθ k signal for 2 QL Sb2Te3. (b) The Δθ k signal for 4 QL Sb2Te3. The symbol denotes circularly right handed polarized and circularly left handed polarized photons.

Full size image

In order to investigate the critical QL thickness at which the 2D insulator to 3D-TI crossover occurs, we have measured the pump-polarization (helicity) dependence of the Δθ k signal for Sb2Te3 samples with thickness in the range from 2–10 QL. Figure 3a presents the peak amplitude of the Δθ k signal for several samples with differing numbers of QL, which exhibits a periodic oscillation with variation of the pump polarization. It is interesting to note that for the 2 QL thick sample, the oscillation displayed a two-cycle oscillation with a periodicity of π, while robust changes in the oscillatory pattern are observed for samples with thickness greater than 4 QL in the form of a four-cycle oscillation with a periodicity π/2. The abrupt change in periodicity at 4 QL clearly suggests the presence of a crossover.

Figure 3
figure 3

Helicity dependence of the Kerr-rotation signal. (a) The peak amplitude of the Δθ k signal as a function of the pump-polarization state (helicity) for Sb2Te3. The solid line is a fit to the data using Eq. 3. The oscillatory pattern of the Kerr rotation shows a drastic change in periodicity at 4 QL and 10 QL for Sb2Te3. (b) The same plots for amorphous GeTe films. The periodicity (π) is the same for a 2 QL Sb2Te3 sample in (a). (c) The Kerr rotational amplitude measured as a function of pump power for two different QWP angles at 45° and 25°, corresponding to circularly and elliptically polarized pump pulses, respectively. The solid line represents a linear fit.

Full size image

To confirm the behavior in the periodicity of the helicity dependence of the Δθ k signal for Sb2Te3 samples with a thickness of 4 QL or greater arises from a 2D insulator to 3D-TI crossover, we have carried out measurements on amorphous GeTe, a trivial insulator, for samples with a range of thicknesses (Fig. 3b). The helicity dependence of the Δθ k signal always exhibits an oscillating pattern with a periodicity of π with increasing amorphous GeTe thickness. The periodicity (i.e. π) of the helicity dependence of the Δθ k signal is the same with that observed in crystalline GaAs20, which belongs to normal insulators as well as for the 2 QL Sb2Te3 sample (Fig. 3a). This is in accordance with the fact that topological surface states are absent in the surface of the 2 QL thick Sb2Te3 sample, and thus it shows similar behavior to the normal insulator GeTe.

The measured Δθ k signal is linearly dependent on the pump power \(I=|\overrightarrow{E}(\omega {)|}^{2}\) as shown in Fig. 3c. The quadratic nature of the Δθ k signal with \(|\overrightarrow{E}(\omega )|\) supports the hypothesis that the transient magnetization \(\overrightarrow{M}\) induced on the TI can be treated as phenomenologically similar to a nonlinear process observed in a magnetic medium15,17,19. Since our experimental technique utilizes a reflection geometry, the observed phenomena should be given by the specular IFE and specular OKE, which have been reported to occur in semiconductor and metallic systems21,22. There, the semiconductor GaAs exhibited a π oscillation, while Al metal exhibited a π/2 oscillation as a function of the polarization state (helicity) of the pump light20. The π and π/2 periodicities were assigned to specular IFE and specular OKE, respectively, and the relation between Δθ k and third-order nonlinear susceptibility χ(3) was presented21,22. For the case of a TI, on the other hand, strong SOC leads to enhancement of the second-order susceptibility χ(2) from the surface state23, resulting in a cascading OKE via χ′(3) = χ(2) · χ(2)3,24,25. Thus, the oscillatory pattern of the Δθ k signal with helicity can be attributed to a combination of the specular IFE and specular OKE, or χ(3) + χ′(3) = χ(3) + χ(2) · χ(2). As such, the Kerr rotation in the case of a TI can be described by referring the expression for the specular IFE and specular OKE21,22,

$${\rm{\Delta }}{\theta }_{k}=\frac{32{\pi }^{2}{I}_{pump}}{c\mathrm{|1}+n{|}^{2}}(L\,\sin \,4\alpha +C\,\sin \,2\alpha )+D,$$
(3)

where

$$L=Re(\frac{{\chi }^{\mathrm{(2)}}\cdot {\chi }^{\mathrm{(2)}}}{2n{\mathrm{(1}-n)}^{2}}),$$
(4)
$$C=-Im(\frac{{\chi }^{\mathrm{(3)}}}{n{\mathrm{(1}-n)}^{2}}).$$
(5)

Here, α is the angle of the quarter wave plate (QWP; λ/4 plate), I pump is the pump intensity, c is the speed of light, and D is related to a polarization-independent background. It should be noted that the TIs here have a rhombohedral structure with space group D53d (R3m)6, leading to 81 χ(3) tensor components and 27 χ(2) components4,5. Therefore, Eqs (4) and (5) are simplified forms, rather than the reduced form available for higher symmetry crystal structures. The experimental observations can be well fit with Eq. (3) as shown by the solid line in Fig. 3a,b. The π period oscillation term C sin 2α results from the specular IFE while the π/2 oscillation component L sin 4α is associated with the specular OKE as the cascading OKE.

Note that the contribution of the specular OKE dominates the specular IFE in a 10 QL Sb2Te3 film as shown in Fig. 4. The helicity dependence of the specular IFE and specular OKE are phenomenologically similar to the circular photogalvanic effect (CPGE ∝ sin 2α) and the linear photogalvanic effect (LPGE ∝ sin 4α)26 and Eq. (3) is similar to the phenomenological expression for the induced photocurrent extensively observed in TIs27 and transition-metal dichalcogenides28. However, here we are observing uniquely nonlinear light-matter interactions appearing only within the duration of the optical pulse (on the order of less than 60 fs), after which an induced photocurrent will flow on a timescale of 1 ps29,30. Note that our 1.5 eV photon pump can excite electrons close to the Dirac point of the second SS for Sb2Te331, and therefore, circular-polarized pump pulse excites the spin-polarized electron population. Since our pump pulse was incident onto a sample at a slightly off-normal angle, it can create net magnetization in the second SS. Thus, under our experimental conditions, spin-polarized photocurrents can be generated, and may be related to the CPGE and LPGE phenomena4,5. Although we have provided a simple picture for the observed phenomena in TIs based on specular IFE and specular OKE, a comprehensive theoretical treatment is needed to elucidate the relationship between such nonlinear optical effects and the CPGE and LPGE phenomena on the surface of TIs.

Figure 4
figure 4

Helicity dependence of the Kerr-rotation signal. The individual contribution of the specular IFE and specular OKE signals to the Δθ k signal are shown for a 10 QL Sb2Te3 sample as a function of the quarter-wave plate (QWP) angle. The black solid curve is a fit to Eq. (3). The dashed green and blue curves exhibit the contributions correspond to specular OKE and specular IFE, respectively, obtained using the fit values.

Full size image

Similar spin dynamics have also been observed in a magnetic ionic liquid material, although the magnitude of the OKE was much smaller than that of the IFE32. We argue that the larger contribution from the specular OKE signal from the 10 QL Sb2Te3 film (Fig. 4) can be attributed to a surface state origin. Given that Eqs (4) and (5) imply a half magnitude of L compared to C, if χ(3) = χ′(3) = χ(2) · χ(2), our observation of L with twice the magnitude of C (Fig. 4) strongly indicates \({\chi }^{\mathrm{(3)}}\ll {\chi ^{\prime} }^{\mathrm{(3)}}={\chi }^{\mathrm{(2)}}\cdot {\chi }^{\mathrm{(2)}}\), where χ(2) is non-zero only in the surface region. In addition, enhancement of χ(2) may be induced by inversion symmetry breaking at the surface due to Dirac plasmon-related dc electric fields23.

The extracted L value which can be attributed to the specular OKE is plotted as a function of thickness for Sb2Te3, Bi2Te3 and GeTe amorphous samples as shown in Fig. 5. Experimentally, one should thus observe an abrupt change in L at 4 QL for Sb2Te3. This sudden change in the L value indicates that a crossover from a 2D insulator to 3D-TI has occurred. We argue that the abrupt change in L is associated with a change in band topology due to the presence of strong SOC, resulting in the formation of a Dirac cone33. The contribution of L to the Δθ k signal is very small for 2 and 3 QL, indicating that a gap opens at the Dirac point. Similarly, we observe an abrupt change in L for 3 QL for Bi2Te3 (see Figs 5 and 6). Thus, the crossover of TIs observed by the present method is consistent with the results reported in STM experiments10 and ab initio band-structure calculations (see Methods and Supplementary Fig. S1)11. On the other hand, the L value almost linearly increases with sample thickness for amorphous GeTe, as it is a trivial insulator, and therefore the bulk contribution associated with χ(3) increases. Therefore the contribution of the SSs (χ′(3) = χ(2) · χ(2)) on L to the Δθ k signal is negligible, which suggest the absence of a Dirac cone in amorphous GeTe. Note that the extracted C value which can be attributed to the specular IFE was not correlated with the sample thickness, and therefore, we conclude that it may be due to photoinduced magnetization of the Dirac surface states of a TI, associated with χ(3).

Figure 5
figure 5

Detection of the crossover from a 2D insulator to a 3D-TI. The L values extracted from a fit of Eq. (3) are plotted as a function of thickness for the TIs, Sb2Te3, Bi2Te3, and amorphous GeTe. The L values for the Sb2Te3 and Bi2Te3 samples exhibit a strong variation with thickness, whereas for amorphous GeTe the signal remains constant with thickness. The thick lines represent guides for the eyes.

Full size image
Figure 6
figure 6

Helicity dependence of the Kerr-rotation signal. The peak amplitude of the Δθ k signal as a function of the helicity, obtained for Bi2Te3 samples with different thicknesses. The oscillatory pattern shows change in the periodicity from (π) to (π/2) at and above 3 QL for Bi2Te3. The solid line represents a fit to the data using Eq. (3).

Full size image

To check if our observations are also applicable to non-topological materials, i.e., materials which exhibit SOC-induced Rashba effects34, we have measured a crystalline GeTe sample (4.3 nm and 8.6 nm thick), where the inversion symmetry is broken due to the noncentrosymmetric lattice, i.e., χ(2) ≠ 0. As a result, we observed π/2 oscillation signals for crystalline GeTe, which is different from the case of amorphous GeTe (Fig. 3). Note that a sample at least several nm thick is required to preserve the ferroelectricity in crystalline GeTe35,36. Thus, our π/2 oscillation signals can be found in ferroelectric Rashba semiconductors, like crystalline GeTe with non-topological surface states. Therefore, specular OKE is most likely arises from strong spin-orbit coupling.

Conclusion

To conclude, we have experimentally explored the topological magneto-optical response from TIs using a time-resolved Kerr rotation technique. In particular, we have focused on p-type Sb2Te3 since it is generally difficult to investigate its topological properties by conventional techniques such as ARPES. Our experimental results demonstrate that the helicity dependence of the Δθ k signal dramatically changes when the thickness of the sample is varied. By analyzing the helicity-dependent Δθ k signal based on the combined action of nonlinear magneto-optical effects, the specular IFE and specular OKE, we are able to discern the formation of a Dirac cone in a film several QL thick. Our method thus makes it possible to predict the presence of a Dirac cone (or Rashba splitting) in air and at room temperature, i.e. without ultra-high vacuum and sophisticated equipment, using a fundamental nonlinear optical effect that can be applied to a wide range of TIs and related 2D materials. The cascading nonlinear magneto-optical effects are of particular significance for p-type TIs, such as interfacial phase-change materials37, which may become the foundation of the next-generation of ultra-high-speed phase change random access memory.

Methods

Fabrication of X2Te3(X = Bi, Sb) films

The samples used in this study were highly-oriented X2Te3 (X = Bi, Sb) polycrystalline films possessing a rhombohedral crystal structure with space group D53d (R3m)6 and a thickness between 2 to 10 QL (2–10 nm) for Sb2Te3 and 1 to 10 QL (1–10 nm) for Bi2Te3. The samples were fabricated by self-organized van der Waals epitaxy using helicon-wave RF magnetron sputtering on (100) Si substrates at 230 °C38. For comparison, we prepared amorphous and crystalline GeTe films 4.3 and 8.6 nm thick, which were grown at room temperature and at 230 °C, respectively. The crystal structure of the TIs is comprised of alternating layers of X (Bi, Sb) and Te atoms along the z-direction. The alternating arrangement of five layers forms one QL, which is roughly equivalent to 1 nm and has been confirmed by TEM measurements. Within each QL, X (Bi, Sb) and Te atoms are coupled by covalent bonds, whereas the coupling between QL is via van der Waals interactions. The crystal quality was checked by using coherent phonon spectroscopy39, which demonstrated the presence of both the \({A}_{1g}^{1}\) and \({A}_{1g}^{2}\) modes, which correspond to the inter- and intra-layer optical modes, respectively40.

Femtosecond magneto-optical measurements

To investigate the critical thickness (i.e. the critical number of QL) that triggers a crossover from a 2D insulator to 3D-TI, or vice-versa, we have employed a time-resolved Kerr-rotation measurement using a novel pump-probe reflection technique. The time-resolved Kerr measurements were performed using near-infrared optical pulses with pulses generated by a Ti: sapphire laser oscillator with a pulse duration of about 20 fs, a central wavelength of 830 nm and a repetition rate 80 MHz. The optical penetration depth at 830 nm was estimated from the absorption coefficient to be ~14 nm, which is greater than the sample thickness. Thus the optical excitation is homogeneous over the entire sample thickness, and therefore the effects of the penetration depth do not play a role in the observed crossover-like behavior in the present study. The average powers of the pump and the probe beams were fixed at 120 mW and 2 mW respectively. The pump and the probe beam were co-focused onto the sample to a spot size of about 70 μm with an incident angle of about 15° and 10° with respect to the sample normal, respectively. The maximum photo-generated carrier density induced by a single 200 μJ cm−2 pump-pulse was estimated to be n exc  ≈ 5.1 × 1019 cm−3. The probe polarization was p-polarized while the polarization of the pump beam was modulated from linear polarized (0°), to left-circular-polarized (45°), to linear-polarized (90°), to right-circular-polarized (135°) and back to linearly-polarized (180°) and so on by varying the quarter-wave-plate (QWP, λ/4 plate) angle. The change in the Kerr-rotation of the probe pulse was recorded using balanced silicon photo-diodes as a function of the pump-probe delay over a time range up to 15 ps that was introduced by a shaker operated at 19.5 Hz25. The measured signal was accumulated over 700 scans to improve the signal to noise ratio. The measurements were carried out in air at room temperature.

Density functional theory (DFT) simulation

The WIEN2k code was used for band structure calculations41. R MT K max of 7.0 was used for the plane wave component between augmentation spheres. 8 × 8 × 1 Monkhorst-Pack grids were used42. The energy convergence criterion was 0.1 mRy. The SOC term was included. The DFT-D correction was included in order to take account of the van der Waals force43.