## Introduction

The study of long-period moiré superlattices formed in van der Waals heterostructures has emerged as a central topic in condensed matter physics1. After the observation of correlated insulator and superconductivity in twisted bilayer graphene(TBG) with flat bands2,3,4,5,6,7,8,9,10,11, it was proposed that there are moiré-mediated flat bands in twisted transition metal dichalcogenide heterobilayers and homobilayers12,13. Recently, correlated insulating phases and possible superconductivity signatures were discovered in twisted bilayer WSe2 at twist angles between 4 to 514 and twisted double-bilayer WSe215.

Besides the correlated phases, it was shown that the flat bands of the moire superlattices also exhibit non-trivial topological properties. For example, at 3/4 filling in hBN-aligned TBG, the degeneracy of the bands with non-trivial Chern numbers is lifted by electron-electron interactions and results in quantum anomalous Hall states which were observed recently16,17,18. These observations clearly demonstrate that the topological properties of the flat bands also have important consequences on the nature of the correlated phases.

Similar to TBG, it was pointed out that flat bands with non-trivial Chern numbers can be generated in twisted bilayer WSe2 (tWSe2) and a couple of topological insulating phases were predicted at a wide range of twist angles13. As in the case of TBG, the topology of the bands would affect the nature of the insulating phase when the bands are half filled and the interaction effects are strong. In this work, we propose that the nonlinear response in electric field can be used to unveil the topological properties of the flat bands and identify the topological phase transition point through the measurements of the nonlinear Hall (NLH) effect.

The NLH effect is a fascinating phenomenon recently proposed by Sodemann and Fu19, and experimentally observed in bilayer and multilayer WTe220,21. It is the generation of a transverse DC current and a transverse voltage with frequency 2ω when an AC current of frequency ω is applied, which has potential applications in rectifications and second harmonic generations. The effect originates from the non-vanishing dipole moment of the Berry curvature of the bands which characterizes the second-order nonlinear hall susceptibility. In pristine tWSe2, regardless of how large the Berry curvatures of the bands are, the three fold rotational symmetry forces the Berry curvature dipole to be zero and the NLH effect vanishes. However, we demonstrate here that a small strain breaks the three-fold rotational symmetry and generates a large Berry curvature dipole. Importantly, these symmetry breaking strain effects have been observed recently in experiments22. Moreover, the Berry curvature dipole is strongly enhanced and has opposite signs across the topological phase transition point when the top two valence bands touch each other and exchange Berry curvatures23. As the topological phase transition can be tuned by a vertical displacement field, the measurement of the NLH effect can serve as a probe of the topological phase transition in tWSe2. Moreover, the strong enhancement and the highly tunability of the NLH effect by a displacement field renders tWSe2 and related moire materials available platforms for rectification and second harmonic generations applications.

The rest of our paper is organized as follows. First, we present the continuum model of tWSe2 which takes into account a uniaxial strain in the bottom layer induced by substrate or external modulation, which breaks the three-fold rotational symmetry such that nonzero Berry curvature dipole can be created. Second, we calculate the Berry curvature dipole in the presence of strain. We show that the strained tWSe2 exhibits large nonlinear Hall response. Third, we explore the behavior of the NLH response near the topological phase transition induced by a displacement field. We find that the Berry curvature dipole is strongly enhanced near the topological phase transition point and it changes sign across the transition point.

## Results

### Continuum model of strained tWSe2

We consider a AA stacking bilayer WSe2 with lattice constant a0 in a single layer and at a twist angle θ. A schematic figure is shown in Fig. 1a. The moiré superlattice, which has a moiré lattice constant $${L}_{M}={a}_{0}/2\sin \frac{\theta }{2}$$, folds the energy bands and gives rise to the so-called moiré Brillouin zone (see Fig. 1b). These moiré energy bands originating from the K+ or K valleys can be described by the continuum Hamiltonian13$$H={\sum }_{\xi }\int d{{{{{{{\boldsymbol{r}}}}}}}}{\psi }_{\xi }^{{{{\dagger}}} }({{{{{{{\boldsymbol{r}}}}}}}}){\hat{{{{{{{{\mathcal{H}}}}}}}}}}_{\xi }({{{{{{{\boldsymbol{r}}}}}}}}){\psi }_{\xi }({{{{{{{\boldsymbol{r}}}}}}}})$$ with

$${\hat{H}}_{\xi }({{{{{{{\boldsymbol{r}}}}}}}})=\left(\begin{array}{cc}{\hat{H}}_{b,\xi }+{\Delta }_{b}({{{{{{{\boldsymbol{r}}}}}}}})&{T}_{\xi }({{{{{{{\boldsymbol{r}}}}}}}})\\ {T}_{\xi }^{{{{\dagger}}} }({{{{{{{\boldsymbol{r}}}}}}}})&{\hat{H}}_{t,\xi }+{\Delta }_{t}({{{{{{{\boldsymbol{r}}}}}}}})\end{array}\right).$$
(1)

Here, ξ = ± is the valley index denoting whether the bands originate from the K+ or K valleys of the monolayer Brillouin zone. The electron creation operators are denoted as $${\psi }_{\xi }^{{{{\dagger}}} }=({\psi }_{b,\xi }^{{{{\dagger}}} },{\psi }_{t,\xi }^{{{{\dagger}}} })$$, where t and b label the top and bottom layers respectively. It is important to note that due to the large Ising spin-orbit coupling in 2H-structure WSe2, the top valence band of monolayer WSe2K+ and K valleys are fully spin polarized and have opposite spin. Therefore, the spin and valley indices are locked together and the spin-index is dropped. As a result, the Hamiltonian of a single layer at valley ξ can be written as

$${\hat{H}}_{l,\xi }=-\frac{{\hslash }^{2}}{2{m}^{* }}{\left(\hat{{{{{{{{\boldsymbol{k}}}}}}}}}-{{{{{{{{\boldsymbol{K}}}}}}}}}_{l,\xi }^{m}\right)}^{2}-l\frac{{V}_{z}}{2},$$
(2)

where l = + 1(−1) labels the bottom(top) layer, m* is the effective mass of valence band, and Vz is the staggered layer potential generated by the vertical displacement field. The intra-layer moiré potential and the coupling between the two layers are denoted as Δl(r) and Tξ(r) respectively which can be written as:

$${\Delta }_{l}({{{{{{{\boldsymbol{r}}}}}}}})=V\mathop{\sum}\limits_{i=1,2,3}{e}^{i({{{{{{{{\boldsymbol{g}}}}}}}}}_{{{{{{{{\boldsymbol{i}}}}}}}}}\cdot {{{{{{{\boldsymbol{r}}}}}}}}+l\psi )}+h.c.,$$
(3)
$$\qquad{T}_{\xi }({{{{{{{\boldsymbol{r}}}}}}}})=w(1+{e}^{-i\xi {{{{{{{{\boldsymbol{g}}}}}}}}}_{{{{{{{{\boldsymbol{2}}}}}}}}}\cdot {{{{{{{\boldsymbol{r}}}}}}}}}+{e}^{-i\xi ({{{{{{{{\boldsymbol{g}}}}}}}}}_{{{{{{{{\boldsymbol{1}}}}}}}}}+{{{{{{{{\boldsymbol{g}}}}}}}}}_{{{{{{{{\boldsymbol{2}}}}}}}}})\cdot {{{{{{{\boldsymbol{r}}}}}}}}}).$$
(4)

Here, V and ψ, characterize the amplitude and phase of the moire potentials respectively, w characterizes the tunneling strength between the top and bottom layers, and the moiré reciprocal lattice vectors are $${{{{{{{\boldsymbol{{g}}}}}}}_{i}}}=\frac{4\pi }{\sqrt{3}{L}_{M}}\left(\cos \frac{2(i-1)\pi }{3},\sin \frac{2(i-1)\pi }{3}\right)$$. We adopt the model parameters from ref. 13 (a0, m*, w, V, ψ) = (3.32Å, 0.44me, 9.7meV, 8.9 meV, 91), which are estimated from the first principle calculations.

It is important to note that the Berry curvature dipole transforms as a pseudo-vector which vanishes in the presence of a three-fold rotational symmetry as discussed in detail in the next section. In realistic systems, the three-fold symmetry in tWSe2 can be broken by the substrate induced or externally applied strain. The strain induced symmetry breaking has been observed experimentally22. In general, the physical properties of tWSe2 only depend on the relative deformation between the two layers. Therefore for all the calculations in this work, the strain is only applied to the bottom layer of WSe2, while the strain in the top layer is set to zero. Specifically, the strain tensor $${{{{{{{\mathcal{E}}}}}}}}$$ is a two-dimensional matrix, which can be written as24:

$${{{{{{{\mathcal{E}}}}}}}}=\epsilon \left(\begin{array}{cc}{\cos }^{2}\varphi -\nu \,{\sin }^{2}\varphi &(1+\nu )\cos \varphi \sin \varphi \\ (1+\nu )\cos \varphi \sin \varphi &{\sin }^{2}\varphi -\nu \,{\cos }^{2}\varphi \end{array}\right).$$
(5)

The angle φ denotes the in-plane direction of the uniaxial strain direction with respect to the zigzag edge of the sample and ϵ characterizing the strength of strain, and ν = 0.19 is the poisson ratio for WSe225.

In general, the strength ϵ can be estimated to 0.5%~1%, but the direction φ can be random and difficult to determine. We highlight the breaking of the C3 rotational symmetry due to strain is the key of the nonlinear Hall response, while the details of the strain are not essential for our discussion. For simplicity, we use the model that there is no strain in the top layer and the strain is applied to the bottom layer.The strain has two important effects shifting the Dirac point for bottom layer WSe2 to $${{{{{{{{\boldsymbol{D}}}}}}}}}_{b,\xi }=(I-{{{{{{{{\mathcal{E}}}}}}}}}^{T}){{{{{{{{\boldsymbol{K}}}}}}}}}_{b,\xi }-\xi {{{{{{{\boldsymbol{A}}}}}}}}$$ and generating an effective gauge field $${{{{{{{\boldsymbol{A}}}}}}}}=\frac{\sqrt{3}}{2{a}_{0}}\beta ({\epsilon }_{xx}-{\epsilon }_{yy},-2{\epsilon }_{xy})$$. Here, β is taken as 2.30 in our calculations according to previous first principle calculations for strained monolayer WSe224,26. As a result, the continuum Hamiltonian of this strained tWSe2 is obtained by replacing $${\hat{H}}_{b}$$ in Eq. (1) as

$${\hat{H}}_{b,\xi }^{s}=-\frac{{\hslash }^{2}}{2{m}^{* }}{\left(\left(I+{{{{{{{{\mathcal{E}}}}}}}}}^{T}\right)\left(\hat{{{{{{{{\boldsymbol{k}}}}}}}}}-{{{{{{{{\boldsymbol{D}}}}}}}}}_{b,\xi }^{m}\right)\right)}^{2}-\frac{{V}_{z}}{2}.$$
(6)

To estimate the range of strain ϵ within which our model is valid, we expect that the strain induced shifting of the Dirac point would be smaller than the separation of the Dirac points due to twisting. With a twist angle θ, the separation of the Dirac point K in momentum space is ΔK = Kθ, and the shift of the K point by uniaxial strain is ΔKs ≈ ϵK. Therefore, the strain effect can be treated as a perturbation to the moiré super-lattice when ϵθ. In our calculations below, we assume that the strain induces a 0.5% change in the lattice constant of the bottom layer WSe2 along the direction of the strain such that ϵ ≈ 0.2θ.

In Fig. 2a, we show the moiré energy bands of tWSe2 with twist angle θ = 1. 4 with and without strain. The top two valence bands originating from the K+ valley of the monolayer Brillouin zones carry finite Chern numbers C = 1 and C = − 1 respectively when Vz = 0. Due to time-reversal symmetry, the top two valence bands originating from the K valleys carry Chern numbers C = − 1 and C = 1 respectively. As the two valleys do not couple in momentum space, one can define a Z2 topological invariant as $${Z}_{2}=| ({C}_{{K}_{+}}-{C}_{{K}_{-}})/2|$$ to describe the topological properties of the bulk bands, where CK is the Chern number of the top valence band originating from the K valleys of the monolayer Brillouin zone. In Fig. 2b, a staggered potential of Vz = 5 meV induces a topological phase transition and the system becomes topologically trivial. The band structure with and without strain are depicted. It is important to note that the strain with ϵ = 0.5% is not sufficient to close the band gap and the topological properties of the flat bands are not changed by strain. Moreover, the energy dependence of the density of states for Vz = 0, 5 meV are also shown on the right side of Fig. 2, which is important to determine the occupation number of holes nh as discussed below.

### Nonlinear Hall response

In this section, we consider the NLH response for tWSe2. The NLH effect is characterized by the generation of a transverse voltage using a charge current in time-reversal invariant systems without external magnetic fields or magnetic orders. Moreover, this effect is nonlinear in nature and exhibits a quadratic current-voltage relation. More specifically, when an electric field $${{{{{{{\boldsymbol{E}}}}}}}}(t)=\frac{1}{2}(\varepsilon {e}^{i\omega t}+{\varepsilon }^{* }{e}^{-i\omega t})$$ with the amplitude vector ε and frequency ω is applied, the Hall current has both rectified and second-harmonic components $${J}_{y(x)}^{(0)}={\chi }_{yxx(xyy)}{\varepsilon }_{x(y)}{\varepsilon }_{x(y)}^{* }$$ and $${J}_{y(x)}^{(2)}={\chi }_{yxx(xyy)}{\varepsilon }_{x(y)}{\varepsilon }_{x(y)}$$, where χ is the nonlinear Hall susceptibility. As shown in ref. 19,27,28, the nonlinear Hall susceptibility can be written as:

$${\chi }_{yxx(xyy)}=\mp \frac{{e}^{3}\tau }{2{\hslash }^{2}(1+i\omega \tau )}{D}_{x(y)}$$
(7)

where τ is the relaxation time. The Berry curvature dipole Di are elements of the Berry curvature dipole pseudo-vector. It is important to note that in a two-dimensional system, the Berry curvature dipole transforms as a pseudo-vector as described by the vector Di (i = x, y). In other words, Di must be invariant under crystal point group operations. For twisted tWSe2 with or without the displacement field, the system respects a three-fold rotational symmetry C3z inherited from monolayer 2H-structure WSe2. Due to the time-reversal symmetry, Ωn(k + K) = − Ωn( − k − K), vn(k + K) = − vn( − k − K), each valley has an equal contribution, allowing us to consider the K+ valley for the sake of simplicity. For K+ valley, C3z symmetry ensures $${\Omega }_{n}({{{{{{{\boldsymbol{k}}}}}}}})={\Omega }_{n}({C}_{3}{{{{{{{\boldsymbol{k}}}}}}}})={\Omega }_{n}({C}_{3}^{2}{{{{{{{\boldsymbol{k}}}}}}}})$$ and $$\mathop{\sum }\nolimits_{i = 0}^{2}{{{{{{{{\boldsymbol{v}}}}}}}}}_{n}({C}_{3}^{i}{{{{{{{\boldsymbol{k}}}}}}}})=0$$. Therefore, Di vanishes if C3z symmetry is preserved. One way to obtain finite Di is to take into account the strain effects which break C3z. The strain can be induced by the substrate which couples to tWSe2 or it can be induced externally as shown in the above sections. As we will see in the next section, only a very small strain is needed to induce a strong NLH response in tWSe2.

### Topological phase transition and NLH effect

With the formalism discussed above, we can now explore the NLH effects in tWSe2. By applying a unaxial strain with ϵ = 0.5% along the direction of the zigzag edge and at twist angle 1. 4, Dx as a function of the vertical displacement field Vz and the occupation number of holes per moire unit cell nh are depicted in Fig. 3a. The occupation number nh is obtained by integrating the density of states from the top of the valence to the Fermi energy including two valleys. It is clear from Fig. 3a that the Berry curvature dipole is generally large, in the order of 1Å.

Importantly, the Berry curvature dipole is strongly enhanced near the bottom of the top valence band when nh is close to 2 (this happens when the Fermi energy is near the bottom of the top valence band) and Vz close to 4.2 meV. Specifically, the bands originating from the K+ (or K) valley have nontrivial topological invariant Z2 = 1. By applying a displacement field which introduces a potential difference between the top and the bottom layers, the top valence band and the nearby valence band can touch near the K points and exchange Berry curvatures, resulting in a change of Z2 invariant from 1 to 0. The Vz dependence of the Berry curvature dipole across the phase transition with fixed nh is shown in Fig. 3b. It is clear that Berry curvature dipole is strongly enhanced near the critical Vz ≈ 4.2 meV when nh is close to 2. The enhancement of Berry curvature dipole comes from two contributions, one is the enhancement of density of states due to the band flatness, and the other is the large Berry curvature near band edge. This value is several times larger than the optimal Berry curvature dipole measured experimentally in other systems which is about 1Å20,21,29. Moreover, the Berry curvature dipole changes sign across the phase transition point. This is caused by the exchange of Berry curvatures near the K points before and after the topological phase transition as shown in Fig. 3d, e. To illustrate the sign change of Berry dipole more clearly, we show the Berry dipole flow viΩ in Supplementary note 1.

Furthermore, the Vz dependence of the Berry curvature dipole with fixed nh is shown in Fig. 3c when strain is applied along the armchair edge direction. Similar enhancement and sign change in the Berry curvature dipole near the topological phases transition can be observed. In the supplementary note 2, the Vz and nh dependence of the Berry curvature dipole with different twist angles and strain are presented and behaviors similar to Fig. 3a are observed. Therefore, the NLH measurements can be used to indicate possible topological phase transitions in tWSe2. Note that in practice to confirm the sign switching of NLH signal is induced by the topological phase transition rather than other mechanisms, it must be accompanied by other measurements that can probe the bulk topology or edge excitations.

## Discussion

It is important to note that the appearance of NLH effect studied in this work is very general. We expect a finite NLH response as long as the three-fold rotational symmetry is broken intrinsically by the substrate22 or by an externally applied strain. The calculated Berry dipole in tWSe2 is about 1 ~ 3Å, which is relatively large in usual materials. In other strained systems, such as strained MoS2 and strained bilayer graphene30,31, the strain-induced Berry dipole is only 10−2Å. For bilayer and few-layer WTe228,32, the Berry dipole is about 0.2Å, 1Å respectively.

Furthermore, we point out that strained tWSe2 and other moiré materials are excellent candidate materials for studying NLH effects due to their low symmetry, non-trivial Berry curvature and high tunability of experimental parameters33,34. Especially, the C3z symmetry breaking strain have been widely observed in moiré materials. In our other recent work collaborating with experimentalists35, it was found that the twisted monolayer-monolayer, bilayer-bilayer and trilayer-trilayer WSe2 samples, all of them exhibit strong nonlinear hall response, roughly 1000 times larger than those of bilayer or few-layer WTe220,21. This clearly shows that the moiré superlattices behave as an ideal setup in realizing the nonlinear Hall effects, although the understanding of such a giant nonlinear hall response in the moiré superlattices is still open and worth further works in the future.

Our calculations are based on the Boltzmann approach in ref. 19, which requires that the inelastic scattering time τin and the band width W should satisfy W/τin. Although the bandwidth is reduced at orders of meV by the moiré potential in tWSe2, we assume the Boltzmann approach is still valid in this case. This assumption is based on the observation that τin is usually around 10 ps for 2-D van der Waals materials36, corresponding to an energy scale /τin ~0.07 meV that is still much smaller than the moiré band width.

The electron-electron interaction in general is expected to play an important role in moiré superlattices. Recently, correlated insulating phase is observed when the filling factor nh is close to 114. In principle, the interaction can modify the single-particle band structures and further change the Berry curvature dipole. In the experiment35, the nonlinear Hall response is strongly enhanced at half filling of the first moiré band, which cannot be explained in single-particle level and correlation effects must have contributions to the NLH response. Near half-filling of the first moiré band, the nonlinear Hall signal shows a sharp peak which can originate from a mass-diverging type continuous Mott transition. Also the symmetry-breaking ground states as discussed in6 may also get involved and interplay with NLH. For example, the C3z rotational symmetry can be broken in the presence of the nematic phase, which is essential in generating a finite NLH in tWSe2. The interplay between the interaction effects and nonlinear Hall effect is of great interest and is worth a more detail study in the future.

In addition, the topological phase transition happens near nh ≈ 2. For a fixed displacement field Vz, the Berry curvatures near the band edge of top two valence bands are opposite and as a result, the Berry dipoles change sign when filling factor nh across 2. For a fixed nh, by applying Vz which introduces a potential difference between the top and the bottom layers, the top valence band and the nearby valence band can touch near the K points and exchange Berry curvatures, resulting in the sign change of Di. This is the origin of four wings of the "butterfly" in Fig. 3a. Near the phase transition regime, we expect the system to be metallic and can be described by a Fermi liquid even in the presence of electron-electron interaction. Nevertheless, the NLH measurements can reveal the topological properties of the bands away from the Fermi energy that shed light on the nature of correlated phases.

## Methods

### Berry curvature and dipole

In strictly two dimensions, the Berry curvature dipole is given by

$${D}_{i}=-\int \frac{d{{{{{{{\boldsymbol{k}}}}}}}}}{{(2\pi )}^{2}}\mathop{\sum}\limits_{n,\xi =\pm 1}{v}_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }^{i}{\Omega }_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }\frac{\partial f({E}_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi })}{\partial E},$$
(8)

where n as the band index and vi as the Fermi velocity of the Bloch state, and the Berry curvature can be obtained from Bloch wavefunctions as:

$${\Omega }_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }=i\langle {\partial }_{{{{{{{{\boldsymbol{k}}}}}}}}}{u}_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }|\times |{\partial }_{{{{{{{{\boldsymbol{k}}}}}}}}}{u}_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }\rangle .$$
(9)

The Berry curvature dipoles should be integrated over whole moiré Brillouin zone and both valleys are involved in the result. Also the Chern number for an isolated band can be evaluated from Berry curvature:

$${C}_{n\xi }=\frac{1}{2\pi }\int d{{{{{{{\boldsymbol{k}}}}}}}}{\Omega }_{n{{{{{{{\boldsymbol{k}}}}}}}},\xi }$$
(10)