Relaxation effects in transition metal dichalcogenide bilayer heterostructures

Abstract

While moiré structures in twisted bilayer transition metal dichalcogenides (TMDCs) have been studied for over a decade, the importance of lattice relaxation effects was pointed out only in 2021 by DiAngelo and MacDonald¹, who reported the emergence of a Dirac cone upon relaxation. TMDCs of group 6 transition metals MX₂ (M = Mo, W, X = S, Se) share layered structures with pronounced interlayer interactions, exhibiting a direct band gap when exfoliated to a two-dimensional (2D) monolayer. As their heterolayers are incommensurable, moiré structures are present in the bilayers even if stacked without a twist angle. This study addresses the challenge of accurately modeling and understanding the structural relaxation in twisted TMDC heterobilayers. We show that the typical experimental situation of finite-size flakes stacked upon larger flakes can reliably be modeled by fully periodic commensurate models. Our findings reveal significant lattice reconstruction in TMDC heterobilayers, which strongly depend on the twist angle. We can categorize the results in two principal cases: at or near the untwisted configurations of 0° and 60°, domains with matching lattice constants form and the two constituting layers exhibit significant in-phase corrugation—their out-of-plane displacements are oriented towards the same direction in all local stackings—while at large twist angles—deviating from the 0° and 60°—the two layers show an out-of-phase corrugation. In particular, we reveal that the lattice reconstruction results from the competition between the strain energy cost and the van der Waals energy gain. Additionally, our systematical study highlights structural disparities between heterostructures composed of different or identical chalcogen atoms. Our research not only confirms the reliability of using periodic commensurate models to predict heterostructure behavior but also enriches the understanding of TMDC bilayer heterostructures.

Introduction

Stacked two-dimensional (2D) crystals with a twist angle can give rise to large-scale moiré patterns characterized by periodically varying interlayer stacking alignments². These moiré patterns have led to the emergence of various quantum phenomena, capable of significantly altering electronic band structures beyond expectations, even if in most systems the interaction is solely given by the weak van der Waals (vdW) interaction. For example, a graphene bilayer with a ‘magic’ twist angle of 1.05° exhibits flat bands, superconductivity and enhanced electron-electron interactions^3,4,5, which are absent in monolayers and other bilayer configurations with different twist angles. Superlattice effects with strong impact on the electronic structure have been reported for bilayer transition metal dichalcogenides (TMDCs)^1,6,7,8. Moreover, moiré patterns naturally exist in vdW heterostructures (vdWHSs) even without interlayer twist, due to the distinct lattice constants of their constituting layers. These moiré patterns also undergo changes as the twist angle varies. Examples of exciting moiré physics, including moiré excitons, have been reported in various untwisted/twisted TMDC vdWHSs (MoSe₂/WS₂⁹, MoSe₂/WSe₂¹⁰, and WSe₂/WS₂¹¹). While moiré structures have been a subject of study for over a decade, recent attention has been drawn to the significance of lattice relaxation effects specifically in twisted TMDC systems^1,6,7. In order to fully understand the optical and electronic properties of these vdWHSs, the structural changes at the scale of the moiré lattice¹² need to be considered. However, it is difficult to conduct theoretical studies of such large systems, which can encompass 100,000´s of atoms. Several groups have carried out geometry optimization of twisted TMDC vdWHSs. Geng et al.^13,14. for instance, employed density-functional theory (DFT) to relax MoS₂/MoTe₂ with twist angles of 0° and 4.54°. Shabani et al.¹⁵ calculated WSe₂/MoSe₂ at twist angles of ~3° and ~61.7° by a continuum mechanical model combined with stacking fault energy. Similar work was undertaken by Enaldiev et al.¹⁶ for WS₂/MoS₂ and WSe₂/MoSe₂ at different twist angles. Rodríguez et al.¹⁷ investigated twisted MoS₂/MoSe₂ by combining experiments and molecular dynamics simulations. Vitale et al.¹⁸ employed a force field to investigate TMDC vdWHS with large twist angles. Nielsen et al.⁸ studied the domain formation across different TMDC heterostructures. In this work, we systematically studied the lattice reconstruction including domain formation and out-of-plane corrugation of twisted TMDC vdWHS by combining MoS₂, WS₂, MoSe₂ and WSe₂ monolayers with twist angles ranging from 0° to 60°.

To begin, we offer a brief introduction about the local stacking alignments in bilayer TMDC moiré structures. We then show that the typical experimental situation of finite-size flakes stacked upon larger flakes can reliably be modeled by fully periodic commensurate models. This validates the periodic supercells which are commonly used in literature and which we will also utilize throughout the rest of the paper. Subsequently, we present the results of relaxation for all commensurate TMDC vdWHS without interlayer twist. We classify these vdWHS systems into two groups: The first one includes MoS₂/WS₂ and MoSe₂/WSe₂, where the chalcogen atoms are the same in both layers. Due to similar lattice constants, a significantly larger moiré scale is observed. The second group comprises systems with different chalcogen atoms in the layers, leading to notable disparities in the lattice constants between the two layers of the vdWHS. Interestingly, the moiré lattice sizes of all four vdWHS systems within this group (MoS₂/MoSe₂, MoS₂/WSe₂, WS₂/MoSe₂, WS₂/WSe₂) are remarkably similar, regardless of the metal sites involved. Therefore, in the main text we take MoS₂/MoSe₂ and MoS₂/WS₂ as representatives of the two groups and report the details of the remaining four cases in the Supporting Information. Continuing, we investigate the impact of a twist angle on the atomic structure of the vdWHS. Besides fully relaxing these structures, we calculate individual energy contributions, including interlayer vdW energy and strain energy. We provide detailed insight into the structural variations of the untwisted/twisted vdWHS moiré superlattice by demonstrating the distribution of interlayer distance, out-of-plane/in-plane displacement, vdW and strain energy contributions. We suggest two critical twist angles away from 0°/60° in each twisted vdWHS, above which the two consisting layers possess out-of-phase corrugation, while below which the domain formation is evident, with the two layers possessing significant in-phase corrugation.

Results and discussion

Incommensurate systems

Each layer of the vdWHS has a distinct lattice constant, resulting in the presence of a natural moiré pattern even in the unrelaxed structures. We observe a variety of local stacking alignments within the moiré superlattice, including high-symmetry stackings² and low-symmetry transition regions. As an illustrative example, we consider the MoS₂/MoSe₂ vdWHS without interlayer twist and generate the model as detailed in “Methods” section. Figure 1a illustrates the unrelaxed structure of MoS₂/MoSe₂ vdWHS at a twist angle of θ = 0° which includes 3 types of high-symmetry stackings: The X atoms are eclipsed by the X’ atoms in ${R}_{h}^{h}$ high-symmetry stacking; in high-symmetry stacking ${R}_{h}^{X}$ or ${R}_{h}^{M}$ they are located at the centers of triangles formed by the three nearest-neighbor X’ atoms. All of these high-symmetry stacking regions possess hexagonal rotational symmetry. The boundaries separating high-symmetry stackings are regarded as transition stacking regions, where the X atoms are not located at the symmetrical points of the triangles of X’ atoms. In the case of a vdWHS with a twist angle of θ = 60° three H-type high-symmetry stackings are present in Fig. 1b.

Fig. 1: The initial geometric configuration of MoS₂/MoSe₂ van der Waals heterostructures before relaxation.

The top view of an unrelaxed moiré superlattice at twist angles θ of (a) 0° and (b) 60° is provided. The high-symmetry local stacking alignments are marked, and labeled in the same way as in Supplementary Fig. 11. Upon relaxation (see Fig. 4), these regions further evolve to minimize energy. The arrow denotes the diagonal direction. c The interlayer distance d and the binding energy E_b (defined in “Methods”) of six high-symmetry stackings and the relaxed MoS₂/MoSe₂ vdWHS at a twist angle θ = 0 (60)°. d of vdWHS is defined as the average of shortest and longest distance. vdWHS supercells are energetically favorable compared to the high-symmetry stackings and exhibit configuration-dependent interlayer distance. A 0.015% strain is applied to each constituting layer for generating these bilayer models.

To compare the impact of varying local stackings on the moiré system, a comparison of the interlayer distance and energy (as outlined in “Methods”) of all six representative stackings is presented in Fig. 1c. The relaxed vdWHS supercells have the lowest energy compared to high-symmetry stackings, resulting in the inherent presence of a moiré superlattice even without interlayer twist. Among these, ${R}_{h}^{h}$ and ${H}_{h}^{M}$ demonstrate the highest energy and most significant interlayer spacing, while other stackings are characterized by stronger interlayer interactions during relaxation. These varying interlayer spacings and energy density (distribution of binding energy) of different local stackings cause mechanical perturbation, resulting in out-of-plane corrugation and stacking region reconstruction, with a large-scale periodicity corresponding to the size of the moiré cell.

In the heterostructures of 2D materials with different lattice constants, our model generation involves compressing one layer and stretching the other to establish a commensurate superlattice as it is also commonly done in literature. Yet, it is not clear how this affects the resulting structural details. In order to explore whether boundary conditions play an instrumental role in the atomic reconstruction of twisted vdWHS, we calculated systems with a periodic layer onto which we stacked finite, rhombus-shaped flakes, subsequently rotated by various initial twisting angles. Then we compare their structural features to those of commensurate systems. Considering that the edges affect the out-of-plane displacement and moiré length, we cut off the edge to isolate the central regions with a side length that is 80% of the full flakes. An illustrative example of this can be seen in Supplementary Fig. 2, where we observe the same stacking domains as those in commensurate systems. Then we examine the moiré length and the relaxed twist angle within these central areas. The moiré pattern observe in the flakes is quantified by the vdW energy distribution, which is stacking configuration-dependent, e.g., in Fig. 1c. The moiré superlattice constant—the characteristic length scale of the pattern—is calculated by averaging the distance between the atomic positions at which the highest density of vdW energy is observed. The twist angle upon relaxation is measured by the angle between moiré superlattice vector and the unit cell lattice vector of the periodic layer underneath. A more detailed discussion on the methodology can be found in the Supplementary Information and in Supplementary Fig. 4 in particular.

A fundamental question arises regarding the behavior of vdWHS when the individual layers possess nearly identical lattice constants: do these systems prefer to undergo lateral strain to form one uniform stacking, or do they form moiré patterns? We observe that the answer depends on the size of the consisting flakes. For heterostructures composed of large flakes, we observe the emergence of multiple low-energy stacking configurations, or domains, which effectively act as pinning points, to halt lateral movements of the layers relative to each other. In these cases, the system prefers to minimize its free energy by adopting several locally stable configurations rather than forming one uniform, globally strained layer. For example, a MoS₂/WS₂ (periodic layer/flakes) system at θ = 0° is shown in Supplementary Fig. 3, where we observe multiple low-energy stacking with matching lattice constant. Conversely, when the flake size is only large enough to accommodate a single domain, we find that the system favors a uniform stacking configuration. In this case, the energetics do not support the formation of moiré patterns due to the absence of multiple pinning points that would otherwise facilitate the heterostructure’s relaxation into one lower energy stacking alignment. Notably, the experimental fabrication methods employed play a substantial role in determining the final structural configuration of these heterostructures. Our simulations mirror the conditions in the transfer method of fabrication^19,20. Other techniques, such as chemical vapor deposition²⁰ (CVD), differ significantly in their approach to layer deposition and might actually lead to large-scale, high-symmetry stacking areas without reconstructions or the formation of moiré patterns²¹.

Further, we present detailed results for systems featuring a periodic MoS₂ layer with overlying finite MoSe₂ flakes. These flakes, which vary in side-lengths from 20 to 130 nm, were stacked at different initial angles, as shown in Fig. 4. Yet, in Fig. 2a one can also see that small flakes can relax to a different local minimum in the potential energy surface and that upon relaxation the twist angle between periodic substrate and finite flake can change dramatically. For the larger systems the interaction between the layers is strong enough to fix the flakes, keeping the initial interlayer twist. This is however not only a theoretical problem but can also occur in experiments in which, e.g., the mobility of the strain solitons^22,23 was already shown. From our results we deduce that experiments on flakes with lateral dimensions of 80 nm or more for MoS₂/MoSe₂ systems at different twist angles, can reliably be described by fully periodic models (see also results for more twist angles in Supplementary Fig. 5). Furthermore, the sizes of the final moiré pattern converge for large flakes as shown in Fig. 2b, c. While in twisted MoS₂/WS₂ (periodic layer/flakes) systems where the flake size exceeds 190 nm, the moiré superlattice constants converge to the values in corresponding commensurate models and the twist angle upon relaxation matches the initial stacking angle, as detailed in Supplementary Information and Supplementary Fig. 8.

Fig. 2: The effect of flake size on twist angles and moiré superlattice constant in MoS₂/MoSe₂ (periodic layer/flakes) systems.

The numerical determined (a) twisted angle and (b, c) moiré superlattice constant after relaxation versus the side-length of MoSe₂ flakes. The horizontal dashed lines in (a) and the legend indicate the initial stacking angle before relaxation. A vertical dashed line across all panels marks convergence size (80 nm) beyond which the properties are considered size-independent for all stacking angles. In (b) and (c), the shaded regions highlight the range of moiré superlattice constants deduced from commensurate systems. This range reflects the lattice constants derived from commensurate systems where the twist angles are in agreement with those post-relaxation twist angles in flakes systems, when the flake size exceeds its convergence point. The overlap of the shaded regions and data points indicates the similarity between flake systems and commensurate systems, further illustrated in Supplementary Fig. 7.

We now discuss the observed variations in moiré length and corrugation in those models. The systems have been shown to be incommensurate, which means that M or X are not precisely on the symmetry sites but change slightly from one moiré superlattice to the next. This incommensurability is expected to generate repeating second-order (or possibly higher) moiré patterns. In these patterns, the alignment of symmetry sites approaches the ideal configuration more closely in certain regions than in others. As shown in Supplementary Fig. 6, the moiré superlattice size exhibits variations, with a deviation of 3 Å—the length of a unit cell. This misalignment impacts the local interlayer interaction, leading to fluctuations in the corrugation of the heterobilayers. The phenomenon of corrugation variation, for instance, has been experimentally observed in graphene grown on Ir(111)²⁴. Interestingly, despite these variations, the average periodic length and corrugation measurements align well with those predicted by first-order, fully periodic system models. This is further evidenced by Supplementary Fig. 7, where a significant agreement in moiré length and out-of-plane corrugation between the flakes (with side-lengths above 80 nm) and commensurate systems is observed. Therefore, the commensurate fully periodic models can be employed to study van der Waals heterostructures in the following discussion.

Commensurate systems

Next, we conducted a comprehensive study of a generically twisted TMDC vdWHS. The structural reconstruction strongly depends on the moiré lattice size, and thus on twist angle. As Supplementary Fig. 10 indicates, the four types of vdWHSs featuring different chalcogen species (MS₂/M’Se₂, where M, M’ = Mo, W) possess similar moiré lattice sizes at the same twist angle, while the remaining two types sharing the same chalcogen species (MoX₂/WX₂, where X = S, Se) possess large-scale moiré superlattices. Figure 3 illustrates the out-of-plane corrugation shapes in all these vdWHS at θ = 0°. Notably, MS₂/M’Se₂ exhibit similar structural features due to their comparable moiré lattice size, while MoX₂/WX₂ with much larger moiré superlattices exhibit more varying structures. Therefore, we select MoS₂/MoSe₂ (solid blue curve in Fig. 3) and MoS₂/WS₂ (solid green curve) with different twist angles as representative examples. The results for the rest heterostructures are reported in Supplementary Figs 17–20.

Fig. 3: A comparison of the out-of-plane corrugation patterns among various relaxed MX₂/M’X'₂ vdWHSs (M = Mo, W; X = S, Se; where MX₂ is the bottom layer and M’X'₂ is the top layer).

The shape of out-of-plane corrugation along the diagonal of the moiré superlattice at twist angle θ = 0° is illustrated. ${Z}_{{\rm{ML}},{\rm{bottom}}}$ is the out-of-plane displacement of metal atoms in bottom layer as denoted in Methods and $Z$ is the diagonal length. Top layer possesses similar corrugation. Insert: Zoomed-in portion of the overlaid curves.

Figure 4 reveals the delicate interplay between structural deformations and energy losses/gains. As shown in Fig. 4a, the heterostructure undergoes structural changes in terms of interlayer distance and out-of-plane displacement, with a clear direct relation between interlayer distance and interlayer energy. This is notably evident in regions of large interlayer distance and strain energy. These deformations lead to expanding low-energy stackings and contracting transition regions between high-symmetry stackings, resulting in a lower energy of the hetero-system and the formation of domains. In these domains, the two layers possess matching lattice constants and are separated by nodes of high-energy stackings and solitons²², where stacking alignment changes rapidly and strain accumulates. Similar observations have been reported for other twisted structures^22,25,26. To quantitatively describe these regions, we define the domain or node area as regions where the lattice mismatch of the local stackings is less than 15% and the interlayer distance deviates by no more than 1.5% from that of the corresponding the high-symmetry stackings. Our findings reveal that the ${R}_{h}^{M}$ and ${R}_{h}^{X}$ domains constitute approximately 20% of the entire superlattice while node (${R}_{h}^{h}$) accounts for a smaller proportion of about around 3%.

Fig. 4: Interplay between structural deformations and energy losses/gains in MoS₂/MoSe₂ vdWHS at θ = 0° and 60°.

Panels (a–d) and (e–h) show the results at a twist angle of 0° and 60°, respectively. Panels (a, e) illustrate the distribution of interlayer distance d and average out-of-plane displacement ${\bar{Z}}_{{\rm{tbL}}}$, as defined in Method and Fig. 10. Panels (b, f) display the distribution of the in-plane displacement vector of metal atoms in each layer, where the magnitude $\left|{\bf{u}}\right|$ is represented by the color map, and the vector direction is indicated by the arrows. Panels (c, g) show the distribution of calculated strain energy, and panels (d, h) present the contribution of vdW energy of each constituting layer, expressed in energy per formula unit (f.u.).

Further evidence of these structural adaptations is observable in the in-plane deformations, as depicted in Fig. 4b. The MoS₂ layer possesses a smaller lattice constant compared to MoSe₂. It becomes locally compressed at the ${R}_{h}^{h}$ stacking regions while stretched at ${R}_{h}^{M}$ and ${R}_{h}^{X}$ stackings. These changes provoke the heterostructure to enlarge the area of ${R}_{h}^{M}$ and ${R}_{h}^{X}$ stacking, resulting in two domains. Conversely, MoSe₂, with a larger lattice constant, presents an opposing strain distribution. The residual tensile strain localizes to ${R}_{h}^{h}$ stacking. To relieve this strain, the heterobilayer undergoes a three-dimensional reconstruction in the out-of-plane direction.

The interaction between the two layers in a vdWHS system can be considered as a sum of two components: an attractive interlayer energy and a destabilizing strain energy. To gain a deeper understanding of the origin of such a domain-soliton pattern and the strong spatially modulated strain, we calculated and plotted both components in Fig. 4c, d. The strain energy (Fig. 4) accumulates at the high-energy stacking ${R}_{h}^{h}$ (nodes) and the transition solitons – it exhibits a correlation with the in-plane deformations and the curvature, which is most pronounced at nodes and solitons as shown in Supplementary Fig. 1. The presence of a significant lattice mismatch between MoS₂ and MoSe₂ explains the non-zero curvature at domains. The vdW energy is position dependent due to the influence of the interlayer moiré pattern. It strongly couples to the interlayer distance (Fig. 4a), with maxima and minima occurring in the ${R}_{h}^{h}$ and ${R}_{h}^{M}$ stackings, respectively. As the interlayer energy depends on the local stacking, it is in turn controlled by the nonuniform in-plane deformation. Upon relaxation, the in-plane deformation and the related formation of low-energy domains dominate the energy balance. However, the domain expansion is only possible to a certain extent, controlled by the strain-stress relationship of the 2D crystals comprising the layers. The transition regions of varying, low-symmetry stackings are topologically necessary. As they are energetically unfavorable in terms of both strain and interlayer interaction, their spatial extension is minimized.

At a twist angle of θ = 60° (Fig. 4e–h) the MoS₂/MoSe₂ vdWHS presents three H-type high-symmetry stackings with varying binding energy differences (cf. Fig. 1c). Among these stackings, the most stable one is ${H}_{h}^{h}$, forming a larger triangular domain accounting for 22% of the entire superlattice. The slightly less stable stacking is ${H}_{h}^{X}$, which forms a smaller triangular domain constituting 15% of the area. The marginally lower stability of ${H}_{h}^{X}$, compared to ${H}_{h}^{h}$, suggests the system’s sensitivity to its stacking configuration. This indicates the intricate interplay between the vdWHS’s geometry and its stability, where minor alterations lead to substantial changes in the stability levels. Lastly, high-energy, high-symmetry nodes and transition solitons are observed as they are essential from a topological perspective.

To examine the microscopic mechanism of the sensitivity of angle bending energy and bond stretching energy to the strain energy, we calculated separate radial distribution functions (RDFs) for different pair-wise combinations of chemical elements, as indicated in Fig. 5. The distribution of those bond lengths and pairwise distance throughout the superlattice is displayed in Supplementary Fig. 12. The RDF shapes for both Mo-X (where X represents a chalcogen) bonds exhibit greater heights and sharper peaks, whereas they are broader for Mo-Mo and X-X pairs. This demonstrates that the bond lengths of fewer Mo-X bonds deviate from the equilibrium distance within the monolayer. The presence of two peaks for Mo-S bonds or Mo-Se bonds is attributed to the stretching (expansion) of Mo-S (Mo-Se) bonds at nodes, as shown in Supplementary Fig. 12. This behavior, consistent with the in-plane displacement distribution, results in high-energy stackings shrinking. While the bond length is insensitive to structural corrugation, bond angles are more flexible, as shown in Fig. 5c—they display a broad peak. This phenomenon has been observed earlier for strained MoS₂²⁷. By comparing the two layers, MoS₂ exhibits a larger variation in the bond angle distribution spectra, yet a sharper peak in Mo-S bond distance during relaxation. This aligns with their respective mechanical properties: MoS₂ monolayers possess a higher in-plane Young’s modulus and Poisson’s ratio^28,29, but lower bending rigidity³⁰ compared to MoSe₂ monolayers.

Fig. 5: Changes in the interatomic distances and angles for MoS₂/MoSe₂ vdWHS at θ = 0°.

The distributions of (a) bond lengths, (b) pairwise distance and (c) bond angles are shown for the optimized structure. The y-axis represents the probability density analyzed through a kernel density estimation (KDE) approach. The dashed lines marking the distances and angles within the unit cell. In (b) and (c), each line corresponds to two identical angles or distances.

MS₂/MSe₂ (M = Mo, W) vdWHSs with small twist angles exhibit similar layer-asymmetric corrugation and stacking-dependent in-plane displacement distribution. Figure 6 displays selected results for the MoS₂/MoSe₂ vdWHS with twist angle variations of ±2° from their high-symmetry stackings at 0° and 60°. When the twist angle increases from the untwisted reference stacking of 0° or decreases from 60°, we observe that the size of superlattice and the low-energy domains decrease, while the proportion of high-energy nodes increases. These shifts lead to higher total energy of the whole systems. Figure 6b reveals a decrease in the magnitude of corrugation with increasing twist angles relative to the untwisted reference stackings. The connection between twist angles and corrugation magnitude bears significance for potential electronic applications. Similar to the behavior observed in twisted bilayer graphene³¹, the in-plane displacement in twisted MoS₂/MoSe₂ vdWHS (Fig. 6c) produces vortices surrounding the high-energy stackings, where atoms in the top and bottom layers rotate in opposite directions. Conversely, the untwisted ones show linear displacement. Atoms in the low-energy stackings are essentially unaffected by in-plane relaxations. The reducing out-of-plane and in-plane displacements with increasing twist angles result in lower strain energy. This phenomenon of spatially varying strain energy and vdW energy is common in vdWHS systems^26,32. Figure 6d, e further support this observation by showing that decreasing vdW energy corresponds to reduced strain energy as the twist angle increases (decreases) from 0° (60°). Consequently, in-plane displacement and out-of-plane corrugation decrease, underscoring the intricate interplay among twist angles, stacking configurations, and the resulting energy landscapes within these systems.

Fig. 6: Structural properties of MoS₂/MoSe₂ vdWHS with different twist angles.

Distribution of (a) interlayer distance d between metal atoms across both layers; (b) the average out-of-plane of metal atoms in both layers; (c) the in-plane displacement of metal atoms within the MoS₂ layer, (d) calculated strain energy and (e) contribution of vdW energy from all atoms in MoS₂ layer. MoSe₂ has similar behavior with lower magnitude.

Figure 7a illustrates the corrugation of the relaxed MoS₂/MoSe₂ vdWHS as a function of twist angles. We can identify two regions: for small angles deviating from high-symmetry stackings, both layers exhibit pronounced corrugation, while for larger angles, corrugation diminishes and even becomes negligible. Our findings reveal that the extent of corrugation varies with θ (and thus the moiré period), exhibiting a distinct behavior. In the yellow region, out-of-plane displacements of two consisting layers are oriented towards the same direction in high-symmetry stackings, while in purple regions they are oriented in the opposite direction in high-symmetry stacking regions (as illustrated in Supplementary Fig. 13). The results indicate the presence of a critical twist angle (5° deviating from 3 R/2H stackings in twisted MoS₂/MoSe₂ vdWHS), below which significant corrugation occurs. Furthermore, The MoS₂ layer exhibits stronger corrugation than the MoSe₂ layer, due to the lower bending rigidity of MoS₂ monolayers³⁰. Figure 7b reveals an interplay between twist angles and domains/nodes. As the twist angle increases (or decreases) from 0° (or 60°), a continuous decrease in the proportion of domain areas can be observed. This decrease persists until domain areas reach 0 within the purple area. Meanwhile, the nodes show minimal variation as twist angles change and make up less than 6% of the whole superlattice. Specifically, at a twist angle close to 0°, the ${R}_{h}^{M}$ and ${R}_{h}^{h}$ domains possess similar energy levels, leading to comparable domain areas. Conversely, as the twist angle approaches 60°, the ${H}_{h}^{h}$ domain is the most energetically favorable, resulting in a significantly larger domain than that of the ${H}_{h}^{X}$. Similar to the variation of out-of-plane corrugation, Supplementary Fig. 14 illustrate a decrease in curvature in both MoS₂ and MoSe₂ layers, corresponding to a reduction in out-of-plane relaxation, eventually leading to a state of zero curvature at twist angles of ±12°. Notably, for the same twist, the curvature is most pronounced at the nodes while much lower at the domains. Figure 7c shows the calculated binding energy ${E}_{{\rm{b}}}$ and strain energy ${E}_{{\rm{strain}}}$ of vdWHS as a function of $\theta$. The areas characterized by high corrugation directly link with domain formation and stabilization of larger low-energy stacking areas of the bilayer system. The formation of low-energy stackings requires transition regions, where the system has to adopt unfavorable stackings due to symmetry, and where the system is locally strained. The strain energy in the transition regions is compensated by the stronger interlayer attraction.

Fig. 7: The impact of twist angle on the structural properties of twisted MoS₂/MoSe₂ vdWHS.

a Magnitude of out-of-plane corrugation ∆Z (denoted in Fig. 10) of metal atoms in each constituting layer, (b) the proportion of different domain and node areas relative to the total area, (c) binding energy and strain energy as a function of twist angle θ. Yellow/purple regions show different types of corrugation, where the out-of-plane displacements of each layer are oriented towards the same (in-phase, yellow) or opposite (out-of-phase, purple) directions in all local stackings, corresponding to the Supplementary Fig. 13. The critical angles are ~5° and 55°, respectively.

The above observation generally applies to a wide variety of twisted TMDC vdWHSs, as similar periodic perturbations arise from either the mismatch of two layers with differing lattice constants or from interlayer twists in these hetero-systems. MoX₂/WX₂ have much larger moiré superlattice due to their comparable lattice constants, and exhibit distinct structural properties. Figure 8 displays examples of MoS₂/WS₂ vdWHSs with various small twist angles. At twist angles near 0°, ${R}_{h}^{M}$ and ${R}_{h}^{X}$ stackings, featuring nearly identical interlayer distance and energy (cf. Supplementary Fig. 23), expand to two large, flat, triangular domains, while ${R}_{h}^{h}$ regions shrink and form nodes. At twist angles near 60°, ${H}_{h}^{h}$ is the lowest-energy stacking, and therefore forms the hexagonal domain. Consequently, strain is intensely concentrated within nodes and along solitons. As the twist angle deviates further from 0° or 60°, node and soliton sizes increase while the domains become smaller. This variation leads to differing patterns of out-of-plane corrugation, in-plane displacement, as well as the distribution of strain and van der Waals energy.

Fig. 8: Structural properties of MoS₂/WS₂ vdWHS with different twist angles.

Distribution of (a) interlayer distance d between metal atoms across both layers; (b) the average out-of-plane of metal atoms in both layers; (c) the in-plane displacement of metal atoms within the MoS₂ layer, (d) calculated strain energy and (e) contribution of vdW energy from all atoms in MoS₂ layer. WS₂ has similar behavior with lower magnitude.

Figure 9a illustrates the structural features of MoS₂/WS₂ vdWHS as function of twist angles. Similar to the twisted MoS₂/MoSe₂ system, two regions emerge. For small twists, the corrugation is significant in both layers, and the out-of-plane displacements are oriented in the same direction at local stackings. When the twist angle in-/decrease from 0°/60°, the magnitude of out-of-plane deformation exhibits a continuous decrease compared to the untwisted structures. For larger angles, the out-of-phase corrugation becomes marginal or even negligible (as illustrated in Supplementary Fig. 15). The critical angles are around 3°, smaller than those observed in MoS₂/MoSe₂ vdWHS. The relationship between twist angles and the domain/node areas is explored in Fig. 9b. Results demonstrate that as the twist angle increasing from 0°, the proportion of both ${R}_{h}^{M}$ and ${R}_{h}^{X}$ remains nearly the same, and it decreases monotonically. This suggests that the presence of a twist angle disrupts the formation of large commensurate domains. The node exhibits minimal variation and make up less than 2% of the entire superlattice. A similar behavior is noted for angles decrease from 60°. The proportion of the most energy-favorable ${H}_{h}^{h}$ domain decreases, while ${H}_{h}^{X}$ and node show non-monotonic behaviors as the twist angle changes: Initially, as the twist angle decreases, the proportion of ${H}_{h}^{X}$ and node first increases because the mismatch caused by twist disrupts the formation of ${H}_{h}^{h}$. As the twist angle gradually approaches the purple region, a subsequent decline becomes evident in the proportion of the ${H}_{h}^{X}$ and the node area. The Gaussian curvature at different areas is depicted in Supplementary Fig. 16. Due to the negligible lattice mismatch, the curvature within these domains is nearly zero, resulting in the formation of large, flat domains. Curvature is more pronounced at nodes where strain accumulated. Figure 9c presents binding energy and strain energy. The competition between strain energy and vdW interaction causes a non-monotonic behavior of the strain energy as a function of twist angles. As the angle increases from 3 R stackings, corrugation decreases while strain energy increases until it reaches a maximum at a certain angle. Further angle increase leads to a decrease in both strain energy and corrugation. Similar behavior is observed for angles near 60°.

Fig. 9: The impact of twist angle on the structural properties of twisted MoS₂/WS₂ vdWHS.

a Magnitude of out-of-plane corrugation ∆Z (denoted in Fig. 10) of metal atoms in each constituting layer, (b) the proportion of different domain and node areas relative to the total area, (c) binding energy and strain energy as a function of twist angle θ. Yellow/purple regions show different types of corrugation, where the out-of-plane displacements of each consisting layer are oriented towards the same (in-phase, yellow) or opposite (out-of-phase, purple) directions in all local stackings (cp. Supplementary Fig. 15). The critical angles are ~ 3° and 57°, respectively.

In the preceding study, we focus on counterclockwise rotations, denoted as positive θ > 0. When considering clockwise rotations, where θ < 0, a chiral counterpart to the counterclockwise configuration emerges. This results in a mirror-symmetry to the counterclockwise configuration, regardless of whether the moiré patterns are commensurate or incommensurate. To evaluate this chiral symmetry, we conducted a comparison of the results obtained from clockwise and counterclockwise rotations. Supplementary Figure 21 reveals that, within the accuracy limit of our calculations, the binding energy remains unaffected by the twist direction. Similarly, the average corrugation of both enantiomeric structures is in close agreement. However, a difference emerges when considering the in-plane displacements (Supplementary Fig. 22): the atoms exhibit a chiral displacement pattern in counterclockwise/clockwise rotated structures.

Conclusion

Our study on TMDC vdWHs consisting of MoS₂, MoSe₂, WS₂ and WSe₂ monolayers reveals that structural features and energy contributions are closely related to interlayer twist angles and specific stacking configurations. In particular, we observe substantial lattice reconstruction involving out-of-plane corrugation and in-plane displacement in TMDC vdWHSs due to the competing influences of strain and van der Waals interactions. Specifically, as the twist angle approaches 0°, two types – ${R}_{h}^{M}$ and ${R}_{h}^{X}$ – stacking domains form, while at/near 60° the ${H}_{h}^{h}$ stacking domains become prominent, separated by nodes and solitons. The balance between these energy contributions largely depends on the twist angle between two layers, thereby leading to a variety of structural features and behaviors. For instance, as the twist angle deviate from 0°/60°, we observe a consistent reduction in the proportion of domain areas and the magnitude of corrugation. Notably, our findings suggest critical twist angles away from 0°/60°, below which the domain formation is evident, with the two layers possessing significant in-phase corrugation, while above which the two layers show out-of-phase corrugation.

Furthermore, we demonstrate structural disparities between heterostructures with different or identical chalcogen atoms. The four types of MS₂/M’Se₂ (M, M’ = Mo, W) possess similar moiré lattice sizes at the same twist angle, while the remaining two types of MoX₂/WX₂ (X = S, Se) possess large-scale moiré superlattices due to their comparable lattice constants.

In simulations of heterostructures consisting of two layers with distinct lattice constants, one layer is typically compressed while the other one is stretched, in order to create a commensurate superlattice. To investigate the influence of the boundary conditions on the atomic reconstruction of twisted heterostructures, we modeled systems with a periodic layer onto which we stacked flakes twisted by various angles. For heterostructures composed of large flakes, we always observe multiple domains, particularly for the MoX₂/WX₂ at twist angle θ = 0°/60°. By comparing the flake systems with fully periodic commensurate systems, we deduce that structures created by the transfer method with flakes of lateral size above 80 nm for MS₂/M’Se₂ or 190 nm for MoX₂/WX₂ can reliably be described by fully periodic models.

In summary, we demonstrate the interplay between lattice reconstructions, energy contributions, and twist angles in TMDC heterostructures. Understanding these relationships is crucial for designing next-generation heterostructures with customized functionalities. Furthermore, the similarity between finite flakes and commensurate systems highlights the robustness and reliability of large-scale, fully periodic commensurate models in predicting behaviors in van der Waals heterostructures.

Methods

Model generation

Based on the fully optimized TMDC monolayers, we first generated unrelaxed twisted and untwisted bilayer models. Zero twist refers to the 3R-like configuration, with parallel monolayers in AA stacking. The rotation axis is arbitrarily chosen to be perpendicular to both layers and passes through the chalcogen atom. Then, the moiré lattice vectors can be derived from¹²

$$\left[\begin{array}{c}{\widetilde{{\bf{a}}}}_{1}^{{\rm{T}}}\\ {\widetilde{{\bf{a}}}}_{2}^{{\rm{T}}}\end{array}\right]={{\bf{M}}}_{{\rm{a}}}\left[\begin{array}{c}{{\bf{a}}}_{1}^{{\rm{T}}}\\ {{\bf{a}}}_{2}^{{\rm{T}}}\end{array}\right]$$

(1)

$$\left[\begin{array}{c}{\widetilde{{\bf{b}}}}_{1}^{{\rm{T}}}\\ {\widetilde{{\bf{b}}}}_{2}^{{\rm{T}}}\end{array}\right]={\bf{R}}{{\bf{M}}}_{{\rm{b}}}\left[\begin{array}{c}{{\bf{b}}}_{1}^{{\rm{T}}}\\ {{\bf{b}}}_{2}^{{\rm{T}}}\end{array}\right]$$

(2)

where ${{\bf{M}}}_{{\rm{a}}}$ and ${{\bf{M}}}_{{\rm{b}}}$ are $2\times 2$ transformation matrices, ${{\bf{a}}}_{i}$ and ${{\bf{b}}}_{i}$ ($i=1,\,2$) are basis vectors of the two constituting layers, ${\widetilde{{\bf{a}}}}_{i}$ and ${\widetilde{{\bf{b}}}}_{i}$ are the basis vectors of the new moiré lattice of each layer, and the rotation matrix ${\bf{R}}=[\begin{array}{cc}\cos \theta &\sin \theta \\ -\sin \theta & \cos \theta \end{array}]$.

Due to the rotation and reflection symmetries of the hexagonal lattice of each monolayer, the moiré structures are of hexagonal symmetry, which results in

$${{\bf{M}}}_{j}=\left[\begin{array}{cc}{p}_{j} & {q}_{j}\\ {-q}_{j} & {p}_{j}-{q}_{j}\end{array}\right]$$

(3)

where $j\in \{a,b\}$ and ${p}_{j},\,{q}_{j}{\mathbb{\in }}{\mathbb{Z}}$. A commensurate rotation would occur if ${\widetilde{{\bf{a}}}}_{i}={\widetilde{{\bf{b}}}}_{i}$, which only happens in exceptional cases. For the general incommensurate case we introduced a tolerance related to the maximum strain applied to each layer (one layer is stretched while the other one is compressed) and thus transformed it to a commensurate state³³: $\frac{\left|{\widetilde{{\bf{a}}}}_{i}-{\widetilde{{\bf{b}}}}_{i}\right|}{\max ({\widetilde{{\bf{a}}}}_{i},{\widetilde{{\bf{b}}}}_{i})\,}\, <\, \varepsilon$. In this work, we only considered first-order moiré superlattice models with strain ($\varepsilon$) less than 0.5% (and in most cases less than this, see Supplementary Fig. 9). We do not encompass the analysis of second-order (or higher) moiré patterns, where each supercell contains crystallographically inequivalent sub-moiré lattices—the alignment of symmetry sites approaches the ideal configuration more closely in certain regions than in others.

Geometry optimization and Structure Analysis

Full geometry optimizations (atomic positions and lattice constants) were performed using the LAMMPS package^34,35, employing the Stillinger-Weber (SW)^36,37 and Kolmogorov-Crespi (KC)^38,39 force fields to capture the intralayer and interlayer interaction, respectively. Energy minimization was carried out using the conjugate gradient method, where the convergence criteria of relative energy change (the absolute energy change divided by the energy magnitude) was ${10}^{-18}$ and the upper bound of residual forces on any atom was ${10}^{-8}$ eV/Å. Then, we analyzed the optimized structures by characterizing the key quantities as shown in Fig. 10: interlayer distance $d(x,y)$, out-of-plane displacement of each layer ${Z}_{{\rm{ML}}}(x,y)$, average out-of-plane displacement ${\bar{Z}}_{{\rm{tbL}}}(x,y)$ and magnitude of corrugation $\triangle Z$. The interlayer distance $d(x,y)$ is defined as the distance between the two adjacent surfaces at each lateral position $(x,y)$. These surfaces are determined by interpolating the positions of the metal atoms within each layer. The out-of-plane displacement of each layer ${Z}_{{\rm{ML}}}(x,y)$ is defined as ${Z}_{{\rm{ML}}}(x,y)=h(x,y)-{h}^{0}$, where $h(x,y)$ is the height (z coordinate) of the metal-atom surface at $(x,y)$. The reference height ${h}^{0}$ is a constant, given by the mid-range value ${h}^{0}=({h}^{\max }+{h}^{\min })/2$, where ${h}^{\max }$ and ${h}^{\min }$ are the global maximum and minimum heights, respectively. The average out-of-plane displacement is ${\bar{Z}}_{{\rm{tbL}}}(x,y)={h}_{{\rm{tbL}}}(x,y)-{h}_{{\rm{tbL}}}^{0}$, where ${h}_{{\rm{tbL}}}(x,y)=({h}_{{\rm{top}}}(x,y)+{h}_{{\rm{bottom}}}(x,y))/2$ and ${h}_{{\rm{tbL}}}^{0}$ is the constant reference height as also defined above. This definition leads to ${\bar{Z}}_{{\rm{tbL}}}\left(x,y\right)=0$ for perfect out-of-phase corrugation while an in-phase one will lead to a finite average out-of-plane displacement (cf. Fig. 10e, sketching an in-phase or out-of-phase corrugation). The magnitude of corrugation of each layer $\triangle Z$ indicates the overall vertical variation, defined as $\triangle Z={h}^{\max }-{h}^{\min }$, the difference between the maximum and minimum of downward/upward out-of-plane displacement of metal atoms after the geometry optimization.

Fig. 10: An introductory visualization to support the quantitative definitions detailed in the Methods section.

a The 3D view of the relaxed MoS₂/MoSe₂ vdWHS without interlayer twist. b Schematic of out-of-plane corrugation during geometry optimization using the coordinates of Mo atoms. ${Z}_{{\rm{ML}}}$ represents upward/downward height displacement of each constituting layer. The interlayer distance is defined as the distance between the two adjacent surfaces generated through interpolation. c The 3D view of atomic positions and the profile of the out-of-plane displacement ${Z}_{{\rm{ML}}}$ using Mo atoms, along with a 2D projection of the interlayer distance $d$. d The out-of-plane displacement obtained by scanning the atomic height along the diagonal of the vdWHS, as indicated by the arrow in (c), alongside the interlayer distance and the average corrugation ${Z}_{{\rm{tbL}}}$ (the variation of the midpoint height of top and bottom layer relative to the midrange height of the entire structure). e Sketch of heterostructures with either in-phase or out-of-phase corrugation. A 0.015% strain is applied to each constituting layer for generating the bilayer model.

It is important to note that $d(x,y)$, ${Z}_{{\rm{ML}}}(x,y)$, and ${\bar{Z}}_{{\rm{tbL}}}(x,y)$ are dependent on the lateral position, while $\triangle Z$ is a global constant of the structure. However, to emphasize the general structural features, we will simplify our notation without explicitly mentioning their $(x,y)$ dependence in the following discussion.

In addition to the structural features, we also calculated the binding, strain and vdW energies. The binding energy is defined as ${E}_{{\rm{b}}}=({E}_{{\rm{BL}}}-{E}_{{\rm{ML}},{\rm{bottom}}}-{E}_{{\rm{ML}},{\rm{top}}})/N$, where $N$ is the number of atoms in the moiré superlattice, and ${E}_{{\rm{BL}}}$ and ${E}_{{\rm{ML}},{i}}$ are the total energies of the vdWHS and each freestanding constituting layer, respectively. The strain energy is defined as ${E}_{{\rm{strain}}}=\sum \triangle {V}_{1}({r}_{{ij}})+\sum \triangle {V}_{2}({r}_{{ij}},{r}_{{jk}},{\theta }_{{ijk}})$, i.e., the variation of the sum of bond stretching energy and angle bending energy with respect to that in the flat layer; while the vdW energy is defined as ${E}_{{\rm{vdW}}}=1/2{\sum }_{j\ne i}{V}_{{ij}}$, i.e., the KC interaction potential between the two layers.

The force field was validated by performing a series of DFT benchmark calculations. We took MoS₂/MoSe₂ vdWHS with different twist angles as examples and calculated their interlayer distance and out-of-plane displacement using the FHI-aims⁴⁰ software. The exchange-correlation functional was approximated by generalized gradient approximation (GGA) by Perdew-Burke-Ernzerhof (PBE)⁴¹. Tkatchenko-Scheffler (TS) vdW corrections⁴² were applied. The convergence criteria of the charge density was ${10}^{-3}{e}/{a}_{0}$ in self-consistent calculations and that of residual forces on atoms was $0.05\,{\rm{eV}}/{\rm{\mathring{\rm A} }}$ for geometry optimization. The tight tier 1 basis sets and the Γ point approximation have been used, as the unit cells of vdWHSs include 400-2000 atoms. The force-field and DFT exhibit good agreement concerning the out-of-plane displacement field as depicted in Supplementary Fig. 25, where the maximal difference between the two method is a mere 0.03 ${\rm{\mathring{\rm A} }}$. The interlayer distances indicate slight disparities (i.e., maximal difference of 0.1 ${\rm{\mathring{\rm A} }}$); nevertheless, the variation as a function of twist angles is similar (Supplementary Fig. 24). It is noteworthy that the TS approach consistently overestimates the interlayer distances, for instance, with an overestimation magnitude of 0.1 ${\rm{\mathring{\rm A} }}$ in the 2H-MoS₂ system⁴³. Hence, the force-field method is sufficient to capture the interlayer interaction of the hetero-systems.