Tetrahedral triple-Q magnetic ordering and large spontaneous Hall conductivity in the metallic triangular antiferromagnet Co_1/3TaS₂

Abstract

The triangular lattice antiferromagnet (TLAF) has been the standard paradigm of frustrated magnetism for several decades. The most common magnetic ordering in insulating TLAFs is the 120° structure. However, a new triple-Q chiral ordering can emerge in metallic TLAFs, representing the short wavelength limit of magnetic skyrmion crystals. We report the metallic TLAF Co_1/3TaS₂ as the first example of tetrahedral triple-Q magnetic ordering with the associated topological Hall effect (non-zero σ_xy(H = 0)). We also present a theoretical framework that describes the emergence of this magnetic ground state, which is further supported by the electronic structure measured by angle-resolved photoemission spectroscopy. Additionally, our measurements of the inelastic neutron scattering cross section are consistent with the calculated dynamical structure factor of the tetrahedral triple-Q state.

Introduction

Discovering magnetic orderings with novel properties and functionalities is an important goal of condensed matter physics. While ferromagnets are still the best examples of functional magnetic materials due to their vast spectrum of technological applications, antiferromagnets are creating a paradigm shift for developing new spintronic components^1,2. A more recent case is the discovery of Skyrmion crystals induced by a magnetic field in different classes of materials. These chiral states, which result from a superposition of three spirals with ordering wave vectors that differ by a $\pm$120° rotation about a high-symmetry axis, can produce a substantial synthetic magnetic field that couples only to the orbital degrees of freedom of conduction electrons^3,4. When conduction electrons propagate through a skyrmion spin texture, they exhibit a spontaneous Hall effect. The origin of this effect becomes transparent in the adiabatic limit. Due to the exchange interaction, the underlying local moment texture aligns the spin of a conduction electron that moves in a loop, inducing a Berry phase in its wavefunction equal to half of the solid angle spanned by the local moments enclosed by the loop. This phase is indistinguishable from the Aharonov-Bohm phase induced by a real magnetic flux, and each skyrmion generates a flux quantum in the corresponding magnetic unit because the spins span the full solid angle of the sphere (4π).

The triangular lattice Heisenberg model is a textbook example that can host diverse quantum states with small variations of short-range exchange interactions. The generic ground state for nearest-neighbor (NN) antiferromagnetic interactions is the three-sublattice 120° structure shown in Fig. 1a. This spiral structure is characterized by an ordering wave vector located at $\pm$K-points of the hexagonal Brillouin zone (Fig. 1d). Adding a relatively small second NN antiferromagnetic interaction gives rise to the two-sublattice collinear stripe spin configuration shown in Fig. 1b, whose ordering wave vector is one of the three M-points of the Brillouin zone (see Fig. 1e). Remarkably, for S = ½, these two phases seem to be separated by a quantum spin liquid state^{5,6,7,8,9,10,11} whose nature is not yet fully understood.

Fig. 1: The Tetrahedral triple-Q state and crystal structure of Co_1/3TaS₂.

a–c Three fundamental antiferromagnetic orderings for a triangular lattice system. The red-shaded regions denote each magnetic unit cell. d–f Positions of the magnetic Bragg peaks (red circles) in momentum space generated by a–c A black (red) hexagon corresponds to a crystallographic (magnetic) Brillouin zone. The green and blue circles in d–e denote the magnetic Bragg peaks from the other two magnetic domains. g A crystallographic unit cell of Co_1/3TaS₂. h The temperature-dependent magnetization of single-crystal Co_1/3TaS₂ with H//c.

However, small effective four-spin interactions can induce a fundamentally different chiral antiferromagnetic order in triangular lattice antiferromagnets. This state is the triple-Q version of the stripe order, where the three different M-ordering wave vectors (see Fig. 1f) coexist in the same phase giving rise to a non-coplanar four-sublattice magnetic ordering (see Fig. 1c). The spins of each sublattice point along the all-in or all-out principal directions of a regular tetrahedron. Theoretical studies suggest that this state can appear naturally in metallic TLAFs, where effective four-spin interactions arise from the exchange interaction between conduction electrons and localized spin degrees of freedom^12,13. This state was predicted to appear in the Mn monolayers on Cu(111) surfaces by density functional theory¹⁴, and it was observed in the hcp Mn monolayers on Re(0001) using spin-polarized scanning tunneling microscopy^15,16. However, it has not been reported yet in bulk systems.

The tetrahedral ordering is noteworthy for its topological nature, as it can be viewed as the short-wavelength limit of a magnetic skyrmion crystal¹⁷. The three spins of each triangular plaquette span one-quarter of the solid angle of a sphere, implying that each skyrmion (one flux quantum) is confined to four triangular plaquettes. As illustrated in Fig. 1c, the two-dimensional (2D) magnetic unit cell of the tetrahedral ordering consists of eight triangular plaquettes, meaning there are two skyrmions per magnetic unit cell. In other words, the tetrahedral triple-Q ordering creates a very strong, effective magnetic field of one flux quantum divided by the area of 4 triangular plaquettes in the adiabatic limit. Notably, this ordering does not have any net spin magnetization. However, the emergent magnetic field couples to the orbital degrees of freedom of the conduction electrons, giving rise to a uniform orbital magnetization and a large topological Hall effect characterized by scalar spin chirality (${\chi }_{{ijk}}=\langle {{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j}\times {{{{{{\boldsymbol{S}}}}}}}_{k}\rangle$)^3,4. Thus, this spin configuration is the simplest textbook example where non-trivial band topology is induced in the absence of relativistic spin-orbit coupling (the non-coplanar configuration generates an effective gauge field that couples to the orbital degrees of freedom of the conduction electrons). Moreover, the tetrahedral ordering can provide a potential route to realize the antiferromagnetic Chern insulator by properly adjusting the Fermi level of the system, as suggested in Refs. ^12,13,18.

This work reports a four-sublattice tetrahedral triple-Q ordering as the only scenario known to the authors that is consistent with our data for the metallic triangular antiferromagnet Co_1/3TaS₂. Our key observations are the coexistence of long-range antiferromagnetic ordering with wave vector q_m = (1/2, 0, 0), a weak ferromagnetic moment (M_z$({{{{{\bf{H}}}}}}=0)$), and a non-zero AHE (${{{{{{\rm{\sigma }}}}}}}_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$) below T_N2, which rules out the possibility of single-Q and double-Q ordering^19,20,21,22. Based on the crystalline and electronic structure of Co_1/3TaS_2, we also provide a theoretical conjecture about the origin of the observed ordering_, consistent with our angle-resolved photoemission spectroscopy (ARPES) data. Moreover, the calculated low-energy magnon spectra of the tetrahedral ordering agree with the spectra measured by inelastic neutron scattering. Finally, we discuss the robustness of the tetrahedral ordering against an applied magnetic field in Co_1/3TaS₂.

Results and discussion

Co_1/3TaS₂ is a Co-intercalated metal comprising triangular layers of magnetic Co²⁺ ions (Fig. 1g). Previous studies on Co_1/3TaS₂ in the 1980s reported the bulk properties of a metallic antiferromagnet with S = 3/2 (a high-spin d⁷ configuration of Co²⁺)^23,24,25, including a neutron diffraction study that reported an ordering wave vector q_m = (1/3, 1/3, 0) characteristic of a 120° ordering²⁵. More recently, an experimental study on single-crystal Co_1/3TaS₂ observed a significant anomalous Hall effect (AHE) comparable to those in ferromagnets below 26.5 K, which is the second transition temperature (T_N2) of the two antiferromagnetic phase transitions at T_N1 = 38 K and T_N2 = 26.5 K (see Fig. 1h)²⁶. Based on the 120° ordering reported in ref. ²⁵ and a symmetry argument, the authors of ref. ²⁶ suggested that the observed AHE, ${{{{{{\rm{\sigma }}}}}}}_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)\, \ne \, 0,$ can be attributed to a ferroic order of cluster toroidal dipole moments. However, our latest neutron scattering data reported in this work reveals an entirely different picture: Co_1/3TaS₂ has a magnetic structure with ordering wave vectors of the M-points (q_m = (1/2, 0, 0) and symmetry-related vectors) instead of q_m = (1/3, 1/3, 0). The most likely scenario for such distinct outcomes would be a difference in Co composition; while we confirmed that our experimental results are consistently observed in Co_xTaS₂ with 0.299(4) < x < 0.325(4), the reported composition value for ref. ²⁵’s sample is x = 0.29 (see ref. ²⁴). Additional Co disorder, beyond variations in composition, could also be a contributing factor. However, the limited data available for the sample in ref. ²⁵ prevents us from forming a definitive assessment in this regard (see Supplementary Notes). In any case, the theoretical analysis in ref. ²⁶ based on ref. ²⁵ is not valid for q_m = (1/2, 0, 0) since a different ordering wave vector leads to a qualitatively different scenario. This forced us to conduct a more extensive investigation and come up with a scenario consistent with our comprehensive experimental datasets.

Figure 2a, b show the neutron diffraction patterns of powder and single-crystal Co_1/3TaS₂ for T < T_N. Magnetic reflections appear at the M-points of the Brillouin zone for both T < T_N2 and T_N2 < T < T_N1, implying that the ordering wave vector is q_m = (1/2, 0, 0) or its symmetrically equivalent wave vectors. It is worth noting that this observation is inconsistent with the previously reported wave vector q_m = (1/3, 1/3, 0)²⁵; i.e., Co_1/3TaS₂ does not possess a 120° magnetic ordering. The magnetic Bragg peaks at the three different M points connected by the three-fold rotation along the c-axis (C_3z) have equivalent intensities within the experimental error (Fig. 2c). This result suggests either single-Q or double-Q ordering with three equally weighted magnetic domains or triple-Q ordering.

Fig. 2: Magnetic ground state of Co_1/3TaS₂ revealed by neutron diffraction.

a The powder neutron diffraction pattern of Co_1/3TaS₂, measured at 60 K (Gray squares) and 3 K (Red circles). A weak λ/2 signal is highlighted explicitly (see Methods). The solid black line is the diffraction pattern of Co_1/3TaS₂ simulated with the spin configuration shown in h, or equivalently i. The gray (red) vertical solid lines denote the position of nuclear (magnetic) reflections. The full diffraction data can be found in Supplementary Fig. 3. b The single-crystal neutron diffraction pattern at 5 K, demonstrating magnetic Bragg peaks located at the M points. c The intensities of the three magnetic Bragg peaks originating from three different ordering wave vectors. d The temperature-dependent intensities of some magnetic Bragg peaks in single-crystal Co_1/3TaS₂. e, f The temperature dependence of ${\sigma }_{{xy}}$(H = 0) and ${M}_{z}$(H = 0) in Co_1/3TaS₂, measured after field cooling under 5 T. g, h The refined magnetic structures for g 26.5 K < T < 38 K and h T < 26.5 K. i The triple-Q counterpart (tetrahedral) of the single-Q (stripe) ordering shown in h. Note that the spin configurations in h and I give the same powder diffraction pattern.

First, we analyzed the neutron diffraction data based on group representation theory and Rietveld refinement, assuming a single-Q magnetic structure: ${{{{{{{\bf{M}}}}}}}_{i}^{\nu }{{{{{\boldsymbol{=}}}}}}{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }\cos {{{{{\boldsymbol{(}}}}}}{{{{{{\bf{q}}}}}}}_{{{\rm{m}}}}^{\nu }{{{{{\boldsymbol{\bullet }}}}}}{{{{{{\bf{r}}}}}}}_{i}{{{{{\boldsymbol{)}}}}}}$ with $\nu=$ 1, 2, and 3 (see Supplementary Text and Supplementary Tables 2 and 3). As a result, we found that the spin configurations for T < T_N2 and T_N2 < T < T_N1 correspond to Fig. 2g, h, respectively, where each configuration belongs to the ${\varGamma }_{2}+{\varGamma }_{4}$ ($\alpha {{{{{{\bf{V}}}}}}}_{22}^{\nu }$ + $\beta {{{{{{\bf{V}}}}}}}_{41}^{\nu }$, see Supplementary Table 3) and ${\varGamma }_{2}$ (${{{{{{\bf{V}}}}}}}_{22}^{\nu }$) representations—see Supplementary Notes. The refinement result yielded an ordered moment of 1.27(1)${{{{{{\rm{\mu }}}}}}}_{{{{{{\rm{B}}}}}}}$/Co²⁺ at 3 K.

However, a single-Q spin configuration with q_m = (1/2, 0, 0) possesses time-reversal symmetry (TRS) combined with lattice translation ($\equiv {{{{{{\rm{\tau }}}}}}}_{1a}T$, see Fig. 2g, h), which strictly forbids the finite ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$ and ${M}_{{{{{{\rm{z}}}}}}}({{{{{\bf{H}}}}}}=0)$ observed at T < T_N2 (Fig. 2e, f). Similarly, double-Q spin configurations in a triangular lattice are not compatible with finite ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$ and ${M}_{{{{{{\rm{z}}}}}}}({{{{{\bf{H}}}}}}=0)$ due to its residual symmetry relevant to chirality cancellation¹⁹. Triple-Q ordering is, therefore, the only possible scenario that can resolve this contradiction since it allows for finite ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$ and ${M}_{{{{{{\rm{z}}}}}}}({{{{{\bf{H}}}}}}=0)$ due to broken ${{{{{{\rm{\tau }}}}}}}_{1a}T$ symmetry^19,20,21. In general, determining whether the magnetic structure of a system is a single/double-Q phase with three magnetic domains or a triple-Q ordering requires advanced experiments. However, q_m = (1/2, 0, 0) is a special case where a triple-Q state can be easily distinguished from the other possibilities by probing non-zero TRS-odd quantities incompatible with the symmetry of single/double-Q structures.

The most symmetric triple-Q ordering that produces the same neutron diffraction pattern as that of Fig. 2h is illustrated in Fig. 2i. This is precisely the four-sublattice tetrahedral ordering shown in Fig. 1c, except that Co_1/3TaS₂ has an additional 3D structure with an AB stacking pattern. Such a triple-Q counterpart can be obtained through a linear combination of three symmetrically equivalent single-Q states (${{{{{{{\bf{M}}}}}}}_{i}^{\nu }{{{{{\boldsymbol{=}}}}}}{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }\cos {{{{{\boldsymbol{(}}}}}}{{{{{{\bf{q}}}}}}}_{{{\rm{m}}}}^{\nu }{{{{{\boldsymbol{\bullet }}}}}}{{{{{{\bf{r}}}}}}}_{i}{{{{{\boldsymbol{)}}}}}}$ with $\nu=$ 1, 2, 3) connected by the three-fold rotation about the c axis²² (see Supplementary Notes for more explanation). However, an arbitrary linear combination of these three components yields a triple-Q spin configuration with a site-dependent magnitude of ordered moments on the Co sites, i.e., |${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}$| depends on i (see Supplementary Fig. 7). A uniform |${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}$| is obtained only when the Fourier components ${{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }$ of the three wave vectors ${{{{{{\bf{q}}}}}}}_{{{\rm{m}}}}^{\nu }$ are orthogonal to each other¹², i.e., ${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}={\sum }_{\nu={{{{\mathrm{1,3}}}}}}{{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }\cos ({{{{{{\bf{q}}}}}}}_{{{\rm{m}}}}^{\nu }{{\cdot }}{{{{{{\bf{r}}}}}}}_{i})$ with Δ_v⊥Δ_v′ for $\nu \ne {\nu }^{{\prime} }.$ We note, however, that the magnitude of the ordered moments (|${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}$|) does not have to be the same for quantum mechanical spins. Nevertheless, as explained in detail in the Supplementary Information, only non-coplanar triple-Q orderings corresponding to equilateral ($\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }\right|{{{{{\boldsymbol{=}}}}}}\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{{\nu }^{{\prime} }}\right|$, Fig. 2h) or non-equilateral ($\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }\right|{{{{{\boldsymbol{\ne }}}}}}\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{{\nu }^{{\prime} }}\right|$) tetrahedral configurations are consistent with our Rietveld refinement of neutron diffraction data.

In the high-temperature ordered phase at T_N2 < T < T_N1, the triple-Q ordering that yields the same neutron diffraction pattern as that of the single-Q ordering shown in Fig. 2g is collinear, giving rise to highly nonuniform |${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}$| (see Supplementary Fig. 7). Such a strong modulation of |${{{{{{\bf{M}}}}}}}_{i}^{{{{{{\rm{tri}}}}}}}$| is unlikely for S = 3/2 moments weakly coupled to conduction electrons (see discussion below). In addition, the collinear triple-Q ordering allows for finite M_z(H = 0) and ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$^19,20,21,22, while they are precisely zero within our measurement error for T_N2 < T < T_N1. On the other hand, the single-Q ordering shown in Fig. 2g is more consistent with M_z(H = 0) = ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$ = 0 in the temperature range T_N2 < T < T_N1 due to its ${{{{{{\rm{\tau }}}}}}}_{1a}T$ symmetry. Therefore, the combined neutron diffraction and anomalous transport data indicate that a transition from a collinear single-Q to non-coplanar triple-Q ordering occurs at T_N2. Interestingly, as we will see below, our theoretical analysis captures this two-step transition process.

We now examine the feasibility of the tetrahedral triple-Q ground state in Co_1/3TaS₂. Notably, in contrast to typical triple-Q orderings reported in other materials^27,28,29, this state emerges spontaneously in Co_1/3TaS₂ without requiring an external magnetic field. It was proposed on theoretical grounds that this state could arise in a 2D metallic TLAF formulated by the Kondo lattice model^12,13,18,30:

$$H=-t{\sum }_{\langle i,j\rangle }{c}_{i\alpha }^{{{\dagger}} }{c}_{j\alpha }-J{\sum }_{i}{{{{{{\boldsymbol{S}}}}}}}_{{i}}\cdot {c}_{i\alpha }^{{{\dagger}} }{{{{{{\boldsymbol{\sigma }}}}}}}_{\alpha \beta }{c}_{i\beta }.$$

(1)

From a crystal structure perspective, Co_1/3TaS₂ is an ideal candidate to be described by this model. The nearest Co-Co distance (5.74 Å) is well above Hill’s limit, and the CoS₆ octahedra are fully isolated ((Fig. 3a). This observation suggests that the Co 3d bands would retain their localized character, while itinerant electrons mainly arise from the Ta 5d bands. Our density functional theory (DFT) calculations confirm this picture and reveal that the density of states near the Fermi energy has a dominant Ta 5d orbital character (Supplementary Fig. 11). In this situation, a magnetic Co²⁺ ion can interact with another Co²⁺ ion only via the conduction electrons in the Ta 5d bands. Thus, in a first approximation, the Co²⁺ 3d electrons can be treated as localized magnetic moments interacting via exchange with the Ta 5d itinerant electrons^23,25 (see Fig. 3b). Indeed, the Curie-Weiss behavior observed in Co_1/3TaS₂ provides additional support for this picture; the magnitude of the fitted effective magnetic moment indicates S~1.35, close to the single-ion limit of Co²⁺ in the high spin S = 3/2 configuration²⁶. However, our neutron diffraction measurement indicates a significant suppression of the ordered magnetic moment S ~ 0.64 (assuming that a g-factor is 2). Similar reductions of the ordered moment have been reported in other metallic magnets comprising 3d transition metal elements, and they are generally attributed to a partial delocalization of the magnetic moments. In addition, interactions between Co local moments and itinerant electrons from Ta 5d band can lead to partial screening of the local moments. Another possible origin of the reduction of the ordered moment is quantum spin fluctuations arising from the frustrated nature of the effective spin-spin interactions.

Fig. 3: Stabilization mechanism and dynamical properties of the tetrahedral order in Co_1/3TaS₂.

a The in-plane crystal structure of Co_1/3TaS₂ demonstrates isolated CoS₆ octahedrons (purple-colored) and a long NN Co-Co distance. b The exchange interaction between Co local moments and conduction electrons from TaS₂ layers leads to an effective RKKY interaction between the local moments. c The Fermi surface of a 2D TLAF with 3/4 filling (shaded hexagons) and the Fermi surface of Co_1/3TaS₂ measured by ARPES. d The magnon spectra of Co_1/3TaS₂ at 5 K along the (00 L) direction. E_i = 7.9 and 14 meV data are plotted. e Antiferromagnetic NN interlayer coupling (J_c) of Co_1/3TaS₂, which is necessary for explaining the data in d and the refined spin configuration (Fig. 2g–i). f Const-E cuts of the INS data measured at 5 K (<T_N2). An energy integration range for each plot is $\pm$0.2 meV. The E = 1 and 1.5 meV (2.0~3.0 meV) plots are based on the E_i = 5 (7.9) meV data. In addition to bright circular spots centered at six M points (=linear modes), a weak, diffuse ring-like scattering which we interpret as the quadratic mode predicted by spin-wave theory, appears for E > 1.5 meV. g The calculated INS cross-section of the tetrahedral triple-Q ordering with J₁S² = 3.92 meV, J_cS² = 2.95 meV, J₂/J₁ = 0.19, and K_bq/J₁ = 0.02 (see Supplementary Materials). h The calculated INS cross-section of the single-Q ordering with three domains, using J₁S² = 3.92 meV, J_cS² = 2.95 meV, J₂/J₁ = 0.1, and K_bq/J₁ = 0. The line-shaped signal in h has a much higher intensity than in f or g. The simulations in g, h include resolution convolution (see Supplementary Fig. 9), and their momentum and energy integration range are the same as f. i Low-energy magnon spectra measured with E_i = 3.5 meV at 5 K, showing the energy gap of the linear magnon mode.

Tetrahedral ordering can naturally emerge when the Fermi surface (FS) is three-quarters (3/4) filled^12,13,18,31 because the shape of the FS is a regular hexagon (for a tight-binding model with nearest-neighbor hopping), whose vertices touch the M-points of the first Brillouin zone (Fig. 3c). In this case, there are three nesting wave vectors connecting the edges of the regular hexagon and the van Hove singularities at different M points, leading to magnetic susceptibility of the conduction electrons (${{{{{\rm{\chi }}}}}}({{{{{\bf{q}}}}}},\, {E}_{{{{{{\rm{F}}}}}}})$) that diverges as ${\log }^{2}\left|{{{{{\bf{q}}}}}}{{{{{\boldsymbol{-}}}}}}{{{{{{\bf{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{\nu }\right|$ at ${{{{{\bf{q}}}}}}\in$ M-points³⁰. This naturally results in a magnetic state with ordering wave vectors corresponding to the three M points (i.e., triple-Q). While perfect nesting conditions are not expected to hold for real materials, ${{{{{\rm{\chi }}}}}}({{{{{\bf{q}}}}}},\, {E}_{{{{{{\rm{F}}}}}}})$ is still expected to have a global maximum at the three M points for Fermi surfaces with the above-mentioned hexagonal shape¹⁸. As shown on the right side of Fig. 3c, our ARPES measurements reveal that this is indeed the case of the FS of Co_1/3TaS₂, indicating that the filling fraction is close to 3/4 and that the effective interaction between the Co²⁺ magnetic ions is mediated by the conduction electrons.

The above-described stabilization mechanism based on the shape of the Fermi surface suggests that the effective exchange interaction between the Co²⁺ magnetic ions and the conduction electrons is weak compared to the Fermi energy. Under these conditions, it is possible to derive an effective RKKY spin Hamiltonian by applying degenerate second-order perturbation theory in J/t^30,31. Since the RKKY model includes only bilinear spin-spin interactions, the single-Q (stripe) and triple-Q (tetrahedral) orderings remain degenerate in the classical limit. Moreover, by tuning the mutually orthogonal vector amplitudes ${{{{{{\boldsymbol{\Delta }}}}}}}_{\nu }$ while preserving the norm ${\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{1}\right|}^{2}{{{{{\boldsymbol{+}}}}}}{\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{2}\right|}^{2}{{{{{\boldsymbol{+}}}}}}{\left|{{{{{{\boldsymbol{\Delta }}}}}}}_{3}\right|}^{2}$, it is possible to continuously connect the single-Q (stripe) and triple-Q (tetrahedral) orderings via a continuous manifold of degenerate multi-Q orderings. The classical spin model’s accidental ground state degeneracy leads to a gapless magnon mode with quadratic dispersion and linear Goldstone modes. The accidental degeneracy is broken by effective four-spin exchange interactions that gap out the quadratic magnon mode, which naturally arises from Eq. (1) when effective interactions beyond the RKKY level are taken into consideration^14,17,30,31. The simplest example of a four-spin interaction favouring the triple-Q ordering is the bi-quadratic term ${K}_{{{{{{\rm{bq}}}}}}}{({{{{{{\bf{S}}}}}}}_{i}\cdot {{{{{{\bf{S}}}}}}}_{j})}^{2}$ with ${K}_{{{{{{\rm{bq}}}}}}} > 0$. This was explicitly demonstrated using classical Monte-Carlo simulations with a simplified phenomenological J₁-J₂-J_c-K_bq model (see Fig. 3a and 3e), as shown in Supplementary Fig. 8. This model successfully captures the tetrahedral ground state and manifests a two-step transition process at T_N2 and T_N1 with an intermediate single-Q ordering, which is consistent with our conclusion based on experimental observations. The latter outcome can be attributed to thermal fluctuations favouring the collinear spin ordering^32,33. In addition, previous DFT studies on TM_1/3NbS₂ (TM = Fe, Co, Ni), which is isostructural to Co_1/3TaS₂, also revealed that a non-coplanar state could be energetically more favorable than collinear and co-planar states³⁴.

Before analyzing the magnon spectra in detail, it is worth considering the interlayer network of Co_1/3TaS₂ with AB stacking. Along with the refined magnetic structure (Fig. 2g–i), the steep magnon dispersion shown in Fig. 3d indicates non-negligible antiferromagnetic NN interlayer exchange: ${J}_{{{{{{\rm{c}}}}}}}{S}^{2} \sim$2.95 meV (see Fig. 3e). However, the finite value of J_c does not change the competition between stripe and tetrahedral orderings analyzed in the 2D limit. More importantly, the antiferromagnetic exchange J_c forces the tetrahedral spin configuration of the B layer to be the same as that of the A layer (see Supplementary Fig. 8a). Therefore, all triangular layers have the same sign of the scalar chirality ${\chi }_{{ijk}}$ (or skyrmion charge), resulting in the realization of 3D ferro-chiral ordering. In this 3D structure, each magnetic unit cell of Co_1/3TaS₂ includes four skyrmions.

The low-energy magnon spectra (<3 meV) of Co_1/3TaS₂ measured by INS are presented in Fig. 3f. In addition to the linear (Goldstone) magnon modes, which appear as bright circular signals centered at the M points, a ring-like hexagonal signal connecting six M points was additionally observed with weaker intensity. This can be interpreted as the trace of the quadratic modes since they should be present together with linear modes at low energy (see Supplementary Fig. 9). Since the signal is only present for E > 1.5 meV, we infer that the quadratic mode is slightly gapped. We used linear spin-wave theory to compare the measured INS spectra with the theoretical spectra of both single-Q and triple-Q orderings. Despite the simplicity of the J₁-J₂-J_c-K_bq model, the calculated magnon spectra of the tetrahedral ordering (Fig. 3g) successfully describe the measured INS spectra. The magnon spectra of the stripe order are also presented in Fig. 3h for comparison. The intensity of the quadratic magnon mode is much stronger than that of the triple-Q spectra, in apparent disagreement with our INS data. A complete comparison between our data and the two calculations is shown in Supplementary Fig. 10. Additionally, the linear magnon modes of Co_1/3TaS₂ are also slightly gapped (~0.5 meV, see Fig. 3i). This feature can be explained for tetrahedral ordering only by considering both exchange anisotropy and higher order corrections in the 1/S expansion (see Supplementary Notes).

Finally, we discuss the behavior of the tetrahedral ordering in Co_1/3TaS₂ in response to out-of-plane and in-plane magnetic fields. Figure 4c illustrates the field dependence of the measured anomalous Hall conductivity (${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$) and M_z for H // c, showing evident hysteresis with a sign change at $\pm {H}_{{{{{{\rm{c}}}}}}1}$. Indeed, as already discussed in ref. ²⁶, M_z cannot characterize the observed ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$. Instead, as explained in the introduction, it is ${\chi }_{{ijk}}$ that characterizes ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ for the tetrahedral ordering. Therefore, based on our triple-Q scenario (see Fig. 4a, b), the sign change at $\pm {H}_{{{{{{\rm{c}}}}}}1}$ can be interpreted as the transition between tetrahedral orderings with positive and negative values of the scalar chirality ${\chi }_{{ijk}}$. In addition, the coexistence of weak ferromagnetic moment and large ${\sigma }_{{xy}}({{{{{\bf{H}}}}}}=0)$ is expected because, as we explained before, the real-space Berry curvature of the tetrahedral triple-Q ordering generates both the orbital ferromagnetic moment (of conduction electrons) and spontaneous Hall conductivity. However, the experimental techniques used in this study cannot discriminate between the spin and orbital contributions to M_z(H = 0). While it will be interesting to identify the nature of this weak ferromagnetic moment, this is left for future studies.

Fig. 4: The effect of the out-of-plane and in-plane magnetic field on the tetrahedral triple-Q ordering in Co_1/3TaS₂.

a, b A time-reversal pair of the tetrahedral spin configuration, having ${\chi }_{{ijk}}$ opposite to each other. The blue and orange arrows in a depict the generic canting of the tetrahedral ordering in Co_1/3TaS₂ by an out-of-plane magnetic field. c Comparison between the measured ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ (orange) and M_z (red) under the out-of-plane field at 3 K. d Comparison between the measured ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ (orange) at 3 K and ${\chi }_{{ijk}}$ (red) calculated from the M_z data in c. e Intensities of the three magnetic Bragg peaks in Fig. 2c under an external magnetic field along the a* direction. Error bars in e represent the standard deviation of the measured intensity.

We also compared the field dependence of the measured ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ and ${\chi }_{{ijk}}$ calculated from our M_z(H) data based on the canting expected in the tetrahedral order of Co_1/3TaS₂ (blue and orange arrows in Fig. 4). As shown in Fig. 4d, ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ decreases slightly in response to both positive and negative magnetic fields, consistent with the calculated ${\chi }_{{ijk}}$ of the tetrahedral ordering with mild canting. However, the model cannot capture the sudden decrease of ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ due to a meta-magnetic transition at $\pm {H}_{{{{{{\rm{c}}}}}}2}$, indicating that this transition changes the tetrahedral spin configuration. Additional neutron diffraction is required to identify the new ordering for |H| > H_c2.

The effect of an in-plane magnetic field was investigated using single-crystal neutron diffraction. Figure 4e shows field-dependent (H // ${{{{{{\bf{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{1}$ = (1/2, 0, 0)) intensities of the three magnetic Bragg peaks, each originating from three different ${{{{{{\bf{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{\nu }$ of the tetrahedral ordering. Interestingly, the equal intensity of the three peaks remains almost unchanged by a magnetic field up to 10 T, indicating robustness of the tetrahedral ordering against an in-plane magnetic field. This should be contrasted with triple-Q states found in other materials, which are induced by a finite magnetic field and occupy narrow regions of the phase diagram due to the ferromagnetic nature of the dominant exchange interaction^27,28,29.

In summary, we have reported a tetrahedral triple-Q ordering in Co_1/3TaS₂, as the only magnetic ground state compatible with our bulk properties (non-zero ${\sigma }_{{{{{{\rm{xy}}}}}}}({{{{{\bf{H}}}}}}=0)$ and M_z(H = 0)) and neutron diffraction data (q_m = (1/2, 0, 0) or its symmetry-equivalent pairs). Moreover, we provide a complete theoretical picture of how this exotic phase can be stabilized in the triangular metallic magnet Co_1/3TaS₂, which is further corroborated by our measurements of electronic structure and long-wavelength magnetic excitations using ARPES and INS. We further investigated the effect of external magnetic fields on this triple-Q ordering, which demonstrates its resilience against the fields. Our study opens avenues for exploring chiral magnetic orderings with the potential for spontaneous integer quantum Hall effect³⁰.

Note added: After the submission of this work, we came across another neutron diffraction work on Co_1/3TaS₂ published recently³⁵. While they have drawn the same conclusion of magnetic structure as our tetrahedral triple-Q magnetic ordering, our study further delves into the microscopic aspects of this magnetic ground state, specifically in relation to electronic structure and the properties of a geometrically frustrated triangular lattice. We also substantiate our findings through the ARPES and INS measurements.

Methods

Sample preparation and structure characterization

Polycrystalline Co_1/3TaS₂ was synthesized by the solid-state reaction. A well-ground mixture of Co (Alfa Aesar, >99.99%), Ta (Sigma Aldrich, >99.99%), and S (Sigma Aldrich, >99.999%) was sealed in an evacuated quartz ampoule and then sintered at 900 °C for 10 days. A molar ratio of the three raw materials was $\alpha$:3:6 with 1.05 <$\,\alpha$ < 1.1, which is necessary to obtain the resultant stoichiometric ratio of Co close to 1/3. Single-crystal Co_1/3TaS₂ was grown by the chemical vapor transport method with an I₂ transport agent (4.5 mg I₂/cm³). The pre-reacted polycrystalline precursor and I₂ were placed in an evacuated quartz tube and then were heated in a two-zone furnace with a temperature gradient from 940 to 860 °C for 10~14 days.

We measured the powder X-ray diffraction (XRD) pattern of Co_1/3TaS₂ using a high-resolution (Smartlab, Rigaku Japan) diffractometer, which confirmed the desired crystal structure without any noticeable disorder²⁶. In particular, the intercalation profile of Co atoms was carefully checked by the (10 L) superlattice peak pattern in the low-$2{{{{{\rm{\theta }}}}}}$ region; see ref. ²⁶. The powder neutron diffraction experiment further corroborated this result (see the relevant subsection below). Finally, single crystals were examined by XRD (XtaLAB PRO, Cu K_α, Rigaku Japan) and Raman spectroscopy (XperRam Compact, Nanobase Korea), confirming the high quality of our crystals, e.g., the sharp Raman peak at 137 cm⁻¹ (ref. ²⁶).

The composition x of Co_xTaS₂ single-crystal was confirmed primarily by energy-dispersive X-ray (EDX) spectroscopy (Quantax 100, Bruker USA & EM-30, Coxem Korea). We measured 12 square areas of 300${{{{{\rm{\mu }}}}}}{{{{{\rm{m}}}}}}\times$ 300${{{{{\rm{\mu }}}}}}{{{{{\rm{m}}}}}}$ wide for every single piece, yielding homogeneous x centered at ~0.320 with a standard deviation of ~0.004. The homogeneity of x was further verified by the spatial profile (~1.5 ${{{{{\rm{\mu }}}}}}{{{{{\rm{m}}}}}}$ resolution) of the EDX spectra, which is very uniform, as shown in Supplementary Fig. 1. The obtained x from EDX was again cross-checked by the composition measured by inductively coupled plasma (ICP) spectroscopy (OPTIMA 8300, Perkin-Elmer USA), which is almost the same within a measurement error bar²⁶.

Bulk property measurements

We measured the magnetic properties of Co_1/3TaS₂ using MPMS-XL5 and PPMS-14 with the VSM option (Quantum Design USA). To measure the spontaneous magnetic moment (Fig. 3d), a sample was field-cooled under 5 T and then measured without a magnetic field. Transport properties of Co_1/3TaS₂ were measured by using four systems: our home-built set-up, PPMS-9 (Quantum Design, USA), PPMS-14 (Quantum Design, USA), and CFMS-9T (Cryogenic Ltd, UK). To observe the temperature dependence of the anomalous Hall effect (Fig. 3d), we field-cooled the sample under ±9 T and then measured the Hall voltage without a magnetic field. The measured Hall voltage was anti-symmetrized to remove any longitudinal components. Hall conductivity (${\sigma }_{{xy}}$) was derived using the following formula:

$${\sigma }_{{xy}}=\frac{{\rho }_{{xy}}}{{{\rho }_{{xx}}}^{2}+{{\rho }_{{xy}}}^{2}}.$$

(2)

Anomalous Hall conductivity ${\sigma }_{{xy}}^{{{{{{\rm{AHE}}}}}}}$ was derived by first subtracting a normal Hall effect from measured ${\rho }_{{xy}}$ and then using Eq. 2.

Powder neutron diffraction

We carried out powder neutron diffraction experiments of Co_1/3TaS₂ using the ECHIDNA high-resolution powder diffractometer ($\lambda$ = 2.4395 Å) at ANSTO, Australia. To acquire clear magnetic signals of Co_1/3TaS₂, which are weak due to the small content of Co and its small ordered moment, we used 20 g of powder Co_1/3TaS₂. The quality of the sample was checked before the diffraction experiment by measuring its magnetic susceptibility and high-resolution powder XRD. The neutron beam at ECHIDNA contains weak λ/2 harmonics (~0.3%), yielding additional nuclear Bragg peaks (Supplementary Fig. 3). We also performed Rietveld refinement and magnetic symmetry analysis using Fullprof software³⁶. The results are summarized in Supplementary Tables 1–3 and Supplementary Figs. 2–4.

Single-crystal neutron diffraction

We carried out single-crystal neutron diffraction under a magnetic field using ZEBRA thermal neutron diffractometer at the Swiss spallation neutron source SINQ. We used one single-crystal piece (~16 mg) for the experiment, which was aligned in a 10 T vertical magnet (Oxford instruments) with (HHL) horizontal (Supplementary Fig. 5a). The beam of thermal neutrons was monochromatic using the (220) reflection of germanium crystals, yielding a neutron wavelength of $\lambda$ = 1.383 Å with >1% of λ/2 contamination. Bragg intensities as a function of temperature and magnetic field were measured using a single ³He-tube detector in front of which slits were appropriately adjusted.

Angle-resolved photoemission spectroscopy (ARPES) measurements

The ARPES measurements were performed at the 4A1 beamline of the Pohang Light Source with a Scienta R4000 spectrometer³⁷. Single crystalline samples with a large AHE were introduced into an ultra-high vacuum chamber and were cleaved in situ by a top post method at the sample temperature of ~20 K under the chamber pressure of $\sim 5.0\times {10}^{-11}$ Torr. A liquid helium cryostat maintained the low sample temperature. The photon energy was set to 90 eV to obtain a high photoelectron intensity for the states of Co 3d characters. The total energy resolution was ~30 meV, and the momentum resolution was ~0.025 ${\mathring{\rm A} }^{-1}$ in the measurements.

Single-crystal inelastic neutron scattering

We performed single-crystal inelastic neutron scattering of Co_1/3TaS₂ using the 4SEASONS time-of-flight spectrometer at J-PARC, Japan³⁸. For the experiment, we used nearly 60 pieces of the single-crystal with a total mass of 2.2 g, which were co-aligned on the Al sample holder with an overall mosaicity of ~1.5° (Supplementary Fig. 5b). We mounted the sample with the geometry of the (H0L) plane horizontal and performed sample rotation during the measurement. The data were collected with multiple incident neutron energies (3.5, 5.0, 7.9, 14.0, and 31.6 meV) and the Fermi chopper frequency of 150 Hz using the repetition-rate-multiplication technique³⁹. We used the Utsusemi⁴⁰ and Horace⁴¹ software for the data analysis. Based on the crystalline symmetry of Co_1/3TaS₂, the data were symmetrized into the irreducible Brillouin zone to enhance statistics.

Classical Monte-Carlo simulations

To investigate the magnetic phase diagram of our spin model, we performed a classical Monte-Carlo (MC) simulation combined with simulated annealing. We used Langevin dynamics⁴² for the sampling method of a spin system. To search for a zero-temperature ground state, a large spin system consisting of 30 × 30 × 4 unit cells (7200 spins) with periodic boundary conditions was slowly cooled down from 150 K to 0.004 K. The thermal equilibrium was reached at each temperature by evolving the system through 5000 Langevin time steps, with the length of each time-step defined as dt = 0.02/(J₁S²). Finally, we adopted the final spin configuration at 0.004 K as the magnetic ground state, which was further confirmed by checking whether the same result was reproduced in the second trial.

For finite-temperature phase diagrams, we used a 30 × 30 × 6 supercell (10,800 spins). After waiting for 600~2000 Langevin time steps for equilibration, 600,000~2,000,000 Langevin time steps were used for the sampling. Since thermalization and decorrelation time strongly depend on the temperature, the length of equilibration and sampling time steps were set differently for different temperatures. From this result, we calculated heat capacity (C_V), staggered magnetization (M_stagg), and total scalar spin chirality $({\chi }_{{ijk}})$ using the following equations:

$${C}_{V}=\frac{\left\langle {\epsilon }^{2}\right\rangle -{\left\langle \epsilon \right\rangle }^{2}}{{k}_{B}{T}^{2}},$$

(3)

$${M}_{{{{{{\rm{stagg}}}}}}}=\frac{g{\mu }_{B}}{N}\mathop{\sum}\limits_{i}{\left(-1\right)}^{2\pi ({{{{{{\boldsymbol{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{\nu }{{\cdot }}{{{{{{\boldsymbol{r}}}}}}}_{{{{{{\boldsymbol{i}}}}}}})}\left\langle {{{{{{\bf{S}}}}}}}_{i}\right\rangle,$$

(4)

$${\chi }_{{ijk}}=\frac{1}{{S}^{3}}\frac{\mathop{\sum}\limits_{\varDelta }\left\langle {{{{{{\bf{S}}}}}}}_{\varDelta 1}{{{{{\boldsymbol{\bullet }}}}}}{{{{{\boldsymbol{(}}}}}}{{{{{{\bf{S}}}}}}}_{\varDelta 2}{{{{{\boldsymbol{\times }}}}}}{{{{{{\bf{S}}}}}}}_{\varDelta 3}{{{{{\boldsymbol{)}}}}}}\right\rangle }{{N}_{t}},$$

(5)

where $\left\langle A\right\rangle$ is an ensemble average of A estimated by the sampling, $\epsilon$ is the total energy per Co²⁺ ion, g is the Lande g-factor, ${{{{{{\bf{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{\nu }$ ($\nu=1,2,3$) is the ordering wave vector of two-sublattice stripe order (see Supplementary Note), $\Delta$ is the index for a single triangular plaquette on a Co triangular lattice consisting of three sites (${\Delta }_{1},$ ${\Delta }_{2}$, ${\Delta }_{3}$), and N (N_t) is the total number of Co²⁺ ions (triangular plaquettes). When calculating M_stagg, one should first identify which ordering wave vector the spin system chose among the three ${{{{{{\bf{q}}}}}}}_{{{\rm{m}}}}^{\nu }$ ($\nu=1,2,3$), and then use proper ${{{{{{\bf{q}}}}}}}_{{{{{{\rm{m}}}}}}}^{\nu }$ to calculate ${{{{{{\rm{M}}}}}}}_{{{{{{\rm{stagg}}}}}}}$. For all the simulations, we used S = 3/2. The results of the simulations are shown in Supplementary Fig. 8.

Spin-wave calculations

We calculated the magnon dispersion and INS cross-section of Co_1/3TaS₂ using linear spin-wave theory. For this calculation, we used the SpinW library⁴³. Since we did not align positive and negative ${\chi }_{{ijk}}$ domains of the tetrahedral ordering in our INS experiments, we averaged the INS cross-section of both cases.

DFT calculations

We performed first-principles calculations using ‘Vienna ab initio simulation package (VASP)’^44,45,46 based on projector augmented wave (PAW) potential⁴⁷ and within Perdew-Burke-Ernzerhof (PBE) type of GGA functional⁴⁸ (see Supplementary Fig. 11). DFT + U method^49,50 was adopted to take into account localized Co-3d orbitals properly, where U = 4.1 eV and J_Hund = 0.8 eV were used as obtained by the constrained RPA for CoO and Co⁵¹. For the 2 × 2 magnetic unit cell, we used the Γ-centered 6 × 6 × 6 k-grid. We obtained and used the optimized crystal structure with the force criterion for the relaxation fixed at 1 meV/Å. The plane-wave energy cutoff was set to 500 eV.