Non-coplanar helimagnetism in the layered van-der-Waals metal DyTe3

Van-der-Waals magnetic materials can be exfoliated to realize ultrathin sheets or interfaces with highly controllable optical or spintronics responses. In majority, these are collinear ferro-, ferri-, or antiferromagnets, with a particular scarcity of lattice-incommensurate helimagnets of defined left- or right-handed rotation sense, or helicity. Here, we report polarized neutron scattering experiments on DyTe3, whose layered structure has highly metallic tellurium layers separated by double-slabs of dysprosium square nets. We reveal cycloidal (conical) magnetic textures, with coupled commensurate and incommensurate order parameters, and probe the evolution of this ground state in a magnetic field. The observations are well explained by a one-dimensional spin model, with an off-diagonal on-site term that is spatially modulated by DyTe3’s unconventional charge density wave (CDW) order. The CDW-driven term couples to antiferromagnetism, or to the net magnetization in an applied magnetic field, and creates a complex magnetic phase diagram indicative of competing interactions in this easily cleavable van-der-Waals helimagnet.


Main text
Magnetism in layered materials, held together by weak van-der-Waals interactions, is an active field of research spurred on by the discovery of magnetic ordering in monolayer sheets of ferromagnets and antiferromagnets [1][2][3][4].At the frontier of this field, helimagnetic layered systems, where magnetic order has a fixed, left-or right-handed rotation sense, have been predicted to host complex spin textures [5,6] and to serve as controllable multiferroics platforms, where magnetic order is readily tuned by electric fields or currents [7][8][9][10].
However, most layered van-der-Waals magnets are commensurate ferro-, antiferro-, or ferrimagnets [3,4]; the rare helimagnets provided to us by nature are often modulated along the stacking direction, with relatively simple spin arrangement in individual layers (Table E1).
In the quest for helimagnetism in layered structures with weak van-der-Waals bonds, we focus on rare earth tritellurides RTe 3 (R: rare earth element).These materials form a highly active arena of research regarding the interplay of correlations and topological electronic states [11][12][13][14][15]. Their structure, which can be exfoliated down to the thickness of a few monolayers [16,17], is composed of metallic tellurium Te 2 double-layers and covalently bonded, nearly insulating RTe slabs, with characteristic square net motifs in both (Fig. 1 a) [18].Tellurium 5p electrons are are highly localized in Te 2 square net bilayers (ac plane), in which they form highly dispersive bands with elevated Fermi velocity [16,[19][20][21].This quasi two-dimensional electronic structure amplifies correlation phenomena, such as the formation of charge-density wave (CDW) order [22] and superconductivity [23,24], while also hosting protected band degeneracies [16,25].In view of intense research efforts on RTe 3 , it is remarkable that their magnetism has never been discussed in detail; in particular, no full refinement of magnetic structures is available [26][27][28][29][30][31].
Here, we report on helimagnetic, cone-type orders of DyTe 3 using polarized elastic neutron scattering.We reveal the magnetic texture in real space, its evolution with temperature, and its relationship to charge-density wave formation.In DyTe 3 , dysprosium moments are arranged in square net bilayers, where each ion's nearest neighbour (NN) is located in the respective other layer (Fig. 1 b).As all zero-field magnetic orders of DyTe 3 are uniform along the crystallographic a-axis, we approximate each square net bilayer as an effective zigzag chain of magnetic rare earth ions.On such chains, our experiment shows that pairs of NN ions have cones pointing along the same direction, followed by a flip of the cone axis (Fig. 1 c, which illustrates half a magnetic unit cell).The coupling between two DyTe bilayers, i.e. between two zigzag chains, is antiferromagnetic.Despite this complex cone arrangement, the magnetic structure defines a fixed sense of rotation, or helicity.Our Monte Carlo simulations show that charge-density wave order in rare earth tellurides modulates the magnetic exchange interactions and drives lattice-incommensurate magnetism.We further discuss magnetocrystalline anisotropy in this layered structure, with an unconventional combination of metallic and covalent bonds.Helimagnetism of Dy rare earth moments with 4f 9 magnetic shell emerges despite the naive expectation of strong preference for easy-axis or easy-plane anisotropy for 2S+1 L J = 6 H 15/2 , with large orbital angular momentum L = 5.

Magnetic properties of DyTe 3
Some essential magnetic properties of DyTe 3 are apparent already from the magnetic susceptibility χ in Fig. 1 d.In the high temperature regime, anisotropy in the Curie-Weiss law indicates easy-plane behaviour of magnetic moments, favouring the ac plane with uniaxial anisotropy constant K 1 = 22(1) kJ m −3 (Methods).At low temperatures, the strongest enhancement of χ occurs when the magnetic field H is along the c-axis, i.e. parallel to the zigzag direction defined in Fig. 1 c.We may deduce that the magnetic moments are aligned, predominantly, along the a and b axes.All susceptibility curves show maxima around T χ = 4.5 K, quite far above the onset of three-dimensional, long-range magnetic order, as shown in the following.
We characterize the phase transition in DyTe 3 using thermodynamic and transport probes in Fig. 2 a-c.The specific heat C(T ) shows a two-peak anomaly, describing the transitions from the paramagnetic (PM) regime to phase II at T N2 = 3.85 K and to phase I at T N1 = 3.6 K. Below T N1 , the resistivities in the ac basal plane drop abruptly, suggesting a clear correlation between the behaviour of freely moving conduction electrons and the magnetic structure.The presence of a partial charge gap in the electronic structure, related to magnetic ordering, is inferred from an increase of the ratio of resistivities ρ a and ρ c .Simulataneously, as discussed in the following, strong neutron scattering intensity appears below T N2 at two independent positions in reciprocal space, c.f. Fig. 2 c.The magnetic scattering intensity rises abruptly upon cooling below T N2 .
To obtain this neutron data, a single-domain crystal of DyTe 3 is mounted on an aluminium holder and is pre-aligned by means of Laue x-ray diffraction.More quantitatively, we determine the crystallographic directions in DyTe 3 using the crystallographic extinction rule (Methods).Figure 2 d describes the geometry of our neutron scattering experiment.
The scattering plane that includes the incoming and outgoing neutron beams k i and k f , is spanned by the b-and c-axes.Hence, reflections of the type Q = k f − k i = (0KL) can be detected, as in Fig. 2 e, where a line scan along (01L) provides sharp magnetic intensity.Three types of magnetic peaks Q = G + q, with G a reciprocal lattice vector, are observed: An incommensurate reflection at q cyc = (01L cyc ) with L cyc = 0.21, a commensurate (C) reflection q AFM = (01L AFM ) with L AFM = 0.5, and a higher harmonic (3Q) reflection, corresponding to thrice the length of q cyc ; this describes an anharmonic distortion of the texture (Methods).We now focus on the ground state (phase I), before addressing the evolution or splitting of magnetic reflections in the high-temperature phase II.

Ground state magnetic structure model
We reveal the helimagnetic structure in the ground state of DyTe 3 using polarized neutron scattering.As shown in Fig. 2 d, the incident neutron spins were polarized perpendicular to the scattering plane.We employ a magnetized single-crystal analyzer to select the energy and spin state of the scattered neutrons (Methods).The scattering processes in which the neutron spins are reversed (remain unchanged) is referred to as spin-flip, SF (non-spinflip, NSF).For SF scattering, it is required that magnetic moments m have a component perpendicular to the spin of the incoming neutron.This means that SF and NSF scattering detect components of m within (m b , m c ) and perpendicular to (m a ) the scattering plane, respectively.
Polarization analysis of the magnetic reflections shows that q cyc and q AFM relate to different vector components of the ordered magnetic moment (Figs. 3 a,b and e,f ).We find no hint of SF scattering at q AFM , demonstrating collinear antiferromagnetism with magnetic moments exclusively along the a-direction.The incommensurate part q cyc , in contrast, has no NSF intensity and roughly equal SF signals at various positions in reciprocal space (Fig. 3 e,f and insets).As neutron scattering detects the part of m that is orthogonal to Q, comparison of magnetic reflections situated at nearly orthogonal directions in momentum space suggests m b and m c components of roughly equal length.
We determine the quantitative relationship between magnetic moments within a DyTe bilayer (within an effective zigzag chain), by comparing the observed and calculated mag-netic structure factors under the constraints imposed by polarized neutron scattering, c.f. Fig. E7.The analysis for q AFM demonstrates up-up-down-down type ordering along the zigzag chain, visualized from two perspectives in Fig. 3 c,d.At q cyc , the refinement yields a cycloid with a phase delay δ between the upper and lower sheets in a zigzag chain, see Fig. 3 g.In effect, pairs of nearly parallel magnetic moments are followed by a significant rotation of the moment direction.The coupling between zigzag chains is antiferromagnetic, as imposed by the K = 1 component in both q cyc and q AFM .Superimposing the three components m a , m b , and m c , we realize the noncoplanar, helimagnetic cone texture of Fig. 1 c that is, to our knowledge, unique in both insulators and metals.In Extended Data, we discuss the presence of magnetic domains in the sample and how the occurrence of higher harmonic reflections further supports our magnetic structure model.

Phase diagram and thermal fluctuations
We are ready, now, to consider the evolution of magnetic order in DyTe 3 upon heating.Fig- ure 4 a shows a contour map of the magnetic susceptibility χ (Methods), where the external magnetic field is applied along the in-plane direction [101], i.e.H ∥ (a + c).We focus on the zero-field regime: starting from a ground state of reduced magnetic susceptibility (phase I), heating of the sample leads to a loss of ordered moment m c along the c-axis, and fan-type order with antiferromagnetic layer stacking (Fig. 4 b, phase II).This structure is derived from neutron diffraction in Figs.E6 and E7.We stress that, although Fig. 4 b illustrates a single zigzag chain, the coupling between chains remains antiferromagnetic in phase II.The loss of one component of m -m c in this case -is quite common in materials with competing interactions [32].Only the antiferromagnetic component m a survives further heating into the paramagnetic regime, where we observe a weak, diffuse neutron signal along the (0, K, 1/2) line, with rapid decay of the coherence length just above T N2 .

Theoretical modeling
We now argue that cone-type magnetism in DyTe 3 is realized through (i) a spatial modulation of near-neighbour exchange interactions J 1 , J 2 in presence of charge-density wave (CDW) order and (ii) unconventional single-ion anisotropy.We turn first to (i), that is the role of the CDW in stabilizing noncoplanar helimagnetism in DyTe 3 .We use an unbiased numerical technique and a 1D chain model to reproduce key features of the modu-lated magnetic order, neglecting the material's three-dimensionality (Methods).In DyTe 3 , atomic distances in a DyTe square net bilayer (in a zigzag chain) are spatially modulated by the CDW in the adjacent Te 2 sheets [11-14, 18, 22-24, 31, 33, 34].Assuming harmonically oscillating nearest-and next-nearest neighbour interactions J 1 = J 1,CDW sin(q CDW • x), J 2 = J (0) 2 + J 2,CDW sin(q CDW • x) with wavenumber q CDW = 0.29, the 1D model robustly reproduces two types of magnetic reflections q AFM = 0.5 and q cyc = 0.5 − 0.29 = 0.31.In good consistency with experiment Fig. 4 d shows the intensity I cyc of the latter reflection normalized by the former, with a dashed line indicating the threshold to 5 % intensity ratio.
Next, consider the local environment of a single dysprosium ion in Fig. 4 e.Te-B ions form covalent bonds with the central Dy, while the point charges of Te-A are effectively screened by itinerant electrons in the conducting tellurium slab.We model the sequence of crystal electric field (CEF) states for the 4f 9 shell of dysprosium as a function of the effective crystal field charge q situated on Te-A and Te-B ions (Fig. E10).Fig. 4 f illustrates two limiting cases: When Te-A and Te-B contribute equally to the CEF, the 4f 9 charge cloud is compressed along the b-direction, with |J b = ±15/2⟩ dominating the ground state wavefunction, and with effective out-of-plane magnetic anisotropy for magnetic moments.
Even for intermediate charge 0 < q(Te−A)/q(Te−B) < 1, Dy's site symmetry m2m still demands that CEF eigenstates have finite magnetic moment either along the c-axis or in the basal plane, but not both.We require exchange interactions E ex between magnetic moments to mix the CEF eigenstates and to allow for tilting of magnetic moments.Taking this to heart, the CEF Hamiltonian of a point charge model is diagonalized for the orthorhombic environment of Dy; the exchange interaction E ex is included as an effective magnetic field (Methods).As a result, we obtain an anisotropic free energy density described by two parameters K 1 cos 2 (θ) + K 2 sin 2 (θ) cos 2 (ϕ), where θ, ϕ are spherical coordinates with respect to the b and c crystal axes, respectively.Fig. 4 g,h testify to a transition from easy-axis to easy-plane anisotropy through a sign change of K 1 at intermediate charge ratio (pink line); two green lines bound the regime where easy-axis (easy-plane) anisotropy is not strong enough to prevent tilting of m along directions intermediate between b-axis and the ac plane.
As noted above, the experimental data suggests weak in-plane anisotropy, or K 1 > 0.
Further constraining E ex in agreement with T N2 , we identify the black box in Fig. 4 g,h to capture a parameter range well consistent with experiment.Here, the model yields K 2 > 0, meaning m a is preferred over m c , a suitable condition for the formation of conical magnetic order.

Discussion
As compared to transition metal dichalcogenides, where the magnetic ion is buried inside a rather symmetric block layer [35,36], RTe 3 harbors more complex structural features, with magnetic ions at the boundary between metallic and insulating blocks.This mixed covalent / metallic environment for the magnetic ion is key to realizing the present scenario: it facilitates coupling between magnetic ions and a charge density wave (CDW) on the metallic tellurium square net, and -at the same time -generates unconventional magnetocrystalline anisotropy.In fact, the present charge-transfer phenomenology is partially inspired by work on thin films of magnetic metals on insulating substrates [37], on electric field control of magnetocrystalline anisotropy [38], and on the behaviour of magnetic materials when charge transfer is induced by oxidation at the surface [39].
As compared to complex cone-type orders in insulating multiferroics, the combination of antiferromagnetic and cycloidal components in DyTe 3 is unique.For example, Mn 2 GeO 4 has cones arrayed on one-dimensional chains, with uniform cone direction along the chain [40].
In another metallic system, EuIn 2 As 2 , jump in the rotation sense of a helimagnetic texture have recently been identified, with a short magnetic period [41].In contrast, the rotation of moments in DyTe 3 proceeds in nearly parallel pairs, without abrupt jumps in the cycloidal component of the texture.We expect helimagnetic orders of the type observed here to be common in layered materials.and especially in rare earth tellurides and selenides.Here, rich magnetic phase diagrams from coupling between CDW and magnetic order have been generally observed [42,43].For one, DyTe 3 represents a simple example, where a combination of nearest and next-nearest neighbour exchange realize helimagnetism, when fertilized by a CDW.On the other hand, this complex magnetic order, its relationship to a straincontrollable CDW [34], and its (likely) rich excitation spectrum certainly warrant further research.
Indeed, the lowest-energy, Goldstone mode of a typical helimagnet corresponds to a spatial shift of the magnetic texture, termed phason excitation [44].In DyTe 3 , the magnetic and CDW phasons [45] are expected to be closely intertwined, as evident from the robust L cyc (T ) in Fig. E8.Such locking between low-energy modes may have implications for dynamic responses, further enriching the spectrum of elementary excitations in RTe 3 that has drawn intense scrutiny recently [15].

Concluding remarks
An important open question is the stability of helimagnetism in few-layer devices of DyTe 3 , where the cleavage plane, as well as the center of structural inversion, are situated between tellurium bilayers.As a fundamental building block of the structure, we consider a DyTe slab sandwiched by Te square nets -that is half a unit cell in Fig. 1 a.Being screened from top and bottom by metallic tellurium layers, we expect no qualitative change of the local crystal field environment of Dy in the few-layer limit.However, the absence of an inversion center for odd numbers of layers, and its presence for even numbers of layers, may have a profound effect on magnetic ordering and the presence or absence of (right-or left-handed) helicity domains in the sample, considering the presence or absence of Dzyaloshinskii-Moriya interactions [46].Most appealingly, DyTe 3 is a potential platform for spin-Moiré engineering in solids, where complex magnetic textures can be designed by combining and twisting two or more helimagnetic sheets.Here, a plethora of noncoplanar spin textures can be engineered at will [6,47], while highly conducting tellurium square net channels may serve as a test bed for of emergent electromagnetism in a tightly controlled setting [48,49]   agreement with previous work [18].We found it challenging to obtain high-quality powder x-ray data from crushed single crystals, which tend to include traces of Te flux on their surface and form thin flakes, even when thoroughly ground in a mortar.We also verified the stoichiometric chemical composition of our crystals by energy-dispersive x-ray spectroscopy (EDX).Cleaved single crystals have a reddish-brown surface; but even in vacuum, the colour of the surface changes to silver-metallic, and then to black, after two weeks or so.A red hue can be recovered by renewed surface cleaving.

Magnetization measurements and crystal alignment
We use a commercial magnetometer with T = 2 K base temperature and a maximum magnetic field of 7 T (MPMS, Quantum Design, USA).The measurement is carried out using a rectangular-shaped single crystal of mass m = 1.22 mg, with carefully aligned edges along the a and c crystal axes.By means of a single crystal diffractometer (Malvern Panalytical Empyrean, Netherlands), we confirm the extinction rule h+k = even in space group Cmcm.
It is difficult to distinguish a and c axes in this orthorhombic, yet nearly tetragonal structure by eye or with the help of the Laue diffractogram.Temperature dependent susceptibility χ(T ) is measured in a DC magnetometer with 1000 Oe applied field; there is no observable difference between field-cooled and zero-field cooled magnetization traces.A demagnetiza-tion correction is carried out according to the standard expression H int = H ext −N M, where H ext , M, and N are the externally applied magnetic field, the bulk magnetization, and the dimensionless demagnetization factor.The latter is calculated by approximating the crystal as an oblate ellipsoid [50].For the H − T phase diagram in Fig. 4, the [101] direction is aligned within ±3 • and bulk magnetization is measured in discrete field steps, for selected temperatures.Fig. 4 shows data for decreasing magnetic field ∂H/∂t < 0. Note that hysteresis occurs at all phase transitions shown in Fig. 4 a, indicating their first-order nature.The magnetic anisotropy energy is expressed as where θ is the angle between M and the b-axis.Utilizing the free energy expression ) and the Curie-Weiss law χ α = C/(T − Θ α CW ), where Θ α CW is the Curie-Weiss temperature along the α ∈ (a, b, c) direction and C = 2.077(1) K is the Curie constant of DyTe 3 , we obtain (1) K is the difference between the Curie-Weiss temperatures in the ac-plane and along the b-axis (c.f.Fig. 1 d, inset).Specific heat was recorded using a relaxation technique in a Quantum Design PPMS cryostat, in zero magnetic field.
For specific heat anomalies in applied magnetic field, we employed the AC calorimetry technique in a custom-built setup.Anisotropy of the resistivity, as in Fig. 2 At 2.3 K and in unpolarized condition, we also collected integrated intensities I from ω scans of 6 lattice-commensurate and 11 incommensurate, independent magnetic Bragg reflections.These I are converted to magnetic structure factors by correcting for scale factor and Lorentz factor.On the basis of polarized scattering in Fig. 3, we assume that commensurate and incommensurate magnetic modulations correspond to a collinear antiferromagnetic magnetic modulation with magnetic moments parallel to the a-axis, and a cycloidal magnetic modulation with magnetic moments in the bc-plane, respectively.We model the commensurate component as shown in Fig. 3 c, consistent with the observation that magnetic peaks appear when K and L are half and odd integers, respectively.This magnetic structure results in two magnetic domains owing to the symmetry of the crystal structure, as shown in Fig. E7 a.We assume equal volume fractions for these two magnetic domains and calculate magnetic structure factors [51] with the analytic approximation of the magnetic form factor of Dy 3+ [52].By comparing calculated and observed magnetic structure factors, the commensurate moment magnitude -assumed to be uniform -is estimated to be µ AFM = 6.49± 0.36µ B .
As for the incommensurate component, we determine the moment amplitude and the phase differences between the four Dy ions (j = 1 − 4) in the chemical unit cell, shown in Fig. 3 g.The phase of the magnetic modulation at the j'th atom is described as q cyc • (R k + r j ) + ϕ j , where R l is a vector pointing to the origin of chemical unit cell number l, r j is the fractional coordinate of the j'th Dy atom in the unit cell, and ϕ j accounts for the phase shift -depending on the atomic site.Magnetic reflections are observed only when K is an odd integer, so that two atoms with the same z coordinate are antiferromagnetically coupled: ϕ 1 -ϕ 3 = ϕ 2 -ϕ 4 = π.Therefore, there remains one adjustable parameter ±δ = ϕ 1 -ϕ 2 = ϕ 3 -ϕ 4 , corresponding to two domains with (assumed) equal volume fraction and coupled to the commensurate modulation; see Figs.E1, E2.
The existence of third harmonic reflections indicates a slightly distorted cycloidal magnetic modulation.The present analysis assumes merely weak difference in size between m b and m c components, so that all Dy sites have the same magnetization amplitudes.A fit of data and model yields the amplitude of the proper-cycloidal magnetic modulation µ cyc = 6.18 ± 0.14µ B and the phase shift δ ≈ 2π/5.The calculation of the scattering intensity in Fig. 2 f, which includes the third harmonic reflection, takes into account instrumental resolution broadening (Fig. E3), anharmonicity of the cycloidal magnetic structure component, and the presence of two magnetic domains (Fig. E7).
Contrary to all other integrated intensities, the temperature dependence in Fig. 2 c was obtained from L-scans of magnetic scattering.

Crystal electric field calculations
We use the software package PyCrystalField [53] for the calculation of crystal electric field energies via the point charge model in the limit of strong spin-orbit interactions.The calculation is based on published fractional coordinates of Dy and Te ions within the crystallographic unit cell [18,22], with a ∼ 0.15 % tensile strain along the a-axis, lifting tetragonal symmetry and yielding finite K 2 .In Fig. E10, we vary the effective crystal electric field originating from Te-A (on the metallic Te 2 slab) by changing its point charge, while keeping the total charge in the environment of Dy unchanged.An unperturbed, diagonal Hamiltonian matrix is constructed from the energies in Fig. E10, and the operator of total angular momentum J = L + S is also expressed in the basis of these CEF eigenstates.Adding an effective exchange term E ex J α (α = a, b, c are vector components), the total Hamiltonian is diagonalized and the expectation value of J a , J b , J c is evaluated in the respective ground state.The anisotropy constants are approximated, as (2) so that K 1 < 0 for easy-axis anisotropy along the b-axis and K 2 > 0 if a-axis orientation is energetically preferred over the c-axis.Here, ⟨J α ⟩ α is shorthand for ⟨ψ 0,α |J α | ψ 0,α ⟩, where |ψ 0,α ⟩ is the ground state of the total Hamiltonian when an exchange field of magnitude E ex is applied along the α-direction.
The anisotropic part of the charge density is exaggerated 20× in Fig. 4 according to the expression 20 • (R − R 0 ) + R 0 , where R 0 corresponds to 10 Bohr radii.More details are given in Figs.E10,E11.

Monte Carlo calculations
Classical Monte Carlo simulations are performed for a minimal Heisenberg spin chain model with antiferromagnetic second-neighbour J 2 interactions based on the SpinMC code [54].
To simulate the effects of the charge density wave, sinusoidally modulated strengths of the first-and the second-neighbour exchange interactions are added to the minimal model as J 1,CDW • sin(z q CDW ) and J 2,CDW • sin(z q CDW ), respectively, where q CDW = 2π/λ CDW is the wave vector of the charge density wave, assumed to be q CDW = 0.3.For the calculated phase diagram in Fig. 4, a chain with 100 unit cells is constructed, where each cell contains two spins to mimic the Dy bilayer structure in DyTe 3 .The total magnetic structure factor α=x,y,z S α (q)S α (−q) is calculated through fast Fourier transform over 3 • 10 4 measurement sweeps after 10 4 thermalization sweeps (inverse temperature β/ |J 2 | = 20).c, Charge density wave (CDW) on tellurium layers [18], where orange (grey) spheres are distorted (undistorted) ionic positions.Coupling of Dy (violet) and CDW drives lattice-incommensurate magnetic order through spatially modulated exchange constants J i,CDW , i = 1, 2, normalized to the uniform second-neighbour exchange J g,h Anisotropy constants K 1 and K 2 calculated for 4f 9 multiplet in DyTe 3 (Methods).The ordinate describes the relative weight of CEF charges q on Te-A and Te-B sites.Green lines bound an intermediate regime of weak K 1 , where spin tilting and conical order are allowed.with metal-to-insulator transition (MIT), and spin textures are classified into coplanar (CP) and noncoplanar (NCP).DyTe 3 is the only metallic compound that is (a) incommensurate with the underlying lattice, and (b) has a component of the modulation vector q perpendicular to the stacking direction.Moreover, it is a rare example of noncoplanar (NCP) magnetism in a bulk vdW compound.

Extended Data and Figures
Compound Space group q-vector Transport Magnetism Ref.
[2] C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y. Wang, Z. Qiu, R. Cava, S. Louie, J. Xia, and X. Zhang, Discovery of intrinsic ferromagnetism in two-dimensional van der Waals crystals, Nature 546, 265-269 (2017).[3]K. Burch, D. Mandrus, and J.-G.Park, Magnetism in two-dimensional van der Waals mate-Methods Sample preparation and characterization Single crystals are grown from tellurium self-flux following the recipe in Ref. [43]: We set elemental Dy and Te at a ratio of 1 : 21.65 in an alumina crucible, which in turn is sealed in a quartz tube in high vacuum.The raw materials are heated to 450 • C for 36 hours and then to 780 • C in 96 hours, where the melt remained for 48 hours, followed by cooling to 450 • C at a rate of 1.375 • C/hour.The final product is centrifuged after renewed heating to 500 • Celsius, so that plate-shaped single crystals of typical dimensions 5 × 5 × 1 mm 3 are obtained.The face of each plate is perpendicular to the b-axis of DyTe 3 's orthorhombic unit cell, and facet edges tend to be parallel to either a or c.The existence of impurity phases above 1 % volume fraction is ruled out by single-crystal x-ray diffraction on cleaved surfaces in a Rigaku SmartLab X-ray powder diffractometer.The experiment yields lattice constants of a = 4.27(2) Å, b = 24.433(1)Å, and c = 4.27(2) Å at room temperature, in good , was recorded on exfoliated flakes of thickness ∼ 100 µm using the Montgomery technique.Electric contacts are made with Ag paste (Dupont) and deteriorate with time.To maintain excellent contact resistance ∼ 1 Ω, it is crucial to immediately cool the contacted crystal in vacuum, after depositing the silver paste.The sample and contact quality is robust at low temperatures for at least two weeks.Elastic neutron scattering We performed unpolarized and polarized neutron scattering experiments using the POlarized Neutron Triple-Axis spectrometer (PONTA) installed at the 5G beam hole of the Japan Research Reactor 3 (JRR-3).A single crystal of DyTe 3 is set in an aluminium cell and mounted in a 4 He closed cycle refrigerator with (0, K, L) horizontal scattering plane.For the unpolarized measurements, the spectrometer is operated in two-axis mode with horizontal beam collimation of open-80 ′ -80 ′ .The unpolarized incident neutron beam with energy of 14.7 meV is obtained by a pyrolytic graphite (002) monochromator.For polarized measurements, the spectrometer is operated in triple-axis mode with horizontal beam collimation of open-80 ′ -80 ′ -open.A Heusler (111) single crystal monochromator fixes the spin-polarization of incident neutrons, with energy 13.7 meV.The spin direction of incident neutrons is further controlled by a spin flipper, guide fields, and a Helmholtz coil, with the flipper placed between monochromator and sample.We also employ a Heusler (111) crystal analyzer to select the energy and spin states of scattered neutrons, separating spin-flip (SF) and non-spin flip (NSF) intensities.The spin polarization of the incident neutron beam (P 0 ) was 0.823, as measured by the (002) nuclear Bragg reflection of our single crystal.We collected integrated intensities of 9 independent nuclear Bragg reflections by ω scans in unpolarized condition in the paramagnetic phase.After applying the Lorentz factor correction, λ 3 / sin 2Θ with neutron wavelength λ and scattering angle 2θ for the Bragg reflections, we obtain the structure factors |F obs | for the nuclear reflections.Comparing these to structure factors |F cal | calculated from the structural parameters in Ref. [18], the scale factor s is defined via |F obs | 2 = s |F cal | 2 .The calculation of the nuclear structure factor follows Ref. [51].

FIG. 1 .FIG. 2 . 4 FIG. 3 .FIG. 4 .
FIG. 1. Conical helimagnetism in the layered square-lattice antiferromagnet DyTe 3 .a, Crystallographic unit cell with insulating DyTe bilayers and metallic Te bilayers, with natural cleaving plane (dashed).b, Magnetic exchange interactions in a single DyTe double-square net bilayer, with antiferromagnetic nearest neighbour (NN) and next-nearest neighbour (NNN) exchange couplings J 1 and J 2 .c, Zigzag chain illustration of double-square net structure in DyTe 3 .Conical, noncoplanar helimagnetism is resolved in the zero-field ground state (T = 2.3 K) by neutron scat-tering.The cone direction, parallel to the a-axis, alternates both between pairs of magnetic sites in a zigzag chain, and between stacked zigzag chains.This texture causes polarization along the b-axis, i.e. perpendicular to square net bilayers (yellow arrow).Note: The full magnetic unit cell extends two times further along the chain direction (Fig.E2).d, Weak anisotropy of the magnetic susceptibility χ in DyTe 3 .The softest direction is H ∥ c, consistent with the modulation direction of the magnetic order in panel c.The inset shows Curie-Weiss temperatures and temperatures T χ of maximal χ, measured in a small magnetic fields along three crystallographic directions.
An incommensurate component in the Fourier transform of a 1D zigzag chain model appears in Monte Carlo simulations.e, Local environment of Dy in DyTe 3 , with covalent (metallic) bonds to Te-B (Te-A) depicted by solid (dashed) lines, respectively.f, Charge density (CD) of Dy 4f 9 shell under the influence of crystal electric fields (CEF) from nearest neighbour ions, with exaggerated non-spherical part.If metallic and covalent bonds cause CEF of roughly equal strength (if metallic bonds are screened), oblate |J b = ±15/2⟩ (prolate |J b = ±1/2⟩) is the lowest energy CEF doublet.This favours out-of-plane (in-plane) magnetization, respectively.Colour on CD isosurfaces indicates amplitude |E| of the local CEF.
FIG. E1.Two possible magnetic structures in the ground state of DyTe 3 .Neutron scattering does not constrain the combination of cycloidal (m b , m c ) and antiferromagnetic (AFM, m a ) components, which can be either in-phase (left) or out of phase (right).The AFM and cycloidal components of in-phase (out of phase) combinations are depicted, in panels b,c (in panels f, g), respectively.Only a single zigzag chain is shown, although there is antiferromagnetic coupling between subsequent zigzag chains along the b-axis, c.f. Fig. 1. d,h Magnetic texture in a single DyTe magnetic square net bilayer, corresponding to the simplified zigzag-chain picture in panels a,e.Colour on each ionic site (arrows) illustrate the b-axis (the ac-plane) component of the magnetic moments.Dashed lines are guides to the eye, describing the square net in the upper sheet of the DyTe bilayer slab.The bright (dark) highlight marks the size of the magnetic unit cell (the size of half a magnetic unit cell, as shown in panels a-c, e-g, and in the figures of the main text).
FIG. E2.Projection of neighbouring moments between two possible magnetic structures in the ground state of DyTe 3 .Only a single zigzag chain is shown; the full magnetic unit cell consists of two chains, coupled antiferromagnetically along the b-axis.Neutron scattering does not constrain the combination of cycloidal (m b , m c ) and antiferromagnetic (AFM, m a )

FIG
FIG. E3.Calibration of instrument resolution at 5G-PONTA at the JRR-3 neutron reactor source.Each pair of (red, blue) data points corresponds to the full width at half maximum (FWHM) of a nuclear lattice reflection.The shape of the resolution ellipsoid in the b * -c * plane is defined by the width of an ω-scan (w T , transverse width) and the width of a ω − 2θ scan (w L , longitudinal width), respectively.These parameters depend on the sample shape, crystal quality, the momentum transfer |Q|, and the performance of the instrument.Blue and red dashed lines are a linear fit, and a second order polynomial fit to the data, respectively.These fits can be used for estimation of the instrument resolution at arbitrary positions in the bc scattering plane, e.g. in Fig.4 c. Error bars correspond to statistical uncertainties of Gaussian fits to the nuclear reflections.
FIG. E7.Magnetic structure refinement for the ground state of DyTe 3 .Antiferromagnetic FIG. E8.Incommensurate fan-like component in phase II of DyTe 3 .a, Line scans of magnetic intensity through the incommensurate reflection on the (01L) line, with clear temperature dependence.Shaded Gaussian curves at T = 3.7 K indicate a double-Gaussian fit in the regime of phase coexistence between phases I and II.High temperature data was multiplied by a scalar factor to enhance visibility.The magnetic intensity vanishes entirely at T N2 = 3.85 K, with no indications of diffuse scattering in the thermally disordered regime.b,c, In phase II at T = 3.8 K, we report polarization analysis of neutron scattering intensities for two representative reflections (01L cyc )and (09L cyc ).The absence of non-spin flip (NSF) intensity at L = 0.19 in both panels indicates m a = 0; finite spin-flip (SF) intensity at (0, 1, 1.19), paired with zero intensity at (0, 9, 0.19), is consistent with collinear sinusoidal magnetic order, where moments are exclusively along the b-axis.g, Illustration of deduced incommensurate magnetic component in phase II, with antiferromagnetic coupling between double-square net bilayer.

FIG. E11 .
FIG. E11.Mixing of crystal field states in DyTe 3 by exchange interactions.a-c, The maximal and minimal eigenvalues of Ĵa and d-f of Ĵb calculated for the crystal field doublets ψ ± 0,1,2 .The charge ratio (x-coordinate) describes the ratio of effective charges assigned to Te-A and Te-B tellurium ions corresponding to metallic and ionic bonds around Dy, respectively.A matrix representation of the operators Ĵa and Ĵb was calculated on each respective subspace spanned by a Kramers pair of states, and subsequently diagonalized.There is a transition from dominant ψ 0 Ĵb ψ 0 = ±1/2 (easy-plane) to ±15/2 (easy-axis) for the ground state |ψ 0 ⟩, when reducing the effective crystal electric field charge for Te-A on metallic bonds.The first excited state behaves similarly, while the second excited state generally covers a broader range of Ĵb eigenvalues.Red and grey shaded areas are defined as in Fig. E10.In the red region, the character of the ground state (GS, ψ ± 0 , left column) changes from predominant |J b = ±15/2⟩ to mainly |J b = ±1/2⟩ character.It is not possible to generate sizable in-plane (b-axis) magnetic moment using any linear combination of ψ + 0 , ψ − 0 at relative weight of CEF charges q on Te-A and Te-B sites > 0.5 (< 0.4).

TABLE E1 .
Magnetic properties of van-der Waals systems with complex magnetic order.Electrical transport properties are categorized into metals (M), insulators (I), and materials