Introduction

Anomalous Hall effect (AHE) is a traditional and experimentally convenient tool for identifying phases that spontaneously break \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry1,2. The AHE refers to a non-dissipative antisymmetric component of the electrical conductivity tensor, that is odd under \({{{{{{{\mathcal{T}}}}}}}}\) and that can be generated by certain magnetic orderings1,3. Among those, the most common and arguably best understood is the ferromagnetic ordering where the broken symmetries allowing for the AHE are related to the net internal magnetization of the crystal3. A common model of ferromagnetism is a collective order in the spin space accompanied by an isotropic partial-wave (s-wave) form of the electronic structure in the momentum space3,4. In contrast, anisotropic higher-order partial-wave forms of magnetically ordered phases were elusive and much less is known about their responses4,5,6,7. In fact, compensated magnetic orderings with a vanishingly small net magnetization have remained outside the scope of the research of the spontaneous \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking responses for more than a century1,3. Indeed, these responses can be absent in the conventional compensated antiferromagnets whose spin arrangement on the crystal has a symmetry combining \({{{{{{{\mathcal{T}}}}}}}}\) with a translation (\({{{{{{{\bf{t}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\)-symmetry – see Fig. 1a) or with inversion (\({{{{{{{\mathcal{P}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\)-symmetry)1.

Fig. 1: Unconventional d-wave magnetism and crystallographic structure of Mn5Si3.
figure 1

a Conventional antiferromagnet with \({{{{{{{\boldsymbol{t}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\) symmetry combining translation with time-reversal. b Unconventional d-wave magnetism with broken \({{{{{{{\boldsymbol{t}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\) symmetry. (Opposite magnetization-density isosurfaces calculated from first principles are marked in red and blue.) c Spin-split Fermi surface cut of an anisotropic d-wave form calculated from first-principles. The Néel vector is along the [\(2\bar{2}01\)] crystal direction ([111] direction in the 3-component a − b − c notation), and we plot spin projection on the [\(2\bar{1}\bar{1}0\)] x-axis ([100] a-axis). d, e Top and side view, resp., of the hexagonal crystal structure of the Mn5Si3 epilayers with marked in-plane a and out-of-plane c lattice constants. f Transmission electron microscopy image of the Mn5Si3 epilayer grown on a Si substrate. g Optical micrograph of the lithographically patterned Hall bar, and orientation of the crystal and the applied magnetic field B.

Full size image

However, over the past decade, two types of crystal structures were predicted to host the spontaneous \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking responses, including a spontaneous AHE, that are not related to a net internal magnetization of the crystal8,9: (i) The first type is geometrically frustrated structures, such as kagome, pyrochlore, or triangular lattices10,11,12, where the experimentally observed spontaneous AHE10,11,12 was related to a non-collinear magnetic ordering11 or a spin-liquid state candidate12. (ii) For the second type of crystals with a collinear magnetic order, termed altermagnetic13,14, the distinctive feature are non-relativistic spin symmetries where the opposite-spin sublattices are connected by real-space rotation transformations and not by translation or inversion1,9,13,14. In contrast, conventional collinear ferromagnets (ferrimagnets) and antiferromagnets have exclusively distinct symmetries13,14: ferromagnets (ferrimagnets) have only one spin lattice (or opposite-spin sublattices not connected by any symmetry transformation), and antiferromagnets have opposite-spin sublattices connected by a real-space translation or inversion. The spontaneous anomalous Hall response in altermagnets has then been related, when including relativistic spin-orbit coupling, to a compensated collinear magnetic order with a vanishingly small (zero non-relativistic) magnetization1,9,13,14. The general characteristic of the unconventional magnetism in altermagnets is a strong \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking and alternating spin polarization in both real-space crystal structure and momentum-space electronic structure, with or without the presence of the weak relativistic magnetization13,14,15,16,17. The alternating spin polarization has suggested to refer to this phase as altermagnetism13,14. Note that, in general, the \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry-breaking responses in altermagnets do not require relativistic spin-orbit coupling13,14. In the specific case of the AHE, however, additional symmetry breaking by the spin-orbit coupling is required in collinear magnets, including altermagnets1,9,14,18. Experimental confirmation of altermagnetic band structure was recently published19,20,21,22.

Remarkably, for certain crystal symmetries, the altermagnetic phase has been predicted to take a form of an unconventional d-wave magnet9,14,23. Unlike the earlier suggested realizations via Fermi-liquid instabilities in strongly-correlated materials4,6,7, here the d-wave magnetism is generated by a robust crystal potential and an unconventional real-space spin-density ordering13. Remarkably, it can also host \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry-breaking responses of comparable strength to the conventional s-wave ferromagnetism13,14. Besides the AHE, the predicted responses in these unconventional d-wave magnets also include analogues of the non-relativistic spin-polarized currents that underpin the prominent giant-magnetoresistance and spin-torque phenomena in ferromagnetic spintronic devices1,9,13,24,25,26,27,28,29.

In the experimental part of our paper, we present a discovery of a spontaneous anomalous Hall conductivity of 5–20 S/cm in epitaxial thin films of Mn5Si3 with a vanishingly small net magnetic moment. Our characterization measurements show that the Mn5Si3 epilayers have a hexagonal crystal structure without canonical geometric frustration. The observed unconventional combination of a spontaneous anomalous Hall response and a vanishingly small net magnetization is, therefore, not related in our Mn5Si3 epilayers to phases stabilized by the first type of crystal structures with geometric frustration. This turns our attention in the theory section to the second type of crystal structures with the d-wave altermagnetic phase. Below the magnetic-ordering temperature, the crystal structure of Mn5Si3 is known from previous studies to result in a sizable magnetic moment on two fifths of the Mn atoms, as highlighted in Fig. 1a, b30,31. Our first-principles calculations show that without strong correlations, the unconventional d-wave magnetism of these magnetically ordered Mn atoms in the direct real space (Fig. 1b), and the corresponding d-wave spin polarization in the reciprocal momentum space (Fig. 1c), generate a vanishingly small net magnetization and a sizable spontaneous anomalous Hall conductivity of the microscopic Berry-curvature mechanism1, consistent with our measurements. We acknowledge that while the proposed magnetic order aligns with our experimental observations, future efforts should be focused on directly observing the spin structure. This is discussed in detail in the Supplementary Note 5. In the final discussion section we point out that our unconventional d-wave magnetism candidate, realized in a crystal comprising abundant and only weakly relativistic elements, points towards research and applications of magnetic band-topology, non-dissipative electronics, valleytronics or spintronics unparalleled within the framework of the conventional ferromagnetic, antiferromagnetic and paramagnetic phases.

Results

We start the experimental part by discussing the structural characterization of Mn5Si3 in the room-temperature paramagnetic phase. Earlier studies of bulk crystals determined that the space group of Mn5Si3 is P63/mcm, with a hexagonal unit cell containing two formula units30,31,32. The unit cell has sixteen atoms: four Mn atoms (Mn1) at a Wyckoff position 4d, and six Mn atoms (Mn2) and six Si atoms at a Wyckoff position 6g. The crystal structure motif of Mn5Si3, shown in Fig. 1d, e, is characterized by a distorted octahedron [Mn1Si6] with Si occupying its vertices and Mn1 in the center, and a distorted octahedron [□(Mn2)6] with Mn2 at the vertices and no atoms in its interior30. Since the distances of Mn atoms in pairs Mn1–Mn1, Mn1–Mn2 and Mn2–Mn2 are substantially different30, the exchange interactions between Mn atoms do not exhibit the canonical geometric frustration33.

For our study, we have prepared thin films of Mn5Si3 by molecular beam epitaxy on top of a Si(111) substrate. In Fig. 1f, we present a room-temperature transmission electron microscopy (TEM) image showing the (0001) orientation of our Mn5Si3 films with a thickness of 12 nm which were used to fabricate microdevices for electrical transport measurements (Fig. 1g). The TEM measurements, complemented by X-ray diffraction (XRD) shown in the Supplementary Information Figs. 1 and 2 (and also Methods), indicate high crystal quality of the epilayers, with an in-plane hexagonal symmetry. They confirm that our thin films have the same crystal structure motif as previously observed in the bulk samples. Apart from the same crystal-structure motif, there are important differences between the overall crystal structure of the bulk and our thin-film samples that stem from the epitaxial strain and the epitaxial constraints. The Mn5Si3 epilayers on the Si(111) substrate are constrained to a hexagonal crystal lattice in the whole studied temperature range and, therefore, the films do not undergo the structural transitions observed in bulk. In the following paragraphs, we elaborate on this point in more detail.

In Fig. 2a, b we show temperature-dependent lattice constants of Mn5Si3, and we start the discussion by first recalling the behavior of Mn5Si3 as reported earlier in the bulk samples30. The lattice constants a and b, that are equal in the room-temperature paramagnetic phase, show two anomalies in bulk Mn5Si3: one at T1 ≈ 100 K and the other one at T2 ≈ 70 K (Fig. 2a). At the higher critical temperature T1, the crystal undergoes an orthorhombic distortion that lifts the degeneracy between the a and b lattice parameters. When further decreasing the temperature to the lower critical point T2, a monoclinic distortion results in one of the two lattice parameters abruptly increasing while the other one is decreasing, which is also accompanied by an increase of the lattice parameter c (Fig. 2b).

Fig. 2: Temperature dependent lattice parameters and resistivity.
figure 2

a, b In-plane and out-of-plane lattice constants, resp., obtained in our Mn5Si3 epilayers from temperature dependent X-ray diffraction (red), and compared to previously reported data for bulk Mn5Si330 (green). c Temperature-dependent longitudinal resistivity ρ and its derivative dρ/dT. d Temperature-dependent longitudinal magnetoresistance recorded with a magnetic field of 0.5,1, 2, and 5 T applied along the [0001] crystal direction.

Full size image

The structural transitions have counterparts in anomalies at T1 and T2 previously detected in the magnetic susceptibility, specific heat and longitudinal resistivity of the bulk samples30,31,34. Earlier neutron scattering measurements on the bulk samples30,33,35,36,37 revealed that at T1, the othorhombic crystal distortion is accompanied by an onset of a collinear antiferromagnetic ordering of two-thirds of the Mn2 atoms. The antiferromagnetic propagation vector (0, 1/2, 0) corresponds to a doubling of the unit cell along the b-axis, as compared to the paramagnetic phase30,37. The resulting \({{{{{{{\bf{t}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\)-symmetry of this conventional antiferromagnetic phase13,24 is consistent with the absence of a spontaneous AHE signal1, as experimentally confirmed in the bulk samples (or thick polycrystalline films)31,34.

Below T2, the neutron studies in bulk Mn5Si3 showed that the magnetic phase becomes non-collinear (non-coplanar)30,36,37. This second magnetic transition can be suppressed, and the collinear antiferromagnetic phase recovered by an applied magnetic field30. The strength of the critical field increases with decreasing temperature, reaching approximately 1 T at 60 K31,34,36,37. A spontaneous Hall resistivity of ≈0.02–0.04 μΩ cm measured below T2 in the bulk samples (or thick polycrystalline films) was ascribed31,34 to a topological Hall effect38,39 arising from the low-temperature non-collinear magnetic order in Mn5Si3. Consistently, the topological Hall signal was suppressed by applied magnetic fields of strengths comparable to the above critical fields obtained in the temperature-dependent neutron measurements30,31,34.

We now compare the established temperature-dependent phenomenology in bulk Mn5Si3 to our measurements in the thin-film epilayers. As expected, the in-plane lattice parameters a and b of our epilayers, constrained by the substrate, show no transitions (Fig. 2a), and their weak temperature dependence closely follows the weakly decreasing in-plane lattice parameter with decreasing temperature of the Si substrate. In contrast, the out-of-plane lattice parameter c of the Mn5Si3 film is not fixed by the substrate, and we observe an anomaly analogous to the T2 transition observed in the bulk samples (Fig. 2b).

Note that in the case of Mn5Si3 on Si(111), the value of the in-plane lattice constant is governed primarily by the mismatch in the thermal expansion coefficients of the epilayer and the substrate. During cooling after growth, the mismatch in the thermal expansion coefficients, which are around 2.6 × 10−6 K−1 and 23 × 10−6 K−1 in Si and Mn5Si3, respectively, causes an in-plane tensile strain. At room temperature and below we, therefore, find the in-plane lattice constant in our epilayers to be considerably larger than the bulk value. Consistently, the out-of-plane lattice constant in the epilayers is smaller than in bulk Mn5Si3. In contrast to the thermal-expansion mismatch, the nominal mismatch of 3.7% between room-temperature in-plane lattice constants of the separate Si(111) and Mn5Si3(0001) crystals plays a more minor role as it is partially accommodated by a thin MnSi interfacial layer between the Si substrate and the Mn5Si3 epilayer (see Methods for more details).

In Fig. 2c, d we plot resistivity measurements of our microdevices (Fig. 1g) patterned from the thin-film Mn5Si3 epilayers. They show a metallic resistivity of the order of magnitude that is consistent with earlier studies of thicker films31,34. Consistent with the bulk phenomenology, we detect the T2 anomaly in the resistivity of our thin films, as shown in Fig. 2c. We also point out that our observation in Fig. 2d of a strong magnetoresistance below T2, contrasting with a negligible magnetoresistance over a broad temperature range above T2, is reminiscent of the sensitivity to the magnetic field of the non-collinear component of the magnetic order in the low-temperature phase of the bulk samples.

Unlike the T2 transition, we observe no counterparts of the bulk anomaly at T1 ≈ 100 K in either the structural characterization or resistivity measurements of our Mn5Si3 epilayers. However, as seen in Fig. 2c, d, we detect a second anomaly in the resistivity of the thin films at ≈240 K, accompanied by an enhanced magnetoresistance above this temperature.

To explore the phases of our Mn5Si3 epilayers over the broad temperature range we performed magnetometry and Hall measurements, summarized in Fig. 3. At 300 K, the magnetization is linear in the external magnetic field Bz, that we applied along the out-of-plane [0001] crystal axis (Fig. 3a). At lower temperatures, a weak non-linearity is observed at small fields. In Supplementary Fig. 3 we show control SQUID measurements of a bare Si(111) substrate (with no deposited epilayer), exhibiting a similar weak low-field non-linearity. The important observation in Fig. 3a is that the remanent zero-field magnetization remains below ~0.01 μB per unit cell at all temperatures, as highlighted in the inset of Fig. 3a.

Fig. 3: Magnetization and anomalous Hall effect.
figure 3

a Magnetization as a function of the magnetic field applied along the [0001] crystal direction at different temperatures. The inset highlights a vanishingly small remanent magnetization within error of 5 mμB/u.c. The plot shows the total magnetization of the measured sample, i.e., also including the substrate. The inset also shows the total measured magnetization. However, when recalculating from A/m to mμB/u.c., we considered that the remanent signal at zero field is due to the Mn5Si3 film alone and has no contribution from the Si substrate (we also considered the same for the small field range around zero plotted in the inset). b Field and temperature dependent anomalous Hall resistivity. The ordinary Hall effect, which is linear in the applied magnetic field, was subtracted (see text). c Decomposition of the anomalous Hall resistivity measured at 50 K into a topological Hall component \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{T}}}}}}}}}\), and a component ascribed to the unconventional d-wave magnetism, \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) (see Methods and Supplementary Fig. 4).). d The decomposition of the spontaneous (zero-field) anomalous Hall resistivity as a function of temperature. e Spontaneous anomalous Hall conductivity corresponding to the component ascribed to the unconventional d-wave magnetism.

Full size image

From the measured total Hall resistivity, \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{tot}}}}}}}}}\), we first extracted the component that is linear in Bz, and gives an ordinary Hall coefficient, RH ≈ 1 − 4 × 10−10 m3 C−1. Assuming a single-band model, it corresponds to a metallic carrier density, n ~ 1022 cm−3. The remaining anomalous component of the Hall resistivity is given by, \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{A}}}}}}}}}={\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{tot}}}}}}}}}-{R}_{{{{{{{{\rm{H}}}}}}}}}{B}_{z}\) (see Methods and Supplementary Fig. 4). Remarkably, despite the vanishingly small remanent magnetic moment, we observe a sizable spontaneous AHE signal at zero field over a broad range of temperatures.

The AHE exhibits a large coercivity of ≈2–3 T and, overall, it is not diminished by strong fields. Nevertheless, below T2 and within a ~1 T field-range, we observe a weak contribution to the anomalous Hall resistivity that is non-monotonic in the field. To highlight this feature, we decompose in Fig. 3e the anomalous Hall resistivity into two contributions: \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{A}}}}}}}}}={\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{T}}}}}}}}}+{\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}.\) The \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{T}}}}}}}}}\) component appears below T2, has a spontaneous value reaching 0.09 μΩcm, and vanishes at ~1 T. Its phenomenology is thus consistent with the topological Hall effect identified in the bulk Mn5Si3 samples31,34,40.

In Supplementary Figs. 5, 6, we compare the field-dependence of the AHE with the longitudinal magnetoresistance. A strong negative magnetoresistance is observed below T2, consistent with the presence of the \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{T}}}}}}}}}\) contribution to the AHE that has been associated with the deviation of the magnetic order from the fully collinear state. Above T2 where the \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{T}}}}}}}}}\) contribution is absent and the magnetic order is expected to be collinear, we observe the correspondingly weaker magnetoresistance.

Note that the observed large coercive field of ≈2–3 T at which the AHE reverses is consistent with the absence of a strong net magnetic moment, as detected by SQUID, and with the corresponding weak Zeeman coupling in our compensated magnet. The observed increase of the reorientation field (coercivity) with increasing temperature (see also Supplementary Fig. 7) is another signature that contrasts with the conventional ferromagnetic phenomenology. In the collinear compensated magnets, the increasing reorientation field with increasing temperature was already reported in earlier studies and ascribed to a complex and highly anisotropic response to the applied magnetic field41. This was associated, besides the magnetic anisotropy and exchange interaction, with the effect of the Zeeman coupling of the field-induced or weak-relativistic net magnetic moment.

The \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) contribution to the measured anomalous Hall resistivity in our thin-film epilayers is detected below a transition temperature of ≈ 240 K (Fig. 3d), which coincides with the temperature of the second anomaly observed in the resistivity and magnetoresistance measurements in Fig. 2c, d. The \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) component dominates the anomalous Hall resistivity over the entire temperature range down to the lowest measured temperature of 10 K, i.e., also below T2 (Fig. 3d). The zero-field spontaneous value of \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) reaches 0.2–0.7 μΩ cm (see also Supplementary Fig. 8 for data measured on other Mn5Si3 epilayer samples).

In Fig. 3f we plot the spontaneous anomalous Hall conductivity component \({\sigma }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\approx {\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}/{\rho }^{2}({B}_{z}=0)\) over the whole measured temperature range. The magnitude of \({\sigma }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) reaches values between 5 and 20 S/cm, depending on the studied Mn5Si3 epilayer (see also Supplementary Fig. 8).

To further explore the origin of the \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) signal, we prepared a series of films with a varying nominal thickness, which gives a varying crystal quality. For film thicknesses ≳ 30 nm, spurious phases are formed in our Mn5Si3 films. We have, therefore, focused on lower thicknesses and parametrized the quality of the crystals by XRD measurements. In Supplementary Fig. 8 we show that the magnitude of \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) decays with lowering the crystal quality, which we characterize by the ratio of intensities of Mn5Si3 and MnSi X-ray diffraction peaks, and the signal is absent in polycrystalline films. We also show that there is no correlation between \({\rho }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\) and the negligible net magnetic moment measured across the series of samples with varying crystal quality. Finally, we show in Supplementary Fig. 9 measurements in rotating 4 T fields that highlight an unconventional anisotropy of the AHE in our Mn5Si3 films.

To summarize the experimental part of our work, we observe in our Mn5Si3 thin-film epilayer a sizable spontaneous AHE signals below ≈240 K, accompanied by a vanishingly small remanent net magnetization. This excludes AHE mechanisms analogous to conventional ferromagnets or due to a field-induced magnetization. The structural characterization of our films implies that exchange interactions between Mn atoms do not exhibit the canonical geometric frustration. This allows us to further exclude AHE mechanisms associated with compensated magnetically-ordered or spin-liquid phases generated by the geometrically frustrated lattices. We detect a component in our spontaneous Hall response with analogous phenomenology that in bulk Mn5Si3 samples was ascribed to the topological Hall effect. In contrast to the bulk samples, however, it only represents a weak contribution to the total spontaneous Hall signal measured in our thin-film epilayers. Moreover, the dominant contribution in our samples persists also at temperatures well above T2 ≈ 70 K, at which the topological Hall effect disappears in both bulk and our thin films.

The origin of the dominant contribution to the spontaneous AHE in our Mn5Si3 epilayers is, therefore, unconventional. The following section presents our theory analysis showing that the above structural and electric characterizations, combined with the observation of the spontaneous AHE signal and a vanishingly small net magnetization, are consistent with the formation of the unconventional d-wave magnetic phase (Fig. 1b, c).

Theory

Earlier density functional theory (DFT) calculations33 showed that for the two thirds of Mn2 atoms contributing to the magnetic ordering in Mn5Si3, the strongest exchange coupling, in the notation of Fig. 1a, is between crystal sites I and II, and between sites III and IV. These exchange interactions tend to stabilize the collinear antiparallel ordering33, consistent with the transition from the paramagnetic to the antiferromagnetic phase observed in the neutron measurements on bulk samples at T1 ≈ 100 K30,37.

In our thin-film epilayers, the spontaneous anomalous Hall signal occurring below ≈ 240 K evidences a transition to a \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry broken phase. On one hand, the close similarity between the crystal-structure motifs identified in our thin films and in the bulk samples suggests that the leading exchange interactions in the thin films are again between the Mn2 sites I and II, and sites III and IV. This implies that a good candidate for the \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry broken phase below ≈ 240 K in the thin films, illustrated in Fig. 1b, has the same antiparallel ordering between the sites I and II, and the sites III and IV, as in the bulk (Fig. 1a). On the other hand, the doubling of the unit cell, and the resulting \({{{{{{{\bf{t}}}}}}}}{{{{{{{\mathcal{T}}}}}}}}\)-symmetry of the conventional antiferromagnetic phase1,13,24 observed in the bulk samples (Fig. 1a)30,37, is excluded in our thin films by the experimentally detected spontaneous anomalous Hall signal1,13,24. Therefore, the candidate phase of the thin films below ≈240 K (Fig. 1b), consistent with the earlier theoretical and experimental works in bulk samples, and with the complete set of our structural characterization, resistivity and Hall measurements, shares with the bulk samples the antiparallel ordering of the Mn2 magnetic moments while, simultaneously, keeping the same size of the unit cell upon the transition from the paramagnetic to the magnetically ordered phase.

In Fig. 1b we highlight on our DFT real-space magnetization densities, and in Figs. 1c and 4a, b on DFT momentum-space Fermi surfaces, that the candidate magnetic ordering corresponds to the unconventional compensated collinear magnetic phase of the d-wave form13,24. In real space, the candidate magnetic ordering shows the defining characteristics of the unconventional phase, dubbed altermagnetic: Namely the lack of translation or inversion and, in the non-relativistic limit, the presence of rotation symmetry transformations connecting opposite-spin sublattices. The rotation symmetries protect the compensated nature of the magnetic phase, i.e. the precisely zero net spontaneous magnetization in the non-relativistic limit, while allowing for the \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking and alternating spin splitting in the band structure13,24.

Following the general classification of non-relativistic collinear magnetic phases based on the spin-group formalism13,24, the spin-dependent band structure of our candidate unconventional magnetic phase in Mn5Si3 is described by a spin Laue group 2m2m1m. The group is generated by the following three symmetry transformations: a real-space inversion connecting same-spin sublattices, and real-space two-fold rotations around the x and y axes (\({{{{{{{{\mathcal{C}}}}}}}}}_{2x}\) and \({{{{{{{{\mathcal{C}}}}}}}}}_{2y}\)) connecting opposite-spin sublattices. These spin Laue group symmetries result in two orthogonal spin-degenerate nodal planes crossing the Γ (M) point of the Brillouin zone. When making a closed loop in the momentum space around the Γ (M) point in a plane orthogonal to the spin-degenerate nodal planes, the spin makes a discrete 180∘ reversal when passing through each nodal plane. This results in the d-wave form of the non-relativistic spin-dependent band structure13,24, as highlighted on the kz = 0 Fermi-surface cut in Fig. 4a.

Fig. 4: First-principle theory of Mn5Si3 electronic structure and comparison chart of compensated magnets with anomalous Hall effect.
figure 4

Spin-polarized Fermi surface calculated without (a) and with (b) spin–orbit coupling (SOC). In the latter case, the Néel vector is along the [\(2\bar{2}01\)] crystal direction ([111] direction in the 3-component a–b–c notation), and we plot spin projection on the [\(2\bar{1}\bar{1}0\)] x-axis ([100] a-axis).c, Hexagonal Brillouin zone notation. Calculated non-relativistic non-magnetic (d) and magnetic (e) energy bands. The red and blue coloring corresponds to the spin up and down projection, respectively. f Calculated anomalous Hall conductivity σH. g Comparison diagram of selected anomalous Hall systems in the parameter space of critical temperature vs. abundance of the least abundant chemical element in the material59. Gray colored circles mark systems with the Hall effect mechanism that requires the application of an external field. Blue colored circles mark systems with the spontaneous anomalous Hall effect. The magenta colored circle marks out present study of Mn5Si3. Data in panel (g) are taken from a review article in ref. 1.

Full size image

By comparing Fig. 4a, b, we see that the relativistic spin-orbit coupling generates only a weak perturbative correction in the Mn5Si3 Fermi surfaces. The d-wave form is preserved, and only the discrete 180∘ spin reversals when passing through the non-relativistic spin-degenerate nodal planes are replaced in the presence of the spin-orbit coupling by a continuous 180∘ spin reorientation. Note that in the relativistic calculations we considered the magnetic order vector pointing along the [\(2\bar{2}01\)] crystal direction (for more details see the discussion below on the DFT AHE calculations and Supplementary Note 4).

In Fig. 4c–e we show the Brillouin zone notation and non-relativistic band structure calculations. In the non-magnetic bands, we observe a number of van Hove singularities around the Fermi level. Additonally, in the kz = π plane, the bands are four-fold degenerate. (The origin of the degeneracy is the off-centered mirror plane orthogonal to the z-axis42.) These band structure features make Mn5Si3 highly susceptible to the emergence of magnetic ordering6,13,23.

The stability of the unconventional d-wave magnetic phase can be illustrated based on zero-temperature DFT calculations of the total energy. The d-wave magnetic phase has the total energy smaller than the paramagnetic (ferromagnetic) phase by 0.95 (0.96) eV per unit cell. Apart from the significantly higher total energies, we also recall that the two reference phases are inconsistent with the set of our experimental results. Namely, the ferromagnetic phase allows for a spontaneous AHE that, however, originates in this phase from its sizable net magnetization. The paramagnetic phase has a zero net magnetization, but also a zero spontaneous AHE.

This brings us to our DFT calculations of the AHE conductivity generated by the compensated magnetic order of the unconventional d-wave phase. In our calculations, we consider the intrinsic microscopic AHE mechanism due to Berry curvature in the relativistic band-structure of a perfect crystal without extrinsic disorder3,43. The focus on the intrinsic AHE is justified by the systematic studies of the microscopic mechanisms in ferromagnets3. The studies showed that the extrinsic (skew scattering) contribution becomes significant only in samples with conductivities above 106 Ω−1 cm−13, which is much higher than the conductivity of our Mn5Si3 films.

In Fig. 4f, we plot the DFT AHE conductivity as a function of the position of the Fermi level. The calculations are performed for the magnetic order vector pointing along the crystal direction [\(2\bar{2}01\)] ([111] in the 3-component a–b–c notation) between the in-plane [\(2\bar{2}00\)] and normal-to-the-plane [0001] crystal axes. This off high-symmetry direction is chosen because it gives in our DFT calculations a lower total energy than the in-plane or normal-to-the-plane axes (see Supplementary Note 4). Moreover, the magnetic point group \(\bar{1}\) for the Néel vector along the [\(2\bar{2}01\)] direction allows for a spontaneous anomalous Hall vector component along the [0001] crystal axis, i.e. along the normal to the thin-film plane, which makes it detectable in our experimental geometry. In contrast, AHE is excluded by symmetry in the magnetic point group mmm which corresponds in our case to the theoretically identified [0001] hard axis of the N’eel vector. Also consistently with our measurements and DFT calculations, no spontaneous AHE would be detected for the Néel vector within the (0001)-plane (c-plane), (\(2\bar{1}\bar{1}0\))-plane (a-plane) or (\(0\bar{1}10\))-plane (b-plane) because in these cases the Hall vector, if allowed, is constrained by symmetry to the (0001)-plane of the thin film.

Our calculations in Fig. 4f illustrate that the spontaneous AHE conductivity, arising from the strong \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking in the electronic structure by the compensated collinear magnetic order of the unconventional d-wave phase, combined with the relativistic Berry-curvature mechanism, can reach values comparable to the AHE in common ferromagnets3. We obtain sizable \({\sigma }_{{{{{{{{\rm{H}}}}}}}}}^{{{{{{{{\rm{U}}}}}}}}}\approx 5-20\) Scm−1 within a ~ 100 meV energy window around the Fermi level. These theoretical values are consistent with our measurements.

Discussion

We have presented our discovery in epitaxial thin-film Mn5Si3 of an unconventional combination of a sizable spontaneous AHE and a vanishingly small net remanent magnetization. Among the experimentally established or theoretically proposed mechanisms, our set of characterization experiments, AHE measurements and DFT calculations is consistent with the formation of the recently predicted unconventional collinear compensated magnetic phase13,24. Here we point out that our work complements, in a fundamentally distinct way, parallel experimental AHE studies of other candidate materials of this unconventional phase, namely of RuO2 and MnTe18,44. In both RuO2 and MnTe, non-magnetic atoms play a central role in breaking the translation and inversion symmetries, while preserving a rotation symmetry, connecting the opposite-spin sublattices9,18,44. Mn5Si3 is principally distinct, as here these basic crystal-symmetry conditions for the formation of the unconventional phase are fulfilled by the spin and crystal arrangement of the magnetic atoms alone. The emergence of the magnetic state we propose has the potential to inspire further research efforts. This is discussed in detail in the Supplementary Note 5.

We also point out that in the experiments on RuO2, the magnetic order vector was reoriented by an applied magnetic field from the zero-field direction to allow for the AHE18. In contrast, the measured AHE signal in our thin-film Mn5Si3 is spontaneous, i.e., is observed at zero applied magnetic field. In MnTe, the detected AHE signal was also spontaneous44 as in the present study. However, the AHE in MnTe was ascribed to a higher-order (g-wave) form of the unconventional magnetic phase44. The high-order partial-wave forms can be less favorable as they, e.g., exclude by symmetry giant-magnetoresistive or spin-torque phenomena that are based on non-relativistic spin-dependent conductivities13,14,24. In contrast, these prominent \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry-breaking spintronic responses are allowed in the unconventional compensated collinear magnets of the d-wave form, and have been predicted to reach comparable strengths to ferromagnets13,14,24. This underlines the foreseen impact in spintronics of our discovery of the spontaneous \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry breaking response in a d-wave magnet candidate13,24.

The potential implications of the emerging unconventional d-wave magnetism go, however, well beyond the field of spintronics24. One area, highlighted by the specific band structure of the d-wave phase of Mn5Si3, is related to the spin splitting of alternating sign at time-reversal invariant momenta (TRIMs). In Mn5Si3, the spin-split TRIMs are the ± M1 and ± M2 points in the Brillouin zone, as seen in the DFT band structure calculations in Fig. 4 and confirmed by the spin-group symmetry analysis24. The TRIMs in centro-symmetric crystals are known to encode the information on whether the systems can host topological magnetic phases and phenomena45. Specifically, they are directly relevant for the axion insulators, topological magneto-electrics, Weyl fermions or the quantum AHE46. Moreover, spin polarized valleys centered around the TRIMs of the d-wave magnet represent \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry broken counterparts of the relativistic spin-split valleys in non-magnetic systems, where the \({{{{{{{\mathcal{T}}}}}}}}\)-symmetry excludes spin splitting at TRIMs. The unconventional d-wave phase may thus also open a new research path of unconventional magnetic valleytronics24.

Next, we look at the d-wave magnet candidate Mn5Si3 from a more practical perspective. In Fig. 4g we show a diagram comparing Mn5Si3 with representative non-collinear and collinear magnets in which the combination of the AHE and the vanishingly small (weak) net magnetization has been experimentally identified. The axes describe the abundance of elements forming the crystals and the magnetic transition temperature. We see that our Mn5Si3 d-wave magnet candidate represents a combination of an exceptional abundance of the involved elements, and a sizable transition temperature.

Finally, we point out that the lighter more abundant elements have a weaker relativistic spin-orbit coupling and tend to make electronic-correlation effects less prominent than heavier elements. Mn5Si3 is thus an example showing that the unconventional d-wave magnetism can be a robust phase not relying on complex strongly relativistic or correlated physics13,24.

Methods

Epitaxial crystal growth

We have grown the epilayers by ultrahigh-vacuum molecular beam epitaxy (MBE) with a base pressure less than 10−10 Torr. We have cleaned the Si(111) substrate surface by using a modified Shiraki method47. We have formed a final oxide layer chemically to protect the Si surface against oxidization in ambient air. This thin oxide layer was then thermally removed by annealing at 900 °C during a few min in the MBE chamber. Subsequently, a 10 nm-thick Si buffer layer was deposited at 600 °C to ensure a high-quality starting surface. The surface of the sample was monitored in situ by the reflection high energy electron diffraction (RHEED) technique that revealed an atomically flat surface with a well-developed (7 × 7) reconstruction (see Supplementary Note 1 and Supplementary Fig. 1). We decreased the growth temperature to 170 °C for the subsequent deposition of Mn and Si. We have evaporated high-purity Mn and Si by using a conventional high-temperature effusion sublimation cells. We have calibrated the cell fluxes by using RHEED oscillations and a quartz microbalance to achieve the desired stoichiometry of the layers with a total growth rate in the range of 0.1–0.2 Å/s. The first monolayers exhibited the typical signature of a Mn5Si3-type crystal, a \((\sqrt{3}\times \sqrt{3})\)R30° reconstruction48. Crystal quality was further improved by thermal annealing with its quality degree monitored by RHEED pattern (see Supplementary Fig. 1). Different growth parameters (including the nominal thickness the Mn/Si layers, the Mn and Si deposition rate and the growth temperatures) were optimized to minimize the presence of the spurious MnSi phase. We note that the Curie temperature of MnSi is around 30K and therefore, cannot contribute to the measured signal up to 240K. The same is valid for typical Mn-based oxides which have typically low critical temperature. We show the amount of the spurious phase in our five different samples in Supplementary Fig. 8. We prepared a reference MnSi sample, as discussed in Supplementary Note 3, Supplementary Fig. 10 and we performed reference magnetometry magneto-transport measurements as shown on Supplementary Fig. 11.

Transmission electron microscopy and X-ray diffraction

TEM investigations were performed at an accelerating voltage of 300 kV on a JEOL JEM-3010 instrument with a spatial resolution of 1.7 Å. The transmission electron microscopy (TEM) cross-section specimens were prepared by using a dual-focused ion beam (FEI Helios 600 NanoLab) milling through a liftout technique. (The TEM analyses summarized in Supplementary Fig. 1 confirmed the epitaxial relationships (Mn5Si3(0001)[10-10]//MnSi(111)[11-2]//Si(111)[1-10]) and reveals the location of MnSi at the interface between the Si substrate and Mn5Si349. The lattice mismatch of 3.7 percent between Si(111) and Mn5Si3 is partially accommodated by the formation of a thin layer of interfacial MnSi and an array of interfacial dislocations. In Supplementary Note 3 we summarize measurements on a control thin epitaxial film of MnSi deposited on Si(111)50. They confirm a negligible role of the MnSi seed layer in our Mn5Si3/Si(111) films on the measured AHE.

XRD measurements at room temperature were realized using a high brilliancy rotating anode, Rigaku RU-200BH equipped with an image plate detector, Mar345. The radiation used was Cu Kα, λ = 1.5418Å and the beam size was 0.5 × 0.5 mm2. The high-intensity Mn5Si3 0002 reflection in the XRD data recorded at 300K and shown in Supplementary Fig. 2 evidenced the preponderant formation of the Mn5Si3 hexagonal phase that grows along the c-axis.

Temperature-dependent XRD experiments from which we extracted the lattice constants of our epilayers shown in Fig. 2a, b were performed at CRISTAL beamline of Soleil synchrotron in the Bragg-Brentano geometry using a Siemens D500 diffractometer. The experimental error bar of the data is approximately the size of the dots plotted in Fig. 2a, b. The diffraction-peak intensity in these XRD measurements is much larger compared to the laboratory XRD experiment, as illustrated in Supplementary Fig. 2. Cooling of the sample was provided by a closed-cycle refrigerator (CCR, Sumitomo Heavy Industries), and He exchange gas ensured equalization of the temperature between the cold-finger, thermometer and sample. Cu-Kα1,2 radiation and a linear detector were used to speed up the data recording51. Additional low-temperature XRD measurements have been carried out at 12.7 keV using a 6-circle diffractometer with an angular accuracy better than 0.001∘. A 2D XPAD detector and an Advanced Research System closed-cycle cryostat were used in this setup.

Magnetotransport and anomalous and topological Hall extraction

We have patterned the Hall bars by standard optical lithography and Argon plasma etching. In Supplementary Fig. 5, we show the raw transversal and longitudinal resistivity data, measured simultaneously. In Supplementary Figs. 4 and 6, we show the subtraction of the linear slope, i.e. the ordinary Hall effect. The measured data were separated into symmetric and antisymmetric components. In Fig. 4a we show only the antisymmetric part. This procedure removes the small constant offset in transverse resistivity caused by tiny misalignments of the Hall contacts and the even contribution to the transverse signal. The even contribution presumably originates from anisotropic magnetoresistance due to the low symmetry52,53. The anomalous and topological Hall resistivities were extracted by fitting a cosh function. The anomalous Hall contribution is taken as the amplitude of the cosh fit. We show the result in Fig. S3. The additional bump-like features correspond to the topological Hall signal31. The recalculated amplitude of the AHE reaches 5-20 S/cm and correlates with the quality of the crystal, as shown in Supplementary Fig. 8. The error of the magneto-transport is caused by thermal noise and current source noise and it is negligible. We provide more details on magneto-transport measurements in Supplementary Note 2.

Magnetometry measurements

For the magnetic characterization of the Mn5Si3 thin films, a Quantum Design MPMS7-XL SQUID magnetometer with a reciprocating sample option has been used. The unpatterned sample was cleaned prior to the measurement and mounted using plastic straws. The field-dependent magnetization has been measured at different temperatures for magnetic field strengths between ± 5 T (cp. Fig. 4b) applied out of the sample plane. The signal is dominated by the diamagnetism of the silicon substrate, this diamagnetic contribution is, however, negligible in the small magnetic field (inset). The error of the magnetometry measurements is relatively large because of subtracting the signal from the substrate and the sample holder and we estimate it to be 5 mμB/u.c.

Magnetic relativistic density functional theory calculations

The density functional theory calculations were performed using the VASP package54 employing the projector augmented plane wave method55. We have set the energy cut-off of the plane wave basis at 520 eV, used the PBE exchange-correlation functional56, and the wavevector grid 9 × 9 × 12. For the calculations presented in the main text we have used the in-plane high-temperature lattice constant a=6.902Å30 and the c-lattice constant corresponding to the bulk collinear phase at T=70 K and our epilayers at T=170 K (4.795Å). Fermi surface calculations of the Mn5Si3 are shown in Supplementary Fig. 12.

Berry curvature calculations of Hall conductivity

We have constructed a maximally localized Wannier function and the effective tight-binding model by using the Wannier90 code57. We have calculated the intrinsic anomalous Hall conductivity in WannierTools package58 by employing the Berry curvature formula. We have used a fine-mesh of 320 × 320 × 240 Brillouin zone sampling points and have checked the convergence. Berry curvatures in the Mn5Si3 are shown in Supplementary Fig. 13.