Orbital character of the spin-reorientation transition in TbMn₆Sn₆

Abstract

Ferromagnetic (FM) order in a two-dimensional kagome layer is predicted to generate a topological Chern insulator without an applied magnetic field. The Chern gap is largest when spin moments point perpendicular to the kagome layer, enabling the capability to switch topological transport properties, such as the quantum anomalous Hall effect, by controlling the spin orientation. In TbMn₆Sn₆, the uniaxial magnetic anisotropy of the Tb³⁺ ion is effective at generating the Chern state within the FM Mn kagome layers while a spin-reorientation (SR) transition to easy-plane order above T_SR = 310 K provides a mechanism for switching. Here, we use inelastic neutron scattering to provide key insights into the fundamental nature of the SR transition. The observation of two Tb excitations, which are split by the magnetic anisotropy energy, indicates an effective two-state orbital character for the Tb ion, with a uniaxial ground state and an isotropic excited state. The simultaneous observation of both modes below T_SR confirms that orbital fluctuations are slow on magnetic and electronic time scales < ps and act as a spatially-random orbital alloy. A thermally-driven critical concentration of isotropic Tb ions triggers the SR transition.

Introduction

Two-dimensional (2D) kagome metals possess both flat electronic bands and Dirac band crossings that provide an adaptable framework for the emergence of a variety of correlated and topological phases¹. For example, the Haldane model predicts that a Chern insulator may be generated (without an applied magnetic field) in a ferromagnetic (FM) kagome layer when spin-polarized Dirac crossings are gapped by spin-orbit coupling (SOC)^2,3. Generating this topological phase in real (3D) materials requires that the Dirac bands are composed of orbitals with 2D character (such as ${d}_{{x}^{2}-{y}^{2}}$) in order to minimize coupling between kagome layers. For a 2D Dirac crossing, the Chern gap is largest when spins point perpendicular to the kagome layer.

This scenario has generated interest in RMn₆Sn₆ (R166) compounds where the magnetic anisotropy of the rare-earth ion (R) plays the crucial role of controlling the spin orientation of the FM Mn kagome layers. The family of R166 is known for their combination of unique magnetic instabilities^{4,5,6,7,8,9,10,11,12} and the topological character of their electronic bands^13,14,15. Thus, they provide a promising framework for highly tunable magnetic topological insulators and semimetals.

Hexagonal R166 compounds consist of alternating Mn kagome and R triangular layers, and starkly different properties are found for magnetic and non-magnetic R ions. For the latter, the magnetism in compounds such as Y166 is defined by FM Mn kagome layers with weak easy-plane anisotropy. Complex incommensurate helical magnetism is formed by competing FM and antiferromagnetic (AF) coupling between kagome layers, resulting in a rich field and temperature phase diagram demonstrating the topological Hall effect^6,10,11,12. For magnetic R ions, strong AF R-Mn coupling provides a collinear 3D ferrimagnetic network where the orbital ground state of the R ion determines either a uniaxial (Tb), easy-cone (Ho,Dy), or easy-plane (Gd, Er) magnetic anisotropy¹⁶. The variability of the magnetic anisotropy for the different rare-earth ions originates from orbital ground states enabled by large fourth-order terms in the crystalline electric field (CEF) potential in the R166 structure¹⁶. It was recognized that the unique uniaxial magnetism of Tb166 is supremely effective at gapping spin-polarized Dirac band crossings, potentially creating a two-dimensional Chern insulator that is a realization of the spinless Haldane model¹³. However, Tb166 remains metallic, and achieving quantized Hall conductivity of a Chern insulator requires methods to control the Fermi energy by chemical substitution or gating of thin film samples.

These materials are also characterized by instabilities in the moment direction. In Tb166, a spin-reorientation (SR) transition from uniaxial to easy-plane collinear ferrimagnetism occurs above T_SR = 310 K^17,18. This first-order SR transition may also be driven by a magnetic field applied along the magnetic hard axis. The prospects for rare-earth engineering of R166 compounds, which utilize mixed orbital moments, anisotropies, and exchange couplings, promises to create magnetic instabilities necessary to enable switching of topological magnets¹⁵.

To better understand the role of R orbital states and their temperature evolution, we investigate the intrinsic nature of the SR transition in Tb166 using inelastic neutron scattering (INS). This transition has been described in terms of the competition between easy-plane Mn magnetic anisotropy and a temperature-dependent uniaxial Tb anisotropy^19,20,21,22. In this description, fluctuations of the Tb moment direction with increasing temperature weaken the uniaxial Tb anisotropy, ultimately causing the SR transition²³.

Here, our INS data provide an insightful microscopic picture of the SR transition that is consistent with this scenario. At low temperatures, the observation of a propagating Tb spin wave mode near 25 meV confirms both the strong uniaxial ground state anisotropy of the Tb ion and AF coupling between the Tb and Mn sublattices. Above T_SR, the Tb mode appears below 10 meV, which is consistent with an isotropic Tb ion. At intermediate temperatures, the simultaneous appearance of both the upper (uniaxial) and lower (isotropic) Tb modes are fingerprints of unusual Tb orbital dynamics. Within a random-phase approximation (RPA) description of the local Tb CEF states coupled to Mn, one expects a spectrum of orbital exciton modes (propagating, magnon-like quasiparticles comprised of Tb CEF excitations) to appear at high temperatures. However, the observation of only two Tb modes suggests a simple two-state orbital model consisting of a uniaxial ground state and an isotropic excited state. Thermal occupancy of the isotropic state weakens the magnetic anisotropy and drives the SR transition. Classical Monte Carlo simulations show that the simultaneous observation of both modes is consistent with a random, quasi-static distribution of these orbital states, suggesting that Tb orbital quantum states have a longer lifetime than the ps time scale of the Tb spin waves.

Results

Experimental data

Tb166 has been shown to host a complex spin wave dispersion characterized by large intralayer ferromagnetic Mn-Mn exchange coupling and competing Mn-Mn and Mn-Tb interlayer couplings with excitations that extend beyond 200 meV²⁴. Figure 1 shows INS measurements of the low-energy dynamical susceptibility of intralayer excitations along the (H,0,1), (H,0,2), and (H,0,3) directions as a function of energy for different temperatures. At low temperatures (≤100 K), well within the uniaxial ferrimagnetic ground state, two branches are observed at low energies (<30 meV), as shown in Fig. 1a and b. The very steep branch with a spin gap of Δ = 6.5 meV corresponds to acoustic magnons that propagate principally within the Mn kagome layer. The narrow branch near 25 meV corresponds to spin waves propagating within the Tb triangular layer (upper Tb mode).

Fig. 1: Temperature dependence of the low-energy spin excitations in TbMn₆Sn₆.

The evolution of the low-energy ferrimagnetic spin waves, plotted as χ^″(q, E)/E. Inelastic neutron scattering (INS) data are shown along the a (H, 0, 1) and b (H, 0, 2) directions at T = 100, 150, 200, 240, 300, and 350 K through the spin-reorientation (SR) transition at T_SR = 310 K. Insets at higher energy in a and b correspond to data along (H, 0, 3). At 100 K, the dashed and solid lines correspond to the ground state dispersion as obtained from fits to the Heisenberg model described in Ref. ²⁴. Calculations of the spin dynamics in the random-phase approximation (RPA) are shown along the c (H, 0, 1) and d (H, 0, 2) directions at the same temperatures as the data. In a and b in the SR-phase at 350 K, the dashed and solid lines correspond to the RPA dispersions obtained using the Heisenberg parameters and a magnetically isotropic Tb ion. Thinner lines at other temperatures are guides to the eye.

Upon increasing the temperature towards T_SR, Fig. 1a and b show that several phenomena are observed. First, the spin gap gradually closes to zero at T_SR. Second, the Tb mode at 25 meV begins to soften and its intensity is strongly reduced, vanishing at T_SR. Third, broad excitations associated with orbital excitons begin to appear below 10 meV at 150 K and grow in strength. The low-energy features coalesce to form a collective spin wave branch between 5 meV and 10 meV above T_SR which we identify as the lower Tb mode.

Figure 2 shows the evolution of the spin gap at q = (0,0,2), where the intensity of the Mn acoustic magnon is very strong, upon approach to the SR transition. Up to 100 K, the low-energy dispersion is largely the same as the ground state dispersion at T = 7 K reported in Ref. ²⁴ and is shown as the dashed line in Fig. 2a. Above 100 K, the spin gap begins to close. Figure 2b shows the low-energy spectra cut through (0,0,2) that illustrate more clearly that the gap follows a roughly linear reduction above 100 K (Fig. 2d). As described below, the thermally-driven reduction of the spin gap is associated with spin fluctuations that decrease the average Tb sublattice magnetization, consistent with neutron diffraction measurements²⁵, and should become gapless in the easy-plane phase above T_SR. Figure 2c and d show that the lower Tb mode develops with a significant gap at (1,0,2) of almost 6 meV at 150 K, which softens as T_SR is approached but retains a 4 meV gap in the SR-phase at 350 K.

Fig. 2: Temperature dependence of the spin gap in TbMn₆Sn₆.

a The evolution of the acoustic mode intralayer spin wave dispersion along q = (H, 0, 2) at T = 100, 150, 200, 240, 300, and 350 K through the spin-reorientation (SR) transition at T_SR = 310 K. The dashed line at T = 100 K corresponds to the ground state dispersion as obtained from fits to the Heisenberg model described in Ref. ²⁴. b The inelastic neutron scattering spectra at (0,0,2) are plotted as χ^″(q, E)/E for several temperatures up to (solid lines) and above T_SR (dashed lines). c INS spectra at (1,0,2) for several temperatures. d The temperature dependence of the spin gap energy at (0, 0, 2) and (1, 0, 2) as obtained from the maxima in panels b and c. The dashed lines in d correspond to random-phase approximation (RPA) calculations in the uniaxial phase. Observation of dynamical features in the hashed area are obscured by the instrumental elastic resolution. Error bars are statistical in origin and represent one standard deviation.

Figure 2b shows that warming above T_SR (from 300 K to 350 K) leads to an immediate two-fold reduction of the susceptibility. As the dipole scattering factor for INS measures the magnetic moment (m) components perpendicular to q = (0,0,2), this reduction reflects the change in the moment direction on the transverse spin fluctuations since ${{{{{{{\bf{m}}}}}}}}\parallel \hat{z}$ gives ${\chi }^{{\prime\prime} }(T < {T}_{SR})={\chi }_{xx}^{{\prime\prime} }+{\chi }_{yy}^{{\prime\prime} }=2{\chi }_{xx}^{{\prime\prime} }$ and ${{{{{{{\bf{m}}}}}}}}\perp \hat{z}$ gives ${\chi }^{{\prime\prime} }(T > {T}_{SR})={\chi }_{xx}^{{\prime\prime} }$.

Figure 3a shows the temperature dependence of the Tb modes through the M-point of the hexagonal Brillouin zone boundary at (1/2, 0, L) as a function of L. At the zone boundary, the acoustic Mn mode is absent and the Tb modes along this direction are dispersionless, allowing an unobstructed observation of the crossover from the Tb upper mode to the lower mode with the coexistence of the two modes at intermediate temperatures. This is more clearly seen in Fig. 3b, which displays the dynamical susceptibility at (1/2,0,3) as a function of temperature and energy.

Fig. 3: Temperature dependence of the zone-boundary spin excitations in TbMn₆Sn₆.

a The evolution of the upper and lower Tb spin wave modes along the (1/2, 0, L) direction at T = 100, 150, 200, 240, 300, and 350 K through the spin-reorientation (SR) transition at T_SR = 310 K. Inset shows data along the (3/2, 0, L) direction. b The inelastic neutron scattering data at (1/2,0,3) as a function of energy and temperature with intensities plotted as χ^″(E). The filled symbols indicate that the energy of the upper mode drops in a similar fashion to the spin gap. c The integrated susceptibility of the upper Tb mode as a function of temperature compared to random-phase approximation (RPA) calculations. d, e The structure factor of the lower mode at 6 meV along (1/2, 0, L) at 300 K and 350 K, respectively. The solid red lines are calculations of the dipole factor times the Tb magnetic form factor for transverse excitations with m_Tb∥[0,0,1] (d) and m_Tb⊥[0, 0, 1] (e) averaged over equivalent domains. Sharp excitations seen at L = ± 6 in panels a, d, and e are phonons. Error bars are statistical in origin and represent one standard deviation.

The susceptibility of the upper and lower Tb modes has a minimum at (1/2, 0, L = 0) when T < T_SR, as shown in Fig. 3a and d. This minimum is again a consequence of the sensitivity of the dipole factor to the Tb moment direction (m_Tb). Calculations of the dipole factor times the Tb magnetic form factor clearly indicate that both the upper and lower Tb modes are transverse excitations with ${{{{{{{{\bf{m}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}\parallel \hat{z}$ when T < T_SR. Upon warming above T_SR, Fig. 3a and e indicates a susceptibility maximum at (1/2, 0, 0), consistent with the form factor for transverse excitations of an in-plane moment configuration with ${{{{{{{{\bf{m}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}\perp \hat{z}$.

Model Hamiltonian

The evolution of the upper and lower Tb modes is similar to the appearance of orbital excitons that are reported to occur in other rare-earth magnets, such as PrTl₃^26,27 and TbSb²⁸, and also transition metal compounds^29,30. These excitons arise from the thermal population of low-lying CEF states which are predominantly local in character, but propagate from site-to-site due to exchange coupling. Here, we analyze the spin excitations in Tb166 using a similar RPA approach as outlined in the above references.

A minimal description of the magnetic Hamiltonian is given by ${{{{{{{\mathcal{H}}}}}}}}={{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Mn}}}}}}}}}+{{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{ex}}}}}}}}}$ and includes terms that describe the local CEF states of the Tb ion with total angular momentum J = 6 and spin S = 3 (${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}$), the single-ion anisotropy of the Mn ion with spin s = 1 (${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Mn}}}}}}}}}$), and the isotropic (Heisenberg) exchange couplings within and between the Mn-Mn and Mn-Tb magnetic sublattices (${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{ex}}}}}}}}}$).

The Heisenberg Hamiltonian is

$${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{ex}}}}}}}}}=\mathop{\sum}\limits_{i,j}{{{{{{{{\mathcal{J}}}}}}}}}_{ij}^{MM}{{{{{{{{\bf{s}}}}}}}}}_{i}\cdot {{{{{{{{\bf{s}}}}}}}}}_{j}+{{{{{{{{\mathcal{J}}}}}}}}}^{MT}\mathop{\sum}\limits_{\langle i < j\rangle }{{{{{{{{\bf{s}}}}}}}}}_{i}\cdot {{{{{{{{\bf{S}}}}}}}}}_{j}$$

(1)

where ${{{{{{{{\mathcal{J}}}}}}}}}_{ij}^{MM}$ represent various, and quite strong, intralayer and interlayer magnetic couplings between Mn spins (s), as described in detail in Ref. ²⁴. Here, ${{{{{{{{\mathcal{J}}}}}}}}}^{MT} > 0$ is the AF coupling between neighboring Mn and Tb spins (S) that result in tightly bound Mn-Tb-Mn collinear ferrimagnetic trilayers.

DFT calculations¹⁶ and the analysis of magnetization data near the SR-transition^19,21 suggest that a Tb CEF Hamiltonian

$${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}={B}_{2}^{0}{{{{{{{{\mathcal{O}}}}}}}}}_{2}^{0}+{B}_{4}^{0}{{{{{{{{\mathcal{O}}}}}}}}}_{4}^{0}$$

(2)

where ${{{{{{{{\mathcal{O}}}}}}}}}_{l}^{0}$ are the Steven’s operators, is sufficient to explain the SR transition. Figure 4a shows the level scheme for arbitrary CEF parameters using Segal and Wallace notation³¹ where ${B}_{2}^{0}=W(1-|y|)$ and $60{B}_{4}^{0}=Wy$. The CEF states are labeled by $\left|{m}_{J}\right\rangle$ which are good quantum numbers for ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{Tb}}}}}}}}}$. Determination of the CEF and exchange parameters and a figure showing the magnetic structure and interactions can be found in Supplementary Notes 1 and 2 and Supplementary Fig. 1.

Fig. 4: Crystal field model for TbMn₆Sn₆.

a Crystal electric field (CEF) energy levels for a J = 6 Tb ion with dominant ${B}_{2}^{0}$ and ${B}_{4}^{0}$ terms. The orange arrow indicates the experimental values with ${B}_{2}^{0}=-0.0347$ meV and ${B}_{4}^{0}=-0.00143$ meV. b The Zeeman splitting of the CEF levels due to the molecular field acting on the Tb ion. The vertical line shows the estimated ground state molecular field strength of ${B}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}=22.2$ meV and the blue arrow labels the only dipole-allowed transition out of the ground state. Orbital states are labeled as $\left|{m}_{J}\right\rangle$. c Tb state populations as a function of temperature, where the molecular field decreases according to the measured Mn sublattice magnetization²⁵. T_SR indicates the spin-reorientation transition.

In Fig. 4a, the $\left|\pm 6\right\rangle$ CEF ground state is consistent with the uniaxial character of the Tb ion at low temperatures. We now consider the local Tb orbital dynamics by treating ${{{{{{{{\mathcal{H}}}}}}}}}_{{{{{{{{\rm{ex}}}}}}}}}$ within a mean-field approximation. A molecular field Hamiltonian acting on the Tb site is given by ${{{{{{{{\mathcal{H}}}}}}}}}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}={B}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}{S}_{z}$, where ${B}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}=12{{{{{{{{\mathcal{J}}}}}}}}}^{MT}\langle {s}_{z}\rangle$ and points along the − c-axis (opposite to the Tb moment direction) in the ferrimagnetic ground state. ${{{{{{{{\mathcal{H}}}}}}}}}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}$ creates a Zeeman splitting of $\left|\pm {m}_{J}\right\rangle$ CEF states, as shown in Fig. 4b. The Heisenberg parameters in Supplementary Table 1 provide ${B}_{MF}^{{{{{{{{\rm{Tb}}}}}}}}}=$ 22.2 meV at low temperatures, as indicated by the vertical dashed line in Fig. 4b. The large molecular field results in a Zeeman-like spectrum that is distorted by the CEF potential.

In typical magnets, increasing the temperature leads to fluctuations that decrease the average magnetization, causing a concomitant decrease in the molecular field. What is unique about Tb166 is that the Mn sublattice magnetization initially drops very slowly with increasing temperature²⁵ due to the strong ${{{{{{{{\mathcal{J}}}}}}}}}^{MM}$ couplings. As a result, the strength of the molecular field is maintained while orbital excited states on the Tb ion become thermally populated. As shown in Fig. 4b and c, this initially leads to thermal occupancy of the $\left|-5\right\rangle$ state, which deviates from the linear spin-wave theory approximation and should result in damping, as observed in the upper Tb mode beginning around 100 K. As the temperature is increased further, thermal depopulation of the $\left|-6\right\rangle$ ground state becomes severe, reaching only about 50% occupancy at 250 K, which explains the weakening of the upper Tb mode intensity that is comprised of excitations out of the ground state. The thermally excited CEF states will also generate orbital excitons which disperse due to ${{{{{{{{\mathcal{J}}}}}}}}}^{MT}$.

RPA calculations

We can test these ideas using a mean-field RPA approach,^27,28 which couples thermally excited CEF levels through the magnetic exchange interactions listed in Supplementary Table 1. Figure 1c and d shows results from RPA calculations of the dynamical susceptibility at various temperatures along the (H, 0, 1) and (H, 0, 2) directions. This also includes RPA calculations performed in the easy-plane state at T > T_SR where the molecular field lies in the ab-plane, resulting in a different spectrum of local Tb states.

The RPA results demonstrate several of the observed features described above, including closure of the spin gap [see Figs. 1 and 2d], decreasing strength of the upper Tb mode (Fig. 3c), and the appearance of orbital excitons below 10 meV. These comparisons also reveal several limitations of the RPA approach. Most notably, RPA predicts narrowing (reduced dispersion) and hardening of the upper Tb mode with increasing temperature, whereas the INS data indicate narrowing and softening along with the typical development of lifetime broadening. One source of discrepancy is that RPA underestimates the decrease of the Mn sublattice magnetization with temperature, thereby overestimating the molecular field and its temperature dependence. More importantly, RPA predicts the appearance of many modes (eg. near 15 meV), whereas only two Tb modes are observed in the INS data. These disagreements are also likely a limitation of RPA, which truncates higher-order mode-mode coupling terms that result in decay and which should become increasingly important at higher temperatures when Tb thermal fluctuations are large.

Orbital quantum alloy model

Here, we define a simple effective two-state orbital model that ensures the appearance of only two Tb modes. The ground state singlet Ψ_u has uniaxial anisotropy and the excited state Ψ_i is highly degenerate (2J = 12 states) and magnetically isotropic, as shown in Fig. 5b. Normally, magnetic isotropy is recovered when all CEF states have equal occupancy, most commonly occurring from Boltzmann statistics at high temperatures (see Fig. 4c). Whereas a uniaxial ground state Ψ_u describes the low-temperature data consisting only of the upper Tb mode, the assumption of completely isotropic Tb ions describes the SR-phase quite well, where only the lower Tb mode is found in RPA calculations, as shown in Fig. 5a. After the renormalization of ${{{{{{{{\mathcal{J}}}}}}}}}^{MT}$ because of reduction of the Mn sublattice magnetization by 20%, excellent agreement with the INS data in the SR-phase is achieved. In this two-state model, the energy difference of the upper and lower mode is just the anisotropy energy, ${E}_{{\Psi }_{u}}-{E}_{{\Psi }_{i}}\approx 2{K}_{1}/J=15$ meV, where K₁ is the Tb anisotropy constant determined from INS data (see Supplementary Note 2).

Fig. 5: Quantum alloy model for TbMn₆Sn₆.

a Random-phase approximation (RPA) calculations of the excitations in the spin-reorientation phase for an isotropic Tb ion with ${B}_{2}^{0}={B}_{4}^{0}=0$. b Instantaneous configuration of a quantum alloy on the Tb triangular lattice. Magnetically uniaxial (Ψ_u) and isotropic (Ψ_i) quantum states of the Tb ion are split by the magnetic anisotropy energy (2K₁/J). c Classical Monte Carlo simulations of a quenched mixed anisotropy alloy with uniaxial fractions f = 1, 1/2, and 0.

To explain the INS data at intermediate temperatures where both modes are observed, the orbitally isotropic excited state must be present below T_SR. At a given instant, a Tb ion can be found in either of two different orbital quantum states, Ψ_u and Ψ_i, respectively, as shown in Fig. 5b, with a “concentration” determined by Boltzmann statistics. These orbital states must be long-lived on the ~THz time-scale of the spin wave frequencies, essentially behaving as a quenched magnetic binary alloy with mixed anisotropy (e.g., such as a Tb and Gd substitutional alloy). Surprisingly, classical Monte Carlo spin dynamics simulations of a quenched magnetic alloy capture the essential features of our observations as the concentration of the alloy changes, as shown in Fig. 5c.

Discussion

Microscopic CEF and magnetic exchange parameters obtained from INS data confirm an intuitive picture of the SR-transition in Tb166 where thermal softening of the uniaxial Tb anisotropy eventually drops below the Mn easy-plane anisotropy energy. INS data also reveal details of how the orbital dynamics drive this transition. RPA calculations that account for local Tb CEF orbital states and their exchange coupling can capture many essential features of the observed orbital dynamics that result in the closing of the spin gap, the disappearance of the upper Tb mode, and the appearance of dispersive low-energy excitations comprised of orbital excitons. The RPA calculations outline a rich spectra of these excitons which belie a much simpler observed spectrum only consisting of coexisting upper and lower Tb modes. We can reconcile this simpler observation by implementing a two-state orbital model for Tb, where a temperature-dependent fraction, f(T), of the Tb ions are magnetically uniaxial and 1 − f(T) are isotropic on average. The isotropic state can be considered as a quasi-classical coherent state formed from a superposition of angular momentum states (ostensibly due to mode-mode coupling) which replaces the discrete CEF level spectrum of excited states with a single, highly degenerate level. In this scenario, the material behaves as a two-state quantum orbital alloy.

For thermodynamic properties, this distinction has no consequences due to ensemble averaging. We predict, for example, that the SR transition occurs at f(T_SR) ≈ 30% when the average magnetic anisotropy of Tb and Mn ions is zero (see Fig. 4c and Supplementary Notes 2 and 3 and Supplementary Figs. 4 and 5). This is in rough correspondence with the appearance of the SR-phase in Gd_1−xTb_xMn₆Sn₆ mixed anisotropy alloys with x ≈ 0.2³².

However, there are essential differences for the dynamical properties (such as spin waves) which are fast on the time scale of the orbital fluctuations. On these time scales, the quantum alloy behaves as a quenched random alloy and translational symmetry is broken by the instantaneous configuration of orbital states. In topological electronic materials, electronic time scales are very fast, and band structure calculations should consider quasi-static quantum alloy configurations which include broken translational symmetry. Whereas a classical binary alloy, such as Gd_1−xTb_xMn₆Sn₆, is only found in one of these random configurations and its phase diagram will be affected by percolation and domain effects, the quantum alloy will explore all configurations over time. All quantum materials properties must account for the configuration of quantum states in some way (e.g. Born-Oppenheimer and frozen phonon approximations), but Tb166 is special, since there are effectively two distinct magnetic/orbital quantum states. This makes Tb166 a simple, magnetic quantum binary alloy. More generally, the tunable state configuration of quantum alloys (with temperature, for example) can be used to accelerate our understanding of how disorder affects magnetic and electronic states under adiabatic conditions.

Methods

Neutron scattering

Tb166 crystallizes in the HfFe₆Ge₆-type structure with hexagonal space group P6/mmm (No. 191) and Mn, Sn1, Sn2, Sn3, and Tb ions, respectively, sitting at the 6i, 2e, 2d, 2c, and 1b Wyckoff positions, as shown in Supplementary Fig. 1. Neutron diffraction data were taken on the BT7 triple-axis spectrometer at the NIST Center for Neutron Research with a PG(002) monochromator and analyzer fixed at an energy of 14.7 meV (λ = 2.359 Å)³³. Vertical focusing was employed for the monochromator and both the monochromator and analyzer were in the flat horizontal focusing configuration. PG filters were used before and after the sample to reduce λ/2 neutrons in the beam. The Söller collimation before the monochromator, before the sample, after the sample, and before the analyzer were 25’-50’-50’-25’ full-width-at-half maximum, respectively. The sample used was a 30 mg single crystal with an 8′ intrinsic mosaic. Momentum transfers are reported in hexagonal coordinates as q[r. l. u. ] = (H, K, L) where $q[{{{{{\text{\AA }}}}}}^{-1}]=(4\pi H/(a\sqrt{3}),4\pi K/(a\sqrt{3}),2\pi L/c)$. The sample was sealed in an aluminum can with helium exchange gas to promote temperature uniformity and was aligned in the reciprocal space (H, H, L) scattering plane in hexagonal notation.

The onset of magnetic order with propagation vector k = (0, 0, 0) was observed at T_C = 420 K by the increase in intensity at integer (H, K, L) Bragg peak positions, as shown in Supplementary Fig. 2 and in agreement with previous neutron diffraction measurements⁷. The spin-reorientation transition, T_SR, can be seen just above 300 K as the sudden decrease in intensity of this and other (0, 0, L)-type Bragg peaks. Hysteresis was observed about T_SR, and to ensure this was not an artifact from the sample not equilibrating, a slow temperature sweep of 0.1 K/min was used both on cooling and warming at the (0,0,1) Bragg peak. As the sample is cooled through T_SR, the moments, which are within the ab-plane above T_SR, rotate towards the c-axis below T_SR adopting the magnetic space group P6/mm’m’ (No. 191.240) where both the Tb and Mn magnetic sublattices again are antiparallel but are confined to the c-axis (see Supplementary Fig. 1). This results in a sudden decrease of the (0,0,1) Bragg peak because neutrons are only sensitive to the components of ordered magnetic moments which are perpendicular to the scattering vector, q. The SR-transition extinguishes the intensity as the moments orient parallel to q = (0, 0, L), as shown in Supplementary Fig. 2(c) and (d). Additional scans along high symmetry directions were performed at 400 K, 300 K, and 150 K to check for the appearance of unexpected commensurate or incommensurate magnetic Bragg peaks away from integer (H, K, L) positions and no new peaks were observed. Below T_SR the temperature evolution of the Bragg peaks, such as (1,1,1) peak shown in Supplementary Fig. 2(a), evolve smoothly within the uniaxial ferrimagnetic ground state.

Inelastic neutron scattering was performed on the Wide Angular-Range Chopper Spectrometer (ARCS) located at the Spallation Neutron Source at Oak Ridge National Laboratory using single crystals of Tb166 grown from excess Sn using the flux method, with details described in Ref. ²⁴. An array of 9 crystals with a total mass of 2.564 grams was co-aligned with the hexagonal (H,0,L) scattering plane set horizontally, and attached to a top-loading closed-cycle refrigerator. The data were collected at various temperatures using an incident energy of E_i = 30 meV with an elastic resolution full-width-at-half-maximum of 1.4 meV. The sample was rotated around the vertical axis to increase the q coverage.

The INS data are presented in terms of the orthogonal hexagonal vectors (1,0,0) and (−1,2,0). The measured INS intensities are proportional to the spin-spin correlation function $S({{{{{{{\bf{q}}}}}}}},E)={\sum }_{\alpha \beta }({\delta }_{\alpha \beta }-{\hat{q}}_{\alpha }{\hat{q}}_{\beta }){S}_{\alpha \beta }({{{{{{{\bf{q}}}}}}}},E)$, where E is the energy transfer and ${\hat{q}}_{\alpha }$ is the cartesian component of the (unit) momentum transfer vector. In order to remove the trivial temperature dependence caused by the thermal population factor, the INS data are reported as the dynamical susceptibility ${\chi }^{{\prime\prime} }({{{{{{{\bf{q}}}}}}}},E)=S({{{{{{{\bf{q}}}}}}}},E){[1-\exp (-E/{k}_{B}T)]}^{-1}$ or as χ^″(q, E)/E. To improve statistics, the data have been symmetrized with respect to the crystallographic space group P6/mmm.

Random-phase approximation calculations

The RPA approach begins by considering the local-ion Green’s functions

$${g}_{\alpha \beta }^{i}(\omega )=\mathop{\sum}\limits_{m,n}\frac{({f}_{m}-{f}_{n})\langle n|{J}_{\alpha }^{i}|m\rangle \langle m|{J}_{\beta }^{i}|n\rangle }{\omega -i\gamma+{\epsilon }_{m}-{\epsilon }_{n}}.$$

(3)

where $\left|n\right\rangle,{\epsilon }_{n}$, f_n are the eigenstates, eigenvalues, and thermal occupancies obtained from the local-ion Hamiltonians ${{{{{{{{\mathcal{H}}}}}}}}}_{i}+{{{{{{{{\mathcal{H}}}}}}}}}_{MF}^{i}$ for i = Tb or Mn, respectively. The sites are coupled through the pairwise exchange interactions, giving a set of coupled linear equations for the propagating Green’s function

$${G}_{\alpha \beta }^{ij}({{{{{{{\bf{q}}}}}}}},\omega ) \,=\, {g}_{\alpha \beta }^{i}(\omega ){\delta }_{ij}+\mathop{\sum}\limits_{k}{{{{{{{{\mathcal{J}}}}}}}}}^{ik}({{{{{{{\bf{q}}}}}}}})\mathop{\sum}\limits_{\gamma }{g}_{\alpha \gamma }^{i}(\omega ){G}_{\gamma \beta }^{kj}({{{{{{{\bf{q}}}}}}}},\omega )$$

(4)

where ${{{{{{{{\mathcal{J}}}}}}}}}^{ik}({{{{{{{\bf{q}}}}}}}})={\sum }_{lmn}{{{{{{{{\mathcal{J}}}}}}}}}^{ik}\exp [-i{{{{{{{\bf{q}}}}}}}}\cdot ({{{{{{{{\bf{d}}}}}}}}}_{i}-{{{{{{{{\bf{d}}}}}}}}}_{k}-{{{{{{{{\bf{R}}}}}}}}}_{lmn})]$. The dynamical magnetic susceptibility (weighted by the magnetic cross-section) is related to the summation of the imaginary part of the pairwise Green’s functions,

$${\chi }_{\alpha \beta }^{{\prime\prime} }({{{{{{{\bf{q}}}}}}}},\omega ) \propto ({\delta }_{\alpha \beta }-{\hat{q}}_{\alpha }{\hat{q}}_{\beta })\\ \times \mathop{\sum}\limits_{ij}[{g}^{i}{F}^{i}({{{{{{{\bf{q}}}}}}}})][{g}^{j}{F}^{j}({{{{{{{\bf{q}}}}}}}})]{{{{{{{\mathcal{I}}}}}}}}m[{G}_{\alpha \beta }^{ij}({{{{{{{\bf{q}}}}}}}},\omega )].$$

(5)

Here gⁱ and Fⁱ(q) are the Lande g-factor and magnetic form factor, respectively. The unpolarized INS data measures the summation of ${\chi }_{\alpha \beta }^{{\prime\prime} }$ over α and β.

Classical Monte Carlo simulations

Classical Monte Carlo (MC) simulations were performed using the UppASD package³⁴. 31 × 31 × 4 unit cells containing one Tb and six Mn atoms form the magnetic lattice, giving a total of 26908 spins. We treat the Tb sites as a random alloy with a concentration of f sites having a uniaxial single-ion anisotropy of D^T = − 1.28 meV and 1 − f sites having an anisotropy of D^T = 0. All Mn sites have an easy-plane anisotropy of D^M = 0.44 meV. A uniaxial collinear ferrimagnetic phase was prepared and annealed at 10 K in 40,000 Monte Carlo steps. Spin dynamics simulations were performed for different alloy concentrations, averaging over 10 different ensembles, using Landau-Lifshitz-Gilbert dynamics with 40,000 Monte Carlo time steps of 3 × 10⁻¹⁶ seconds and correlation functions S(r, t) evaluated over 2000 time steps of 6 × 10⁻¹⁵ seconds. The correlation functions are Fourier transformed to obtain S(q, ω). Due to the essential difference between classical and quantum treatment of local Tb states and the resultant spin fluctuations, all MC simulations are done at 10 K. All other simulation parameters are shown in Supplementary Table 1.