Diffusive excitonic bands from frustrated triangular sublattice in a singlet-ground-state system

Abstract

Magnetic order in most materials occurs when magnetic ions with finite moments arrange in a particular pattern below the ordering temperature. Intriguingly, if the crystal electric field (CEF) effect results in a spin-singlet ground state, a magnetic order can still occur due to the exchange interactions between neighboring ions admixing the excited CEF levels. The magnetic excitations in such a state are spin excitons generally dispersionless in reciprocal space. Here we use neutron scattering to study stoichiometric Ni₂Mo₃O₈, where Ni²⁺ ions form a bipartite honeycomb lattice comprised of two triangular lattices, with ions subject to the tetrahedral and octahedral crystalline environment, respectively. We find that in both types of ions, the CEF excitations have nonmagnetic singlet ground states, yet the material has magnetic order. Furthermore, CEF spin excitons from the tetrahedral sites form a dispersive diffusive pattern around the Brillouin zone boundary, likely due to spin entanglement and geometric frustrations.

Introduction

In most magnetic materials, the ground state of electron orbital states of magnetic ions in the crystal electric field (CEF) produced by the surrounding charge anion neighbors is magnetic with a finite moment and nonzero spin^1,2. The long-range magnetic ordered structure and ordering temperature are determined by the magnetic exchange interactions (J), and the ordered moment direction is controlled by (typically small) easy-axis anisotropy gap². However, there are also materials where the ground state of CEF levels of magnetic ions is a nonmagnetic singlet. Here, magnetic ordering and properties are sensitive to the ratio of magnetic exchange J to single-ion anisotropy (SIA, D) which is controlled by the CEF energy level of the first excited state^3,4,5. With negligible magnetic exchange ($J\ll D$), the system is paramagnetic at all temperatures. For a relatively large magnetic exchange ($J\gg D$), magnetic order will be induced through a polarization instability of the singlet-ground state, termed an induced moment^3,4,5,6,7,8. In this case, the basic magnetic excitations that describe the CEF transitions propagating through the lattice are called spin excitons^3,4,5,6,7, analogous to electronic excitons that are bound states of an electron and a hole in a solid⁹. Spin excitons are fundamentally different from spin waves (magnons), which are strongly dispersive collective modes associated with spin precession on the lattice of magnetically ordered materials, and which disappear above the magnetic ordering temperature for isotropic Heisenberg magnets. In most cases, spin excitons originate from CEF levels of a localized single ion. Therefore, they are expected to be dispersionless in reciprocal space and well defined in both the magnetically ordered and paramagnetic states. However, when dispersive spin excitons are observed, the dispersion of these excitations can reveal unique information concerning magnetic exchange interactions between the localized ionic sites (spin–spin entanglement) and their relationship with the magnetically ordered state^{3,4,5,6,7,10,11}.

Here we use thermodynamic and neutron scattering experiments to study stoichiometric honeycomb lattice antiferromagnetic (AF) ordered magnet Ni₂Mo₃O₈, where Ni²⁺ ions form a bipartite honeycomb lattice comprised of two triangular lattices, in the tetrahedral and octahedral crystalline environment, respectively (Fig. 1a–c)^12,13. We find that CEF levels of Ni²⁺ ions from both tetrahedral and octahedral environments have a nonmagnetic spin-singlet-ground state but with very different single-ion anisotropy energy scales for the two sites. Spin excitations of CEF levels (spin excitons) from the Ni²⁺ triangular tetrahedral sites form a diffusive pattern around the Brillouin zone (BZ) boundary in the AF and paramagnetic states in momentum space. Therefore, Ni₂Mo₃O₈ realizes a novel situation in which the exchange energy falls between two very different single-ion energies, leading to a cooperative mechanism for magnetic order and strongly dispersing excitons associated with the larger single-ion anisotropy of the tetrahedral sites. Due to this hierarchy of energy scales, the excitons can persist even when the magnetic order is destroyed above the Néel temperature. In this regime, the excitons are strongly scattered from spin fluctuations and give rise to a distinct mechanism for dispersive diffusive scattering, hosting a unique coexistence of heavy particles (tetrahedral excitons) propagating in a frustrated background of light but dense (octahedral) spins due to spin entanglement and geometric frustrations.

Fig. 1: Structure, Brillouin zone, magnetic Bragg peak, d.c. susceptibility and specific heat of Ni₂Mo₃O₈.

a A schematic of the structure of Ni₂Mo₃O₈ with a-, b-, c-axis labeled. b The octahedral coordinated Ni ions (blue) and the tetrahedral coordinated Ni ions (green) form a quasi-2D honeycomb lattice. c, d Side (along the b-axis) and top (along the c-axis) view of the magnetic structure of Ni₂Mo₃O₈. The black frames indicate the size of magnetic unit cell and black arrows denote magnetic exchange couplings along different directions. e Temperature dependence of the intensities in arbitrary units (a. u.) of two magnetic Bragg peaks at (3/2,0,1) and (1/2,−1/2,0). Inset shows a 2D color plot in the [H, 0, L] plane, in which magnetic Bragg peaks are indicated by red arrows. The vertical error bars are statistical errors of 1 standard deviation. f The d.c. susceptibility with 1–8 T magnetic fields perpendicular to the c-axis. Data are shifted by magnetic field × 0.01 emu / mol T for clarity. Magnetic transition is suppressed in the 8 T field. g The magnetic contribution to the specific heat C_mag with 0-8 T magnetic fields perpendicular to the c-axis. Magnetic transition is suppressed in the 8 T field. h Temperature versus magnetic field phase diagram from d.c. susceptibility and specific data.

Results

Crystal structure, magnetic order, susceptibility, and specific heat of Ni₂Mo₃O₈

The M₂Mo₃O₈ (M = Fe, Mn, Ni, Co, Zn) compounds have drawn increasing attention due to their multiferroic properties^{12,13,14,15,16}. The crystal structure of M₂Mo₃O₈ consists of magnetic bipartite honeycomb M-O layers, separated by sheets of Mo⁴⁺ layers (Fig. 1a), where the Mo⁴⁺ ions inside each layer are trimerized and form a singlet. The two M²⁺ sites have different oxygen coordination, with one site being an MO₆ octahedron and the adjacent one being an MO₄ tetrahedron. In this family, Ni₂Mo₃O₈ was studied as a platform to explore the physics of geometrically frustrated lattice^12,13. Neutron powder diffraction experiments reveal that both the MO₆ octahedron and the MO₄ tetrahedron each form perfect triangular lattices with no inter-site disorder, and the system orders antiferromagnetically with a Néel temperature of T_N = 5.5 K^12,13. The magnetic structure is stripe like within the Ni-O plane, and zig-zag like along the c-axis with different ordered moments for octahedral and tetrahedral Ni sites (Fig. 1c and d). Previous single-ion CEF analysis suggests that both octahedral and tetrahedral Ni sites have nonmagnetic singlet-ground states, and the first excited magnetic doublets are at 7.8 and 23 meV, respectively¹².

Even though the Ni ions appear to form a simple (bipartite) honeycomb lattice in Ni₂Mo₃O₈ (Fig. 1b–d), this compound should be viewed rather as two inter-penetrating triangular lattices formed by NiO₆ octahedron and NiO₄ tetrahedron. In Ni₂Mo₃O₈, magnetic order is assumed to arise from the relatively large magnetic exchange coupling between the octahedral and tetrahedral Ni sites (denoted as J₁ in Fig. 1d) in comparison with the energy of the CEF level of the Ni octahedral site¹². However, there are no inelastic neutron scattering (INS) experiments to date to identify the CEF levels of the Ni octahedral and tetrahedral sites, and prove that the ground state is indeed a singlet. Here we report magnetic susceptibility, heat capacity, X-ray diffraction, and INS experiments on single crystals of Ni₂Mo₃O₈ and NiZnMo₃O₈ grown by the chemical vapor transfer method. Our careful single crystal X-ray diffraction experiments reveal that both the NiO₆ octahedra and NiO₄ tetrahedra form perfect triangular lattices with no inter-site disorder, and there is also no disorder on the Mo site (see Table 1 and Supplementary Information for details). The X-ray and neutron diffraction refinements also show that nonmagnetic dopant ions like Zn²⁺ prefer to occupy the tetrahedral sites¹². Consistent with previous neutron powder diffraction work^12,13, our neutron single crystal diffraction refinements find that the spins of Ni²⁺ ions form a stripy AF order (Fig. 1c and d) below T_N = 5.5 K (Fig. 1e), but with the ordered moments of tetrahedral and octahedral Ni sites being 1.47 and 1.1 μ_B, respectively, different from the previously reported values.

Table 1 Positions of atoms in Ni₂Mo₃O₈ as determined from single crystal X-ray diffraction

Figure 1f shows the temperature dependence of the d.c. susceptibility χ(T) with 1–8 T magnetic fields applied along the [1,1,0] direction perpendicular to the c-axis. The data in the 1 T field (orange line) shows a clear peak around T_N $\approx$ 6 K consistent with neutron data in Fig. 1e, while the data in the 8 T field (blue line) shows no evidence of a magnetic transition. In-plane susceptibility is much larger than the c-axis susceptibility (see Supplementary Fig. 3), indicating easy-plane anisotropy. The Curie-Weiss fitting to the in-plane and c-axis susceptibility above 200 K gives θ_CW⊥ = −454.12 ± 0.53 K and θ_CW,// = −100.52 ± 0.11 K, respectively, suggesting a quasi-2D system where the in-plane magnetic exchange is larger than the c-axis exchange. The difference between our analysis and previous results on single crystals mainly comes from the direction of the in-plane magnetic fields¹³. Figure 1g shows the temperature dependence of the magnetic contribution to the specific heat as a function of applied magnetic fields perpendicular to the c-axis. At zero field (red line), we see a typical λ-shaped transition around T_N $\approx$ 6 K. At 8 T field, there is no evidence of a magnetic transition above 2 K, consistent with Fig. 1f. Figure 1h shows the temperature-field phase diagram from our susceptibility and specific heat data.

Neutron scattering studies of spin waves and CEF levels

To search for the CEF levels of the tetrahedral and octahedral Ni sites and demonstrate that the ground state of Ni₂Mo₃O₈ is indeed a spin singlet, we carried out INS experiments on single crystalline and powdered samples with incident energies (E_i) of 2.5 meV, 3.7 meV, 40 meV, 250 meV, and 1.5 eV. At E_i = 2.5 meV, we see clear dispersive spin waves at 1.7 K (Fig. 2a) with two modes. This is consistent with the expectations from the linear spin-wave theory (LSWT) calculation, since there are four Ni²⁺ sites in the magnetic unit cell, resulting in two doubly-degenerate modes. The small anisotropic spin gap ($< 0.3$ meV) and overall spin-wave energy bandwidth of 1.5 meV are consistent with thermal dynamic data in Fig. 1f–h and T_N = 5.5 K. Figure 2b and c shows a powder averaged INS $S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)$ spectrum, where E and Q are energy and moment transfer, respectively, and a constant-|Q | cut with E_i = 40 meV at temperatures below and above T_N. At 1.7 K, there are at least three excitation peaks from the Ni²⁺ spins at 13.8, 16.9, and 20.3 meV, and no visible excitations from 2 to 10 meV and above 40 meV. These excitations cannot be spin waves since there are already two modes below 1.5 meV. They must arise from the single-ion CEF levels which we analyze using the Stevens operator formalism, in which, due to the ${C}_{3V}\left(3m\right)$ point-group symmetry, both tetrahedral and octahedral Ni sites are described by the Hamiltonian ${H}_{{CEF}}={B}_{2}^{0}{\hat{O}}_{2}^{0}+{B}_{4}^{0}{\hat{O}}_{4}^{0}+{B}_{4}^{3}{\hat{O}}_{4}^{3}$, where ${B}_{2}^{0}$ and ${B}_{4}^{m}$ are the second- and fourth-order crystal field parameters and ${\hat{O}}_{2}^{0}$ and ${\hat{O}}_{4}^{m}$ are the corresponding Stevens operators. As described in ref. ¹², the orbital ground state is the ³A state which has a three-fold degeneracy that is further lifted by the spin-orbit coupling (SOC) producing a singlet and doublet (Fig. 2d). Since the Ni²⁺ ions in NiZnMo₃O₈ tend to occupy octahedral sites, we performed INS experiments on the powder sample of NiZnMo₃O₈ to study the single-ion crystal fields of octahedral sites¹². We find that the intensity of the scattering in the range of 12–21 meV is considerably reduced (see Supplementary Fig. 4), consistent with the percentage of the Ni in the tetrahedral site of NiZnMo₃O₈ determined from neutron powder diffraction. For comparison, the spin excitations in NiZnMo₃O₈ are mostly centered below 2 meV, consistent with the notion that the energy of the CEF doublet levels from the octahedral site is below 2 meV (see Supplementary Fig. 5), which is clearly different from earlier low resolution electron spin resonance measurements and estimation from the point charge model¹². Therefore, the three CEF levels observed in Ni₂Mo₃O₈ at 1.7 K near 17 meV are all from the first excited doublet of the tetrahedral Ni site. Since the magnetic unit cell doubles the structural unit cell in the ordered state and the molecular field from the ordered moments splits the doublets, one would expect up to four excitation modes from the Ni tetrahedral site below T_N, while there is only one excitation above T_N. On cooling below T_N from 10 K, we see a clear splitting of the broad CEF peak at ~17 meV into two peaks around 13 meV and a broad peak around 20 meV, consistent with this picture (Fig. 2c and d). Combined with susceptibility data in Fig. 1f, we construct the Ni CEF levels as shown in Fig. 2d. While both Ni sites have singlet-ground states, the first excited state for the Ni tetrahedral and octahedral sites is at ~17 and ~1 meV, respectively. Since the energy bandwidth of spin waves in the AF ordered Ni₂Mo₃O₈ is less than 2 meV (Fig. 2a), we estimate that the magnetic exchange interactions between the Ni tetrahedral and octahedral sites J₁, and second-neighbor tetrahedral (octahedral) and tetrahedral (octahedral) sites J_T (J_O) to be less than 2 meV (Fig. 1d). Therefore, we identify Ni₂Mo₃O₈ as a spin-singlet-ground-state system with magnetic order being induced by the exchange J₁ comparable to the CEF level of the Ni octahedral site (${J}_{1} > {D}_{O}\approx 0.8$ meV, Fig. 2e and f). For comparison, we note that Co₂Mo₃O₈ has a much higher T_N ~40 K¹⁷ and its upper band of spin waves is around 12 meV¹⁸.

Fig. 2: Powder inelastic neutron spectra and crystal field levels of Ni₂Mo₃O₈.

a Powder average of the low-energy spin waves of Ni₂Mo₃O₈ single crystal at 1.7 K with E_i = 2.5 meV, which cannot be resolved in the spectrum using E_i = 40 meV. b Powder average of the spin excitation spectrum of Ni₂Mo₃O₈ single crystal at 1.5 K with E_i = 40 meV. Black arrows denote the three excitonic levels at 13.8, 16.9, and 20.3 meV. c Temperature dependence of the constant-Q ([1.5–2.5] Å⁻¹) cuts at 1.5 K from data in panel b. Blue arrows denote three peaks at 13.8, 16.9, and 20.3 meV. The horizontal bar is the energy resolution. d Schematics of crystal field levels of the tetrahedral coordinated and the octahedral coordinated Ni ions. e, f Comparison of the magnetic exchange couplings and CEF energy levels for Ni ions in tetrahedral and octahedral sites, respectively. One expects a nonmagnetic ground state for tetrahedral site while a magnetic ordered state is expected in the octahedral sites.

To determine the energy, wavevector, and temperature dependence of the Ni tetrahedral excitonic magnetism in Ni₂Mo₃O₈, we co-aligned high-quality single crystals in the [H, H, 0] × [-K, K, 0] scattering plane (Fig. 3a and b). Figure 3c–e is the E-Q dispersions of the spin excitons at 1.5, 10, and 120 K. We observed magnetic scattering in two separated energy regions, 12–16 and 18–22 meV, at 1.5 K. Above T_N, the scattering below and above 17 meV merge and become dispersive. The dispersion persists up to 120 K, which is one of the signatures of excitons. Figure 3f–k shows reciprocal space maps of the spin excitations in the [H, K, 0] plane in the two energy regions at 1.5, 10, and 120 K. The maps at different temperatures show qualitatively the same features: For $E=20\pm 2$ meV, the scattering show broad peaks centered near the Brillouin zone center Γ points (Fig. 3f–h), while for $E=14\pm 2$ meV, the scattering is like the complementary part of the high energy scattering that forms a diffusive pattern around the zone boundary (Fig. 3i–k). In both cases, the scattering below and above T_N are similar, contrary to the expected broadening of spin waves in momentum space from a magnetically ordered state to a paramagnetic state across T_N. Since spin waves in Ni₂Mo₃O₈ have a band top of ~1.5 meV at 1.7 K (Fig. 2a), the broad dispersive excitations in Fig. 3 f and i at 1.5 K well below T_N cannot arise from spin waves of magnetic ordered Ni²⁺.

Fig. 3: Momentum and temperature dependence of magnetic scattering in Ni₂Mo₃O₈.

a, b 3D and 2D reciprocal space of Ni₂Mo₃O₈, where the high symmetry positions are marked. c–e Energy- momentum plots of the dispersions along the [H, H, 0] direction at 1.5, 10, and 120 K measured with 40 meV incident neutron energy. The data is integrated along the L direction since the scattering above 12 meV has no modulation along the L direction. Dashed lines indicate the energy integration range in f–h, Momentum dependence of the magnetic scattering at 1.5, 10, and 120 K, where high intensity is near the zone center. The energy integration range is 18–22 meV. i–k Momentum dependence of the scattering at 1.5, 10, and 120 K, where high intensity is near the zone boundary. The energy integration range is 12–16 meV. The scattering becomes sharper in the paramagnetic state at 10 K. l–q In-plane magnetic field dependence of $S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)$ at 0 and 5-T at 2 K. The missing data in h-q is due to the narrower detector coverage when Ni₂Mo₃O₈ is in a magnet.

Figure 3l and m shows the in-plane magnetic field dependence of $S(E,{{{{{\boldsymbol{Q}}}}}})$ at energies near the CEF levels of Ni tetrahedron at 2 K for zero and 5-T field-polarized ferromagnetic state, respectively. The wavevector dependence of spin excitations for zero (Fig. 3n) and 5-T (Fig. 3o) field at $E=20\pm 2$ meV and 2 K are similar to zero field data at 1.5 K (Fig. 3f) and 10 K (Fig. 3g), respectively. The situation is similar at $E=14\pm 2$ meV (Fig. 3i at 1.5 K and 3j at 10 K in zero field, Fig. 3p at zero field and 3q at 5-T both at 2 K). This reflects the fact that the paramagnetic and ferromagnetic states have the same periodicities and that the exchange interaction does not appreciably change with field or temperature, whereas the AF state has a different periodicity, as the magnetic unit cell is doubled that of the structural cell in the AF phase. However, the paramagnetic and ferromagnetic states differ in that the applied field splits the excited doublet in the ferromagnetic state while there is no splitting in the paramagnetic state. This splitting is not seen in the data because it is smaller (≈0.1 meV) than the instrument resolution (≈1 meV), whereas the splitting due to the magnetic ordering is expected to be an order of magnitude larger than that produced by the external field.

Discussion

There are several unusual features in the CEF levels of Ni₂Mo₃O₈. First, the band top of low-energy spin waves is below 1.5 meV and the magnetic order is destroyed by an in-plane field of 8 T, indicating ${J}_{1}\ll {D}_{T}\approx 1.2$ meV. In singlet-ground-state systems, one would expect a paramagnetic state at zero temperature, but the system orders below 6 K. Second, spin excitons and spin waves should only hybridize when they have similar energy scales, while excitons should be featureless in Q if CEF levels have much higher energy than spin waves. In Ni₂Mo₃O₈, the excitonic bands at high energy (12-20 meV) are weakly dispersive in energy but show clear Q-dependence.

To understand these results, we consider a $S=1$ XXZ Hamiltonian for Ni₂Mo₃O₈:

$$H={H}_{1}+{H}_{23}+{H}_{{{\rm{SIA}}}},$$

(1)

where the first-nearest neighbor coupling is

$${H}_{1}=\mathop{\sum}\limits_{i\in t}{J}_{1}({{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{i+a}-\gamma \left({{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}\right)\left({{{{{{\boldsymbol{S}}}}}}}_{j}\cdot {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}\right)+d(\hat{z}\times {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a})\cdot {{{{{{\boldsymbol{S}}}}}}}_{i}\times {{{{{{\boldsymbol{S}}}}}}}_{i+a}),$$

(2)

the second- and third- neighbor coupling are

$${H}_{23}=\mathop{\sum}\limits_{ < {ij} > \in t}{J}_{T}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j}+\mathop{\sum}\limits_{ < {ij} > \in o}{J}_{O}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j}+\mathop{\sum}\limits_{ < {ij} > \in \{3{NN}\}}{J}_{3}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j},$$

(3)

and the SIA term is

$${H}_{{{{{{{\mathrm{SIA}}}}}}}}=\mathop{\sum}\limits_{i\in t}{D}_{T}{({S}_{i}^{z})}^{2}+\mathop{\sum}\limits_{i\in o}{D}_{O}{({S}_{i}^{z})}^{2}.$$

(4)

Here ${{{{{{\boldsymbol{S}}}}}}}_{i}$ is the spin operator at site i, $\gamma$ the anisotropic exchange, ${\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}$ is the unit vector linking the spins at site $i$ and $i+a$, ${D}_{T}$ and ${D}_{O}$ are SIA at tetrahedral and octahedral sites, respectively. For $J=0$, the Hamiltonian in Eq. (1) has a unique gapped ground state (${S}_{i}^{z}=0$ on all states). Perturbation for $J\ll D$ preserves the gap and the system remains in a unique, trivial ground state. For $J\gg D$, the single-ion terms are unimportant, and we expect an ordered ground state. Consequently, a quantum phase transition from paramagnetic to ordered state is expected as a function of increasing $J$. We capture the transition by a Curie-Weiss mean-field approach (see Supplementary Information). At the simplest level, if we only consider nonzero ${J}_{1},\gamma$ and ${D}_{T},{D}_{O}$, the critical value of ${J}_{1}$ is found to be

$${\widetilde{J}=J}_{1}(1+\gamma /2)=\frac{\sqrt{{D}_{T}{D}_{O}}}{2}.$$

(5)

This equation shows that order can be induced when the exchange is intermediate between the two single-ion energies. In Ni₂Mo₃O₈, ${D}_{T}\approx$ 16 meV and ${D}_{O}\approx$ 1 meV, according to our CEF analysis. Therefore, the scale of magnetic exchange required to induce moment is largely reduced due to the bipartite nature of the lattice. From Eq. (5), we have $\widetilde{J}\approx$ 2.06 meV, which is consistent with the low-energy scale of the magnetic order. The anisotropic exchange $\gamma$, which is originated from the combined effects of crystal field and SOC, can be used to energetically favor the experimentally observed four-sublattice state. The Dzyaloshinskii-Moriya interaction $d$ is responsible for the small out-of-plane spin canting¹⁹, and a finite value above the threshold ($d\approx \widetilde{J}$) is necessary¹⁹ to stabilize the stripy (as opposed to the Néel) ordered state that is observed experimentally.

The Q-dependent scattering for Ni tetrahedral CEF levels in Fig. 3f, g and i, j suggest short-range ferromagnetic and AF correlations of Ni tetrahedrons, respectively. We first consider three tetrahedral Ni atoms on the vertices of an equilateral triangle and the spins forms a 120° configuration (Fig. 4a). The Fourier transform of the spins on this cluster is:

$${{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{c}}}}}}}\left(h,\,k\right)=\mathop{\sum }\limits_{j=0}^{2}{{{{{{\boldsymbol{S}}}}}}}_{{{{{{\bf{0}}}}}}}{R}_{1/3}{F}_{{Ni}}\left(\left|{{{{{{\boldsymbol{Q}}}}}}}_{{{{{{\boldsymbol{hkl}}}}}}}\right|\right){{\exp }}\left[-i2\pi \left({x}_{j}h+{x}_{y}k\right)\right]={{{{{{\boldsymbol{S}}}}}}}_{{{{{{\bf{0}}}}}}}{\mathbb{M}}{F}_{{Ni}}\left(\left|{{{{{{\boldsymbol{Q}}}}}}}_{{{{{{\boldsymbol{hk}}}}}}{{{{{\boldsymbol{l}}}}}}}\right|\right),$$

(6)

where ${{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{0}}}}}}}$ is the direction of one of the spins, ${R}_{1/3}$ is the 120° rotational matrix, ${F}_{{Ni}}$ is the magnetic form factor of Ni²⁺, and ${\mathbb{M}}$ is the magnetization matrix. The observed scattering $S(E,{{{{{\boldsymbol{Q}}}}}})$ intensity is proportional to ${|\hat{\kappa }\times ({{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{c}}}}}}}\times \hat{\kappa })|}^{2}$, where $\hat{\kappa }\equiv {{{{{\boldsymbol{Q}}}}}}/Q$. Figure 4b shows the calculated structural factor $S({{{{{\boldsymbol{Q}}}}}})$ for the 120° AF spin configuration, where the J_T in Ni tetrahedron triangles dominates. Similarly, the structural factor for a ferromagnetic spin configuration can be calculated by canceling the off-diagonal term of ${\mathbb{M}}$, which is shown in Fig. 4d. The calculated factors fit well with the observed spin excitations in the INS experiments, indicating that spin excitons have spin-spin correlations.

Fig. 4: Structural factors in Ni₂Mo₃O₈ calculated on spin clusters and by flavor wave (FW) method.

a Schematic of spin clusters used to calculate the structural factors. The 120° configurations with approximately zero net spin give rise to the pattern in which high intensity is near the zone boundary. The FM clusters lead to the pattern where high intensity is near the zone center. b–d Comparison of the structural factors by the cluster and flavor wave calculations. Both methods reproduce the scattering in Ni₂Mo₃O₈ at 12–16 meV (b), 18–22 meV (d), and the sum of two (c).

To quantitatively understand the $S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)$ spectrum of the CEF levels, we modeled the INS spectra using an effective Hamiltonian in Eqs. (1–5) to describe a ground state singlet and excited doublet on each octahedral and tetrahedral site with an effective S = 1 spins, SIAs, symmetric and anti-symmetric exchange interactions (see Supplementary Information). The large SIA on the tetrahedral site gives rise to the apparent high energy CEF modes, which disperse due to the exchange couplings. The spectral function of the excitations in the ordered state is calculated using a flavor wave expansion based on the SU(3) representation of the triplet of levels on each site. This method captures the partial suppression of ordered moment by the tendency to single-ion singlet formation, while still describing the ordering and associated spin waves.

Focusing on the high energy excitations, the flavor wave calculation predicts the formation of four bands, which are spin-split and folded due to the AF enlargement of the unit cell. The predominant momentum dependence arises from the AF exchange coupling between the closest pairs of tetrahedral sites (J_T), which leads to high intensity at the lower edge of the band near the zone boundary, and at the upper edge of the band near the zone center (right panels in Fig. 4b–d). The exchange parameters from the calculation are summarized in the Supplementary Information.

Our single crystal X-ray and neutron diffraction refinements find that Ni tetrahedrons in Ni₂Mo₃O₈ form an ideal 2D triangular lattice without the magnetic and nonmagnetic disorder (Table 1). As the energy scale of the CEF spin excitons from Ni tetrahedrons is much larger than the magnetic exchange interactions determined from spin waves of Ni₂Mo₃O₈ (Fig. 2a), the presence of static AF order only slightly modifies the continuum-like Q-dependent scattering by making it less well-defined in the ordered state possibly due to mixing of the dispersive spin waves with CEF levels.

In induced-moment systems with a singlet-ground state^3,4,5,6,7, spin excitons can become highly dispersive and couple strongly to the ground state with a large magnetic exchange coupling J. When the ground state is not a spin singlet but a pseudospin doublet, spin excitons at high energies in some d³ transition-metal oxides can also be dispersive and have unusual properties due to strong SOC. For example, in the classic Mott insulator CoO, where the strength of SOC is comparable to the magnetic exchange coupling²⁰, the Q-dependence of spin excitons at high energies decay faster with Q than the Co²⁺ magnetic form factor, suggesting a breakdown of the localized spin excitons towards spatially extended magnetism²¹. More recently, in A-type AF ordered CoTiO₃ with T_N $\approx$ 38 K²², the dispersive spin excitons around ~27 meV due to SOC become softer and acquire a larger bandwidth on warming from the AF (5 K) to the paramagnetic state (60–120 K)^23,24. For comparison, the ~17 meV CEF doublet in the Ni tetrahedral site in Ni₂Mo₃O₈ induced by SOC (Fig. 2d) is similar in Q-space around the BZ boundary (see Supplementary Fig. 8 for the detailed cuts and comparisons) and narrower in energy bandwidth on warming from the AF (1.5 K) to the paramagnetic (10 and 120 K) state (Fig. 3c–k). While weakly dispersive spin waves above a large spin gap seen in the spin-chain compound Sr₃NiIrO₆²⁵ and the 1D magnet BaMo(PO₄)₂²⁶ also survive to temperatures well above their perspective T_Ns, they originate from magnons (not spin excitons) and do not have the line-shapes in Q-space as we observe in Ni₂Mo₃O₈.

Therefore, our results highlight the novel physics associated with two magnetic species, each on a frustrated triangular lattice, with very different single-ion anisotropies, and expose Ni₂Mo₃O₈ as a promising venue to explore the propagation of spin excitons in a dense highly fluctuating magnetic background. Most importantly, they indicate that CEF levels in an ideal triangular lattice magnet can produce dispersive spin excitons irrespective of static magnetic order, and the origin of this phenomenon is most likely due to spin entanglement and geometric frustrations without invoking the quantum spin liquid paradigm^27,28,29,30.

Methods

Sample growth

Polycrystalline samples of Ni₂Mo₃O₈, NiZnMo₃O₈, and Zn₂Mo₃O₈ were synthesized using a solid-state method. Stoichiometric powders of NiO, ZnO, Mo, and MoO₃ were mixed and pressed into pellets, and then sintered at 1050 °C for 24 h. Single crystalline Ni₂Mo₃O₈ was synthesized using the chemical vapor transport method. Powder X-ray diffraction measurements performed on powder samples and ground single crystals reveal that the samples have a pure phase, with a space group P6₃mc and the lattice parameter a = b = 5.767 Å and c = 9.916 Å. The structural information of the Ni₂Mo₃O₈ single crystal was investigated using a Rigaku XtaLAB PRO diffractometer equipped with a HyPi x-6000HE detector at ORNL. A molybdenum anode was used to generate X-rays with wavelength λ = 0.7107 Å. The samples were cooled by cold nitrogen flow provided by an Oxford N-Helix cryosystem. Single crystal X-ray refinements on Ni₂Mo₃O₈ reveal that the Ni and Mo are in fully occupied positions with no magnetic and nonmagnetic site disorder (Table 1).

Specific heat measurements

Specific heat measurements were conducted using a thermal-relaxation method in a physical property measurement system (Quantum Design).

Neutron diffraction

Neutron powder diffraction experiments were performed at room temperature using the high resolution powder diffractometer BT-1, at the NIST center for neutron research. 5.0 grams of NiZnMo₃O₈ powder was used. Powder neutron refinements reveal that Zn prefers to occupy tetrahedral sites (88.1%) and the rest of the tetrahedral sites are occupied by Ni (11.9%). Detailed results of the refinement are shown in Supplementary Table 1. Single crystal neutron diffraction experiments were carried out using the elastic diffuse scattering spectrometer,CORELLI³¹, at the Spallation Neutron Source, ORNL. One small piece of single crystalline Ni₂Mo₃O₈ was aligned in the [H, 0, L] plane. 53 structural Bragg peaks and 32 magnetic Bragg peaks at 2 K were used for the refinement.

CEF level measurements on powder samples

INS experiments were carried out on polycrystalline Ni₂Mo₃O₈ (6.0 g), NiZnMo₃O₈ (4.24 g), and Zn₂Mo₃O₈ (4.33 g), on the chopper spectrometer, MARI, and the cold neutron multi-chopper spectrometer, LET, at ISIS neutron and muon source. We collected data with 40 meV, 250 meV, and 1.5 eV incident energy (E_i) at 4 K on MARI, and with 1.8, 3.7, and 12.1 meV E_i at 2 and 12 K on LET.

INS experiments on single crystals

We co-aligned more than 200 pieces of single crystals of Ni₂Mo₃O₈ (1.5 g) to carry out inelastic neutron experiments on the cold neutron chopper spectrometer, CNCS, the fine-resolution fermi chopper spectrometer³², SEQUOIA³³ and ARCS³⁴ thermal chopper spectrometers, at the Spallation Neutron Source, Oak Ridge National Laboratory. The sample assembly was aligned in [H, K, 0] scattering plane on CNCS and SEQUOIA. We performed 180° rotational scans at 1.7, 3.5, 4.5, 5.5, and 6.5 K with 2.5 meV E_i on CNCS and at 1.5, 10, and 120 K with 40 meV E_i on SEQUOIA. On ARCS, we aligned the sample in [−K, K, L] scattering plane and measured with 26 and 40 meV E_i at 2 K.

Flavor wave calculations

Each mean-field state of the quantum magnet corresponds to a product state where the wave function on each site lives in the spin-1 Hilbert space. One can design a trial wave function to describe the magnetic order, with the variational parameters determined by minimization of the mean-field energy. The excitations on top of the ground state can then be suitably described by the “flavor waves” making use of the SU(3) flavor rotation in the spin-1 Hilbert space (see Supplementary Information for details). The dispersive excitations are well-defined in both ordered and disordered phases and can exhibit the experimentally observed behaviors in structure factors. In Fig. 4, we have used the parameters

$${D}_{t}=16\;{{{{{{\mathrm{meV}}}}}}},\,{D}_{o}=1.0\;{{{{{{\mathrm{meV}}}}}}},\,d=0.3,\,{{{{{\rm{\gamma }}}}}}=0.5,$$

$${J}_{1}=2.0\;{{{{{{\mathrm{meV}}}}}}},\,\,{J}_{t}=0.5\;{{{{{{\mathrm{meV}}}}}}},\,\,{J}_{o}=0.15\;{{{{{{\mathrm{meV}}}}}}},\,\,{J}_{3}=0.0\;{{{{{{\mathrm{meV}}}}}}}.$$