In most magnetic materials, the ground state of electron orbital states of magnetic ions in the crystal electric field (CEF) produced by the surrounding charge anion neighbors is magnetic with a finite moment and nonzero spin1,2. The long-range magnetic ordered structure and ordering temperature are determined by the magnetic exchange interactions (J), and the ordered moment direction is controlled by (typically small) easy-axis anisotropy gap2. However, there are also materials where the ground state of CEF levels of magnetic ions is a nonmagnetic singlet. Here, magnetic ordering and properties are sensitive to the ratio of magnetic exchange J to single-ion anisotropy (SIA, D) which is controlled by the CEF energy level of the first excited state3,4,5. With negligible magnetic exchange (\(J\ll D\)), the system is paramagnetic at all temperatures. For a relatively large magnetic exchange (\(J\gg D\)), magnetic order will be induced through a polarization instability of the singlet-ground state, termed an induced moment3,4,5,6,7,8. In this case, the basic magnetic excitations that describe the CEF transitions propagating through the lattice are called spin excitons3,4,5,6,7, analogous to electronic excitons that are bound states of an electron and a hole in a solid9. Spin excitons are fundamentally different from spin waves (magnons), which are strongly dispersive collective modes associated with spin precession on the lattice of magnetically ordered materials, and which disappear above the magnetic ordering temperature for isotropic Heisenberg magnets. In most cases, spin excitons originate from CEF levels of a localized single ion. Therefore, they are expected to be dispersionless in reciprocal space and well defined in both the magnetically ordered and paramagnetic states. However, when dispersive spin excitons are observed, the dispersion of these excitations can reveal unique information concerning magnetic exchange interactions between the localized ionic sites (spin–spin entanglement) and their relationship with the magnetically ordered state3,4,5,6,7,10,11.

Here we use thermodynamic and neutron scattering experiments to study stoichiometric honeycomb lattice antiferromagnetic (AF) ordered magnet Ni2Mo3O8, where Ni2+ ions form a bipartite honeycomb lattice comprised of two triangular lattices, in the tetrahedral and octahedral crystalline environment, respectively (Fig. 1a–c)12,13. We find that CEF levels of Ni2+ ions from both tetrahedral and octahedral environments have a nonmagnetic spin-singlet-ground state but with very different single-ion anisotropy energy scales for the two sites. Spin excitations of CEF levels (spin excitons) from the Ni2+ triangular tetrahedral sites form a diffusive pattern around the Brillouin zone (BZ) boundary in the AF and paramagnetic states in momentum space. Therefore, Ni2Mo3O8 realizes a novel situation in which the exchange energy falls between two very different single-ion energies, leading to a cooperative mechanism for magnetic order and strongly dispersing excitons associated with the larger single-ion anisotropy of the tetrahedral sites. Due to this hierarchy of energy scales, the excitons can persist even when the magnetic order is destroyed above the Néel temperature. In this regime, the excitons are strongly scattered from spin fluctuations and give rise to a distinct mechanism for dispersive diffusive scattering, hosting a unique coexistence of heavy particles (tetrahedral excitons) propagating in a frustrated background of light but dense (octahedral) spins due to spin entanglement and geometric frustrations.

Fig. 1: Structure, Brillouin zone, magnetic Bragg peak, d.c. susceptibility and specific heat of Ni2Mo3O8.
figure 1

a A schematic of the structure of Ni2Mo3O8 with a-, b-, c-axis labeled. b The octahedral coordinated Ni ions (blue) and the tetrahedral coordinated Ni ions (green) form a quasi-2D honeycomb lattice. c, d Side (along the b-axis) and top (along the c-axis) view of the magnetic structure of Ni2Mo3O8. The black frames indicate the size of magnetic unit cell and black arrows denote magnetic exchange couplings along different directions. e Temperature dependence of the intensities in arbitrary units (a. u.) of two magnetic Bragg peaks at (3/2,0,1) and (1/2,−1/2,0). Inset shows a 2D color plot in the [H, 0, L] plane, in which magnetic Bragg peaks are indicated by red arrows. The vertical error bars are statistical errors of 1 standard deviation. f The d.c. susceptibility with 1–8 T magnetic fields perpendicular to the c-axis. Data are shifted by magnetic field × 0.01 emu / mol T for clarity. Magnetic transition is suppressed in the 8 T field. g The magnetic contribution to the specific heat Cmag with 0-8 T magnetic fields perpendicular to the c-axis. Magnetic transition is suppressed in the 8 T field. h Temperature versus magnetic field phase diagram from d.c. susceptibility and specific data.


Crystal structure, magnetic order, susceptibility, and specific heat of Ni2Mo3O8

The M2Mo3O8 (M = Fe, Mn, Ni, Co, Zn) compounds have drawn increasing attention due to their multiferroic properties12,13,14,15,16. The crystal structure of M2Mo3O8 consists of magnetic bipartite honeycomb M-O layers, separated by sheets of Mo4+ layers (Fig. 1a), where the Mo4+ ions inside each layer are trimerized and form a singlet. The two M2+ sites have different oxygen coordination, with one site being an MO6 octahedron and the adjacent one being an MO4 tetrahedron. In this family, Ni2Mo3O8 was studied as a platform to explore the physics of geometrically frustrated lattice12,13. Neutron powder diffraction experiments reveal that both the MO6 octahedron and the MO4 tetrahedron each form perfect triangular lattices with no inter-site disorder, and the system orders antiferromagnetically with a Néel temperature of TN = 5.5 K12,13. The magnetic structure is stripe like within the Ni-O plane, and zig-zag like along the c-axis with different ordered moments for octahedral and tetrahedral Ni sites (Fig. 1c and d). Previous single-ion CEF analysis suggests that both octahedral and tetrahedral Ni sites have nonmagnetic singlet-ground states, and the first excited magnetic doublets are at 7.8 and 23 meV, respectively12.

Even though the Ni ions appear to form a simple (bipartite) honeycomb lattice in Ni2Mo3O8 (Fig. 1b–d), this compound should be viewed rather as two inter-penetrating triangular lattices formed by NiO6 octahedron and NiO4 tetrahedron. In Ni2Mo3O8, magnetic order is assumed to arise from the relatively large magnetic exchange coupling between the octahedral and tetrahedral Ni sites (denoted as J1 in Fig. 1d) in comparison with the energy of the CEF level of the Ni octahedral site12. However, there are no inelastic neutron scattering (INS) experiments to date to identify the CEF levels of the Ni octahedral and tetrahedral sites, and prove that the ground state is indeed a singlet. Here we report magnetic susceptibility, heat capacity, X-ray diffraction, and INS experiments on single crystals of Ni2Mo3O8 and NiZnMo3O8 grown by the chemical vapor transfer method. Our careful single crystal X-ray diffraction experiments reveal that both the NiO6 octahedra and NiO4 tetrahedra form perfect triangular lattices with no inter-site disorder, and there is also no disorder on the Mo site (see Table 1 and Supplementary Information for details). The X-ray and neutron diffraction refinements also show that nonmagnetic dopant ions like Zn2+ prefer to occupy the tetrahedral sites12. Consistent with previous neutron powder diffraction work12,13, our neutron single crystal diffraction refinements find that the spins of Ni2+ ions form a stripy AF order (Fig. 1c and d) below TN = 5.5 K (Fig. 1e), but with the ordered moments of tetrahedral and octahedral Ni sites being 1.47 and 1.1 μB, respectively, different from the previously reported values.

Table 1 Positions of atoms in Ni2Mo3O8 as determined from single crystal X-ray diffraction

Figure 1f shows the temperature dependence of the d.c. susceptibility χ(T) with 1–8 T magnetic fields applied along the [1,1,0] direction perpendicular to the c-axis. The data in the 1 T field (orange line) shows a clear peak around TN \(\approx\) 6 K consistent with neutron data in Fig. 1e, while the data in the 8 T field (blue line) shows no evidence of a magnetic transition. In-plane susceptibility is much larger than the c-axis susceptibility (see Supplementary Fig. 3), indicating easy-plane anisotropy. The Curie-Weiss fitting to the in-plane and c-axis susceptibility above 200 K gives θCW = −454.12 ± 0.53 K and θCW,// = −100.52 ± 0.11 K, respectively, suggesting a quasi-2D system where the in-plane magnetic exchange is larger than the c-axis exchange. The difference between our analysis and previous results on single crystals mainly comes from the direction of the in-plane magnetic fields13. Figure 1g shows the temperature dependence of the magnetic contribution to the specific heat as a function of applied magnetic fields perpendicular to the c-axis. At zero field (red line), we see a typical λ-shaped transition around TN \(\approx\) 6 K. At 8 T field, there is no evidence of a magnetic transition above 2 K, consistent with Fig. 1f. Figure 1h shows the temperature-field phase diagram from our susceptibility and specific heat data.

Neutron scattering studies of spin waves and CEF levels

To search for the CEF levels of the tetrahedral and octahedral Ni sites and demonstrate that the ground state of Ni2Mo3O8 is indeed a spin singlet, we carried out INS experiments on single crystalline and powdered samples with incident energies (Ei) of 2.5 meV, 3.7 meV, 40 meV, 250 meV, and 1.5 eV. At Ei = 2.5 meV, we see clear dispersive spin waves at 1.7 K (Fig. 2a) with two modes. This is consistent with the expectations from the linear spin-wave theory (LSWT) calculation, since there are four Ni2+ sites in the magnetic unit cell, resulting in two doubly-degenerate modes. The small anisotropic spin gap (\( < 0.3\) meV) and overall spin-wave energy bandwidth of 1.5 meV are consistent with thermal dynamic data in Fig. 1f–h and TN = 5.5 K. Figure 2b and c shows a powder averaged INS \(S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)\) spectrum, where E and Q are energy and moment transfer, respectively, and a constant-|Q | cut with Ei = 40 meV at temperatures below and above TN. At 1.7 K, there are at least three excitation peaks from the Ni2+ spins at 13.8, 16.9, and 20.3 meV, and no visible excitations from 2 to 10 meV and above 40 meV. These excitations cannot be spin waves since there are already two modes below 1.5 meV. They must arise from the single-ion CEF levels which we analyze using the Stevens operator formalism, in which, due to the \({C}_{3V}\left(3m\right)\) point-group symmetry, both tetrahedral and octahedral Ni sites are described by the Hamiltonian \({H}_{{CEF}}={B}_{2}^{0}{\hat{O}}_{2}^{0}+{B}_{4}^{0}{\hat{O}}_{4}^{0}+{B}_{4}^{3}{\hat{O}}_{4}^{3}\), where \({B}_{2}^{0}\) and \({B}_{4}^{m}\) are the second- and fourth-order crystal field parameters and \({\hat{O}}_{2}^{0}\) and \({\hat{O}}_{4}^{m}\) are the corresponding Stevens operators. As described in ref. 12, the orbital ground state is the 3A state which has a three-fold degeneracy that is further lifted by the spin-orbit coupling (SOC) producing a singlet and doublet (Fig. 2d). Since the Ni2+ ions in NiZnMo3O8 tend to occupy octahedral sites, we performed INS experiments on the powder sample of NiZnMo3O8 to study the single-ion crystal fields of octahedral sites12. We find that the intensity of the scattering in the range of 12–21 meV is considerably reduced (see Supplementary Fig. 4), consistent with the percentage of the Ni in the tetrahedral site of NiZnMo3O8 determined from neutron powder diffraction. For comparison, the spin excitations in NiZnMo3O8 are mostly centered below 2 meV, consistent with the notion that the energy of the CEF doublet levels from the octahedral site is below 2 meV (see Supplementary Fig. 5), which is clearly different from earlier low resolution electron spin resonance measurements and estimation from the point charge model12. Therefore, the three CEF levels observed in Ni2Mo3O8 at 1.7 K near 17 meV are all from the first excited doublet of the tetrahedral Ni site. Since the magnetic unit cell doubles the structural unit cell in the ordered state and the molecular field from the ordered moments splits the doublets, one would expect up to four excitation modes from the Ni tetrahedral site below TN, while there is only one excitation above TN. On cooling below TN from 10 K, we see a clear splitting of the broad CEF peak at ~17 meV into two peaks around 13 meV and a broad peak around 20 meV, consistent with this picture (Fig. 2c and d). Combined with susceptibility data in Fig. 1f, we construct the Ni CEF levels as shown in Fig. 2d. While both Ni sites have singlet-ground states, the first excited state for the Ni tetrahedral and octahedral sites is at ~17 and ~1 meV, respectively. Since the energy bandwidth of spin waves in the AF ordered Ni2Mo3O8 is less than 2 meV (Fig. 2a), we estimate that the magnetic exchange interactions between the Ni tetrahedral and octahedral sites J1, and second-neighbor tetrahedral (octahedral) and tetrahedral (octahedral) sites JT (JO) to be less than 2 meV (Fig. 1d). Therefore, we identify Ni2Mo3O8 as a spin-singlet-ground-state system with magnetic order being induced by the exchange J1 comparable to the CEF level of the Ni octahedral site (\({J}_{1} > {D}_{O}\approx 0.8\) meV, Fig. 2e and f). For comparison, we note that Co2Mo3O8 has a much higher TN ~40 K17 and its upper band of spin waves is around 12 meV18.

Fig. 2: Powder inelastic neutron spectra and crystal field levels of Ni2Mo3O8.
figure 2

a Powder average of the low-energy spin waves of Ni2Mo3O8 single crystal at 1.7 K with Ei = 2.5 meV, which cannot be resolved in the spectrum using Ei = 40 meV. b Powder average of the spin excitation spectrum of Ni2Mo3O8 single crystal at 1.5 K with Ei = 40 meV. Black arrows denote the three excitonic levels at 13.8, 16.9, and 20.3 meV. c Temperature dependence of the constant-Q ([1.5–2.5] Å−1) cuts at 1.5 K from data in panel b. Blue arrows denote three peaks at 13.8, 16.9, and 20.3 meV. The horizontal bar is the energy resolution. d Schematics of crystal field levels of the tetrahedral coordinated and the octahedral coordinated Ni ions. e, f Comparison of the magnetic exchange couplings and CEF energy levels for Ni ions in tetrahedral and octahedral sites, respectively. One expects a nonmagnetic ground state for tetrahedral site while a magnetic ordered state is expected in the octahedral sites.

To determine the energy, wavevector, and temperature dependence of the Ni tetrahedral excitonic magnetism in Ni2Mo3O8, we co-aligned high-quality single crystals in the [H, H, 0] × [-K, K, 0] scattering plane (Fig. 3a and b). Figure 3c–e is the E-Q dispersions of the spin excitons at 1.5, 10, and 120 K. We observed magnetic scattering in two separated energy regions, 12–16 and 18–22 meV, at 1.5 K. Above TN, the scattering below and above 17 meV merge and become dispersive. The dispersion persists up to 120 K, which is one of the signatures of excitons. Figure 3f–k shows reciprocal space maps of the spin excitations in the [H, K, 0] plane in the two energy regions at 1.5, 10, and 120 K. The maps at different temperatures show qualitatively the same features: For \(E=20\pm 2\) meV, the scattering show broad peaks centered near the Brillouin zone center Γ points (Fig. 3f–h), while for \(E=14\pm 2\) meV, the scattering is like the complementary part of the high energy scattering that forms a diffusive pattern around the zone boundary (Fig. 3i–k). In both cases, the scattering below and above TN are similar, contrary to the expected broadening of spin waves in momentum space from a magnetically ordered state to a paramagnetic state across TN. Since spin waves in Ni2Mo3O8 have a band top of ~1.5 meV at 1.7 K (Fig. 2a), the broad dispersive excitations in Fig. 3 f and i at 1.5 K well below TN cannot arise from spin waves of magnetic ordered Ni2+.

Fig. 3: Momentum and temperature dependence of magnetic scattering in Ni2Mo3O8.
figure 3

a, b 3D and 2D reciprocal space of Ni2Mo3O8, where the high symmetry positions are marked. ce Energy- momentum plots of the dispersions along the [H, H, 0] direction at 1.5, 10, and 120 K measured with 40 meV incident neutron energy. The data is integrated along the L direction since the scattering above 12 meV has no modulation along the L direction. Dashed lines indicate the energy integration range in fh, Momentum dependence of the magnetic scattering at 1.5, 10, and 120 K, where high intensity is near the zone center. The energy integration range is 18–22 meV. ik Momentum dependence of the scattering at 1.5, 10, and 120 K, where high intensity is near the zone boundary. The energy integration range is 12–16 meV. The scattering becomes sharper in the paramagnetic state at 10 K. lq In-plane magnetic field dependence of \(S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)\) at 0 and 5-T at 2 K. The missing data in h-q is due to the narrower detector coverage when Ni2Mo3O8 is in a magnet.

Figure 3l and m shows the in-plane magnetic field dependence of \(S(E,{{{{{\boldsymbol{Q}}}}}})\) at energies near the CEF levels of Ni tetrahedron at 2 K for zero and 5-T field-polarized ferromagnetic state, respectively. The wavevector dependence of spin excitations for zero (Fig. 3n) and 5-T (Fig. 3o) field at \(E=20\pm 2\) meV and 2 K are similar to zero field data at 1.5 K (Fig. 3f) and 10 K (Fig. 3g), respectively. The situation is similar at \(E=14\pm 2\) meV (Fig. 3i at 1.5 K and 3j at 10 K in zero field, Fig. 3p at zero field and 3q at 5-T both at 2 K). This reflects the fact that the paramagnetic and ferromagnetic states have the same periodicities and that the exchange interaction does not appreciably change with field or temperature, whereas the AF state has a different periodicity, as the magnetic unit cell is doubled that of the structural cell in the AF phase. However, the paramagnetic and ferromagnetic states differ in that the applied field splits the excited doublet in the ferromagnetic state while there is no splitting in the paramagnetic state. This splitting is not seen in the data because it is smaller (≈0.1 meV) than the instrument resolution (≈1 meV), whereas the splitting due to the magnetic ordering is expected to be an order of magnitude larger than that produced by the external field.


There are several unusual features in the CEF levels of Ni2Mo3O8. First, the band top of low-energy spin waves is below 1.5 meV and the magnetic order is destroyed by an in-plane field of 8 T, indicating \({J}_{1}\ll {D}_{T}\approx 1.2\) meV. In singlet-ground-state systems, one would expect a paramagnetic state at zero temperature, but the system orders below 6 K. Second, spin excitons and spin waves should only hybridize when they have similar energy scales, while excitons should be featureless in Q if CEF levels have much higher energy than spin waves. In Ni2Mo3O8, the excitonic bands at high energy (12-20 meV) are weakly dispersive in energy but show clear Q-dependence.

To understand these results, we consider a \(S=1\) XXZ Hamiltonian for Ni2Mo3O8:


where the first-nearest neighbor coupling is

$${H}_{1}=\mathop{\sum}\limits_{i\in t}{J}_{1}({{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{i+a}-\gamma \left({{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}\right)\left({{{{{{\boldsymbol{S}}}}}}}_{j}\cdot {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}\right)+d(\hat{z}\times {\hat{{{{{{\boldsymbol{e}}}}}}}}_{a})\cdot {{{{{{\boldsymbol{S}}}}}}}_{i}\times {{{{{{\boldsymbol{S}}}}}}}_{i+a}),$$

the second- and third- neighbor coupling are

$${H}_{23}=\mathop{\sum}\limits_{ < {ij} > \in t}{J}_{T}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j}+\mathop{\sum}\limits_{ < {ij} > \in o}{J}_{O}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j}+\mathop{\sum}\limits_{ < {ij} > \in \{3{NN}\}}{J}_{3}{{{{{{\boldsymbol{S}}}}}}}_{i}\cdot {{{{{{\boldsymbol{S}}}}}}}_{j},$$

and the SIA term is

$${H}_{{{{{{{\mathrm{SIA}}}}}}}}=\mathop{\sum}\limits_{i\in t}{D}_{T}{({S}_{i}^{z})}^{2}+\mathop{\sum}\limits_{i\in o}{D}_{O}{({S}_{i}^{z})}^{2}.$$

Here \({{{{{{\boldsymbol{S}}}}}}}_{i}\) is the spin operator at site i, \(\gamma\) the anisotropic exchange, \({\hat{{{{{{\boldsymbol{e}}}}}}}}_{a}\) is the unit vector linking the spins at site \(i\) and \(i+a\), \({D}_{T}\) and \({D}_{O}\) are SIA at tetrahedral and octahedral sites, respectively. For \(J=0\), the Hamiltonian in Eq. (1) has a unique gapped ground state (\({S}_{i}^{z}=0\) on all states). Perturbation for \(J\ll D\) preserves the gap and the system remains in a unique, trivial ground state. For \(J\gg D\), the single-ion terms are unimportant, and we expect an ordered ground state. Consequently, a quantum phase transition from paramagnetic to ordered state is expected as a function of increasing \(J\). We capture the transition by a Curie-Weiss mean-field approach (see Supplementary Information). At the simplest level, if we only consider nonzero \({J}_{1},\gamma\) and \({D}_{T},{D}_{O}\), the critical value of \({J}_{1}\) is found to be

$${\widetilde{J}=J}_{1}(1+\gamma /2)=\frac{\sqrt{{D}_{T}{D}_{O}}}{2}.$$

This equation shows that order can be induced when the exchange is intermediate between the two single-ion energies. In Ni2Mo3O8, \({D}_{T}\approx\) 16 meV and \({D}_{O}\approx\) 1 meV, according to our CEF analysis. Therefore, the scale of magnetic exchange required to induce moment is largely reduced due to the bipartite nature of the lattice. From Eq. (5), we have \(\widetilde{J}\approx\) 2.06 meV, which is consistent with the low-energy scale of the magnetic order. The anisotropic exchange \(\gamma\), which is originated from the combined effects of crystal field and SOC, can be used to energetically favor the experimentally observed four-sublattice state. The Dzyaloshinskii-Moriya interaction \(d\) is responsible for the small out-of-plane spin canting19, and a finite value above the threshold (\(d\approx \widetilde{J}\)) is necessary19 to stabilize the stripy (as opposed to the Néel) ordered state that is observed experimentally.

The Q-dependent scattering for Ni tetrahedral CEF levels in Fig. 3f, g and i, j suggest short-range ferromagnetic and AF correlations of Ni tetrahedrons, respectively. We first consider three tetrahedral Ni atoms on the vertices of an equilateral triangle and the spins forms a 120° configuration (Fig. 4a). The Fourier transform of the spins on this cluster is:

$${{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{c}}}}}}}\left(h,\,k\right)=\mathop{\sum }\limits_{j=0}^{2}{{{{{{\boldsymbol{S}}}}}}}_{{{{{{\bf{0}}}}}}}{R}_{1/3}{F}_{{Ni}}\left(\left|{{{{{{\boldsymbol{Q}}}}}}}_{{{{{{\boldsymbol{hkl}}}}}}}\right|\right){{\exp }}\left[-i2\pi \left({x}_{j}h+{x}_{y}k\right)\right]={{{{{{\boldsymbol{S}}}}}}}_{{{{{{\bf{0}}}}}}}{\mathbb{M}}{F}_{{Ni}}\left(\left|{{{{{{\boldsymbol{Q}}}}}}}_{{{{{{\boldsymbol{hk}}}}}}{{{{{\boldsymbol{l}}}}}}}\right|\right),$$

where \({{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{0}}}}}}}\) is the direction of one of the spins, \({R}_{1/3}\) is the 120° rotational matrix, \({F}_{{Ni}}\) is the magnetic form factor of Ni2+, and \({\mathbb{M}}\) is the magnetization matrix. The observed scattering \(S(E,{{{{{\boldsymbol{Q}}}}}})\) intensity is proportional to \({|\hat{\kappa }\times ({{{{{{\boldsymbol{S}}}}}}}_{{{{{{\boldsymbol{c}}}}}}}\times \hat{\kappa })|}^{2}\), where \(\hat{\kappa }\equiv {{{{{\boldsymbol{Q}}}}}}/Q\). Figure 4b shows the calculated structural factor \(S({{{{{\boldsymbol{Q}}}}}})\) for the 120° AF spin configuration, where the JT in Ni tetrahedron triangles dominates. Similarly, the structural factor for a ferromagnetic spin configuration can be calculated by canceling the off-diagonal term of \({\mathbb{M}}\), which is shown in Fig. 4d. The calculated factors fit well with the observed spin excitations in the INS experiments, indicating that spin excitons have spin-spin correlations.

Fig. 4: Structural factors in Ni2Mo3O8 calculated on spin clusters and by flavor wave (FW) method.
figure 4

a Schematic of spin clusters used to calculate the structural factors. The 120° configurations with approximately zero net spin give rise to the pattern in which high intensity is near the zone boundary. The FM clusters lead to the pattern where high intensity is near the zone center. bd Comparison of the structural factors by the cluster and flavor wave calculations. Both methods reproduce the scattering in Ni2Mo3O8 at 12–16 meV (b), 18–22 meV (d), and the sum of two (c).

To quantitatively understand the \(S\left(E,{{{{{\boldsymbol{Q}}}}}}\right)\) spectrum of the CEF levels, we modeled the INS spectra using an effective Hamiltonian in Eqs. (15) to describe a ground state singlet and excited doublet on each octahedral and tetrahedral site with an effective S = 1 spins, SIAs, symmetric and anti-symmetric exchange interactions (see Supplementary Information). The large SIA on the tetrahedral site gives rise to the apparent high energy CEF modes, which disperse due to the exchange couplings. The spectral function of the excitations in the ordered state is calculated using a flavor wave expansion based on the SU(3) representation of the triplet of levels on each site. This method captures the partial suppression of ordered moment by the tendency to single-ion singlet formation, while still describing the ordering and associated spin waves.

Focusing on the high energy excitations, the flavor wave calculation predicts the formation of four bands, which are spin-split and folded due to the AF enlargement of the unit cell. The predominant momentum dependence arises from the AF exchange coupling between the closest pairs of tetrahedral sites (JT), which leads to high intensity at the lower edge of the band near the zone boundary, and at the upper edge of the band near the zone center (right panels in Fig. 4b–d). The exchange parameters from the calculation are summarized in the Supplementary Information.

Our single crystal X-ray and neutron diffraction refinements find that Ni tetrahedrons in Ni2Mo3O8 form an ideal 2D triangular lattice without the magnetic and nonmagnetic disorder (Table 1). As the energy scale of the CEF spin excitons from Ni tetrahedrons is much larger than the magnetic exchange interactions determined from spin waves of Ni2Mo3O8 (Fig. 2a), the presence of static AF order only slightly modifies the continuum-like Q-dependent scattering by making it less well-defined in the ordered state possibly due to mixing of the dispersive spin waves with CEF levels.

In induced-moment systems with a singlet-ground state3,4,5,6,7, spin excitons can become highly dispersive and couple strongly to the ground state with a large magnetic exchange coupling J. When the ground state is not a spin singlet but a pseudospin doublet, spin excitons at high energies in some d3 transition-metal oxides can also be dispersive and have unusual properties due to strong SOC. For example, in the classic Mott insulator CoO, where the strength of SOC is comparable to the magnetic exchange coupling20, the Q-dependence of spin excitons at high energies decay faster with Q than the Co2+ magnetic form factor, suggesting a breakdown of the localized spin excitons towards spatially extended magnetism21. More recently, in A-type AF ordered CoTiO3 with TN \(\approx\) 38 K22, the dispersive spin excitons around ~27 meV due to SOC become softer and acquire a larger bandwidth on warming from the AF (5 K) to the paramagnetic state (60–120 K)23,24. For comparison, the ~17 meV CEF doublet in the Ni tetrahedral site in Ni2Mo3O8 induced by SOC (Fig. 2d) is similar in Q-space around the BZ boundary (see Supplementary Fig. 8 for the detailed cuts and comparisons) and narrower in energy bandwidth on warming from the AF (1.5 K) to the paramagnetic (10 and 120 K) state (Fig. 3c–k). While weakly dispersive spin waves above a large spin gap seen in the spin-chain compound Sr3NiIrO625 and the 1D magnet BaMo(PO4)226 also survive to temperatures well above their perspective TNs, they originate from magnons (not spin excitons) and do not have the line-shapes in Q-space as we observe in Ni2Mo3O8.

Therefore, our results highlight the novel physics associated with two magnetic species, each on a frustrated triangular lattice, with very different single-ion anisotropies, and expose Ni2Mo3O8 as a promising venue to explore the propagation of spin excitons in a dense highly fluctuating magnetic background. Most importantly, they indicate that CEF levels in an ideal triangular lattice magnet can produce dispersive spin excitons irrespective of static magnetic order, and the origin of this phenomenon is most likely due to spin entanglement and geometric frustrations without invoking the quantum spin liquid paradigm27,28,29,30.


Sample growth

Polycrystalline samples of Ni2Mo3O8, NiZnMo3O8, and Zn2Mo3O8 were synthesized using a solid-state method. Stoichiometric powders of NiO, ZnO, Mo, and MoO3 were mixed and pressed into pellets, and then sintered at 1050 °C for 24 h. Single crystalline Ni2Mo3O8 was synthesized using the chemical vapor transport method. Powder X-ray diffraction measurements performed on powder samples and ground single crystals reveal that the samples have a pure phase, with a space group P63mc and the lattice parameter a = b = 5.767 Å and c = 9.916 Å. The structural information of the Ni2Mo3O8 single crystal was investigated using a Rigaku XtaLAB PRO diffractometer equipped with a HyPi x-6000HE detector at ORNL. A molybdenum anode was used to generate X-rays with wavelength λ = 0.7107 Å. The samples were cooled by cold nitrogen flow provided by an Oxford N-Helix cryosystem. Single crystal X-ray refinements on Ni2Mo3O8 reveal that the Ni and Mo are in fully occupied positions with no magnetic and nonmagnetic site disorder (Table 1).

Specific heat measurements

Specific heat measurements were conducted using a thermal-relaxation method in a physical property measurement system (Quantum Design).

Neutron diffraction

Neutron powder diffraction experiments were performed at room temperature using the high resolution powder diffractometer BT-1, at the NIST center for neutron research. 5.0 grams of NiZnMo3O8 powder was used. Powder neutron refinements reveal that Zn prefers to occupy tetrahedral sites (88.1%) and the rest of the tetrahedral sites are occupied by Ni (11.9%). Detailed results of the refinement are shown in Supplementary Table 1. Single crystal neutron diffraction experiments were carried out using the elastic diffuse scattering spectrometer,CORELLI31, at the Spallation Neutron Source, ORNL. One small piece of single crystalline Ni2Mo3O8 was aligned in the [H, 0, L] plane. 53 structural Bragg peaks and 32 magnetic Bragg peaks at 2 K were used for the refinement.

CEF level measurements on powder samples

INS experiments were carried out on polycrystalline Ni2Mo3O8 (6.0 g), NiZnMo3O8 (4.24 g), and Zn2Mo3O8 (4.33 g), on the chopper spectrometer, MARI, and the cold neutron multi-chopper spectrometer, LET, at ISIS neutron and muon source. We collected data with 40 meV, 250 meV, and 1.5 eV incident energy (Ei) at 4 K on MARI, and with 1.8, 3.7, and 12.1 meV Ei at 2 and 12 K on LET.

INS experiments on single crystals

We co-aligned more than 200 pieces of single crystals of Ni2Mo3O8 (1.5 g) to carry out inelastic neutron experiments on the cold neutron chopper spectrometer, CNCS, the fine-resolution fermi chopper spectrometer32, SEQUOIA33 and ARCS34 thermal chopper spectrometers, at the Spallation Neutron Source, Oak Ridge National Laboratory. The sample assembly was aligned in [H, K, 0] scattering plane on CNCS and SEQUOIA. We performed 180° rotational scans at 1.7, 3.5, 4.5, 5.5, and 6.5 K with 2.5 meV Ei on CNCS and at 1.5, 10, and 120 K with 40 meV Ei on SEQUOIA. On ARCS, we aligned the sample in [−K, K, L] scattering plane and measured with 26 and 40 meV Ei at 2 K.

Flavor wave calculations

Each mean-field state of the quantum magnet corresponds to a product state where the wave function on each site lives in the spin-1 Hilbert space. One can design a trial wave function to describe the magnetic order, with the variational parameters determined by minimization of the mean-field energy. The excitations on top of the ground state can then be suitably described by the “flavor waves” making use of the SU(3) flavor rotation in the spin-1 Hilbert space (see Supplementary Information for details). The dispersive excitations are well-defined in both ordered and disordered phases and can exhibit the experimentally observed behaviors in structure factors. In Fig. 4, we have used the parameters

$${D}_{t}=16\;{{{{{{\mathrm{meV}}}}}}},\,{D}_{o}=1.0\;{{{{{{\mathrm{meV}}}}}}},\,d=0.3,\,{{{{{\rm{\gamma }}}}}}=0.5,$$