Dynamical ground state in the XY pyrochlore Yb2GaSbO7

The magnetic ground state of the pyrochlore Yb2GaSbO7 has not been established. The persistent spin fluctuations observed by muon spin-relaxation measurements at low temperatures have not been adequately explained for this material using existing theories for quantum magnetism. Here we report on the synthesis and characterisation of Yb2GaSbO7 to revisit the nature of the magnetic ground state. Through DC and AC magnetic susceptibility, heat capacity, and neutron scattering experiments, we observe evidence for a dynamical ground state that makes Yb2GaSbO7 a promising candidate for disorder-induced spin-liquid or spin-singlet behaviour. This state is quite fragile, being tuned to a splayed ferromagnet in a modest magnetic field μ0Hc ~ 1.5 T.


INTRODUCTION
The initial proposal 1 of quantum spin ice 2 behaviour in the pyrochlore Yb 2 Ti 2 O 7 has ignited a plethora of experimental and theoretical studies of Yb 3+ magnetism in various frustrated geometries [3][4][5][6][7][8][9][10][11][12][13][14] . While there have been many candidates for unusual magnetic ground states in other rare-earth pyrochlore systems, some members of this family, including Yb 3+ and Ce 3+based systems, are particularly attractive because quantum fluctuations can be enhanced by their effective spin- 1 2 degrees of freedom arising from a Kramers doublet single-ion ground state that is well-separated from excited crystal field levels [15][16][17][18][19][20] . One less-studied compound within the Yb 3+ pyrochlores is Yb 2 GaSbO 7 , with its non-magnetic B-site containing a mixture of Ga 3+ and Sb 5+ cations. Mixed site pyrochlores were originally studied by the chemistry community [21][22][23][24][25] and have recently generated interest among condensed matter physicists [26][27][28][29][30] . A comparison of muon spin relaxation and Mössbauer spectroscopy experiments on Yb 2 GaSbO 7 and Yb 2 Ti 2 O 7 revealed that, while Yb 2 Ti 2 O 7 had a dramatic change in the relaxation rate of four orders of magnitude near T C = 240 mK, Yb 2 GaSbO 7 remained dynamic down to a relaxation rate plateau near T * = 340 mK 31 . This change in relaxation rate for Yb 2 Ti 2 O 7 has largely been explained 32 but the persistent spin fluctuations in Yb 2 GaSbO 7 still remain a mystery. Naïvely, one would assume that the chemical disorder on the B-site would induce a spin-glass state in the latter compound due to the presence of different nearest neighbour exchange pathways. However, the muon decay asymmetry for Yb 2 GaSbO 7 does not follow the typical functional form for a spin glass with only a single exponential relaxation component down to 50 mK, while no low-temperature transitions were noted in both the specific heat 31 and Ga-NMR 33 .
In this work, we provide experimental evidence for a dynamical magnetic ground state in Yb 2 GaSbO 7 and propose possible explanations for this unusual behaviour. We also construct a comprehensive phase diagram for Yb 2 GaSbO 7 using a combination of AC and DC magnetic susceptibility, heat capacity, and neutron scattering. A modest applied magnetic field μ 0 H c~1 .5 T induces an XY splayed ferromagnetic state, similar to the zero-field ground state observed in pristine Yb 2 Ti 2 O 7 samples [34][35][36][37][38][39] . In magnetic fields below μ 0 H c , however, neutron scattering experiments reveal spin correlations building up upon cooling that can be characterised with a net antiferromagnetic exchange. A key difference between Yb 2 Ti 2 O 7 and Yb 2 GaSbO 7 is the presence of net antiferromagnetic exchange, reflected by a concomitant change in sign of the Curie-Weiss temperature in the susceptibility (θ CW = −1.42 K in Yb 2 GaSbO 7 compared to θ CW = 0.59 K in Yb 2 Ti 2 O 7 ) 40 . Our data suggest that Yb 2 GaSbO 7 in zero field may be close to a phase boundary in the nearest neighbour (n.n.) anisotropic exchange phase diagram for pyrochlore magnets 41 .

Yb 3+ single ion properties
We begin our discussion of Yb 2 GaSbO 7 with the single-ion properties of the magnetic Yb 3+ ions. We first estimated the crystal field parameters for Yb 2 GaSbO 7 using the scaling analysis procedure 42 that has been employed successfully for many other pyrochlore systems. The crystal field parameters used for the scaling were the fitted values for Yb 2 Ti 2 O 7 in Table 1 from ref. 16 . The Yb 2 GaSbO 7 crystal field parameters, eigenfunctions, and eigenvalues obtained from this scaling analysis are presented in Supplementary Tables 1 and 2. We obtain a thermally-isolated crystal field ground state doublet with XY anisotropy for Yb 2 GaSbO 7 with g z = 2.00, g xy = 3.75, and a powder-averaged g 0 ¼ 3:27, suggesting that the single-ion properties for this system are similar to other Yb 3+ pyrochlores. The g z , g xy , and g 0 values yield a crystal field moment μ CEF ¼ and an effective moment μ eff ¼ ffiffi 3 p 2 g 0 ¼ 2:83 μ B . The latter compares favourably to both the value of 3.04(5) μ B (μ 0 H = 0.1 T, Supplementary Table 3) and the previously reported value 31 of 3.15(5) μ B (μ 0 H = 0.004 T), with both values obtained from fitting the low-temperature DC magnetic susceptibility.
Inelastic neutron scattering data were also collected on the neutron spectrometer SEQUOIA at ORNL in an effort to refine the crystal field parameters for Yb 2 GaSbO 7 . While this data confirmed that the ground state doublet was well-isolated from the first excited state at 72 meV (see Supplementary Fig. 2), the B-site mixing led to a broadening of the crystal field excitations that hindered the ability of conventional models to account for the data. This broadening in energy is consistent with the presence of disorder in the local environment surrounding the Yb 3+ ions, as has been previously observed in Tb 2 Sn 2−x Ti x O 7 27 . This disorder also likely causes a distribution in the strength of the magnetic interactions between neighbouring Yb 3+ ions. Notably, the energy broadening observed here prevents us from resolving the two lowest crystal field excitations, which instead yield a single energy peak in the data. This finding is quite different from inelastic neutron scattering results reported previously for defective (i.e. stuffed) single crystalline Yb 2 Ti 2 O 7 16 , which shows only slightly broader crystal field excitations compared to stoichiometric powder samples. Therefore, the disorder induced by the mixed B-site in Yb 2 GaSbO 7 is much larger as compared to other Yb pyrochlores.
Initial characterisation of zero-field ground state Now that the single-ion properties have been discussed for Yb 2 GaSbO 7 and significant disorder has been established in this system, we turn to its cooperative (collective) magnetic properties. A summary of the AC susceptibility data is presented in Fig. 1. A clear peak is observed at T * = 350 mK in the zero-field data shown in Fig. 1a, coinciding with the plateau of the relaxation rate previously observed in muon spin-relaxation measurements 31 . Notably, the temperature of the peak maximum exhibits no frequency dependence, in sharp contrast to expectations for a spin glass. An advantage of the in-house instrument used to measure the AC susceptibility is its ability to capture higher-order harmonics 43,44 , providing another useful way to identify the origin of phase transitions. For example, a paramagnetic to spin-glass transition would only show odd-order harmonics, while ferromagnetic transitions would exhibit both even and odd order harmonics 43 . Therefore, the presence of the second harmonic shown in Fig. 1c provides additional support that the cusp does not arise from a conventional spin-glass transition. Figure 1b, d show the zero-field ground state observed here is fragile, as small DC fields rapidly suppress both the temperature of the cusp and the strength of the second harmonic signal.
To search for evidence of a thermodynamic phase transition near 350 mK, the heat capacity of Yb 2 GaSbO 7 was measured down to 50 mK (Fig. 2). A low-temperature upturn from a nuclear Schottky anomaly 45 arising from hyperfine splitting experienced by a small fraction of Yb 3+ nuclear spins was found in the raw data and it has been subtracted off here (see Supplementary Note 3 and Supplementary Fig. 3). The magnetic specific heat C mag , plotted in Fig. 2b for both 0 and 3 T, was isolated by subtracting off the normalised lattice contribution of Lu 2 GaSbO 7 . The entropy release was then obtained by integrating C mag /T as a function of temperature and is shown in Fig. 2c. Notably, only 87% of the expected Rln(2) entropy is recovered in zero field up to 25 K (see Supplementary Note 3 and Supplementary Fig. 4), although the missing entropy returns by applying a 3 T field. A comparison of the heat capacity measurements on polycrystalline Yb 2 GaSbO 7 and Yb 2 X 2 O 7 (X = Ge, Pt, Ti, Sn) 46-50 is presented in Fig. 2d. No sharp anomaly that is expected for a long-range ordering phase transition is visible in the Yb 2 GaSbO 7 data.
Instead, there is a very weak feature present at a temperature comparable to the AC susceptibility cusp, as shown in Fig. 2d and Supplementary Fig. 3a. However, the broad feature observed in the heat capacity data of other Yb 3+ pyrochlores and generally attributed to the build-up of spin fluctuations is also apparent here. For Yb 2 GaSbO 7 , this feature is centred at 2.3 K in zero field, which is very similar to the value of 2.4 K found for Yb 2 Pt 2 O 7 (a = 10.09 Å) 48 . This observation is consistent with previous literature that has established a correlation between the temperature scale of this broad heat capacity feature and the lattice constant of Yb 3+ pyrochlores 4 . Notably, this broad feature is apparent in the heat capacity data over the entire field range measured (0-9 T) and it shifts up in temperature with increasing field as shown in Supplementary Fig. 3c.
In zero applied magnetic field, the presence of residual entropy and the lack of a sharp anomaly in the specific heat suggest that the Yb 2 GaSbO 7 ground state may be magnetically disordered. This is borne out by our elastic neutron scattering experiments on the HB-1A instrument at ORNL, which found no magnetic signal at k = 0 positions down to 70 mK ( Supplementary Fig. 5), and by our polarised neutron scattering measurements on the D7 diffuse scattering diffractometer discussed below. In particular, the latter measurements show neither new magnetic Bragg peaks, nor depolarisation of the incident beam that would result from weak ferromagnetic ordering.

Field-induced magnetic order
Complementary DC magnetic susceptibility and HB-2A neutron diffraction measurements presented in Fig. 3, and Supplementary Figs. 6 and 7, probed the evolution of the magnetic ground state in applied magnetic fields. It is worth noting that the diffraction data in both zero and applied fields are well-explained by a structural model where the B-site is randomly occupied by a 1:1 ratio of Ga 3+ and Sb 5+ and no additional nuclear Bragg peaks that would be indicative of long-range charge ordering are observed. Below a critical field μ 0 H c~1 .5 T, the susceptibility data show Curie-Weiss behaviour down to 2 K, while the diffraction data reveal no magnetic Bragg peaks down to 1.2 K. By contrast, in applied magnetic fields greater than μ 0 H c , a net moment develops in the susceptibility measurements and k = 0 magnetic Bragg peaks appear in the neutron diffraction data. The k = 0 magnetic structures allowed by symmetry have been discussed several times for the pyrochlore family 15,17,51 and they were all considered in the present case; more details are provided in Supplementary Tables 4 and 5. The best refinement of the 2 T data corresponds to a magnetic structure associated with the Γ 9 irreducible representation that has been identified as the zero-field ground state in many other Yb 3+ pyrochlores [34][35][36][37]48,50,52 . To highlight the magnetic portion of the refinement, we have plotted the difference of the diffraction profiles collected at 2 T and 0 T in Fig. 3c. We find an ordered Yb 3+ moment of 1.4(1) μ B and a spin canting angle (relative to the global [001] direction) of~−11 ∘ . We note that the negative canting angle indicates that the moments move away from their respective local 〈111〉 directions. This magnetic structure is consistent with the net moment observed in the DC susceptibility measurements and arises from a splayed XY ferromagnet, which is the same magnetic structure reported previously for Yb 2 Sn 2 O 7 50 . A schematic of this spin configuration is presented in Fig. 3d.
Evidence for a dynamical ground state in zero field Low-energy inelastic neutron scattering and polarised neutron diffraction experiments provide additional insight into the nature of the zero-field ground state for Yb 2 GaSbO 7 . Inelastic data were collected on the disc-chopper time-of-flight spectrometer DCS at NIST and are plotted in Fig. 4a. The scattering is broad in energy and Q at all measured temperatures. At small values of Q~0.3 Å −1 , the low-energy spectral weight increases with decreasing temperature, suggesting mode softening. Diffuse magnetic scattering data were also measured on the polarised diffuse scattering spectrometer D7 at the ILL, using the 6 pt. xyzpolarisation analysis method 53 to separate out contributions from nuclear coherent, nuclear spin incoherent, and magnetic scattering to the total cross-section (see Supplementary Fig. 8). We focus on the magnetic scattering in the discussion that follows. These data are not energy resolved and effectively integrate the magnetic scattering over energy transfers up to λ. Figure 4b compares the energy-integrated D7 data with the inelastic DCS data integrated over 0.15 ≤ E ≤ 1.50 meV at approximately 50 mK; both datasets were independently converted into absolute intensity units by normalising to the nuclear Bragg scattering in each case. The close agreement between the inelastic DCS data and the energy-integrated D7 data suggests that the magnetic scattering is predominantly inelastic at low temperature. The Yb 3+ magnetic moment was further obtained through the zeroth total moment sum rule 54 , where 2 γro 2 is 13.77 sr b −1 , f(Q) is the Yb 3+ isotropic magnetic form factor, and μ eff is the effective magnetic moment. Integrating the magnetic double differential cross-section over energy and 0.3 ≤ Q ≤ 2.0 Å −1 yields an effective Yb 3+ total moment of μ eff = 3.13(5)μ B from D7 and a dynamic moment of 3.10(2)μ B from DCS, with both values in clear agreement with the value of 3.04(5) μ B obtained from fitting our 0.1 T DC susceptibility data to a Curie-Weiss law between 2 and 15 K. While this comparison may be affected by systematic differences between measurements on two different instruments, the close agreement between the total and dynamic moments strongly suggests that the scattering at base temperature is mainly inelastic. In particular, the fraction of inelastic scattering is significantly enhanced compared to the value of 1 J eff þ1 = 67% expected for a J eff = 1/2 ordered or spin-glass state 55,56 .
To understand the equal-time spin correlations of Yb 2 GaSbO 7 , we performed reverse Monte Carlo (RMC) refinements of the magnetic diffuse scattering measured on D7 at temperatures of 55 mK, 1.2 K, 5.0 K, and 10.0 K. The RMC approach fits configurations of magnetic moments S i directly to experimental data without assuming a model of the underlying interactions 57,58 . Refinements were performed using 3 × 3 × 3 supercells of the crystallographic unit cell and initialised with random moment orientations. At each temperature, two refinements were performed: the first assumed XY moments, while the second placed no constraints (Heisenberg) on the moment distribution. The calculated g-tensor anisotropy is intermediate between Heisenberg and XY limits; however, both limits yielded similar results and so we did not consider the intermediate case. Figure 4c shows fits to data and Fig. 4d shows the temperature dependence of the n.n. and next nearest neighbour (n.n.n.) magnetic correlation functions. The values have been normalised such that 〈S(0) ⋅ S(r)〉 = 1 if all moment pairs separated by distance r were parallel. The equal-time correlations are weakly antiferromagnetic for n.n. and n.n.n.'s at base temperature, but are extremely weak at all measured temperatures- approximately an order of magnitude smaller than for a frustrated classical Heisenberg pyrochlore antiferromagnet at low temperature, where a value of −1/3 would be obtained at the n.n. distance 59 . This simulation result is obtained because the Qdependence of the data deviates only slightly from the square of the magnetic form factor, and is unaffected by the possible magnetic anisotropy. Taken together with the large inelastic spectral weight, it appears consistent with a significant role of quantum fluctuations. From our combined DCS and D7 results, we can ascertain that Yb 2 GaSbO 7 has a predominantly dynamical magnetic ground state in zero field.
To understand further the dynamic properties of Yb 2 GaSbO 7 , we consider the imaginary part of the dynamic magnetic susceptibility. We obtain this quantity from our inelastic DCS data as χ 00 ðωÞ / ½1 À expðÀβ_ωÞ R IðQ; ωÞdQ, where the Q integral was taken over 0.5 ≤ Q ≤ 2.0 Å −1 . A peak at non-zero energy transfer is observed at all measured temperatures in χ″(ω), as shown in Fig.  4e. Successful fitting of the data to the damped harmonic oscillator (DHO) model given by 60 confirmed that this mode is underdamped at all temperatures.
The temperature dependence of the fitted mode energy ω 0 and relaxation rate Γ are both shown in Fig. 4f. The presence of an underdamped mode over a wide temperature range is reminiscent of a singlet-triplet excitation 61 , suggesting the intriguing possibility that chemical disorder drives a random-singlet phase in Yb 2 GaSbO 7 62,63 . Notably, there is no apparent change in the dynamic response measured by neutrons as the 350 mK peak in AC susceptibility is traversed, and the majority of the spectral weight remains inelastic below 350 mK, as discussed above. These results suggest that this AC susceptibility peak may be generated by only a small fraction of the spin system freezing, so that the majority of the spectral weight remains unaffected.

DISCUSSION
Our experimental results for Yb 2 GaSbO 7 are summarised in the field-temperature phase diagram presented in Fig. 5. The phase boundaries were obtained from AC susceptibility (temperature of the peak/cusp maximum observed in the in-phase component of the first harmonic), DC susceptibility (temperature of the minimum in dχ DC dT ), heat capacity (temperature of the broad anomalies), and neutron scattering data from WAND (temperature of increased intensity for order parameter scans of the (113) magnetic Bragg peak, see Supplementary Fig. 9 for some representative data). At first glance, the phenomenology of Yb 2 GaSbO 7 is very similar to Yb 2 Ti 2 O 7 , as some samples of the latter 64 and Yb 2 GaSbO 7 both host a field-induced Γ 9 long-range ordered state and a dynamic, correlated phase in zero field at low temperatures. However, there is now a growing amount of experimental evidence that pristine samples of Yb 2 Ti 2 O 7 exhibit a splayed ice-like ferromagnetic state at low temperatures even in zero field [34][35][36][37] . The nature of this ordering is extremely sensitive to chemical disorder, specific details of sample preparation/single crystal growth, and "stuffing" effects 65 . Very recent neutron scattering measurements 39 suggest that this extreme sensitivity to disorder may arise from the close proximity of Yb 2 Ti 2 O 7 to the Γ 5 antiferromagnet -Γ 9 canted ferromagnet phase boundary of the theoretical phase diagram for  41 . Additional evidence supporting the proximity of Yb 2 Ti 2 O 7 to a phase boundary comes from systematic studies of the magnetic ground states of other Yb 3+ pyrochlores, as the ordered spin configuration evolves from a Γ 9 canted ferromagnet to a Γ 5 antiferromagnet with increasing chemical pressure 17 . The closest chemical analogue to Yb 2 GaSbO 7 in terms of the lattice constant, and therefore spatial separation between neighbouring Yb 3+ ions, is Yb 2 Pt 2 O 7 48,66 . Interestingly, the in-phase, first harmonic component of the AC susceptibility exhibits a cusp with a clear frequency dependence and the specific heat shows a sharp anomaly; both features are centred at 0.3 K in zero field. These results clearly establish that Yb 2 Pt 2 O 7 has a canted ferromagnetic ground state similar to pristine samples of Yb 2 Ti 2 O 7 , although the precise spin configuration for the platinate has not been determined.
We suggest two plausible explanations for the drastic difference in the magnetic behaviour of Yb 2 GaSbO 7 , compared to Yb 2 Ti 2 O 7 and Yb 2 Pt 2 O 7 , despite the similar lattice constant of the latter. First, the intrinsic chemical disorder of Yb 2 GaSbO 7 is likely to generate correspondingly larger disorder in exchange couplings. Intriguingly, this does not cause a conventional spin-glass transition here; instead, our inelastic results are consistent with excitations from a random-singlet ground state. Second, the suppression of conventional magnetic ordering may arise from a fine-tuning of the n.n. superexchange pathways due to the difference in the B-site ions, such that this system is closer to the Γ 5 -Γ 9 phase boundary than any other Yb 3+ pyrochlore magnet studied previously. Multiphase competition has also been discussed for Er 2 Pt 2 O 7 51 , and therefore this phenomenon appears to be a hallmark of many XY pyrochlores. These two effects are not mutually exclusive, and it is possible that disorder and magnetic interactions conspire here to suppress conventional long-range magnetic order or spin-glass formation. There are intriguing similarities between the behaviour of Yb 2 GaSbO 7 and the triangular-lattice quantum-spin-liquid candidate YbMgGaO 4 , in which frustration and chemical disorder are implicated in the behaviour of effective spin- 1 2 Yb 3+ ions 6,67 . Notably, YbMgGaO 4 also shows a low-temperature peak in AC susceptibility 68 that is accompanied by a plateau in the spinrelaxation rate obtained in μSR measurements 69 ; however, inelastic neutron scattering measurements below this transition suggest that, at most, only a small fraction of the spins are frozen 70,71 . These commonalities hint at a common mechanism for spin-liquid-like behaviour in both materials.
In summary, we present evidence for persistent spin dynamics in Yb 2 GaSbO 7 down to 50 mK, which makes this material an intriguing candidate for spin-liquid or frustrated random-singlet behaviour. While the magnetic properties of Yb 2 GaSbO 7 exhibit some similarities to other Yb 3+ pyrochlores, the key difference is the lack of magnetic order for Yb 2 GaSbO 7 down to mK temperatures in the absence of an applied magnetic field. Although single crystals may be difficult to obtain due to the volatile nature of the chemical constituents, inelastic neutron scattering on single crystalline Yb 2 GaSbO 7 is highly desirable. The nature of the dynamical ground state could be fully explored with a zero-field measurement. Furthermore, an experiment in the field-induced ordered state would establish the magnetic Hamiltonian for this material, which is essential information for Fig. 3 Field-induced magnetic order in Yb 2 GaSbO 7 . a DC magnetic susceptibility data of Yb 2 GaSbO 7 for various applied fields. Note the development of a net ferromagnetic moment as the field is increased. b Powder neutron diffraction data from HB-2A for Yb 2 GaSbO 7 measured at T = 1.2 K in zero field. The red curve represents a Rietveld refinement of the data to the pyrochlore structure. No magnetic Bragg peaks are visible in this data. c 2 T −0 T difference plot of the T = 1.2 K neutron powder diffraction data, which ensures that the field-induced magnetic scattering can be isolated. A Rietveld refinement using the Γ 9 canted ferromagnetic structure is superimposed on the data. Uncertainties in the data are derived from Poisson statistics and represent one standard deviation normalised by the detector efficiency correction. d A schematic of the canted ferromagnetic structure, with the inset depicting the spin arrangement on a single tetrahedron.
assessing its proximity to a phase boundary in the theoretical phase diagram for rare earth pyrochlore magnets.

Sample Preparation
Polycrystalline samples of Yb 2 GaSbO 7 and a non-magnetic mixed B-site lattice standard analogue Lu 2 GaSbO 7 were both synthesised by a standard solid state reaction of RE 2 O 3 (RE = Yb or Lu), Ga 2 O 3 , and Sb 2 O 5 , as previously reported by Strobel et al. 22 .
Rietveld refinement of the room-temperature laboratory x-ray diffraction pattern confirmed the presence of single-phase pure RE 2 GaSbO 7 (RE = Yb or Lu), possessing Fd3m symmetry with no discernible impurities. The refined room-temperature lattice constant of a = 10.1047(1) Å for Yb 2 GaSbO 7 agrees with previously reported values 22,72 . A representative x-ray diffraction profile for Yb 2 GaSbO 7 with its corresponding Rietveld refinement superimposed onto the data is presented in Supplementary Fig. 1.   Fig. 4 Evidence for zero-field dynamical ground state in Yb 2 GaSbO 7 . Colour contour plots of the neutron scattering intensity measured on the DCS at (a) 50 mK, 1 K, and 10 K. b Comparison of the normalised scattering intensity measured on DCS to the magnetic cross-section measured on D7 at~50 mK. c Magnetic differential cross-section data from D7 and (d) the associated nearest neighbour and next-nearest neighbour spin correlations extracted from the fits shown in panel (c) and described in the main text. The temperature of the AC susceptibility cusp measured in zero DC field, T AC , is indicated as a reference. For the purposes of clarity, a vertical offset of 0.2 b sr −1 (Yb 3+ ) −1 has been introduced for each successive temperature in (c). e Calculated fits of the imaginary part of the Q-integrated ([0.5, 2.0] Å −1 ) dynamic magnetic susceptibility to the damped harmonic oscillator model 60 (Eq. (2)) with (f) the temperature dependence of its corresponding fitted parameters: Γ and ω o , defining the peak width and centre, respectively. Uncertainties in the data are statistical in origin and represent one standard deviation, while uncertainties in the fit parameters represent standard error.

Bulk characterisation
The field and temperature dependence of the magnetisation of polycrystalline Yb 2 GaSbO 7 was measured using a 9 T Dynacool Physical Property Measurement System (PPMS) (The identification of any commercial product or trade name does not imply endorsement or recommendation by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose.) employing the vibrating sample magnetometer (VSM) option. The heat capacity of Yb 2 GaSbO 7 and its corresponding lattice analogue Lu 2 GaSbO 7 were first measured down to 350 mK with the 3 He option on a 9 T PPMS from Quantum Design, while subsequent measurements employed the dilution refrigerator (DR) option to access a base temperature of 65 mK.
The AC susceptibility of polycrystalline Yb 2 GaSbO 7 was collected with a warming rate of 7.6 mK/min in a zero-field-cooling (ZFC) process at the National High Magnetic Field Laboratory (MagLab) in Tallahassee, Florida using an in-house set-up 44 that facilitated the measurement of both the linear and non-linear components. The raw data (voltage signal) was normalised to both the driving frequency and AC field strength. In general, the magnetisation of a material (with μ 0 set to 1) can be expressed as 43,44 : Applying an AC magnetic field H = H 0 sin(ωt) induces a voltage E in the pick-up coil given by: where A is a numerical factor that depends on the coil dimensions and the filling factor of the sample, and: χ t 0 , χ t 1 H 0 , and 3=4χ t 2 H 2 0 represent the first, second, and third harmonics that we measure. When the applied AC field is small, the first harmonic is essentially equivalent to the linear AC susceptibility χ 0 .

Neutron scattering
High energy inelastic neutron scattering (INS) measurements were performed on the direct-geometry time-of-flight chopper spectrometer SEQUOIA 73 of the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL). Yb 2 GaSbO 7 powder was loaded in a cylindrical Al can, cooled to a base temperature of 5 K using a closed cycle refrigerator, and a powder-averaged (Q, E) spectrum was collected with an incident energy E i = 150 meV. The fine-resolution Fermi chopper operated at a frequency of 600 Hz and the t 0 chopper was spun at 90 Hz. Data normalisation with a vanadium standard ensured that differences in detector efficiencies and solid angle coverage were properly accounted for. The INS intensity from this experiment is plotted as ki kf ∂ 2 σ ∂Ω∂E , where k i and k f are the incident and final neutron momenta respectively, Q is the momentum transfer, E is the energy transfer, and ∂ 2 σ ∂Ω∂E is the double differential cross-section 74 . This quantity is proportional to the powder-averaged dynamical structure factor S(Q, E).
The absence of magnetic order in zero field down to 70 mK was confirmed with the fixed incident energy triple-axis spectrometer HB-1A of the High Flux Isotope Reactor (HFIR) at ORNL (λ = 2.37 Å). Yb 2 GaSbO 7 powder was loaded in a Cu can and elastic scattering was measured both at 70 mK and 800 mK at Q values corresponding to the k = 0 Bragg positions. An excellent signal-to-noise ratio was achieved by generating an incident neutron beam with a double-bounce monochromator system and placing a pyrolytic graphite (PG) crystal analyser for energy discrimination before the single He-3 detector. The incident beam also had extremely low higherorder wavelength contamination due to the use of two PG filters. An energy resolution at the elastic line just over 1 meV (full-width half-maximum) was obtained by a using a collimation configuration of 40 0 -40 0 -40 0 -80 0 .
Neutron powder diffraction (NPD) was performed on the high-resolution powder diffractometer HB-2A 75 of HFIR at ORNL to investigate the evolution of the magnetic ground state for Yb 2 GaSbO 7 in applied magnetic fields up to 4 T and temperatures down to 1.2 K. Pressed powder of Yb 2 GaSbO 7 was loaded in a cylindrical Cu can, and the data were collected with a neutron wavelength of 2.41 Å and a collimation of open-21 0 -12 0 . Rietveld refinements were performed using the FULLPROF software suite 76 and the magnetic structure symmetry analysis was performed using SARAh 77 .
Additional neutron diffraction data were collected on the wide-angle neutron diffractometer WAND (λ = 1.48 Å) at the HFIR. Yb 2 GaSbO 7 powder was loaded in a cylindrical Cu can and data was collected at temperatures between 1.5 and 10 K, with applied magnetic fields between 0 and 4.5 T. Contributions to the phase diagram (Fig. 5) from the WAND data were determined by measuring the temperature dependence of the (113) Bragg peak intensity at fixed applied magnetic fields.
The low-energy magnetic fluctuations of Yb 2 GaSbO 7 were measured on the Disk Chopper Time-of-Flight Spectrometer DCS 78 at the National Institute of Standards and Technology (NIST) Center for Neutron Research (NCNR). Yb 2 GaSbO 7 powder was loaded in a cylindrical Cu can and then placed in an ICE dilution fridge insert of a cryostat. An incident wavelength λ = 4.8 Å in low-resolution mode was chosen, corresponding to a flux of~1 10 6 neutrons/s with an elastic line resolution of approximately 0.125 meV (fullwidth half-maximum) and an accessible Q range of [0.11, 2.45] Å −1 in the elastic channel.
Polarised diffuse neutron scattering experiments were performed on the Diffuse Scattering Spectrometer D7 79 at the Institut Laue-Langevin (ILL). An incident wavelength λ = 4.8 Å was selected by a double-focusing pyrolytic graphite monochromator. Data were collected in non-time-of-flight mode, leading to the extraction of the integrated scattering intensity with energy transfers up to an E i = 3.5 meV. Yb 2 GaSbO 7 powder was loaded in a double-wall cylindrical Cu can and then placed in the dilution fridge insert Fig. 5 Phase diagram of Yb 2 GaSbO 7 as determined by magnetic susceptibility, heat capacity, and neutron scattering measurements. Shaded regions include a paramagnet (PM), an XY splayed ferromagnet (LRO SFM), a correlated, dynamic phase (DYN) and a regime with minority spin freezing (MSF). The inset depicts an enlarged version of the low-temperature, low-field portion of the phase diagram. The points on the phase diagram were derived from fitting the relevant features of the respective datasets to simple functions, and the error bars for the points correspond to the standard errors from these fits. of a cryostat. Data normalisation by a vanadium standard ensured that differences in detector efficiency and solid angle coverage were taken into account. Scattering contributions from an empty and a cadmium-filled sample holder were added together and weighted by the sample transmission to estimate the instrument background. Corrections for polarisation efficiency of the supermirror analysers were made by using the scattering from amorphous quartz. Equal counting times were spent on measuring the scattering along the x, y, and z directions. The 6 pt. xyzpolarisation analysis method 53 was used to separate scattering contributions from the magnetic, nuclear coherent, and nuclear spin incoherent scattering at different momentum transfers Q. The non-spin-flip and spinflip scattering along each of the three directions were measured with a time ratio of 1:4.

DATA AVAILABILITY
Raw bulk characterisation data were generated at the University of Winnipeg, the University of British Columbia, the University of Edinburgh, and the National High Magnetic Field Laboratory. Raw neutron scattering data were generated at the SNS (SEQUOIA), HFIR (HB-1A, HB-2A, and WAND), NIST (DCS), and the ILL (D7). Data included in this study are available from the corresponding authors upon reasonable request.