## Introduction

RuI3 and RuBr3 are recent additions to the RuX3 family (X = Cl, Br, I) of layered Ru-based trihalides (Fig. 1a). The first member, α-RuCl3 (in the following ‘RuCl3’) has attracted considerable attention in recent years as a candidate material for the Kitaev honeycomb model1. RuCl3 is a spin–orbit-assisted Mott insulator2,3,4,5 whose magnetic low-energy degrees of freedom can be described in terms of jeff = 1/2 moments that interact through strongly anisotropic exchange2,6,7,8. While the material enters a so-called zigzag antiferromagnetic order (Fig. 1b) at low temperatures TN ≈ 7 K4,9,10, various experiments at finite temperature11,12,13,14 or at finite magnetic field4,15,16,17,18 have been interpreted as hallmarks of Kitaev physics, a subject which is presently under intensive debate19,20,21,22,23,24.

Recently, a sister compound with a heavier halogen, X = Br, was synthesized25. Analogous to RuCl3, it is insulating and shows zigzag magnetic order, albeit with higher Néel temperature TN = 34 K25. In contrast to RuCl3, the authors of ref. 25 reported a Weiss constant with dominant antiferromagnetic interactions and a direction of the zigzag-ordered moment different from RuCl3 and argued that this deviation suggests closer proximity to the pure Kitaev model.

To complete the RuX3 family, two independent groups have now synthesized RuI3 with the even heavier halogen iodine26,27. In contrast to the two ‘sibling’ compounds, a quasi-metallic behavior was observed in RuI3, questioning the description in terms of localized jeff = 1/2 moments. Even though the dc resistivities measured in RuI3 are orders of magnitude smaller than those of RuCl3 or RuBr3, the reported values of 10−3–10−2Ω cm26 are uncharacteristically large for metals or even typical bad metals28, and practically temperature-independent. While neither of the groups found clear signatures of magnetic ordering26,27, they reported different behaviors of the magnetic susceptibility, which is either found to be temperature-independent26 or with a strong upturn at low temperatures27, suggesting that sample quality plays a crucial role.

In order to understand the apparently distinct behavior of this family of trihalide materials, in this work, we analyze the available experimental data and perform a detailed comparative study of the electronic and magnetic properties of the systems via first-principles calculations and extracted low-energy models. We find that: (i) The behavior of RuI3 is not that far from RuCl3 and RuBr3 and the variations across the series are more quantitative than qualitative. (ii) Pristine samples of RuI3 should be insulating with highly anisotropic magnetic exchange and nearly vanishing conventional Heisenberg interaction. We argue that the reported metallic behavior in RuI3 could have its origin in sample quality. (iii) The magnetism in the Mott insulator RuBr3 has predominantly ferromagnetic interactions, in contrast to what is suggested by the Curie–Weiss analysis of ref. 25. We show that such interactions are consistent with experiment when taking into account spin-orbit coupling effects in the Curie–Weiss behavior.

Our study derives model parameters and magnetic Hamiltonians for the whole RuX3 family from ab-initio, which will be useful for future theoretical studies of these systems. In contrast to the usual model derivations that only include local spin–orbit coupling (SOC) on the magnetic ion2,6,8, our approach includes all SOC effects in the crystal. In fact, we show that SOC from the ligands leads to significant deviations from the Ruthenium-only SOC picture in the case of RuBr3 and RuI3.

## Results and discussion

### Comparative analysis of experiments

In the following, we analyze the reported electrical resistivity, specific heat, and magnetic susceptibility data for RuX3 (X = Cl, Br, I)10,25,26,27,29.

In Fig. 1c we summarize the temperature dependence of the experimental resistivity data10,25,26,27 in all three compounds. In RuI3, the resistivity has a weak27 or almost no26 temperature dependence (Fig. 1c). Traditionally, metals are classified as materials where the resistivity ρ increases with temperature, distinguishing conventional metals (e.g., Cu) as those where in clean samples at temperatures roughly 300–600 K, ρ ~ 10−6 to ~10−5Ω cm, and bad metals as those with resistivities of ~1−10 mΩ cm. This range is shown as a background shading in Fig. 1c. The reported resistivities for RuI3 (ρ ~ 40 mΩ cm27 and ρ ~ 4 mΩ cm26) are high even for bad metals, surpassing the Ioffe–Regel limit by more than an order of magnitude. Even more relevant, the lower-resistivity set of data26 shows no discernible temperature dependence at all, while the data in ref. 27 show a very weak positive derivative dρ/dT, but the absolute value is above anything traditionally considered metallic.

Seemingly, as also pointed out in ref. 27, electron transport in existing RuI3 samples may be contaminated by grain boundaries. One possibility to interpret the measurements is that the pristine material is metallic, but insulating grain boundaries prevent percolation. Then, the in-grain resistivity can be neglected and what is measured is the resistivity of the insulating grain boundaries. In that case, however, thermal activation of carriers in the boundaries should give a positive temperature gradient of the resistivity, which is not observed. The opposite scenario is that of an insulating behavior in the bulk and (possibly bad) metallic one between the grains. In that case, the large resistivity reflects the small relative volume of metallic boundaries, where the transport is dominated by the residual resistivity. This scenario is compatible with the observations. Morphology of the grain boundaries can vary wildly depending on the growth conditions, including but not limited to vacancies, twins, dislocation, and plain chemical dirt. Grain boundaries in semiconductors are often observed to be metallic. Apart from grain boundaries contaminating resistivity measurements, disorder (in form of vacancies, stacking faults, etc.26,27) could promote the bulk metallic phase over the Mott-insulating one, as has been shown for example for the Mott insulator κ-(BEDT-TTF)2Cu[N(CN)2]Cl30. Indeed, in our first-principles calculations discussed below, we find the ideal RuI3 to already be quite close to a Mott–metal transition.

Turning to the sibling compounds RuCl3 and RuBr3, the resistivity (Fig. 1c) decreases with temperature, as expected for Mott insulators, and both systems show an approximate exponential activation gap behavior, $${E}_{{\rm {g}},{{{\rm{eff}}}}}(T)=-{k}_{{{{\rm{B}}}}}{T}^{2}({{{\rm{d}}}}\ln \rho /{{{\rm{d}}}}T)$$, although with a significant blue-shift of the gap with increasing temperatures.

Considering specific heat data in the compounds, the specific heat for RuCl3 displays a well-defined peak at TN ≈ 7 K denoting the onset of the zigzag order, while the onset of long-range magnetic order in RuBr3 is observed by a kink at TN = 34 K25. None of this is observed for RuI326,27. In Table 1 we summarize specific heat parameters reported experimentally25,26,27,31, where γ (β) is the T-linear (T3) contribution to C(T).

In RuI3, the T-linear contribution, even though contaminated by an extrinsic raise at small temperatures in ref. 27 attributed to the nuclear quadrupole moment of Ru, yields γ ~ 15−30 mJ K−2 mol−1 26,27. From our electronic structure calculations of RuI3 shown below, we find that the unrenormalized metallic (i.e., nonmagnetic, not U-corrected) density of states corresponds to γ0 ≈ 3 mJ K−3 mol−1, suggesting a mass renormalization (if this γ is intrinsic) of a factor of 7–12. In the scenario where the metallic grain boundaries take up a sizeable fraction of the sample volume, this renormalization shall be even stronger, encroaching into the heavy fermions domain. This suggests that the origin of the anomalously large residual heat capacity may not be related to intrinsic metallicity. It is worth noting that the T3 term β of C(T), on the other hand, is rather reasonable for the three systems and scales roughly as the harmonic average $${M}_{{{{{\rm{RuX}}}}}_{3}}$$ of the atomic masses (last column in Table 1).

We now turn our attention to magnetic susceptibility measurements. Figure 2a summarizes the powder-averaged measured magnetic susceptibilities χ(T) as reported in refs. 25,26,27,32. At low temperatures, the RuCl3 data32 shows a clear signature of a transition to the ordered magnetic phase at 7 K. For RuBr3, the Néel transition TN ≈ 34 K is less apparent from the susceptibility, but the maximum in dχ/dT is consistent with the distinct transition seen in NMR relaxation measurements25. The experimental report on powder samples of RuBr3 utilized a standard Curie–Weiss (CW) fit, yielding an average Curie–Weiss temperature $${{{\Theta }}}_{{{{\rm{std}}}}}^{{{{\rm{avg}}}}}=-58$$ K25, indicating predominantly AFM interactions. However, as we have recently shown33, the Weiss constants obtained with such a standard CW fit may not anymore reflect the intrinsic exchange couplings in the case of significant SOC in the material, as is the case for the Ru-based trihalides. With SOC, temperature-dependent van-Vleck contributions can arise, which can be effectively captured in a temperature-dependent magnetic moment μeff(T, Δ)33, as shown for Δ = 0.018 eV in Fig. 2b, where Δ can be directly associated to the crystal field splitting resulting from the distorted octahedral environment of Ru. In fact, for the sister compound RuCl3, a standard CW fit would lead to $${{{\Theta }}}_{{{{\rm{std}}}}}^{{{{\rm{avg}}}}}=-20$$ K, whereas an improved CW fit taking into account such van-Vleck-like contributions33 provides CW constants Θ = + 55 K for the magnetic field in the honeycomb plane and Θ = + 33 K for the out-of-plane field, revealing an average CW constant, $${{{\Theta }}}^{{{{\rm{avg}}}}}=\frac{2{{{\Theta }}}^{\parallel }+{{{\Theta }}}^{\perp }}{3}$$, of ≈48K. This indicates predominant ferromagnetic (FM) interactions, as they have become established for the magnetic Hamiltonian in RuCl334,35,36,37.

Considering a similar strategy (see the “Methods” section), we fit the average susceptibility χavg of RuBr325. However, since the crystal-field parameter Δ primarily controls the in-plane vs. out-of-plane anisotropy, and for RuBr3 only powder-averaged data are available, we do not aim at extracting Δ by fitting. Instead, we first fix Δ using our first-principles calculations, enforcing $${\mu }_{{{{\rm{eff}}}}}^{\parallel }/{\mu }_{{{{\rm{eff}}}}}^{\perp }(T=0\,{\rm {K}})$$ g/g, where g/g are taken from quantum chemistry calculations (see Fig. 5a, discussed below). This leads to Δ = 0.018 eV. The best CW fit accounting for the implied μeff(T, Δ = 0.018 eV) (shown in Fig. 2c) yields Weiss constants Θ ≈ 5 K, Θ ≈ 17 K and Θavg ≈ 9 K, which are positive, indicating predominately ferromagnetic interactions for RuBr3, as seen before in RuCl3. In Fig. 2d we further analyze how the best-fit Weiss constants evolve for other choices of Δ. Indeed, for a wide range of reasonable Δ around the first-principles value (indicated by the dashed vertical line), the average Weiss constant Θavg remains positive.

Importantly, for both materials, a standard residual ‘background’ term has to be included in the fitting, which in our case, depending on the material (RuBr3 or RuCl3) ranges from ~−3.5 × 10−﻿﻿4 emu/mol to 1.5 × 10−4. This is of the same order of magnitude as the corresponding term in RuI3 (~3 to 8 × 10−4 emu/mol)26,27. Since in the former cases an intrinsic Pauli origin can be excluded, this observation also casts doubts on a metallic interpretation of this term in RuI3. Actually, the two available susceptibility measurements on RuI3 display different behaviors, one nearly temperature-independent26, and the other27 showing a Curie-like rise at low temperatures, where a standard CW fit yields μeff = 0.53μB and $${{{\Theta }}}_{{{{\rm{CW}}}}}^{{{{\rm{avg}}}}}=-3$$ K27. These differences are consistent with our hypothesis that the measured samples consist of magnetic insulating grains surrounded by metallic boundaries. Then, the samples with larger resistivity data27 hint at larger insulating grains, hence less metallic boundaries are present, leading to the low-temperature Curie-like upturn in the susceptibility, compared to the samples in ref. 26.

### Electronic and magnetic calculations

In the following, we present a comparison of the electronic and magnetic properties of the trihalide RuX3 family obtained from a combination of density functional theory (DFT) and exact diagonalization of ab-initio-derived low-energy models. Details of the calculations are given in the “Methods" section.

Past experience with first-principles calculations for the Ru-based trihalides4,7,38,39,40 indicates that the magnetic order and, to a considerably lesser extent, metallicity is very fragile, with several closely competing different magnetic phases. The ground states may vary depending on small changes in the crystal structure, on the way in which strong correlations are accounted for, and even on tiny details of the computational protocol. With this in mind, it is imperative to compare the calculated properties across the series, using the exact same computational setup.

For the electronic structure calculations, we consider the experimentally reported C2/m4,9 and R$$\bar{3}$$41 structures for RuCl3, and the suggested R$$\bar{3}$$ structures for RuBr325 and RuI327. Structural details of the four models are summarized in the Supplementary Information. For RuCl3, the R$$\bar{3}$$ results are shown in Supplementary Information due to very similar results to the C2/m ones.

Figure 3 shows the relativistic density of states (DOS) obtained within GGA + SOC + U as implemented in Wien2k, where a zigzag magnetic configuration with magnetic moments polarized perpendicular to the ab plane was considered. For the choice of Ueff = UJ we take as a reference the ab initio estimates for the orbitally-averaged Hubbard on-site (Uavg) and Hund’s coupling (Javg) as obtained from constrained random-phase approximation (cRPA) calculations (see the “Methods" section for calculation details). In contrast to previous cRPA estimates for RuCl342, our estimates incorporate all five d orbitals and extend to the complete Ru-based trihalide family. As shown in Fig. 4a, the effective Hubbard interaction parameters decrease with increasing ligand atomic number from Cl to I, which can be attributed to the more delocalized nature of the Ru d orbitals in RuI3 compared to RuCl3 when hybridizing with I instead of Cl.

For RuCl3 a Ueff = 2.7 eV yields both the fundamental and direct gap to be ≈1 eV (Fig. 3a) in agreement with the reported optical gap, apart from the presence of multiplets at 200 meV43. We systematically reduced Ueff to 2.1 eV for RuBr3 and 1.4 eV for RuI3 following the trend given by the cRPA results. With these values, RuBr3 shows a gap of 0.56 eV (Fig. 3b), while RuI3 shows a small gap of 0.1 eV (Fig. 3c). The gap closes in RuI3 when Ueff is further reduced to 1 eV. These results indicate a spin–orbit assisted Mott insulating state in disorder-free RuI3 samples, which is on the verge of a metal–insulator transition. Possibly, as discussed above, a disorder in the experimental samples could act as effective pressure, and bring the samples closer to or over the Mott transition as seen in other Mott insulators30. Note that these results hold regardless of the assumed magnetic pattern in the calculations.

In order to analyze the magnetic structure of the RuX3 compounds, we first consider spin-polarized total energy calculations with VASP in the GGA + SOC + U approximation (see also the “Methods” section). Detailed results are listed in the Supplementary Information. For RuCl3, the calculated energy of the ferromagnetic state EFM is very competitive with the energy of the experimentally observed zigzag ordered state EZZ: EZZEFM ≈ 2 meV/Ru. This observation is consistent with the evidence for a metastable ferromagnetic state in RuCl337,44. Correspondingly, in our effective pseudospin model of RuCl3 discussed below, classically, the energy of the ferromagnet is below that of zigzag, and only by including quantum fluctuations the zigzag ground state is recovered (as in, e.g., ref. 37). For RuBr3 we find an energy minimum for the zigzag ordering in agreement with the experiment. Interestingly, for RuI3 Néel and zigzag orders are energetically almost degenerate ENéelEZZ ≈ 1 meV/Ru, with the rest of magnetic orders we scanned being energetically rather close. All orders show very small and varying magnetic moments for Ru. These results hint at a magnetic frustration.

We proceed with the derivation of magnetic exchange models. In the first place, the magnetic Hamiltonian of RuBr3 has been suggested to be closer to the pure Kitaev limit than in RuCl325, and, secondly, with our proposed scenario of a Mott insulating state for RuI3, the question of its magnetic properties is open. To investigate these issues from the first principles, we derive via the ab-initio projED method45 the pseudospin models $${\mathcal{H}}_{\rm{eff}}={\sum}_{ij}{\bf{S}}_{i}\cdot {\mathbb{J}}_{ij}\cdot {\bf{S}}_{j}$$ of the three RuX3 compounds. Here, S stands for the relativistic pseudospin jeff = 1/2 moment2.

In the conventional parametrization of Kitaev materials, the exchange matrix $${{\mathbb{J}}}_{ij}$$ in R$$\bar{3}$$ symmetry on a nearest-neighbor Z1-bond (defined in Fig. 1) follows the form

$${{\mathbb{J}}}_{ij}=\left(\begin{array}{lll}{J}_{1}+{\nu }_{1}&{{{\Gamma }}}_{1}&{{{\Gamma }}}_{1}^{\prime}+{\eta }_{1}\\ {{{\Gamma }}}_{1}&{J}_{1}-{\nu }_{1}&{{{\Gamma }}}_{1}^{\prime}-{\eta }_{1}\\ {{{\Gamma }}}_{1}^{\prime}+{\eta }_{1}&{{{\Gamma }}}_{1}^{\prime}-{\eta }_{1}&{J}_{1}+{K}_{1}\end{array}\right),$$
(1)

with the isotropic Heisenberg exchange J1, the bond-dependent anisotropic Kitaev exchange K1, the bond-dependent off-diagonal exchange terms Γ1 and $${{{\Gamma }}}_{1}^{\prime}$$ and correction terms η1 and ν1. The latter correction terms are found to be small in our calculated Hamiltonians, and are neglected in what follows. The exchange matrices on X- and Y-bonds follow by respective C3 rotations about the out-of-plane axis ([111] in pseudospin coordinates). Analogously follow the definitions for second and third neighbor exchange terms (or see, e.g., ref. 8).

Using Uavg and Javg from cRPA (Fig. 4a), the complex hopping parameters extracted from full-relativistic DFT (magnitudes shown in Fig. 4b–d) and the projED method, we extracted the exchange constants shown in Fig. 5b–d.

Evaluating the magnetic interactions of the complete RuX3 family, we find a nearest-neighbor ferromagnetic Kitaev interaction K1 to be the dominant in all three compounds. Additionally, a subdominant ferromagnetic nearest-neighbor Heisenberg exchange J1 is present, which is, however, almost vanishing for the iodine case. The symmetric off-diagonal Γ1 interaction is of similar magnitude as J1, changing sign going from Cl and Br to I. $${{{\Gamma }}}_{1}^{\prime}$$, often neglected in the RuCl3 analysis, may become rather important, particularly for RuI3. Further-neighbor interactions are generally smaller than their nearest-neighbor counterparts for all three systems, but increase for larger ligand atomic number and may play, especially in RuI3, an important role.

That the anisotropic interactions do not monotonically increase with stronger spin-orbit coupling of halogen elements can be related to the SOC source. In the original Jackeli-Khaliullin mechanism2, the heavy magnetic ions are solely responsible for SOC effects, which can be well described within the SOC atomic limit. In the case of RuBr3 and RuI3, however, ligand SOC starts to play an important role. To evaluate the interplay of these two SOC sources, we extracted ab initio values for the RuX3 materials and compared them to the SOC atomic limit (see Supplementary Information). We find that in these compounds SOC effects from magnetic ions and ligands do not enhance each other, but do compete. This leads to the observed inhomogeneous behavior of the magnetic anisotropic terms in Fig. 5 as a function of ligand atomic number. Another consequence of this breakdown of the SOC atomic limit is that the established analytic perturbation theory expressions2,6,8 become unjustified in Kitaev materials where SOC arises from both the metal and the ligand elements. RuBr3 and RuI3 are therefore cases where more general approaches, like ours, are indispensable. Another approach would be perturbation theory taking into account ligand orbitals, as recently derived for the S = 3/2 material CrI346.

Along the halogen series Cl–Br–I we observe a decrease for nearest-neighbor couplings (Fig. 5b) and an overall increase in magnitude for second and third neighbors (Fig. 5c, d). This can be understood by consideration of the ligand–metal (pd) hybridization. We quantify the hybridization strength by integrating the DFT(GGA) density of states (DOS) with Ru 4d orbital character in the energy window dominated by the ligand p orbitals (between −7 and −1.05 eV). In spite of respective larger Ru–Ru distances, this can be related to the magnetic exchange by consideration of the ab initio hopping parameters between Wannier d orbitals. As also pointed out in ref. 39, in spite of the stronger hybridization the nearest-neighbor hopping parameters are reduced for heavier ligands, illustrated in Fig. 4b. This is reflected in the magnetic exchange parameters (Fig. 5b) in an overall reduced magnitude in the nearest-neighbor parameters. In contrast, the second and third neighbors show a very different dependence on the halogen element. From the dominant further-neighbor hoppings (Fig. 4c, d), the hoppings show an overall tendency to increase, with few exceptions. Certain further-neighbor magnetic exchange parameters, depending on their relation to the individual hopping parameters, become therefore increasingly important for RuBr3 and especially for the magnetic properties of RuI3.

Finally, we also computed the gyromagnetic g-tensor for the RuX3 family from first principles, in order to relate the pseudospin S of the effective Hamiltonian to the magnetic moment $${\bf{M}}={\mu }_{\rm{B}}{\mathbb{G}}\cdot {\bf{S}}$$. The g-tensor can be approximately characterized by two components, the value parallel to the honeycomb plane, g, and the one perpendicular to it, g, which are shown in Fig. 5a. We consistently find g > g for the whole family, promoting a stronger Zeeman term for in-plane fields.

We now discuss the ramifications of the derived magnetic models for the magnetism in these materials. For RuCl3, we can compare our result to a vast available literature of models that have been shown to reproduce various experimental observations. Indeed, the model presented here in Fig. 5, derived completely from first principles without adjustments or external parameters, is remarkably close to some well-benchmarked recent models19,37,38, and is therefore expected to also describe the material quite well. As we apply the same ab-initio setup for the new members of the RuX3 family, we expect our models to be reliable for them too.

The direction-averaged Weiss constant ($${{{\Theta }}}_{{{{\rm{CW}}}}}^{{{{\rm{avg}}}}}$$ in Table 2) is predicted to be positive across the RuX3 family, characteristic of ferromagnetic exchange interactions. This is in line with our analysis of the experimental magnetic susceptibilities of RuCl3 and RuBr3 above (Fig. 2). While in RuCl3 and RuBr3 a large FM contribution to the Weiss constant comes from a significant FM nearest-neighbor Heisenberg interaction J1, this interaction nearly vanishes for RuI3 (Fig. 5b), leading to a smaller Weiss constant. Furthermore, the small J1 in RuI3 renders the nearest-neighbor interactions to be extremely anisotropic, with a dominant Kitaev interaction K1. While at first glance this might suggest a spin-liquid ground state in RuI3, the increased strength of the further-neighbor interactions in RuI3 (see, e.g., J2, K2 in Fig. 5c) also needs to be considered47.

To find the magnetic ground state properties, we perform exact diagonalization (ED) calculations of the derived jeff = 1/2 models on the 24-site cluster shown in Fig. 6a. In Table 2 and Fig. 6b we summarize the encountered ground states, i.e. zigzag for RuCl3 and RuBr3, and possibly a quantum spin liquid (QSL) in RuI3. This is discussed in detail below.

For RuCl3, the model in Fig. 5 (as well as the R$$\bar{3}$$ model discussed in the Supplementary Information) yields zigzag AFM order, identifiable by a maximum at k = M in the static spin structure factor, shown in Fig. 6b. The computed ordered magnetic moment direction (see the “Methods” section), parametrized by θ and ϕ in Table 2 (compare Fig. 1b), is found to be tilted by θM ≈ 34° out of the plane, in excellent agreement with the recent experiment, where θM = 32 ± 3°36 was reported. Interestingly, on the classical level, the ferromagnetic state is lower in energy than the zigzag state, meaning that the latter only becomes the ground state through quantum fluctuations, as discussed also in ref. 37.

We will now focus on the recently synthesized compounds, starting with RuBr3. The static spin structure factor for the RuBr3 model of Fig. 5 is shown in Fig. 6b, indicating also a zigzag AFM order (k = M and C6-rotated vectors), in agreement with experiment25. However, the calculated tilt angle of the magnetic moment, θM = 32, is more in line with RuCl3 than with the reported measured θM = 64° of RuBr325. The authors of ref. 25 argued that this anomalously large tilt angle indicates an exceptionally strong relative Kitaev coupling, i.e., larger K1/J1 and K11 compared to RuCl3. To investigate to what extent closer proximity to the pure Kitaev model could produce such high tilt angles, we take our RuBr3 Hamiltonian of Fig. 5 as a starting point and tune it towards the pure Kitaev model, where K1 is the only non-zero coupling. This is done by multiplying every exchange coupling except K1 by (1−f) and sweeping f from 0 to 1. As shown in Fig. 6c, the moment indeed rotates further away from the honeycomb plane upon moving towards the pure Kitaev model, however even right before the transition to the Kitaev spin liquid (indicated by the vertical dashed line), θS does not exceed 46. $${\theta }_{{{{\bf{M}}}}}=\arccos \left(\frac{\cos {\theta }_{{{{\bf{S}}}}}}{\sqrt{{{g}_{\parallel }}^{2}{\cos }^{2}{\theta }_{{{{\bf{S}}}}}+{{g}_{\perp }}^{2}{\sin }^{2}{\theta }_{{{{\bf{S}}}}}}}\right)$$, which is to be compared to the neutron diffraction experiment, is even smaller due to the anisotropy g > g in our calculated g-tensor (Fig. 5a). A reconciliation with the reported θM = 64 would therefore require quite drastic changes to the g-tensor anisotropy and/or the exchange parameters. While in the whole J1K1−Γ1 parameter space with Γ1 > 0, no angles of θS beyond ~ 40 are expected in the zigzag phase48, significant negative Γ1 < 0 can in principle lead to θS beyond 60°49. However such terms seem incompatible with the ab-initio results and would likely need strong distortions from the present considered RuBr3 crystal structure to be realized.

More distinct from the other two compounds are our results for RuI3. As discussed above, in our GGA + SOC + U calculations we find a very flat energy landscape of competitive magnetic configurations, indicative of strong magnetic frustration. Fittingly, the ground state from the exact diagonalization of the present exchange model does not show a dominant ordering wave vector in the spin structure factor (see Fig. 6b). Although this is a signature generally associated with quantum spin liquid (QSL) states, we note that in the present model, the Kitaev $${{\mathbb{Z}}}_{2}$$ flux operator yields $$\langle {W}_{{\rm {p}}}\rangle ={2}^{6}\langle {S}_{1}^{x}{S}_{2}^{y}{S}_{3}^{z}{S}_{4}^{x}{S}_{5}^{y}{S}_{6}^{z}\rangle \approx 0.29$$ (site indices refer to Fig. 6a). While this is clearly elevated compared to classical collinear states, where 〈Wp〉 is restricted to $$| \langle {W}_{{\rm {p}}}\rangle | \,\le \,\frac{1}{27}\, < \,0.04$$, it is still significantly below the value of the pure unperturbed Kitaev spin liquid, where 〈Wp〉 = 11. Hence, if the ground state constitutes a QSL state, it is presumably not the $${{\mathbb{Z}}}_{2}$$ Kitaev spin liquid. The precise nature of the encountered magnetically disordered state might be interesting for future studies. It appears to be stabilized by the further-neighbor interactions, as we find a clear ferromagnetic ground state when omitting the second- and third-neighbor interactions in the present model. While a QSL scenario for our full RuI3 model is compelling, we note that finite-size effects in our calculation could play a role. In particular, the finite-size cluster could be incompatible with the supposed correct ordering wave vector of the model, e.g. in the case of an incommensurate ordering vector.

To summarize, we have presented a comparative analysis of the electronic and magnetic properties of the Ru-based trihalide family, including the recently synthesized RuBr3 and RuI3, by combining state-of-the-art ab initio microscopic modeling with analysis of reported resistivity, specific heat, and magnetic susceptibility data. The evolution of the magnetic order and Mott–Hubbard correlations along the halogen series, as well as the possible role of disorder, have been a central part of our study. We conclude that:

1. 1.

All three ideal compounds are spin–orbit-assisted Mott insulators, but their fundamental gap decreases with higher ligand atomic number, Cl → Br → I, with RuI3 coming rather close to a metal–insulator transition.

2. 2.

From DFT total-energy calculations, in ideal, pristine crystals the zigzag magnetic order is even more stable in RuBr3 than in RuCl3, while RuI3 shows significant magnetic frustration. Our ab-initio extracted low-energy models predict RuI3 to feature either an incommensurate magnetic ordered state or a quantum spin liquid, which, interestingly, is possibly of a different kind to the $${{\mathbb{Z}}}_{2}$$ Kitaev spin liquid.

3. 3.

A number of reported experimental observations seem to be adversely affected by the sample quality, in particular by dirty grain boundaries. In fact, most of the observations in RuI3 can be reconciled with theory by assuming insulating grains surrounded by (bad) metallic boundaries. The experimental evidence is consistent with a ‘dirty’ insulator, or a bad metal. Disorder would favor either of these.

4. 4.

In all three systems the dominant nearest-neighbor interaction is FM Kitaev K1, with a subdominant FM Heisenberg Interaction J1, that nearly vanishes for RuI3. We observe a non-monotonous behavior of the magnetic anisotropic terms as a function of ligand atomic number that we trace back to a competition of the SOC effects from magnetic ions and ligands.

5. 5.

RuBr3 has predominantly ferromagnetic interactions, in contrast to what is suggested by standard Curie–Weiss analysis25. Such interactions are consistent with the experimental susceptibility when taking into account high-temperature SOC effects. Our ab-initio magnetic model predicts zigzag order in agreement with the experiment, with a tilting angle of θM = 32 for the magnetic moments, similar to RuCl3, but in contradiction to the reported θM = 6425. We showed that such a large angle cannot be simply explained by proximity to the pure Kitaev model, but would require quite drastic changes to the exchange parameters, such as sizeable negative Γ1 < 0. Those would necessitate strong distortions on the reported RuBr3 crystal structures.

Answering the question posed in the title, our results and analysis strongly suggest that the ideal RuCl3, RuBr3, and RuI3 compounds constitute a family of three Mott-insulating siblings. The challenging task of getting better samples will hopefully help resolve the open issues.

## Methods

### Modified Curie–Weiss fit of RuBr3

We fit the experimental average susceptibility of ref. 25 with four fitting parameters $${\chi }_{0}^{\perp },{\chi }_{0}^{\parallel },{{{\Theta }}}^{\perp },{{{\Theta }}}^{\parallel }$$ using the modified Curie–Weiss formula

$${\chi }^{{{{\rm{avg}}}}}(T)\approx \frac{2}{3}\left({\chi }_{0}^{\parallel }+\,\frac{{C}^{\parallel }(T)}{T-{{{\Theta }}}^{\parallel }}\right)+\frac{1}{3}\left({\chi }_{0}^{\perp }+\,\frac{{C}^{\perp }(T)}{T-{{{\Theta }}}^{\perp }}\right),$$
(2)

where Θ, Θ are the Weiss constants and $${C}^{\alpha }(T)\propto {[{\mu }_{{{{\rm{eff}}}}}^{\alpha }(T,{{\Delta }})]}^{2}$$ is determined through Δ as described in ref. 33. Superscripts and indicate the in- and out-of-honeycomb-plane direction, respectively. The susceptibility is fitted over the temperature range 150–300 K and SOC strength λ = 0.15 eV is taken.

### DFT calculations

To make sure that the calculated features within density functional theory are robust with respect to the choice of the basis set, we have tested the results using two different methods: the projector augmented wave method50,51 as implemented in the VASP code52,53, and the full potential linearized augmented plane-wave (LAPW) basis as implemented in Wien2k54. Throughout the paper, we have used the generalized gradient approximation (GGA55) for the exchange-correlation functional. Hubbard correlation effects were included on a mean-field level in the rotationally invariant implementation of the GGA + U method56. All calculations included spin–orbit coupling (SOC) effects. For VASP we used the Ru_pv pseudopotential, treating Ru p states as valence, and the standard pseudopotentials for the halogens. The Γ-centered 8 × 8 × 8 mesh in the nonmagnetic rhombohedral Brillouin zone was used, or the correspondingly scaled meshes for other structures. The energy cut-off was 350 eV, and the energy convergence criterion 1 × 10−08 eV. For each type of magnetic order, a number of collinear starting configurations with randomly selected Néel vectors were used, and the lowest-energy result was selected as the ground state. Individual results can be found in the Supplementary Information. For Wien2k we chose the plane-wave cutoff $${K}_{\max }$$ corresponding to RK$${}_{\max }=8$$ and a k mesh of 8 × 8 × 2 for the R$$\bar{3}$$ structure in the hexagonal Brillouin zone and 8 × 4 × 6 in the first Brillouin zone of the conventional unit cell for the C/2m structure. The density of states are calculated using a k mesh of 12 × 12 × 3 for the $$\mathrm{R}\bar{3}$$ structure and 12 × 6 × 9 for the C/2m structure. The zigzag configurations are constructed using a conventional cell of the C/2m structure for RuCl3 while a 1 × 2 × 1 supercell of the R$$\bar{3}$$ structures for RuBr3 and RuCl3.

### cRPA calculations

In order to obtain ab-initio estimates for the effective Coulomb interaction for the Ru-trihalide family, we employed the constrained random-phase approximation (cRPA)57,58, as implemented in the FHI-gap code59, based on the Wien2K electronic structure. The low-energy limit of the screened interaction was projected on the five Ru d orbitals, where screening processes in the same window were excluded. Convergence with respect to the discretization of the Brillouin zone and energy cutoff was ensured.

### DFT-based derivation of magnetic models

To derive bilinear exchange parameters for each material, we employed the projED method45, which consists of two steps. First, complex ab-initio hopping parameters between the ruthenium ions are estimated with projective Wannier functions60 applied on full relativistic FPLO61 calculations on a 12 × 12 × 12k mesh. This allows constructing an effective electronic model $${{{{\mathcal{H}}}}}_{{{{\rm{tot}}}}}={{{{\mathcal{H}}}}}_{{{{\rm{hop}}}}}+{{{{\mathcal{H}}}}}_{{{{\rm{U}}}}}$$, where the complex ab-initio hopping parameters enter the kinetic term $${{{{\mathcal{H}}}}}_{{{{\rm{hop}}}}}={\sum }_{ij\alpha \beta }{\sum }_{\sigma {\sigma }^{\prime}}{t}_{i\alpha ,j\beta }^{\sigma {\sigma }^{\prime}}\,{c}_{i\alpha \sigma }^{{\dagger} }{c}_{j\beta {\sigma }^{\prime}}$$ and the cRPA effective Coulomb interaction parameters enter the two-particle term $${{{{\mathcal{H}}}}}_{{{{\rm{U}}}}}={\sum }_{i\alpha \beta \gamma \delta }{\sum }_{\sigma {\sigma }^{\prime}}{U}_{i\alpha \beta \gamma \delta }^{\sigma {\sigma }^{\prime}}\,{c}_{i\alpha \sigma }^{{\dagger} }{c}_{i\beta {\sigma }^{\prime}}^{{\dagger} }{c}_{i\delta {\sigma }^{\prime}}{c}_{i\gamma \sigma }$$. Second, the effective spin Hamiltonian $${{{{\mathcal{H}}}}}_{{{{\rm{eff}}}}}$$ is extracted from the electronic model via exact diagonalization (ED) and projection of the resulting energy spectrum onto the low-energy subspace, mapped onto pseudo-spin operator representation in the jeff picture with the projection operator $${\mathbb{P}}$$: $${\mathcal{H}}_{\rm{eff}}={\mathbb{P}}{\mathcal{H}}_{\rm{tot}}{\mathbb{P}}={\sum}_{ij}{\bf{S}}_{i}\,{{\mathbb{J}}}_{ij}\,{\bf{S}}_{j}$$.

Note that for RuCl3 the exchange constants slightly differ from previously calculated values by some of the authors8,38. The reason for this lies in the following details of the calculation setup: (i) first principles input parameters Uavg and Javg from cRPA in contrast to previous choices, (ii) consideration of all five 4d ruthenium orbitals with the cost of restriction onto two-site clusters, (iii) SOC effects from both Ru3+ and ligands considered through complex hopping parameters in contrast to the atomic limit, and (iv) consideration of the experimental crystal structure in contrast to relaxed ambient pressure structure as it was done in ref. 38.

For the calculation of the gyromagnetic g-tensor, we considered [RuX6]3− molecules within the quantum chemistry ORCA 3.03 package62,63 with the functional TPSSh, basis set def2-TZVP and complete active space for the d orbitals CAS(5,5).

### Exact diagonalization

Exact diagonalization calculations of the jeff = 1/2 models were performed on the 24-site cluster shown in Fig. 6a. To identify possible magnetic ordering, we analyze the static spin structure factor $${\sum }_{\mu = x,y,z}\langle {S}_{-{{{\bf{k}}}}}^{\mu }{S}_{{{{\bf{k}}}}}^{\mu }\rangle$$. For the ordered moment direction, we compute the eigenvector with the maximal eigenvalue of the correlation matrix $${(\langle {S}_{-{{{\bf{k}}}}}^{\mu }{S}_{{{{\bf{k}}}}}^{\nu }\rangle )}_{\mu ,\nu }$$ (μ, ν {x, y, z}) at the ordering wave vector k = Q. This eigenvector then represents the ordered pseudospin direction S49, which relates to the magnetic moment direction $${\bf{M}}\propto {\mathbb{G}}\cdot {\bf{S}}$$, as measured by neutron diffraction, via the anisotropic g-tensor $${\mathbb{G}}$$.