Numerical test of the Edwards conjecture shows that all packings are equally probable at jamming

Abstract

In the late 1980s, Sam Edwards proposed a possible statistical-mechanical framework to describe the properties of disordered granular materials¹. A key assumption underlying the theory was that all jammed packings are equally likely. In the intervening years it has never been possible to test this bold hypothesis directly. Here we present simulations that provide direct evidence that at the unjamming point, all packings of soft repulsive particles are equally likely, even though generically, jammed packings are not. Typically, jammed granular systems are observed precisely at the unjamming point since grains are not very compressible. Our results therefore support Edwards’ original conjecture. We also present evidence that at unjamming the configurational entropy of the system is maximal.

Main

In science, most breakthroughs cannot be derived from known physical laws: they are based on inspired conjectures². Comparison with experiment of the predictions based on such a hypothesis allows us to eliminate conjectures that are clearly wrong. However, there is a distinction between testing the consequences of a conjecture and testing the conjecture itself. A case in point is Edwards’ theory of granular media. In the late 1980s, Edwards and Oakeshott¹ proposed that many of the physical properties of granular materials (‘powders’) could be predicted using a theoretical framework that was based on the assumption that all distinct packings of such a material are equally likely to be observed. The logarithm of the number of such packings was postulated to play the same role as entropy does in Gibbs’ statistical-mechanical description of the thermodynamic properties of equilibrium systems. However, statistical-mechanical entropy and granular entropy are very different objects. Until now, the validity of Edwards’ hypothesis could not be tested directly—mainly because the number of packings involved is so large that direct enumeration is utterly infeasible—and, as a consequence, the debate about the Edwards hypothesis has focused on its consequences, rather than on its assumptions. Here we present results that show that now, at last, it is possible to test Edwards’ hypothesis directly by numerical simulation. To our own surprise, we find that the hypothesis appears to be correct precisely at the point where a powder is just at the (un)jamming threshold. However, at higher densities, the hypothesis fails. At the unjamming transition, the configurational entropy of jammed states appears to be at a maximum.

The concept of ‘ensembles’ plays a key role in equilibrium statistical mechanics, as developed by J. Willard Gibbs, well over a century ago³. The crucial assumption that Gibbs made to arrive at a tractable theoretical framework to describe the equilibrium properties of gases, liquid and solids was that, at a fixed total energy, every state of the system is equally likely to be observed. The distinction between, say, a liquid at thermal equilibrium and a granular material is that in a liquid, atoms undergo thermal motion whereas in a granular medium (in the absence of outside perturbations) the system is trapped in one of (very) many local potential energy minima. Gibbsian statistical mechanics cannot be used to describe such a system. The great insight of Edwards was to propose that the collection of all stable packings of a fixed number of particles in a fixed volume might also play the role of an ‘ensemble’ and that a statistical mechanics-like formalism would result if one assumed that all such packings were equally likely to be observed, once the system had settled into a mechanically stable ‘jammed’ state. The nature of this ensemble has been the focus of many studies^1,4,5,6.

Jamming is ubiquitous and occurs in materials of practical importance, such as foams, colloids and grains when they solidify in the absence of thermal fluctuations. Decompressing such a solid to the point where it can no longer achieve mechanical equilibrium leads to unjamming. Studies of the unjamming transition in systems of particles interacting via soft, repulsive potentials have shown that this transition is characterized by power-law scaling of many physical properties^{7,8,9,10,11,12}. However, the exact nature of both the ensemble of jammed states and the unjamming transition remains unclear.

Here, we report a direct test of the Edwards conjecture, using a numerical scheme for computing basin volumes of distinct jammed states (energy minima) of N polydisperse, frictionless discs held at a constant packing fraction φ. Uniquely, our numerical scheme allows us to compute Ω, the number of distinct jammed states, and the individual probabilities p_{i∈{1, …, Ω}} of each observed packing to occur. Figure 1a shows a snapshot of a section of the system, consisting of particles with a hard core and a soft shell. We obtain jammed packings by equilibrating a hard-disc fluid and inflating the particles instantaneously to obtain the desired soft-disc volume fractions (φ), followed by energy minimization (see Methods). The minimization procedure finds individual stable packings with a probability p_i proportional to the volume v_i of their basin of attraction. Averages computed using this procedure, represented by , would then lead to a bias originating from the different v_i values. Recent advances in numerical methods^13,14,15,16 have now enabled direct computation of v_i, and therefore, an unbiased characterization of the phase space. A summary of the technique is provided in the Methods.

Figure 1: Snapshot of the system studied and illustration of the associated energy landscape at different volume fractions.

a, Snapshot of a jammed packing of discs with a hard core (dark shaded regions) plus soft repulsive corona (light shaded regions). b,c, Illustration of configurational space for jammed packings. The hatched regions are inaccessible due to hard-core overlaps. Single-coloured regions with contour lines represent the basins of attraction of distinct minima. The dark blue region with solid dots indicates the coexisting unjammed fluid region and hypothetical marginally stable packings, respectively. The volume occupied by the fluid V_unj is significant only for finite-size systems at or near unjamming. When φ ≫ φ^∗ (b) the distribution of basin volumes is broad but as φ → φ^∗ (c) the distribution of basin volumes approaches a delta function satisfying Edwards’ hypothesis.

We report a detailed analysis of the distribution of v_i for a fixed number of discs N = 64 (all maximum system sizes in our study were set by the current limits on computing power). We compute v_i using a thermodynamic integration scheme^13,14,15,16, and compute the average basin volume 〈v〉(φ). The number of jammed states is, explicitly, Ω(φ) = V_J(φ)/〈v〉(φ), where V_J(φ) is the total available phase-space volume at a given φ. A convenient way to check equiprobability is to compare the Boltzmann entropy S_B = lnΩ − lnN!, which counts all packings with the same weight, and the Gibbs entropy S_G = −∑ _i^Ωp_i lnp_i − lnN! (refs 17,18,19). The Gibbs entropy satisfies S_G ≤ S_B, saturating the bound when all p_i are equal: p_{i∈{1, …, Ω}} = 1/Ω. As shown in Fig. 2a, S_G approaches S_B from below as φ → φ_N=64^∗(S) ≈ 0.82. Fig. 1b, c schematically illustrates the evolution of the basin volumes as the packing fraction is reduced.

Figure 2: Numerical results obtained by basin-volume calculations for jammed packings of N = 64 discs with a hard core and a soft shell.

a, Gibbs entropy S_G and Boltzmann entropy S_B as a function of volume fraction. S_B is computed both parametrically by fitting with a generalized Gaussian function (Gauss) and non-parametrically by computing a kernel density estimate (KDE) as in ref. 13. The dashed curves are a second-order polynomial fit. b, Scatter plot of the negative log-probability of observing a packing, −ln p_i = F_i + ln V_J(φ), where V_J is the accessible fraction of phase space (see Methods) as a function of log-pressure, Λ. The black solid lines are lines of best fit computed by linear minimum mean square error using a robust covariance estimator and bootstrap (see Methods). c,d, Slopes λ(φ) (c) and intercepts c(φ) (d) of linear fits for equation (16). Solid lines are lines of best fit and error bars refer to the standard error computed by bootstrap²⁸.

To characterize the distribution of basin volumes, we analyse the statistics of v_i along with the pressure P_i of each packing. It is convenient to study F_i ≡ − lnv_i as a function of Λ_i ≡ lnP_i. As shown in Fig. 2b, we observe a strong correlation between F_i and Λ_i, which we quantify by fitting the data to a bivariate Gaussian distribution. The conditional expectation of F given Λ then yields a linear relationship (denoted by solid lines in Fig. 2b) such that , where represents the average over all basins at a given Λ. Previous studies at higher packing fractions¹³ indicate that this relationship is preserved in the thermodynamic limit. Defining f = F/N, we have (see Methods for details)

where . For Edwards’ hypothesis to be valid, we require that in the thermodynamic limit: the distribution of volumes approaches a Dirac delta, which follows immediately from the fact that the variance σ_f² ∼ 1/N (ref. 15) (see Supplementary Information); and F_i needs to be independent of Λ_i, as well as of all other structural observables (see Supplementary Information), and therefore λ(φ) must necessarily vanish. As can be seen from Fig. 2c, d, within the range of volume fractions studied, λ(φ)decreases but saturates to a minimum as φ → φ_N=64^∗(λ). We argue below that the saturation is a finite-size effect. An extrapolation using the linear regime in Fig. 2c indicates that λ → 0 at packing fraction φ_N=64^∗(λ) = 0.82 ± 0.07, remarkably close to where our extrapolation yields S_G = S_B. The analysis of basin volumes, therefore, strongly suggests that equiprobability is approached only at a characteristic packing fraction and that the vanishing of λ(φ) can be used to estimate the point of equiprobability.

We next show that λ(φ) does indeed tend to zero in the thermodynamic limit. We use the fluctuations σ_f², σ_Λ², and the covariance σ_fΛ², obtained from the elements of the covariance matrix of the joint distribution of f and Λ (see Methods for details), to define λ and c as:

From Fig. 2b we observe that the decrease of λ is driven by σ_Λ² increasing to a maximum, while σ_f² and σ_fΛ² decrease (see Supplementary Fig. 2). We expect the main features of these distributions to be preserved as the system size N is increased¹³, which suggests that for larger N, where basin volume calculations are still intractable for multiple densities, the maximum in σ_Λ² can be used to identify φ_N^∗. We have directly measured χ_Λ = Nσ_Λ² using our sampling scheme—which samples packings with probability proportional to the volume of their basin of attraction—for systems of up to N = 128 discs (see inset of Fig. 3a) and finite-size scaling indicates that χ_Λ diverges as φ → φ_N→∞^∗(Λ) = 0.841(3) (see Supplementary Information). The saturation of λ to a minimum as φ → φ_N^∗, for small N, is determined by the fact that χ_Λ diverges only in the thermodynamic limit; a detailed discussion is given in the Methods.

Figure 3: Finite-size scaling analysis.

a, and (inset) χ_Λ ≡ Nσ_Λ², plotted as a function of volume fraction φ. By finite-size scaling (see Supplementary Information) we show that the curves diverge in the thermodynamic limit as φ → φ^J/∗, implying φ_N→∞^∗ = φ_N→∞^J = 0.841(3) (see main text for discussion). For φ ≫ φ_N^J, χ_P approaches a constant value indicating the absence of extensive correlations far from the transition. b, Observed average log-pressure and (inset) probability of obtaining jammed packings by our protocol, as a function of volume fraction φ. By finite-size scaling (see Supplementary Information) we show that as φ → φ_N→∞^J(Λ) = 0.841(3) and p_J collapses for φ → φ_N→∞^J(pJ) = 0.844(2), thus locating the unjamming point. Error bars, computed by BCa bootstrap²⁹, refer to 1σ confidence intervals. The solid lines are generalized sigmoid fits of the form f(φ) = a − (a − b)/(1 + exp(− wΔφ))^1/u. We show only values of φ where the probability of finding a jammed packing is at least 1%, so that the observables are computed over sufficiently large sample sizes.

Interestingly, we find evidence that in the thermodynamic limit, the point of equiprobability, φ_N→∞^∗, coincides with the point at which the system unjams, φ_N→∞^J. We use two characteristics of the unjamming transition to locate φ_N→∞^J: the average pressure of the packings goes to zero, and therefore 〈Λ〉 → −∞ (see Fig. 3b); and the probability of finding jammed packings, p_J, goes to zero (see inset of Fig. 3b). A scaling analysis indicates that 〈Λ〉 → −∞ as φ_N→∞^J(Λ) = 0.841(3), and p_J → 0 as φ_N→∞^J(pJ) = 0.844(2) (see Supplementary Information). We thus find that φ_N→∞^∗ = φ_N→∞^J within numerical error and up to corrections to finite-size scaling²⁰. Our simulations therefore lead to the surprising conclusion that the Edwards conjecture appears to hold precisely at the (un)jamming transition. We note that our earlier simulations, which were performed at densities much above jamming^13,15, did not support the equiprobability hypothesis. The earlier simulations were in fact far enough away from unjamming that the emergence of equiprobability at this point could not be anticipated. Our earlier findings, therefore, do not contradict the more recent ones and are completely consistent with these.

Why is χ_Λ related to the unjamming transition? As the particles interact via purely repulsive potentials, the pressure P is strictly positive, which implies that the fluctuations of P have a floor and go to zero at unjamming. The relative fluctuations can be non-zero, and a diverging χ_P would then imply a diverging χ_Λ. Because of the bounded nature of P (refs 21,22,23), however, χ_P can diverge only at the unjamming transition where (see Methods). We find that χ_P does diverge (Fig. 3a) and finite-size scaling yields φ_N→∞^∗(P) = 0.841(3), in agreement with what has been found for χ_Λ. Returning to the N = 64 case that we have analysed using the basin volume statistics, we find that both χ_P and χ_Λ saturate to their maximum values over similar ranges of φ and our estimate φ_N=64^∗ ≈ 0.82, where S_G = S_B and λ → 0, falls in this region. In addition, the average number of contacts is close to the isostatic value z_N=64^(iso) ≡ 2d − 2/64 ≈ 3.97 (ref. 10) (see Supplementary Information).

Finally, we note that the states in the generalized Edwards ensemble^5,21,24,25 characterized by φ and P have basin volumes that are similar, if not identical, over the full range of φ that we have explored (see scatter plot in Fig. 2b), indicating that equiprobability in the stress–volume ensemble^5,24 is a more robust formulation of the Edwards hypothesis. This observation is consistent with recent experiments²⁶.

Although, the equiprobability of jammed states at a given packing fraction was posited by Edwards for jammed packings of hard particles, our analysis shows that for soft particles, the Edwards hypothesis is valid only for the marginally jammed states at φ_N→∞^∗ = φ_N→∞^J, where the jamming probability vanishes, the entropy is maximized, and relative pressure fluctuations diverge. We have shown not only that there exists a practical ‘Edwardsian’ packing generation protocol, capable of sampling jammed states equiprobably, but we have uncovered an unexpected property of the energy landscape for this class of systems. At this stage, we cannot establish whether the same considerations are valid in three dimensions, although the already proven validity of equation (16) in three dimensions would suggest so¹³. The exact value of the entropy at unjamming, whether finite or not, also needs to be elucidated. The implications for ‘soft’ structural glasses is apparent: at φ^J the uniform size of the basins implies that the system, when thermalized, has the same probability of visiting all of its basins of attraction; hence, there are no preferred inherent structures. This could be a signature of the hard-sphere transition occurring at the same point²⁷. Our approach can, therefore, be extended to spin-glasses and related problems, and it would be clearly very exciting to explore the analogies and differences between ‘jamming’ in various systems for which the configuration space can break up into many distinct basins.

Methods

Packing preparation protocol.

In this section we describe the algorithm that we have used to sample the phase space of jammed packings. This procedure samples each configuration proportionally to the volume of its basin of attraction.

Hard-sphere fluid sampling.

We start by equilibrating a fluid of N hard discs (that serve as the cores of the particles with soft outer shells) at a volume fraction φ_HS in a square box with periodic boundary conditions. The particle radii are sampled from a truncated Gaussian distribution with mean μ = 1 and standard deviation σ = 0.1. We achieve equilibration by performing standard Markov chain Monte Carlo simulations consisting of single-particle displacements and particle–particle swaps, as in ref. 13. To ensure statistical independence, we draw fluid configurations every n_MC steps, where n_MC is (pre-computed by averaging over multiple simulations) the total number of Markov chain Monte Carlo steps necessary for each individual particle to diffuse at least a distance equal to the largest diameter in the system.

Soft shells and minimization.

We next take each equilibrated hard-disc fluid configuration and inflate the particles (instantaneously) with a WCA-like soft outer shell³⁰, to reach the target soft packing fraction φ_SS > φ_HS. Each hard sphere is inflated proportionally to its radius, so that the soft-sphere radius is given by

where d is the dimensionality of the box (2 in our case), and r_SS and r_HS are the soft- and hard-sphere radii respectively. Clearly, this procedure does not change the polydispersity of the sample. The radii are identical across volume fractions and system sizes, and the hard-disc fluid density is chosen so that the radius ratio of hard to soft discs is (0.88/0.7)^1/2 ≈ 1.121.

Next, particle inflation is followed by energy minimization using FIRE^31,32, to produce mechanically stable packings at the desired soft volume fractions φ. This protocol has the advantage of generating packings sampled proportionally to the volume of their basin of attraction. In our simulations, we considered all mechanically stable packings, irrespective of the number of ‘rattlers’. To guarantee mechanical stability we required that the total number of contacts is sufficient for the bulk modulus to be strictly positive, N_min = d(N_nr − 1) + 1 (ref. 33), where N_nr is the number of non-rattlers and d is the dimensionality of the system.

Our implementation of FIRE enforces a maximum step size (set to be equal to the soft-shell thickness) and forbids uphill steps by taking one step back every time the energy increases (and restarts the minimizer in the same fashion as the original FIRE implementation). We use a maximum time step Δt_max = 1, although the maximum step size is directly controlled in our implementation. All other parameters are set as in the original implementation³¹.

HS-WCA potential.

We define the WCA-like potential around a hard core as follows: consider two spherical particles with a distance between the hard cores r_HS, implying a soft-core contact distance r_SS = r_HS(1 + θ), with θ = (φ_SS/φ_HS)^1/d − 1. We can then write a horizontally shifted hard-sphere plus WCA (HS-WCA) potential as

where σ(r_HS) = (2θ + θ²)r_HS²/2^1/6 guarantees that the potential function and its first derivative go to zero at r_SS. For computational convenience (avoidance of square-root evaluations), the potential in equation (4) differs from the WCA form in that the inter-particle distance in the denominator of the WCA potential has been replaced with a difference of squares.

A power series expansion of equation (4) yields

hence, in the limit of no overlap the pair potential is harmonic.

We numerically evaluate this potential, matching the gradient and linearly continuing the function u_HS-WCA(r) for r ≤ r_HS + ɛ, where ɛ > 0 is an arbitrary small constant, such that minimization is still meaningful if hard-core overlaps do occur.

Our choice of potential is based on the fact that: the hard cores greatly reduce the amount of configurational space to explore, replacing expensive energy minimizations (to test whether the random walker has stepped outside the basin) with fast hard-core overlap rejections; and the hard cores exclude high-energy minima (jammed packings) that are not ‘hard-sphere-like’.

Total accessible volume.

The basins of attraction of energy minima tile the ‘accessible’ phase space (schematically shown in Fig. 1b, c). This inaccessible part of the phase space arises due to hard-core constraints and the existence of fluid states (see, for example, ref. 13). The total phase-space volume is equal to V_box^N. The inaccessible part of this volume arising from the hard-core constraints (shown as hatched areas in Fig. 1) is denoted by V_HS, and the part corresponding to the coexisting unjammed fluid states is denoted by V_unj (shown as blue regions with solid dots in Fig. 1b, c). V_unj is significant only for finite-size systems at or near unjamming. We denote the space tiled by the basins of mechanically stable jammed packings by V_J. We then have V_J = V_box^N − V_HS − V_unj. In practice we compute V_J using the following equation

where f_ex(φ_HS) is the excess free energy, that is, the difference in free energy between the hard-sphere fluid and the ideal gas, computed from the Santos–Yuste–Haro equation of state³⁴ as in ref. 13, and p_J(φ_SS) is the probability of obtaining a jammed packing at soft volume fraction φ_SS with our protocol, shown in the inset of Fig. 3b.

Counting by sampling.

We briefly review our approach to computing the number Ω of distinct jammed packings for a system of N soft discs at volume fraction φ. We prepare packings by the protocol described above, which generates jammed structures (energy minima) with probability p_i proportional to the volume of their basin of attraction v_i. We define the probability of sampling the ith packing as

where V_J is the total accessible phase space, such that

Details of the computation of v_i are discussed in refs 13,16. To find Ω, we make the simple observation

from which it follows immediately that

The ‘Boltzmann-like’ entropy, suggested in a similar form by Edwards¹, is then

where the lnN! correction ensures that two systems in identical macrostates are in equilibrium under exchange of particles^17,18,19.

Note that 〈v〉 is the unbiased average basin volume (the mean of the unbiased distribution of volumes). We distinguish between the biased, (as sampled by the packing protocol), and the unbiased, , basin log-volumes distributions (F = −lnv_basin). Since the configurations were sampled proportionally to the volume of their basin of attraction, we can compute the unbiased distribution as

where is the normalization constant, such that

Since small basins are much more numerous than large ones, and grossly under-sampled, it is not sufficient to perform a weighted average of the sampled basin volumes. Instead, to overcome this problem, one can fit the biased measured basin log-volumes distribution with an analytical (or at least numerically integrable) distribution, and perform the unbiasing via equation (12) on the best fitting distribution. Different approaches to modelling this distribution give rise to different analysis methods, which all yield consistent results as shown in ref. 13. In this work we follow ref. 13 and fit using both a (parametric) generalized Gaussian model³⁵ (see Supplementary Equation 21) and a (non-parametric) kernel density estimate with Gaussian kernels^36,37 and bandwidth selection performed by cross-validation^13,38, yielding consistent results in agreement with ref. 13. Before performing the fit we remove outliers from the free-energy distribution in an unsupervised manner, as discussed in the ‘Data analysis’ section of the Methods.

No such additional steps are needed to compute the ‘Gibbs-like’ version of the configurational entropy, in fact

is simply the arithmetic average of the observed volumes: the sample mean of F = −lnv_basin is already correctly weighted because our packing generation protocol generates packings with probability p_i.

Power-law between pressure and basin volume.

A power-law relationship between the volume of the basin of attraction of a jammed packing and its pressure was first reported in ref. 13. In what follows we provide insight into this expression on the basis of this work’s findings. We observe that distributions of basin negative log-volumes, F = −lnv_basin, and log-pressures, Λ = lnP, are approximately normally distributed (see Supplementary Figs 1 and 9). We therefore expect their joint probability to be well approximated by a bivariate Gaussian distribution (when listing a function’s arguments we place parameters that are held constant before the semicolon), with mean μ = (μ_F, μ_Λ) and covariance matrix (ref. 39). This is consistent with the elliptical distribution of points in Fig. 2b. For a given random variable X, with an (observed/biased) marginal distribution , the mean is given by . Similarly, the (biased) conditional expectation of F for a given Λ is then³⁹

This is simply the linear minimum mean square error regression estimator for F, that is, the linear estimator that minimizes . The expectation of the dimensionless free energy (ref. 40) is the average basin negative log-volume at volume fraction φ and log-pressure Λ. Here the average is also taken over all other relevant, but unknown, order parameters Γ, such that . In other words, we write the expectation of F at a given pressure as the (biased) average over the unspecified order parameters Γ. An example of such a parameter would be some topological variable that makes certain topologies more probable than others. Note that F(φ, Λ; Γ) is narrowly distributed around . To simplify the notation we write . We can thus rewrite the power law reported in ref. 13 as

where f = F/N is the basin negative log-volume per particle and λ ≡ 1/κ is the slope of the power-law relation, which depends crucially on the packing fraction φ. The last equality in equation (16) highlights how λ(φ) controls the contributions of the fluctuations of the log-pressures ΔΛ ≡ Λ − μ_Λ(φ) to changes in the basin negative log-volume. Note that one can rewrite the ratio of fluctuations as σ_fΛ²/σ_Λ² = ρ_fΛσ_f/σ_Λ where ρ_fΛ = σ_fΛ²/(σ_fσ_Λ) is the linear correlation coefficient of f and Λ. Finally, we can gain further insight into the power-law dependence by noting that

Data analysis.

Reduced units. While presenting data from our computations, we express pressure and volume in reduced units as P/P^∗ and v/v^∗ respectively. The unit of volume is given by v^∗ ≡ π〈r_HS²〉, where 〈r_HS²〉 is the mean squared hard-sphere radius. The unit of pressure is then P^∗ ≡ ε/v^∗, where ε is the stiffness of the soft-sphere potential, defined in equation (4). The pressure is computed as where is the virial stress tensor and V_box is the volume of the enclosing box.

Summary of calculations.

For the basin-volume calculations we consider systems of N = 64 discs sampled at a range of 8 volume fractions 0.828 ≤ φ ≤ 0.86 and for each φ we measure the basin volume for about 365 < M < 770 samples.

For the finite-size scaling analysis of the relative pressure fluctuations we study system sizes N = 32, 48, 64, 80, 96 and 128 for 48 volume fractions in the range 0.81 ≤ φ ≤ 0.87. For each system size we generate up to 10⁵ hard-disc fluid configurations and compute the pressure for between approximately 10³ and 10⁴ jammed packings (depending on the probability of obtaining a jammed packing at each volume fraction).

Simulations were performed using the open source libraries PELE⁴¹ and MCPELE⁴².

Outlier detection and robust covariance estimation.

Before manipulating the raw data we remove outliers from the joint distribution following the distance-based outlier removal method introduced in ref. 43. This is applied in turn to each dimension, such that we choose to keep only those points for which at least R = 0.5 of the remaining data set is within D = 4σ (compared with the much stricter R = 0.9988, D = 0.13σ required to exclude any points further than |μ − 3σ| for normally distributed data⁴³). On our data sets we find that this procedure removes typically none and at most 0.8% of all data points.

Mean and covariance estimates of are computed using a robust covariance estimator, namely the minimum covariance determinant (MCD) estimator^37,44 with support fraction h/n_samples = 0.99. The MCD estimator defines μ_MCD, the mean of the h observations for which the determinant of the covariance matrix is minimal, and , the corresponding covariance matrix⁴⁵. We use these robust estimates of the location and of the covariance matrix (computed over 1,000 bootstrap samples²⁸) to fit our observations by linear minimum mean square error³⁹ (see Fig. 2).

Before fitting (required to compute Ω), we perform an additional step of outlier detection based on an elliptic (Gaussian) envelope criterion constructed using the MCD estimator. We assume a support fraction h/n_samples = 0.99 and a contamination equal to 10% (ref. 37). We compute S_G and S_B from the resulting data sets. The procedure is strictly unsupervised and allows us to achieve robust fits despite the small sample sizes. We fit using both a (parametric) generalized Gaussian model³⁵ and a (non-parametric) kernel density estimate with Gaussian kernels^36,37 and bandwidth selection performed by cross-validation^13,38.

Error analysis.

Errors were computed analytically where possible and propagated using the ‘uncertainties’ Python package⁴⁶. Alternatively, intervals of confidence were computed by bootstrap for the covariance estimation²⁸ and by BCa bootstrap otherwise using the ‘scikit-bootstrap’ Python package^29,47.

Data availability.

The data that support the plots within this paper and other findings of this study are available from the University of Cambridge data repository doi: http://dx.doi.org/10.17863/CAM.9985.