An Inherent Instability of Efficient Markets

Speculative markets are often described as “informationally efficient” such that predictable price changes are eliminated by traders exploiting them, leaving only residual unpredictable fluctuations. This classical view of markets operating close to an equilibrium is challenged by extreme price fluctuations which occur far more frequently than can be accounted for by external news. Here we show that speculative markets which absorb self-generated information can exhibit both: evolution towards efficient equilibrium states as well as their subsequent destabilisation. We introduce a minimal agent-based market model where the impacts of trading strategies naturally adapt according to their success. This implements a learning rule for the whole market minimising predictable price changes, and an extreme succeptibility at the point of perfect balance. The model quantitatively reproduces real heavy-tailed log return distributions and volatility clusters. Our results demonstrate that market instabilities can be a consequence of the very mechanisms that lead to market efficiency.


A. Model
Each agent i = 1, . . . , N possesses two types of assets which without loss of generality are called money M i (t) and stocks S i (t). N is the total number of agents. In each round, every agent places a market order to either buy or to sell an amount of stocks. Since trading should conserve the amount of traded assets, the price in each round is determined by: with demand δ and supply of stocks ς. This is a fair rule that could be used in a real market with only market orders executed at distinct points in time.
For a market including stochastic limit orders gathered over some period of time, consider the hypothetical price p * (t) at which trades would take place if all agents scaled their orders by a common factor. Then the volume would change, but to preserve market clearing the price is not affected; that is p(t) * = p(t). Therefore, the price is a function of the ratio of demand and supply. After linearisation of this function for small small excess demands the price is proportional to the aforementioned ratio which justifies this choice of the pricing rule also as an approximation of the mean prices obtained from limit orders.
Agents base their decisions on a public information state. In each time step one of D possible states, which is denoted by an index µ(t), is conveyed to the agents. We distinguish two different methods for the generation of these information states at each time step t: For exogenous information, µ(t) are independent identically distributed random variates drawn from a distribution P ext (µ).
For endogenous information, agents possess a memory of the most recent K signs of the log returns r. To eliminate the possibility of a lock, signs for vanishing returns are chosen at random. This information can take one out of D = 2 K possible states: where Θ is the step function, and η is an arbitrarily small symmetric random variable with zero mean. Simulation results do not depend on Var(η) as long as it is small enough.
Each agent's i consequent decision is now determined by a strategy vector whose elements σ µ i initially are drawn randomly out of {0, 1} and then kept constant. These two possible de-cisions correspond to trading an amount m i (t) of money or an amount s i (t) of stocks for the respective other asset in the next round. Orders are placed with a constant use parameter γ: Case σ µ(t) i = 1 (agent i buys stocks): Demand and supply are the sums of all buy and sell orders respectively where is a small positive number. This ensures that prices and returns are always well defined. The cases with zero demand or supply are, however, irrelevant for all practical purposes. A sufficiently small 10 −3 does not influence simulation results to a meaningful degree. All figures were generated using = 10 −10 .
We investigate the effect of a market ecology by dividing the agents into N s speculators and N p = N − N s producers. We focus on markets that are dominated by speculators whose resources are redistributed due to trading: Producers' resources stay constant throughout the game: Thus, only speculators redistribute their wealth and are competitive whereas producers provide a predictable supply of liquidity and stocks. All agents are initially provided with equal amounts of resources M i (0) = S i (0) = 1.

B. Invariant Manifold
We show, that if one distribution of resources (M , S) = (M 1 , . . . , M N , S 1 , . . . , S N ) exists for which the price p(M , S, µ) = p is independent of the information µ, this price is invariant with respect to any resource redistribution due to trading in a purely speculative market.
That is, there is a manifold Q = {(M , S ) | p(M , S , µ) = p ∀ µ} of distributions of stocks for which the price is independent of µ and this manifold is closed with respect to trading according to Eq. (7). For the proof, assume that at some point in time the system is in a suitable state such that Then, denoting the distributions of stocks and money after trading by M i and S i we obtain: = 0 − 0 = 0 (14) C. Completeness of the Strategies As shown above, a sufficient condition for complete suppression of all price changes is finding a resource distribution (M , S) for which the price is independent of the information.
That is, which is equivalent to Eq. (10). To fulfil this criterion, we need enough agents to form a complete basis in the strategy space which has D dimensions. Then, the deviation from p caused by each agent can be cancelled by a superposition of the other agents for every µ.
This can be guaranteed if the number of speculators N s exceeds 2D.
For an insufficient number of speculators, we can still calculate an upper and a lower bound for the variance of the log returns given D and N for a perfect superposition of speculators with exogenous information. Numerical and analytical results for this case are shown in Fig. S1. The mean variance is found to drop dramatically at α = D/N s = 1/2, with an increasingly sharp transition for large D. This phase transition can be understood by considering the probability that a random binary vector can be cancelled by an optimal superposition of N − 1 random binary vectors with positive weights. As an interim step, consider superpositions of random vectors with arbitrary weights. One such vector creates a one dimensional subspace. Adding a second vector expands the dimensionality of the subspace to d 2 = 2 if it is linearly independent of the first one. Adding further vectors one by one, the probability that the ith vector does not lie in a d i−1 -dimensional submanifold is We can therefore iteratively calculate the probability distribution P (d Ns−1 ) of d after adding N s − 1 agents and the probability that one out of N s agents is linearly independent of the others. If a vector is linearly independent of the other agents in d dimensions, it cannot be cancelled by a linear combination of the other agents for all µ. However, it may still be possible to cancel this agent's impact for a subset of all possible µ, i.e. for a smaller subspace. Therefore, the probability that an agent cannot be cancelled in any given time step is The last term weights each summand with the fraction of dimensions in which the agent's impact is not cancelled. Finally, to relate the fraction of not cancelled agents to returns we need to consider the fluctuations prior to any resource redistribution. Since all strategies and µ are chosen randomly, agents initially contribute to the demand or the supply at random.
These fluctuations of demand and supply then follow a binomial distribution with N s trials and equal probability for buying or selling: Since we can approximate the price for small deviations: Therefore, and finally where the logarithm stems from using the logarithm with base 10 in the log return.
Combining Eqns. (18) and (27), we obtain the expected variance of the return for an optimal superposition of agents without the positivity constraint on the resources Since resources cannot be negative, they form a positive cone. Each agent that is linear independent of the others spans a half space. Therefore, 2N s agents are necessary to completely span the strategy space. Yet for small numbers of agents, each agent still represents a full degree of freedom since the probability that two agents lie on the same 1-dimensional submanifold is vanishingly small. However, as the number of agents is increased such that α → 1, an increasingly large number of new agents just converts a halfspace into a full one.
Therefore, Eq. (28) represents a lower limit for the variance of the log returns which is a good description for N s D. An upper limit is obtained by changing Eq. (18) such, that each agents increases d by 1/2. This is a good approximation for N s ≈ 2D. The area in between these limits is shown in Fig. S1 (grey shaded). The lower limit has a phase transition at α = 1 while the upper limit has a phase transition at α = 1/2. A phase transition at α = 1 is already present in Eq. (17). The gradual convergence for the true variance of the system from the lower to the upper limit is captured by a simple heuristic interpolation: For the dark grey line in Fig. S1, the probability for a new linearly independent agent to increase d by one is P 1 = min(1, N s /2 m+1 ) while the probability to increase d by 1/2 is The presented theory describes the numerical results (Fig. S1) for the model with endogenous information very well for α ≤ 1/2. For full markets, the residual error for simulations with only speculators is determined by the numerical precision. When producers are present, the residual error is noticeably higher. This is due to the fact, that producers push the system off the invariant manifold. This error is dependent on the agents' use and vanishes for small γ. Still, predictable producers are cancelled much better than random ones because speculators can successfully predict their choices. For endogenous information (D = 2 K ), the phase transition appears smother and slightly shifted towards larger α. A stronger reduction of average returns for α < 1/2 occurs due to the more localised adaptation: Agents do not adapt to all possible values of µ at the same time.

D. Gradient Descent
We now investigate, how the system evolves towards the invariant manifold. We focus on large numbers of agents and small γ. The resource redistribution due to subsequently trading the two assets for one another is found to be a special case of a learning rule which minimises log-return magnitudes. Even more generally, we consider the error function and show that its gradient with respect to the agents' resources is dominated by terms with the opposite sign as the change in the agents' resources. Therefore, any scaling of the agents' resources which keeps the sign of the return for money and the opposite sign for stocks corresponds to minimising log return magnitudes similar to a gradient descent.
To begin with, consider two subsequent time steps where the information takes the states denoted by µ and µ respectively. We again consider a market consisting of speculators only.
The derivative of the return with respect to the resources of an agent k is ∂r(M, S, µ, µ ) The change in resources after trading twice is We are interested in and continue only with leading terms in γ.
For now, we also assume that agents can only perform roundtrip trades (RT). The general case will be discussed later. Two cases are left: Above, we used p = δ/ς and p = δ /ς . Then, Therefore, the change in the total error function can never be positive if agents only perform roundtrip trades.
On average, this result holds even for the general case. The reason why we have to consider averages is, that agents who buy or sell two times in a row always decrease the amount of money or stocks they own after two time steps. Therefore, these agents' resources are expected to change in the opposite direction of the gradient half of the time. That is, for every given pair of informations (µ, µ ), a quarter of all agents' is expected to have their resources evolve such that future r(µ, µ ) 2 increase. However, the actual influence of these agents is much lower and can be neglected for large systems. Then, demand and supply can be well described as binomial processes as shown above. Here, express demand and supply as: and the relative fluctuations around the mean demand N/2 are only √ N /2 and therefore small for large N. Thus we can expand Eq. (31) for small fluctuations: As the above equation shows, when agents perform roundtrip trades, they contribute a term of order N −1 to the gradient with respect to each asset. When agents buy or sell twice, they only contribute a term of order N −1.5 for one asset. Therefore, the influence of these agents vanishes for sufficiently large N . By a similar argument, approximately a quarter of all agents performs either one of the actions (buy, sell), (sell, buy), (buy, buy), and (sell, sell) while fluctuations can be neglected for large N . Summing up, the expected change in r(µ, µ ) 2 over repeated trades with the same information is always negative given a sufficiently large number of agents.

A. Phase Diagrams for Exogenous Information
For uniformly distributed exogenous information states, a reduction of fluctuations does not generally increase the kurtosis (Fig. S2). This is opposed to the endogenous case shown in Fig. 3 in the main paper. Only very large uses in overcomplete markets cause high kurtoses. Then, even uniformly distributed µ occasionally are not repeated for a sufficiently long time to be "forgotten" by the market.

B. Speculators and Producers
Fig . S3 shows the phase diagram for α versus the amount of producers in the market. As it turns out, a second phase transition with respect to the number of producers is found. This transition is independent of the one for the speculators. Small N p < 0.5 · D only weakly influence return distributions.  For a combination of endo-and exogenous information, results are similar to pure endogenous information as long as the endogenous part dominates. Generally, more exogenous information leads to a stronger reduction of fluctuations, less pronounced volatility clustering, and random time series without visible patterns even for small γ. The scaling of the remaining extreme returns remains unchanged. An example is shown in Fig. S5.

III. SUPPLEMENTARY DISCUSSION
A. Income and the Critical Point Fig. S6 shows the phase transition with respect to α in more detail. As in Fig. S1, we take one parameter set as a reference to which we compare simulations after transients for different parameters. For orientation, the log-return variances (Fig. S6 (a)) and kurtoses (Fig. S6 (b)), which have been discussed earlier, are shown again.
Mean speculator capitals are not constant over time in markets that include producers. For empty markets, the ratio of average speculator and producer capitals quickly evolves towards an equilibrium.
The more agents are added, the longer it takes for C s to saturate. For critical or crowded markets a positive speculator income persists over long times. Then, average speculator capitals after transients can be well described as: The income factor a is shown in Fig. S6 (c) and quantifies how well the speculators can exploit the producers. a is found to be independent of the initial ratio between speculator and producer capitals. It becomes maximal close to the critical point which can be intuitively understood: For empty markets, there is a finite chance for a producer strategy to lie outside of the space spanned by the speculators. Therefore, increasing the number of speculators increases their average income. For crowded markets, producers are already optimally exploited. Then, adding more speculators just distributes the maximal total income over more of them. An analogous maximum can be found in Minority Games (see e.g. ref. 15 in the main paper).   From a purely descriptive point of view, the distribution and autocorrelation of a random process are mathematically distinct features. For example, two processes can have the same probability distribution, but different autocorrelations. However, there are various ways of generating a random process where these two features are closely interdependent. Log returns in particular are sums over many logarithmic price changes. Therefore, if those price changes were independent and identically distributed it follows from the generalised central limit theorem that returns over longer time intervals should follow either a Gaussian or a Lévy stable distribution. Yet, returns have been found to be outside of Lévy regime and still non-Gaussian, as well as remarkably stable as time intervals are increased 1,2 . Since subsequent price magnitudes are not independent, it seems natural to assume that a dynamic volatility is to blame for the slow convergence of long-term returns towards the Gaussian distribution. A possibly related finding is that return distributions become less heavy tailed after normalisation by an estimate of the volatility at each point in time. The extend of this effect depends on the assumptions made about a hidden stochastic volatility process 3,4 .
In our model, jumps to a "surprising" part of the information (attractor) space creates large returns. Such events increase the probability for more large jumps while the market adapts to the new environment. Therefore, volatility in our model is a stochastic variable, and connected to non-Gaussian returns, but this connection is more complex than simply Gaussian noise with a time dependent amplitude. However, a detailed analytical characterisation of this volatility process is outside of the scope of this work. Here, we present two numerical analyses of the model returns.
First, distributions for the model are very stable when returns over more than one time step are calculated. Fig. S7 shows a comparison of single-step returns and cumulated returns over 1000 time steps for the model, and for similarly distributed independent random variates. The return distributions for the model are found to be more stable than the independent ones. Therefore, one time step in the model could also be interpreted as a shorter time interval than one day. This holds especially when increasing N s which leads to a slower decay of correlations of magnitudes over time. For a comparable analysis of high frequency returns see, e.g., 1 . As surrogate log returns (black line), independent random variates were generated using inverse transform sampling. The cdf was chosen such that it follows a Gaussian below a threshold and a power law with a slope of 2.5 above the threshold. At the threshold, the cdf has a continuous derivative.
Second, we consider a simple analysis for how the cdf and local volatility interact  The reduction is not sufficient to make the distribution perfectly Gaussian (dotted grey lines). For independent surrogate returns generated as in Fig. S7, the normalisation has no effect (green lines).