Society's techno-social systems are becoming ever faster and more computer-orientated. However, far from simply generating faster versions of existing behaviour, we show that this speed-up can generate a new behavioural regime as humans lose the ability to intervene in real time. Analyzing millisecond-scale data for the world's largest and most powerful techno-social system, the global financial market, we uncover an abrupt transition to a new all-machine phase characterized by large numbers of subsecond extreme events. The proliferation of these subsecond events shows an intriguing correlation with the onset of the system-wide financial collapse in 2008. Our findings are consistent with an emerging ecology of competitive machines featuring ‘crowds’ of predatory algorithms, and highlight the need for a new scientific theory of subsecond financial phenomena.
As discussed recently by Vespignani1, humans and computers currently cohabit many modern social environments, including financial markets1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25. However, the strategic advantage to a financial company of having a faster system than its competitors is driving a billion-dollar technological arms race6,7,8,9,16,17,18,19 to reduce communication and computational operating times down to several orders of magnitude below human response times26,27 -- toward the physical limits of the speed of light. For example, a new dedicated transatlantic cable18 is being built just to shave 5 milliseconds (5 ms) off transatlantic communication times between US and UK traders, while a new purpose-built chip iX-eCute is being launched which prepares trades in 740 nanoseconds19 (1 nanosecond is 10−9 seconds). In stark contrast, for many areas of human activity, the quickest that someone can notice potential danger and physically react, is approximately 1 second26,27 (1 s). Even a chess grandmaster requires approximately 650 ms just to realize that she is in trouble26,27 (i.e. her king is in checkmate).
In this paper we carry out a study of ultrafast extreme events (UEEs) in financial market stock prices. Our study is inspired by the seminal works of Farmer, Preis, Stanley, Easley and Cliff and co-workers2,3,6,7,8,9 who stressed the need to understand ultrafast market dynamics. To carry out this research, we assembled a high-throughput millisecond-resolution price stream across multiple stocks and exchanges using the NANEX NxCore software package. We uncovered an explosion of UEEs starting in 2006, just after new legislation came into force that made high frequency trading more attractive2. Specifically, our resulting dataset comprises 18,520 UEEs (January 3rd 2006 to February 3rd 2011) which are also shown visually on the NANEX website at www.nanex.net. These UEEs are of interest from the basic research perspective of understanding instabilities in complex systems, as well as from the practical perspective of monitoring and regulating global markets populated by high frequency trading algorithms.
We find 18,520 crashes and spikes with durations less than 1500 ms in our dataset, with examples of each given in Fig. 1A (crash) and 1B (spike). We define a crash (or spike) as an occurrence of the stock price ticking down (or up) at least ten times before ticking up (or down) and the price change exceeding 0.8% of the initial price, i.e. a fractional change of 0.008. We have checked that our main conclusions are robust to variations of these definitions. In order to have a standardized measure of the size of a UEE across stocks, we take the UEE size to be the fractional change between the price at the start of the UEE, and the price at the last tick in the sequence of price jumps in a given direction. Since both crashes and spikes are typically more than 30 standard deviations larger than the average price movement either side of an event (see Figs. 1A and 1B), they are unlikely to have arisen by chance since, in that case, their expected number would be essentially zero whereas we observe 18,520.
Figure 2 shows that as the UEE duration falls below human response times26,27, the number of both crash and spike UEEs increases very rapidly. The fact that the occurrence of spikes and crashes is similar (i.e. blue and red curves almost identical in Fig. 1C and in Fig. 2) suggests UEEs are unlikely to originate from any regulatory rule that is designed to control market movements in one direction, e.g. the uptick regulatory rule for crashes16,17. Their rapid subsecond speed and recovery shown in Figs. 1A and 1B suggests they are also unlikely to be driven by exogenous news arrival. We have also checked that using ‘volume time’ instead of clock time, does not simplify or unify their dynamics. The extensive charts at www.nanex.net, of which Figs. 1A and 1B are examples, show that the total volume traded within each UEE does not differ significantly from trading volumes during typical few-second market intervals, nor do the UEEs originate from one large but possibly mistaken trade.
The horizontal green lines in Fig. 1C show that the UEEs started appearing at different times in the past for individual stock, but then escalated in the build-up to the 2008 global financial collapse (black curve). Moreover, these escalation periods tend to culminate around the 15 September bankruptcy filing of Lehman Brothers. Indeed, the ten stock with the most UEEs (solid green horizontal lines) are all major banks with Morgan Stanley (MS) first, followed by Goldman Sachs (GS). Figure 2 in the SI shows explicitly the escalation of UEEs in the case of Bank of America (BAC) stock. For each stock shown in Fig. 1C, the start and end times of the escalation period (i.e. horizontal green line) are determined by examining the local trend in the arrival rate of the UEEs. In determining these start and end times, we checked various statistical methods such as LOWESS and found them all to give very similar escalation periods to those shown in Fig. 1C. We also find that the occurrence of UEEs is not simply related to the daily volatility, price or volume (see SI Fig. 2 for the explicit case of BAC). Figure 1C therefore suggests that there may indeed be a degree of causality between propagating cascades of UEEs and subsequent global instability, despite the huge difference in their respective timescales. Although access to confidential trade and exchange information is needed to fully test this hypothesis, at the very least Fig. 1C demonstrates a coupling between extreme market behaviours below the human response time and slower global instabilities2,5 above it, and shows how machine and human worlds can become entwined across timescales from milliseconds to months. We have also found that UEEs build up around smaller global instabilities such as the 5/6/10 Flash Crash: although fast on the daily scale, Flash Crashes are fundamentally different to UEEs in that Flash Crashes typically last many minutes () and hence allow ample time for human involvement. Future work will explore the connection to existing studies such as Ref. 28 of market dynamics immediately before and after financial shocks.
Having established that the number of UEEs increases dramatically as the timescale drops below one second, and hence drops below the human reaction time, we now seek to investigate how the character of the UEEs might also change as the timescale drops – and in particular, whether the distribution may become more or less akin to a power-law distribution. Power-law distributions are ubiquitous in real-world complex systems and are known to provide a reasonable description for the distribution of stock returns for a given time increment, from minutes up to weeks13,14,15. Our statistical procedure to test a power-law hypothesis for the distribution of UEE sizes, and hence obtain best-fit power-law parameter values, follows Clauset et al.'s29 state-of-the-art methodology for obtaining best-fit parameters for power-law distributions, and for testing the power-law distribution hypothesis on a given dataset. Following this procedure, we obtain a best estimate of the power-law exponent α, and a p-value for the goodness-of-fit, for the distribution of UEE sizes. Specific details of the implementation, including a step-by-step recipe and documented programs in a variety of computer languages, are given in Ref. 29.
Figure 3 shows a plot of the goodness-of-fit p-value, and the corresponding power-law exponent α, for the distribution of sizes of UEEs having durations above a particular threshold. As this duration threshold decreases, the character of the UEE size distribution exhibits a transition from a power-law above the limit of the human response time to a non-power-law below it -- specifically, the goodness-of-fit p falls from near unity to below 0.1 within a small timescale range in Figs. 3B and 3C. This loss of power-law character at subsecond timescales suggests that a lower limit needs to be placed on the validity of Mandelbrot's claim that price-changes exhibit approximate self-similarity (i.e. approximate fractal behavior and hence power-law distribution) across all timescales30. It can be seen that the transition for crashes is smoother than for spikes: this may be because many market participants are typically ‘long’ the market16 and hence respond to damaging crashes differently from profitable spikes. Not only is the crash transition onset (650 ms) earlier in Fig. 3B than for spikes in Fig. 3C, it surprisingly is the same as the thinking time of a chess grandmaster, even though individual traders are not likely to be as attentive or quick as a chess grandmaster on a daily basis26,27. This may be a global online manifestation of the ‘many eyes’ principle from ecology6 whereby larger groups of animals or fish may detect imminent danger more rapidly than individuals.
Figures 4 and 5 show further evidence for this transition in UEE size character as timescales drop below human response times. Figure 4 shows that the cumulative distribution of UEE sizes for the example of spikes, exhibits a qualitative difference between UEEs of duration greater than 1 second, where p = 0.91 and hence there is strong support for a power-law distribution, and those less than 1 second where p < 0.05 and hence a power-law can be rejected. A similar conclusion holds for crashes. Figure 5 shows the cumulative distribution of sizes for UEEs in different duration windows, with the distribution for the duration window 1200–1500 ms showing a marked change from the trend at lower window values. The following quantities that we investigated, also confirm a change in UEE character in this same transition regime: (1) a Kolmogorov-Smirnov two-sample test to check the similarity of the different UEE size distributions within different duration time-windows (see SI Fig. 5); (2) the standard deviation of the size of UEEs in a given window of duration (see SI Fig. 6); (3) the average and standard deviation in the number of price ticks making up the individual UEEs which lie in a given duration window (see SI Fig. 7); (4) a test for a lognormal distribution for UEE durations (see SI Fig. 8). Figure 9 of the SI confirms that using different binnings for the UEE durations does not change our main conclusions.
Inspired by Farmer and Skouras' ecological perspective6, we analyze our findings in terms of a competitive population of adaptive trading agents. The model is summarized schematically in Fig. 1 of the SI while Refs. 31, 40 and 41 provide full details and derivations of the quoted results below. Each agent possesses several (s > 1) strategies, but only trades at a given timestep if it has a strategy that has performed sufficiently well in the recent past. The common information fed back to the agents at each timestep is a bit-string encoding the m most recent price movements31,32,33,34,35. The key quantity is corresponding to the ratio of the number of different strategies (i.e. strategy pool size which is 2m + 1 in our model31,33) to the number of active agents N. For η > 1, there are more strategies than agents, which is consistent with having many human participants since individual humans have myriad ways of making decisions, including arbitrary guesswork, hunches and personal biases. Hence η > 1 is consistent with having many active human traders, which in turn is consistent with longer timescales (>1 s) since this is where humans can think and act. The η > 1 output, illustrated in Fig. 6B (right panel), does indeed reproduce many well-known features of longer timescale price increments31. The chance that many agents simultaneously use the same strategy and submit the same buy or sell order, is small if η > 1, hence there are very few extreme price-changes -- exactly as observed in our data for > 1 s. Reducing η below 1 corresponds to reducing the strategy pool size below the number of agents, which is consistent with a market dominated by specific high-frequency trading algorithms. As the trading timescale moves into the subsecond regime, the number of pieces of information that can be processed by a machine decreases since each piece of information requires a finite time for manipulation (e.g. storage and recall), which is consistent with a reduction in m and hence a decrease in η since .
Remarkably, decreasing η continually in our model generates a visually abrupt transition in the output with frequent extreme price-changes now appearing (Fig. 6B, left panel), which is exactly what we observed in the data for < 1 s. η < 1 implies more than one agent per strategy on average: crowds of agents frequently converge on the same strategy and hence simultaneously flood the market with the same type of order, thereby generating the frequent extreme price-change events. Although it is quite possible that there are other models that could reproduce a gradual change in the instability as η decreases, the task of reproducing a visually abrupt transition as observed empirically in Fig. 3 (particularly Fig. 3C) is far harder. In addition, our model predicts (1) that the extreme event size-distribution in the ultrafast regime (η < 1) should not have a power law, exactly as we observe; (2) that recoveries as in Figs. 1A and 1B, can emerge endogenously in the regime η < 1 (see Fig. 6C, left panel), again as we observe; and (3) that extreme events can be diverted by momentarily increasing the strategy diversity. To achieve this latter effect, agents simply need to be added with complementary strategies -- shown as complementary colors in the right panel of Fig. 6C -- thereby partially cancelling the machine crowd denoted in red. The fact that the actual model price trajectory can then bypass the potential extreme event (green dashed line in left panel of Fig. 6C) therefore offers hope of using small real-time interventions to mitigate systemic risk.
Although the simplicity of our proposed minimal model necessarily ignores many market details, it allows us to derive explicit analytic formulae for the scale of the fluctuations in each phase, and hence an indication of the risk, if we make the simplifying assumption that the number of agents trading each timestep is approximately N (see Refs. 31, 40 and 41 for details). For η > 1, the scale is given by for general s, while for η < 1 it abruptly adopts a new form with upper bound and lower bound for s = 2. This predicted sudden increase in the fluctuation scale from being proportional to for η > 1, to proportional to N for η < 1, is consistent with the observed appearance of frequent UEEs at short timescales, and specifically the visually abrupt transition that we observe in Fig. 3.
More detailed investigation of the properties of UEEs, and the potential implications for financial market instability, will require access to confidential exchange data that was not available in the present study. However a remarkable new study by Cliff and Cartlidge36 provides some additional support for our findings. In controlled lab experiments, they found that when machines operate on similar timescales to humans36 (longer than 1 s), the ‘lab market’ exhibited an efficient phase (c.f. few extreme price-change events in our case). By contrast, when the machines operated on a timescale faster than the human response time36 (100 milliseconds) then the market exhibited an inefficient phase (c.f. many extreme price-change events in our case).
While our crowd model offers a plausible explanation of the observed transition in Fig. 3, we stress that our purpose in this paper was not to explain the details of the price changes during individual UEEs, nor was it to unravel the underlying market microstructure that might provoke or exacerbate such UEEs. A recent preprint by Golub et al.37 claims that a majority of all UEEs carry the label of ISO (Inter-market Sweep Order37). However this claim does not affect the validity of our findings. Moreover, Ref. 37 does not uncover or explain the visually abrupt transition that we observe in Fig. 3, nor does it invalidate our own crowd model explanation. Irrespective of the underlying order identities, every UEE is the result of a sudden excess buy or sell demand in the market, and our model provides a simple explanation for how sudden excess buy or sell demands are generated, not how they get fulfilled. Indeed it is a common feature of our model output that a large imbalance of buy or sell demand can suddenly appear, producing a UEE as observed empirically. We also note that Golub et al.37 make several strong assumptions in their attempts to label the UEEs, each of which requires more detailed investigation since the resulting identifications are neither unique nor unequivocal. Whether the visually abrupt transition in Fig. 3 is a strict phase transition in the statistical physics sense, also does not affect the validity of our results. The extent to which UEEs were provoked by regulatory and institutional changes around 2006, is a fascinating question whose answer depends on a deeper understanding of the market microstructure along the lines started by Golub et al.37. It may be that ISOs are particularly problematic, but this is still unclear because of the assumptions made in Ref. 37. Once this has been resolved, it should be possible to make definite policy recommendations based on our findings, as well as expanding the study to connect to systemic risk38 and derivative operations39.
The power-law analysis that we use to obtain our main result in Fig. 3, follows the state-of-the-art testing procedure laid out by Clauset et al.29. Our accompanying crowd model considers a simple yet archetypal model of a complex system based on a population of agents competing for a limited resource with bounded rationality. This model has previously been shown to reproduce the main stylized facts of financial markets31. Its dynamics are based on the realistic notion that it is better to be a buyer when there is an excess of sellers or vice versa when in a financial market comprising agents (humans or machines) with short-term, high-frequency trading goals. The formulae given above for the scale of the fluctuations in each phase, are derived explicitly in Ref. 40, and also Refs. 31 and 41.
NJ (Neil Johnson) gratefully acknowledges support for this research from The MITRE Corporation and the Office of Naval Research (ONR) under grant N000141110451. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the above named organizations, to include the U.S. government. We thank Amith Ravindar, Joel Malerba, Zhenyuan Zhao, Pak Ming Hui, Spencer Carran, David Smith, Michael Hart and Paul Jefferies for discussions surrounding this topic and help with assembling datafiles and parts of figures.