Between order and chaos

Journal name:
Nature Physics
Volume:
8,
Pages:
17–24
Year published:
DOI:
doi:10.1038/nphys2190
Received
Accepted
Published online
Corrected online

Abstract

What is a pattern? How do we come to recognize patterns never seen before? Quantifying the notion of pattern and formalizing the process of pattern discovery go right to the heart of physical science. Over the past few decades physics’ view of nature’s lack of structure—its unpredictability—underwent a major renovation with the discovery of deterministic chaos, overthrowing two centuries of Laplace’s strict determinism in classical physics. Behind the veil of apparent randomness, though, many processes are highly ordered, following simple rules. Tools adapted from the theories of information and computation have brought physical science to the brink of automatically discovering hidden patterns and quantifying their structural complexity.

At a glance

Figures

  1. [epsi]-machines for four information sources.
    Figure 1: ε-machines for four information sources.

    a, The all-heads process is modelled with a single state and a single transition. The transition is labelled p|x, where p[0;1]is the probability of the transition and x is the symbol emitted. b, The fair-coin process is also modelled by a single state, but with two transitions each chosen with equal probability. c, The period-2 process is perhaps surprisingly more involved. It has three states and several transitions. d, The uncountable set of causal states for a generic four-state HMM. The causal states here are distributions Pr(A,B,C,D) over the HMM’s internal states and so are plotted as points in a 4-simplex spanned by the vectors that give each state unit probability. Panel d reproduced with permission from ref. 45.

  2. Structure versus randomness.
    Figure 2: Structure versus randomness.

    a, In the period-doubling route to chaos. b, In the two-dimensional Ising-spinsystem. Reproduced with permission from: a, ref. 36, © 1989 APS; b, ref. 61, © 2008 AIP.

  3. Complexity-entropy diagrams.
    Figure 3: Complexity–entropy diagrams.

    a, The one-dimensional, spin-1/2 antiferromagnetic Ising model with nearest- and next-nearest-neighbour interactions. Reproduced with permission from ref. 61, © 2008 AIP. b, Complexity–entropy pairs (hμ,Cμ)for all topological binary-alphabet ε-machines with n=1,…,6 states. For details, see refs 61 and 63.

Change history

Corrected online 17 May 2013
In the version of this Review Article originally published online there were several errors. In the section entitled 'Complicated yes, but is it complex?', seventh paragraph, the subscripts should appear as follows: past X:t; future Xt:; and blocks Xt:t'. All instances of x:l should appear as x0:l. In the section entitled 'Applications', in the third and seventh paragraphs, x: should appear as x0:. In the caption of Fig. 1d, the distributions should be listed as Pr(A, B, C, D), and panel d should have been attributed to ref. 45. These errors have now been corrected in the PDF and HTML versions of the Review Article.

References

  1. Press, W. H. Flicker noises in astronomy and elsewhere. Comment. Astrophys. 7, 103119 (1978).
  2. van der Pol, B. & van der Mark, J. Frequency demultiplication. Nature 120, 363364 (1927).
  3. Goroff, D. (ed.) in H. Poincaré New Methods of Celestial Mechanics, 1: Periodic And Asymptotic Solutions (American Institute of Physics, 1991).
  4. Goroff, D. (ed.) H. Poincaré New Methods Of Celestial Mechanics, 2: Approximations by Series (American Institute of Physics, 1993).
  5. Goroff, D. (ed.) in H. Poincaré New Methods Of Celestial Mechanics, 3: Integral Invariants and Asymptotic Properties of Certain Solutions (American Institute of Physics, 1993).
  6. Crutchfield, J. P., Packard, N. H., Farmer, J. D. & Shaw, R. S. Chaos. Sci. Am. 255, 4657 (1986).
  7. Binney, J. J., Dowrick, N. J., Fisher, A. J. & Newman, M. E. J. The Theory of Critical Phenomena (Oxford Univ. Press, 1992).
  8. Cross, M. C. & Hohenberg, P. C. Pattern formation outside of equilibrium. Rev. Mod. Phys. 65, 8511112 (1993).
  9. Manneville, P. Dissipative Structures and Weak Turbulence (Academic, 1990).
  10. Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27 379–423; 623–656 (1948).
  11. Cover, T. M. & Thomas, J. A. Elements of Information Theory 2nd edn (Wiley–Interscience, 2006).
  12. Kolmogorov, A. N. Entropy per unit time as a metric invariant of automorphisms. Dokl. Akad. Nauk. SSSR 124, 754755 (1959).
  13. Sinai, Ja G. On the notion of entropy of a dynamical system. Dokl. Akad. Nauk. SSSR 124, 768771 (1959).
  14. Anderson, P. W. More is different. Science 177, 393396 (1972).
  15. Turing, A. M. On computable numbers, with an application to the Entscheidungsproblem. Proc. Lond. Math. Soc. 2 42, 230265 (1936).
  16. Solomonoff, R. J. A formal theory of inductive inference: Part I. Inform. Control 7, 124 (1964).
  17. Solomonoff, R. J. A formal theory of inductive inference: Part II. Inform. Control 7, 224254 (1964).
  18. Minsky, M. L. in Problems in the Biological SciencesVol. XIV (ed. Bellman, R. E.) (Proceedings of Symposia in Applied Mathematics, American Mathematical Society, 1962).
  19. Chaitin, G. On the length of programs for computing finite binary sequences. J. ACM 13, 145159 (1966).
  20. Kolmogorov, A. N. Three approaches to the concept of the amount of information. Probab. Inform. Trans. 1, 17 (1965).
  21. Martin-Löf, P. The definition of random sequences. Inform. Control 9, 602619 (1966).
  22. Brudno, A. A. Entropy and the complexity of the trajectories of a dynamical system. Trans. Moscow Math. Soc. 44, 127151 (1983).
  23. Zvonkin, A. K. & Levin, L. A. The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russ. Math. Survey 25, 83124 (1970).
  24. Chaitin, G. Algorithmic Information Theory (Cambridge Univ. Press, 1987).
  25. Li, M. & Vitanyi, P. M. B. An Introduction to Kolmogorov Complexity and its Applications (Springer, 1993).
  26. Rissanen, J. Universal coding, information, prediction, and estimation. IEEE Trans. Inform. Theory IT-30, 629636 (1984).
  27. Rissanen, J. Complexity of strings in the class of Markov sources. IEEE Trans. Inform. Theory IT-32, 526532 (1986).
  28. Blum, L., Shub, M. & Smale, S. On a theory of computation over the real numbers: NP-completeness, Recursive Functions and Universal Machines. Bull. Am. Math. Soc. 21, 146 (1989).
  29. Moore, C. Recursion theory on the reals and continuous-time computation. Theor. Comput. Sci. 162, 2344 (1996).
  30. Shannon, C. E. Communication theory of secrecy systems. Bell Syst. Tech. J. 28, 656715 (1949).
  31. Ruelle, D. & Takens, F. On the nature of turbulence. Comm. Math. Phys. 20, 167192 (1974).
  32. Packard, N. H., Crutchfield, J. P., Farmer, J. D. & Shaw, R. S. Geometry from a time series. Phys. Rev. Lett. 45, 712716 (1980).
  33. Takens, F. in Symposium on Dynamical Systems and Turbulence, Vol. 898 (eds Rand, D. A. & Young, L. S.) 366381 (Springer, 1981).
  34. Brandstater, A. et al. Low-dimensional chaos in a hydrodynamic system. Phys. Rev. Lett. 51, 14421445 (1983).
  35. Crutchfield, J. P. & McNamara, B. S. Equations of motion from a data series. Complex Syst. 1, 417452 (1987).
  36. Crutchfield, J. P. & Young, K. Inferring statistical complexity. Phys. Rev. Lett. 63, 105108 (1989).
  37. Crutchfield, J. P. & Shalizi, C. R. Thermodynamic depth of causal states: Objective complexity via minimal representations. Phys. Rev. E 59, 275283 (1999).
  38. Shalizi, C. R. & Crutchfield, J. P. Computational mechanics: Pattern and prediction, structure and simplicity. J. Stat. Phys. 104, 817879 (2001).
  39. Young, K. The Grammar and Statistical Mechanics of Complex Physical Systems. PhD thesis, Univ. California (1991).
  40. Koppel, M. Complexity, depth, and sophistication. Complexity 1, 10871091 (1987).
  41. Koppel, M. & Atlan, H. An almost machine-independent theory of program-length complexity, sophistication, and induction. Information Sciences 56, 2333 (1991).
  42. Crutchfield, J. P. & Young, K. in Entropy, Complexity, and the Physics of InformationVol. VIII (ed. Zurek, W.) 223269 (SFI Studies in the Sciences of Complexity, Addison-Wesley, 1990).
  43. William of Ockham Philosophical Writings: A Selection, Translated, with an Introduction (ed. Philotheus Boehner, O. F. M.) (Bobbs-Merrill, 1964).
  44. Farmer, J. D. Information dimension and the probabilistic structure of chaos. Z. Naturf. 37a, 13041325 (1982).
  45. Crutchfield, J. P. The calculi of emergence: Computation, dynamics, and induction. Physica D 75, 1154 (1994).
  46. Crutchfield, J. P. in Complexity: Metaphors, Models, and RealityVol. XIX (eds Cowan, G., Pines, D. & Melzner, D.) 479497 (Santa Fe Institute Studies in the Sciences of Complexity, Addison-Wesley, 1994).
  47. Crutchfield, J. P. & Feldman, D. P. Regularities unseen, randomness observed: Levels of entropy convergence. Chaos 13, 2554 (2003).
  48. Mahoney, J. R., Ellison, C. J., James, R. G. & Crutchfield, J. P. How hidden are hidden processes? A primer on crypticity and entropy convergence. Chaos 21, 037112 (2011).
  49. Ellison, C. J., Mahoney, J. R., James, R. G., Crutchfield, J. P. & Reichardt, J. Information symmetries in irreversible processes. Chaos 21, 037107 (2011).
  50. Crutchfield, J. P. in Nonlinear Modeling and ForecastingVol. XII (eds Casdagli, M. & Eubank, S.) 317359 (Santa Fe Institute Studies in the Sciences of Complexity, Addison-Wesley, 1992).
  51. Crutchfield, J. P., Ellison, C. J. & Mahoney, J. R. Time’s barbed arrow: Irreversibility, crypticity, and stored information. Phys. Rev. Lett. 103, 094101 (2009).
  52. Ellison, C. J., Mahoney, J. R. & Crutchfield, J. P. Prediction, retrodiction, and the amount of information stored in the present. J. Stat. Phys. 136, 10051034 (2009).
  53. Ruelle, D. Do turbulent crystals exist? Physica A 113, 619623 (1982).
  54. Varn, D. P., Canright, G. S. & Crutchfield, J. P. Discovering planar disorder in close-packed structures from X-ray diffraction: Beyond the fault model. Phys. Rev. B 66, 174110 (2002).
  55. Varn, D. P. & Crutchfield, J. P. From finite to infinite range order via annealing: The causal architecture of deformation faulting in annealed close-packed crystals. Phys. Lett. A 234, 299307 (2004).
  56. Varn, D. P., Canright, G. S. & Crutchfield, J. P. Inferring Pattern and Disorder in Close-Packed Structures from X-ray Diffraction Studies, Part I:  ε-machine Spectral Reconstruction Theory Santa Fe Institute Working Paper 03-03-021 (2002).
  57. Varn, D. P., Canright, G. S. & Crutchfield, J. P. Inferring pattern and disorder in close-packed structures via ε-machine reconstruction theory: Structure and intrinsic computation in Zinc Sulphide. Acta Cryst. B 63, 169182 (2002).
  58. Welberry, T. R. Diffuse x-ray scattering and models of disorder. Rep. Prog. Phys. 48, 15431593 (1985).
  59. Guinier, A. X-Ray Diffraction in Crystals, Imperfect Crystals and Amorphous Bodies (W. H. Freeman, 1963).
  60. Sebastian, M. T. & Krishna, P. Random, Non-Random and Periodic Faulting in Crystals (Gordon and Breach Science Publishers, 1994).
  61. Feldman, D. P., McTague, C. S. & Crutchfield, J. P. The organization of intrinsic computation: Complexity-entropy diagrams and the diversity of natural information processing. Chaos 18, 043106 (2008).
  62. Mitchell, M., Hraber, P. & Crutchfield, J. P. Revisiting the edge of chaos: Evolving cellular automata to perform computations. Complex Syst. 7, 89130 (1993).
  63. Johnson, B. D., Crutchfield, J. P., Ellison, C. J. & McTague, C. S. Enumerating Finitary Processes Santa Fe Institute Working Paper 10-11-027 (2010).
  64. Lind, D. & Marcus, B. An Introduction to Symbolic Dynamics and Coding (Cambridge Univ. Press, 1995).
  65. Hopcroft, J. E. & Ullman, J. D. Introduction to Automata Theory, Languages, and Computation (Addison-Wesley, 1979).
  66. Upper, D. R. Theory and Algorithms for Hidden Markov Models and Generalized Hidden Markov Models. Ph.D. thesis, Univ. California (1997).
  67. Kelly, D., Dillingham, M., Hudson, A. & Wiesner, K. Inferring hidden Markov models from noisy time sequences: A method to alleviate degeneracy in molecular dynamics. Preprint at http://arxiv.org/abs/1011.2969 (2010).
  68. Ryabov, V. & Nerukh, D. Computational mechanics of molecular systems: Quantifying high-dimensional dynamics by distribution of Poincaré recurrence times. Chaos 21, 037113 (2011).
  69. Li, C-B., Yang, H. & Komatsuzaki, T. Multiscale complex network of protein conformational fluctuations in single-molecule time series. Proc. Natl Acad. Sci. USA 105, 536541 (2008).
  70. Crutchfield, J. P. & Wiesner, K. Intrinsic quantum computation. Phys. Lett. A 372, 375380 (2006).
  71. Goncalves, W. M., Pinto, R. D., Sartorelli, J. C. & de Oliveira, M. J. Inferring statistical complexity in the dripping faucet experiment. Physica A 257, 385389 (1998).
  72. Clarke, R. W., Freeman, M. P. & Watkins, N. W. The application of computational mechanics to the analysis of geomagnetic data. Phys. Rev. E 67, 160203 (2003).
  73. Crutchfield, J. P. & Hanson, J. E. Turbulent pattern bases for cellular automata. Physica D 69, 279301 (1993).
  74. Hanson, J. E. & Crutchfield, J. P. Computational mechanics of cellular automata: An example. Physica D 103, 169189 (1997).
  75. Shalizi, C. R., Shalizi, K. L. & Haslinger, R. Quantifying self-organization with optimal predictors. Phys. Rev. Lett. 93, 118701 (2004).
  76. Crutchfield, J. P. & Feldman, D. P. Statistical complexity of simple one-dimensional spin systems. Phys. Rev. E 55, 239R1243R (1997).
  77. Feldman, D. P. & Crutchfield, J. P. Structural information in two-dimensional patterns: Entropy convergence and excess entropy. Phys. Rev. E 67, 051103 (2003).
  78. Bonner, J. T. The Evolution of Complexity by Means of Natural Selection (Princeton Univ. Press, 1988).
  79. Eigen, M. Natural selection: A phase transition? Biophys. Chem. 85, 101123 (2000).
  80. Adami, C. What is complexity? BioEssays 24, 10851094 (2002).
  81. Frenken, K. Innovation, Evolution and Complexity Theory (Edward Elgar Publishing, 2005).
  82. McShea, D. W. The evolution of complexity without natural selection—A possible large-scale trend of the fourth kind. Paleobiology 31, 146156 (2005).
  83. Krakauer, D. Darwinian demons, evolutionary complexity, and information maximization. Chaos 21, 037111 (2011).
  84. Tononi, G., Edelman, G. M. & Sporns, O. Complexity and coherency: Integrating information in the brain. Trends Cogn. Sci. 2, 474484 (1998).
  85. Bialek, W., Nemenman, I. & Tishby, N. Predictability, complexity, and learning. Neural Comput. 13, 24092463 (2001).
  86. Sporns, O., Chialvo, D. R., Kaiser, M. & Hilgetag, C. C. Organization, development, and function of complex brain networks. Trends Cogn. Sci. 8, 418425 (2004).
  87. Crutchfield, J. P. & Mitchell, M. The evolution of emergent computation. Proc. Natl Acad. Sci. USA 92, 1074210746 (1995).
  88. Lizier, J., Prokopenko, M. & Zomaya, A. Information modification and particle collisions in distributed computation. Chaos 20, 037109 (2010).
  89. Flecker, B., Alford, W., Beggs, J. M., Williams, P. L. & Beer, R. D. Partial information decomposition as a spatiotemporal filter. Chaos 21, 037104 (2011).
  90. Rissanen, J. Stochastic Complexity in Statistical Inquiry (World Scientific, 1989).
  91. Balasubramanian, V. Statistical inference, Occam’s razor, and statistical mechanics on the space of probability distributions. Neural Comput. 9, 349368 (1997).
  92. Glymour, C. & Cooper, G. F. (eds) in Computation, Causation, and Discovery (AAAI Press, 1999).
  93. Shalizi, C. R., Shalizi, K. L. & Crutchfield, J. P. Pattern Discovery in Time Series, Part I: Theory, Algorithm, Analysis, and Convergence Santa Fe Institute Working Paper 02-10-060 (2002).
  94. MacKay, D. J. C. Information Theory, Inference and Learning Algorithms (Cambridge Univ. Press, 2003).
  95. Still, S., Crutchfield, J. P. & Ellison, C. J. Optimal causal inference. Chaos 20, 037111 (2007).
  96. Wheeler, J. A. in Entropy, Complexity, and the Physics of Informationvolume VIII (ed. Zurek, W.) (SFI Studies in the Sciences of Complexity, Addison-Wesley, 1990).

Download references

Author information

Affiliations

  1. Complexity Sciences Center and Physics Department, University of California at Davis, One Shields Avenue, Davis, California 95616, USA

    • James P. Crutchfield

Competing financial interests

The author declares no competing financial interests.

Corresponding author

Correspondence to:

Author details

Additional data