Railroad along Deschutes River in Oregon.

Is cosmic evolution a single track with no choice about the destination?Credit: Getty

Was there ever any choice in the Universe being as it is? Albert Einstein could have been wondering about this when he remarked to mathematician Ernst Strauss: “What I’m really interested in is whether God could have made the world in a different way; that is, whether the necessity of logical simplicity leaves any freedom at all.”

US physicist James Hartle, who died earlier this year aged 83, made seminal contributions to this continuing debate. Early in the twentieth century, the advent of quantum theory seemed to have blown out of the water ideas from classical physics that the evolution of the Universe is ‘deterministic’. Hartle contributed to a remarkable proposal that, if correct, completely reverses a conventional story about determinism’s rise with classical physics, and its subsequent fall with quantum theory. A quantum Universe might, in fact, be more deterministic than a classical one — and for all its apparent uncertainties, quantum theory might better explain why the Universe is the one it is, and not some other version.

In physics, determinism means that the state of the Universe at any given time and the basic laws of physics fully determine the Universe’s backward history and forward evolution. This idea reached its peak with the strict, precise laws about how the Universe behaves introduced by classical physics. Take Isaac Newton’s laws of motion. If someone knew the present positions and momenta of all particles, they could in theory use Newton’s laws to deduce all facts about the Universe, past and future. It’s only a lack of knowledge (or computational power) that prevents scientists from doing so.

Along with this distinctive predictive power, determinism underwrites scientific explanations that come close to the ‘principle of sufficient reason’ most famously articulated by German polymath Gottfried Leibniz: that everything has an explanation. Every state of the Universe (with one obvious exception, which we’ll come to) can be completely explained by an earlier one. If the Universe is a train, determinism says that it’s running on a track, with no option to switch to any other path because different tracks never cross.

Physicists have conventionally liked determinism’s predictive and explanatory power. Others, including some philosophers, have generally been more divided, not least because of how determinism might seem to preclude human free will: if the laws of physics are deterministic, and our actions are just the summation of particle interactions, there seems to be no room for us to freely choose A instead of B, because the earlier states of the Universe will already have determined the outcome of our choice. And if we are not free, how can we be praised or blamed for our actions? Neuroendocrinologist Robert Sapolsky’s 2023 book Determined touches on this fascinating and controversial issue.

Space invaders

The strange behaviours of quantum particles that began to emerge in the twentieth century fundamentally shifted the debate surrounding determinism in physics. The laws of quantum mechanics give only the probabilities of outcomes, which can be illustrated with the thought experiment devised by Austrian physicist Erwin Schrödinger in 1935 (although when he devised it, he was concerned mainly with how the wavefunction represents reality). A cat is trapped in a box with a vial of poison that might or might not have been broken by a random event — because of radioactive decay, for example. If quantum mechanics applied to the cat, it would be described by a ‘wavefunction’ in a superposition of ‘alive’ and ‘dead’. The wavefunction, when measured, randomly jumps to one of the two states, and quantum mechanics specifies only the probability of either possibility occurring. One consequence of the arrival of quantum mechanics was that it seemed to throw determinism out of the window.

But this accepted idea might not be the whole story, as developments in the second half of the twentieth century suggested. The quantum Universe could actually be more deterministic than a classical one, for two reasons. The first is technical. Newton’s laws allow situations in which the past does not determine how things will move in the future. For example, the laws do not provide an upper bound on how much an object can be accelerated, so in theory a classical object can reach spatial infinity in finite time. Reverse this process, and you get what have been called ‘space invaders’ — objects that come from spatial infinity with no causal connection to anything else in the Universe, and which can’t be predicted from any of the Universe’s past states.

In practice, this problem is solved by the universal speed limit, the speed of light, introduced by Einstein’s special theory of relativity. But unruly infinities also plague Einsteinian relativity, which is a classical theory. The equations of general relativity lead to ‘singularities’ of infinite curvature, most notoriously in black holes and at the Big Bang at the beginning of the Universe. Singularities are like gaps in space-time where the theory no longer applies; in some cases, anything can come out of them (or disappear into them), threatening determinism.

Many physicists think that quantum theory can come to the rescue by removing such singularities — for example, by converting the Big Bang into a ‘Big Bounce’, with a Universe that continues to evolve smoothly on the other side of the singularity. If they are right, a theory of ‘quantum gravity’ that fully unifies quantum theory, which predicts the behaviour of matter on the smallest scales, and Einstein’s relativity, which encapsulates the large-scale evolution of the Universe, will smooth out the gaps in space-time and restore determinism.

Supermassive black hole at centre of our galaxy, EHT image.

Space-time ‘singularities’ inside black holes could threaten a deterministic cosmic order.Credit: ESO/SPL

But there is a deeper reason why the quantum Universe might be more deterministic, to which Hartle’s scientific legacies are relevant. With US physicist Murray Gell-Mann, Hartle developed an influential approach to quantum theory, called decoherent histories1. This attempted to explain the usefulness of probabilistic statements in quantum physics, and the emergence of a familiar, classical realm of everyday experience from quantum superpositions. In their picture, the wavefunction never randomly jumps. Instead, it always obeys a deterministic law given by Schrödinger’s equation, which characterizes the smooth and continuous evolution of quantum states. In this respect, it is similar to US physicist Hugh Everett III’s popular ‘many worlds’ interpretation of quantum mechanics, which proposes that the quantum Universe splits into different branches according to the possibilities encoded in the wavefunction whenever anything is measured2. In what follows I assume, as Everett did, that the Universe can be completely described by a quantum wavefunction with no ‘hidden’ variables that operate on a more fundamental level.

Into the quantum cosmos

With Stephen Hawking, Hartle went on to become one of the founders of quantum cosmology, which applies quantum theory to the entire Universe. In a classical Universe, there is freedom in choosing how it all started. Even setting aside the extreme situations mentioned earlier, classical mechanics is deterministic merely in that it lays down many possible evolutionary histories for the Universe, and offers conditional statements about them: if this happens, then that must happen next. To return to the train analogy, a deterministic theory does not, by itself, say why the train is on any one given track out of many: why it is going from A to B via C, rather than from X to Y via Z. We can go back to earlier states to explain the current state, and do that all the way back to the initial state — but this initial state is not explained by anything that precedes it. Ultimately, standard determinism fails to fully satisfy Leibniz’s principle of sufficient reason: when it comes to the initial state, something remains without an explanation.

This failure is not just philosophical. A complete theory of the Universe should predict the phenomena we observe in it, including its large-scale structure and the existence of galaxies and stars. The dynamic equations we have, whether from Newtonian physics or Einsteinian relativity, cannot do this by themselves. Which phenomena show up in our observations depend sensitively on the initial conditions. We must look at what we see in the Universe around us, and use this information to determine the initial condition that might have given rise to such observations.

A theory that specifies deterministic laws of both the Universe’s temporal evolution and its exact initial condition satisfies what English physicist Roger Penrose called ‘strong determinism’ in his 1989 book The Emperor’s New Mind. This is, according to Penrose, “not just a matter of the future being determined by the past; the entire history of the universe is fixed, according to some precise mathematical scheme, for all time”. Let us say that a Universe is strongly deterministic if its basic laws of physics fix a unique cosmic history. If determinism provides a set of non-crossing train tracks, without specifying which one is being used, then strong determinism lays down a single track that has no choice even about where it starts.

A universal wavefunction

Strong determinism is hard to implement in classical physics. You might consider doing it by specifying the initial condition of the Universe as a law. But although the dynamical laws of classical physics are simple, the Universe itself is complex — and so its initial condition must have been, too. Describing the precise positions and momenta of all the particles involved requires so much information that any statement of the initial condition is too complex to be a law.

Hartle suggested3 that quantum mechanics can solve this complexity problem. Because a quantum object’s wavefunction is spread out across many ‘classical’ states (cat alive or cat dead, for instance), you could propose a simple initial condition that includes all the complexities as emergent structures in the quantum superposition of these states. All the observed complexities can be regarded as partial descriptions of a simple fundamental reality: the Universe’s wavefunction. As an analogy, a perfect sphere can be cut into many chunks with complicated shapes, yet they can be put back together to form a simple sphere.

In 1983, Hartle and Hawking introduced4 one of the first (and highly influential) proposals about the quantum Universe’s initial state. Their ‘no boundary’ wavefunction idea suggests that the ‘shape’ of the Universe is like that of a shuttlecock: towards the past, it rounds off smoothly and shrinks to a single point. As Hawking said in a 1981 talk on the origin of the Universe in the Vatican: “There ought to be something very special about the boundary conditions of the Universe, and what can be more special than the condition that there is no boundary?”

In this perspective, the quantum Universe has two basic laws: a deterministic one of temporal evolution and a simple one that picks an initial wavefunction for the Universe. Hence, the quantum Universe satisfies strong determinism. The physical laws permit exactly one cosmic history of the Universe, albeit one described by a wavefunction that superposes many classical trajectories. There is no contingency in what the Universe as a whole could have been, and no alternative possibility for how it could have started. Every event, including the first one, is explained; the entire wavefunction of the Universe for all times is pinned down by the laws. The probabilities of quantum mechanics do not exist at the level of the basic physical laws, but can nonetheless be assigned to coarse-grained and partial descriptions of bits of the Universe.

This leads to a more predictive and explanatory theory. For example, the no-boundary proposal makes predictions for a relatively simple early Universe and for the occurrence of inflation — a period of rapid expansion that the Universe seems to have undergone in its first instants.

There are still many wrinkles to this proposal, not least because some studies have shown that, contrary to initial expectations, the theory might not single out a unique wavefunction for the Universe5,6. But studies in quantum foundations — research that is mostly independent from that of quantum cosmology — could offer yet another method for implementing strong determinism. Several researchers have considered the controversial idea that quantum states of closed systems, including the Universe, need not be restricted to wavefunctions, but instead can come from a broader category: the space of density matrices7–10.

The ultimate theory

Density matrices can be thought of as ‘superpositions of superpositions’, and they provide extra options for the initial condition of the Universe. For example, if we have reasons to adopt the ‘past hypothesis’ — the idea, which seems likely, that the Universe began in a low-entropy state (and its entropy has been increasing steadily since) — and that this theory corresponds to a set of wavefunctions, then we can choose a simple density matrix that corresponds to the uniform mixture of that set. As I have argued10, if we regard the density matrix as the initial state of the Universe and accept that it is specified by a law, then this choice, together with the deterministic von Neumann equation (a generalization of Schrödinger’s equation), can satisfy strong determinism. However, in this case, the laws fix a cosmic history of a quantum Universe that has many evolving branches — a ‘multiverse’.

So how deterministic is the Universe? The answer will depend on the final theory that bridges the divide between quantum physics and relativity — and that remains a far-off prospect. But if Hartle is right, the story of the rise and fall of determinism until now might be the reverse of the conventional tale. From a certain perspective, the quantum Universe is more deterministic than a classical one, providing stronger explanations and better predictions. That has consequences for humans, too, because that makes it harder to appeal to quantum theory to defend free will11. If the quantum Universe is strongly deterministic, then there is no other path to make the Universe than the way it is. The ultimate laws of the quantum cosmos might tell us why it is this one.