In 1998, two teams of astronomers reported that the Universe was pulling itself apart. This came as something of a shock. That the Universe was expanding had been known since the 1920s, but conventional wisdom held that this expansion was slowing and was likely, in the distant future, to come to an all but complete halt. Then, in the late 1990s, observations of distant supernovae showed that the expansion was not slowing down at all. It was speeding up. This discovery was incredibly counterintuitive, recalls Charles Bennett, an astronomer at Johns Hopkins University in Baltimore, Maryland. “I just didn't believe it.”

Credit: D. PARKINS

Within a few years, however, he and almost all his peers could withhold their belief no longer. The observations became stronger. And the expansion provided a way out of a theoretical impasse. Observations of the Big Bang's afterglow made by various groups, including Bennett's, indicated that the Universe's gravity had flattened it out. But other observations suggested that it simply didn't contain enough matter to have that much of a gravitational effect — even when as-yet-undiscovered forms of dark matter were included in the sums (see page 240).

Happily, the theory of relativity requires energy, as well as matter, to have a gravitational effect. And it turned out that the amount of energy needed to drive the acceleration was pretty close to that needed to solve the flatness problem by means of its gravity. 'Dark energy', as it quickly became known, seemed poised to provide great insight into the origin and future of the cosmos, says Michael Turner, a cosmologist at the University of Chicago in Illinois. “This seemed to be the piece that made everything else work.”

But a decade further on, researchers seem to have swapped one theoretical conundrum for a bigger one. Follow-up measurements have revealed little about the nature of dark energy, and theories to explain it have failed to gain traction. And although astronomers are trudging forwards with a battery of new measurements, there is little guarantee that any will solve the problem — and thus no clear consensus on how much effort to put into them. “The issue is: how much information do we get from these future observations?” asks Avi Loeb, an astrophysicist at Harvard University.

Successive images of a patch of sky can reveal exploding supernovae (red spot). Credit: NASA; J. BLAKESLEE, JOHNS HOPKINS UNIV.)

Hidden depths

The big problem is that dark energy is not, in itself, something that astronomers can see. Like dark matter, it is known only by its effects — in this case, the effect it has on the Universe's acceleration. The acceleration is related to dark energy through a quantity known as the 'equation of state'— the ratio of the pressure dark energy exerts to the energy per unit volume involved.

An accelerating expansion means that the equation of state has to be negative. And a value of −1 would mean that dark energy was an unchanging feature of the cosmos — a 'cosmological constant'. Such a constant had been a feature of Einstein's general theory of relativity, one that he had added, ironically, as a way of guaranteeing that the Universe would stay the same size. When Einstein came to accept that the Universe was, in fact, expanding he removed the term, calling it his “greatest mistake”. But if the equation of state had a value of −1, dark energy would fit the cosmological constant bill perfectly. And current measurements make it quite possible that the equation's value is −1.

If dark energy's equation of state is indeed −1, then there's one obvious way to make sense of it, says Leonard Susskind, a cosmologist at Stanford University in California. For decades, physicists have postulated the existence of something known as 'vacuum energy' — a primordial froth of quantum particles that flit in and out of existence in the vacuum of space. This vacuum energy could drive the observed accelerating expansion, and it would do so in a constant manner. Because vacuum energy is an inherent property of space, Susskind explains, an expanding Universe would create more of it, meaning that the ratio of energy density to pressure would never change, their ratio fixed for ever at −1.

There's just one theoretical discrepancy: the vacuum energy as calculated by physicists is more than 10100 times larger than would be needed to explain the relatively weak effects of dark energy as observed by astronomers. If it were as big as physicists suggest, then our Universe would fly apart in the blink of an eye. “Every calculation indicates that vacuum energy should be enormous,” says Turner. “There's no natural way to get such a tiny number.” So most physicists have hoped that some yet-to-be-discovered effect based on some hidden symmetry of nature would cancel out the vacuum energy. Such a hope-it-goes-away approach is used by physicists quite a lot, and can be the only way to make progress in some circumstances. At the same time, applying it to the vacuum energy was, admits Susskind, “completely illogical”.

“And I must say I shared that illogical attitude myself,” he continues almost apologetically. Now, he thinks differently, and is one of those who has proposed a solution of sorts to the conundrum. 'String theories', popular with many particle physicists, make it possible, even desirable, to think that the observable Universe is just one of 10500 universes in a grander 'multiverse', says Susskind. The vacuum energy will have different values in different universes, and in many or most it might indeed be vast. But it must be small in ours because it is only in such a universe that observers such as ourselves can evolve.

This sort of anthropic argument irks many scientists. Critics say such reasoning is almost impossible to verify and doesn't provide any deeper insight into the cosmos. “Anthropics and randomness don't explain anything,” says Paul Steinhardt, a theorist at Princeton University in New Jersey. “I'm disappointed with what most theorists are willing to accept.”

The trouble is that no other approaches are proving any more fruitful. Some suggest that the problem lies with Einstein's idea of gravity, which they then seek to modify in a way that fits in with dark energy. “It would be very fortunate if the dark energy were a modification of gravity,” says Georgi Dvali of New York University, “because it would address fundamental questions of physics.” But others see little mileage in such changes. Leaving aside the cosmos, “it's not so easy to get those theories to be consistent with our Solar System”, says Turner.

Another possibility is that dark energy is some sort of evolving property of the Universe. Some postulate that dark energy is a fifth force (the others being electromagnetism, the two nuclear forces and gravity) that works at the largest scales of the cosmos. Others suspect that it is the aftermath of the inflation that many see following directly on from the Big Bang. Inflation was, after all, a period of extreme expansion — might it not have some sort of 'long tail' that stretched away down cosmic history? These solutions and others, although different conceptually, are equivalent mathematically. And they share a requirement that dark energy changes over time — that its equation of state is not locked in as −1. Such a change would help to explain why dark energy is apparently so weak today, says Steinhardt. And changing values for dark energy might affect other features of the Universe, including some parameters now seen as fundamental constants, in detectable ways, which could be a plus. But critics claim that these ideas require extreme amounts of special pleading.

Starring role

Leonard Susskind: “We could be deeply wrong about cosmology for the next thousand years”. Credit: A. WARREN

In general, the theoretical side of the debate is not a pretty thing. “We've tried a whole bunch of things and nothing has sprung forward,” says Sean Carroll, a theoretical physicist at the California Institute of Technology in Pasadena. What's needed, Carroll says, are a few more good clues.

Astronomers are planning a new generation of dark-energy probes that will refine measurements of the equation of state. They are already pushing ahead with further measurements of type 1a supernovae. These stellar outbursts occur when a stream of material being sucked from a larger star onto a smaller one pushes the smaller star's mass over a threshold, precipitating a massive thermonuclear explosion. Because each star explodes at the same mass threshold, they should all give off the same amount of energy. And so, in absolute terms, each should be as bright as any other. By comparing their relative brightnesses when seen from Earth, it is possible to measure the distance to the explosion with precision, says Saul Perlmutter, the astronomer at Lawrence Berkeley National Laboratory in California who led one of the original dark-energy supernova teams. And by measuring distance in this way and speed by means of the 'red shift' of the supernova's light, astronomers can understand acceleration over time. Perlmutter and others are now working to increase both their understanding of the supernova mechanism and the size of their sample to improve on their original calculations.

Supernovae, although the best understood, are not the only way to measure acceleration. Another option is to study X-rays from distant clusters of galaxies. As in the case of supernovae, a cluster's temperature and brightness should have a standard relationship, so it should be possible to measure the speed at which those at a given distance from Earth are receding, says Steve Allen, an X-ray astronomer at Stanford University.

It is also possible to measure the effects of dark energy in subtler ways. The gravitational field of a cluster or group of galaxies makes light shift towards the blue as it falls into the galaxies' gravity well, and reddens it as it climbs back out. According to Ryan Scranton, an astronomer at the University of Pittsburgh in Pennsylvania, dark energy should affect the way these effects show up in the cosmic microwave background, radiation left over from the Big Bang.

A tangled web

Combining these different sorts of measurement should offer ways of constraining the value of the equation of state better than any single measurement can manage (see 'Closing in on dark energy'). Perhaps the most promising new realm of research, say many in the field, lies in surveys that will look at how the largest structures in the Universe have been shaped or distorted by dark energy. Galaxies are not spread evenly across the cosmos, but instead clump into a three-dimensional cobweb. The structure of that cobweb is sensitive to dark energy. And the sort of error to be expected in measurements of the structure are completely different from those that plague measurements of supernovae, according to Adam Riess, an astronomer at Johns Hopkins University who led the original supernova team that competed with Perlmutter's. That makes the new approach pleasingly independent of the old one. Several ambitious surveys are now being planned to further map the large-scale structure of the Universe (see 'The search for structure').

But none of these techniques can do more than narrow the frustratingly uninformative equation of state down further. To prove that dark energy is a cosmological constant requires showing that the equation of state is indeed −1. Merely showing that it is close doesn't cut it. Astronomers could basically go on measuring dark energy for ever without eliminating other possible theories, says Simon White, director of the Max Planck Institute for Astrophysics in Garching, Germany: “If it's just a constant, then you need infinite accuracy.”

Lawrence Krauss, a theoretical physicist at Case Western Reserve University in Cleveland, Ohio, goes further. If the equation of state is indeed −1, and dark energy is a constant, then the only way to measure it will be through its effect on the Universe's acceleration. “If it is −1, we won't know what dark energy is,” he says. “It doesn't give us any theoretical guidance whatsoever.”

Carlos Frenk, a theoretical physicist at the University of Durham, UK, agrees that probing a single number without a strong theoretical case for doing so is not the way forward. “It's like trying to learn something fundamental about biology by measuring the height of every tree,” he says. “Just measuring something for the sake of measuring it is pointless.” Frenk questions how much money should be spent on such measurements, and Loeb agrees. “One should put money in this direction,” he says, “but not excessive amounts.”

Michael Turner: “Dark energy seemed to be the piece that made everything else work”. Credit: UNIV. CHICAGO

But for all the worries of some theoretical physicists, observational astronomers think that carrying on with the equation of state measurement is the most sensible next step, not least because it is the only one on offer. “My feeling is that we should measure it to the limits,” says Bennett. “We may see things that surprise people, that often happens.” Perlmutter, too, sees room for a few more results to narrow things down rather than shaking them up. “It seems like you'd want to get a couple of boring results before you decide 'we're done',” he says.

Bennett and Perlmutter's enthusiasm for further measurements is evidenced by the fact that they are heading up rival proposals for spacecraft to observe galactic structure (Bennett) and distant supernovae (Perlmutter), seeking to get the money that NASA and the US Department of Energy are considering spending on a dark-energy probe. And even if their endeavours contribute no more than some incremental precision to the debate on dark energy, the observations will still tell astronomers quite a bit about other things in the Universe. A space-based supernova probe, for example, would provide a high-quality survey of infrared objects throughout the sky. “These are not special-purpose instruments,” says Roger Blandford, director of the Kavli Institute for Particle Astrophysics and Cosmology in Stanford, California. “They will revolutionize a whole range of fields.”

And there is always a chance that some other area will reveal the next much-needed clues as to the nature of dark energy. When it starts taking data in 2008, the Large Hadron Collider at CERN, the particle-physics laboratory near Geneva, might conceivably make relevant discoveries about the nature of space-time (see Insight, page 269); “We may learn more from accelerators than we do from the sky,” says Krauss. Similarly, measurements of fundamental constants and gravitation at short distances could have some unexpected connection to the dark-energy problem; and detecting some sort of dark matter might help, too (see 'Welcome to the dark side', page 240).

So far, though, the revolution promised by dark energy's discovery a decade ago hasn't materialized. Although researchers are more certain than ever of the existence of a cosmic push, they know as little about what it means physically as they did in 1998. “Right now there are two possibilities,” says Carroll. “Dark energy is vacuum energy, or it's something else.” Observers are slightly more upbeat. “It feels to me like a very early discussion of all this,” says Perlmutter. Still, he concedes, without a measurement of the equation of state that deviates from −1 it will be difficult to learn much of anything. “If you don't see those ripples,” he says, “it's going to be hard to play the game.”

For now, many in the field are left with a sense of unease: the tantalizing clue they thought they had discovered has turned into an exasperating mystery. And with no clear explanation of something that could be up to three-quarters of everything out there, it's hard not to feel like you're missing a big part of the picture, Susskind says. “We could be wrong about cosmology for the next thousand years. Deeply wrong.”