On 18 June 1962, an 8-metre rocket carried three small X-ray detectors to the edge of space. They spent just under 6 minutes above the altitude of 80 kilometres, high enough for kiloelectronvolt-energy X-rays from space to reach them through the thinned atmosphere. The result of this brief flight by physicist Riccardo Giacconi and his colleagues revolutionized astronomers' view of what the Universe contains.

For 50 years, X-ray astronomy has burgeoned. The field has grown a billion times more sensitive — a feat that took optical astronomy 400 years to achieve — and has opened up such exotica as black holes and dark matter to detailed investigation. It has provided a unique window into 'extreme' places of the Universe, where gas can be 1,000–10,000 times hotter than the surface of the Sun. The question now is whether that progress will continue.

The Milky Way as a combined image of near-infrared (yellow), infrared (red) and X-ray (blue and violet) data collected by NASA's 'Great Observatories'. Credit: CHANDRA X-RAY OBSERVATORY CENTER

In the 1960s, X-rays were known to be produced by the Sun — that had been discovered in 1948. But the Sun's weak X-ray output made it seem futile to try to detect any other star this way. Giacconi justified his flight by telling the US Air Force that he was looking for X-rays from the Moon, but he hoped to find more. Fortunately, the Universe delivered surprises: a strong source of X-rays in the constellation Scorpius and a bright background of X-rays from all over the sky. The researchers were lucky. Had they looked when Sco X-1, as this first source was dubbed, was below the horizon, they would have detected only the bright background noise. Interest in cosmic X-rays might have withered.

Many teams rushed to follow up in the subsequent years. X-ray astronomy papers often dominated the pages of Astrophysical Journal Letters. One important discovery was that the brightness of these cosmic X-ray sources often changed within seconds. Because nothing can change its output faster than light can travel across it, this meant that the objects must be tiny — just light seconds across. Creating so much energy from such a small volume required some not-yet recognized energy source. This turned out to be gravity: the accretion of gas falling down the deep gravitational well of a compact star converts potential energy into millions of degrees of heat.

The next big step was the development of orbiting X-ray instruments, which allowed for exposure times of days rather than minutes. NASA's 1970 Uhuru mission, led by Giacconi, was the first in a small fleet of these. Uhuru and its successors detected hundreds of X-ray sources 1,000–10,000 times fainter than Sco X-1.

But it was the cosmic X-ray background that set the real programme for X-ray astronomy for the next 40 years. Was it caused by hot gas pervading intergalactic space, or by millions of faint and distant sources blending together? Early detectors could not answer this question, because they had to collect X-rays from a large chunk of the sky to detect a signal; discrete sources that might make up the background produced no more than one X-ray per square centimetre of detector each day. To see with greater precision, researchers needed more sensitive telescopes, which they built from nested cylindrical mirrors.

The first such imaging X-ray telescope — NASA's 1978 Einstein Observatory, led by Giacconi — detected radiation 100-fold fainter than any of its predecessors. It found that 20% of the cosmic X-ray background came from active galactic nuclei, later understood to host supermassive black holes in their centres.

Leaps and bounds

By 1987 — the twenty-fifth anniversary of the discovery of Sco X-1 — X-ray astronomy had become 10 million times more sensitive. The next generation of X-ray telescopes came in the early 1990s. Notable among them was Germany's ROSAT, which surveyed the whole sky, cataloguing 100,000 X-ray sources and, by going tenfold fainter still than Einstein, resolving some 60% of the cosmic X-ray background that had an energy lower than 2 kiloelectronvolts (keV).

The current era was ushered in by the launch of two large X-ray observatories in 1999: NASA's Chandra and the European Space Agency's XMM-Newton. XMM has a large collecting area, but Chandra — with an angular resolution ten times better than anything previous — was revolutionary. Giacconi received the Nobel prize in 2002, perhaps stimulated by the outpouring of results from Chandra, which he had been instrumental in getting started.

Chandra has resolved the cosmic X-ray background at all energies up to 7 keV. More than 90% of the background is due to the summed emissions of millions of active galaxies (most of the rest comes from fainter, star-forming galaxies). Chandra and XMM have also delimited the extent of dark matter's interaction with itself, determined properties of dark energy and showed that as much as one-fourth of the energy of a supernova goes into accelerating protons, solving the mystery of the origin of high-energy cosmic rays.

Fifty years on, the founding mysteries of X-ray astronomy have been solved. But our understanding of more complex questions is still primitive. In my own sub-field alone, we don't yet know why active galaxies emit X-rays, where the massive black holes at their centres come from or how a massive black hole accelerates matter in bulk to speeds more than 90% that of light in a tight beam that can extend over millions of light years.

At the same time, X-ray astronomy has the power to test two bases of twentieth-century physics — general relativity and quantum chromodynamics (which describes interactions of the 'strong force' between quarks and gluons) — in the extreme conditions around black holes and neutron stars.

Prospects for a set of 'Greater Observatories' that span the electromagnetic spectrum look bleak.

But for astronomers to answer these questions, X-ray astronomy must become even more sensitive, by collecting more photons with larger mirrors while maintaining the fine resolution seen with Chandra. The US 2010 decadal survey for astronomy ranked a larger-area X-ray mission fourth among major space missions, but that is low enough, given the restricted US science budget, to put a new X-ray observatory on the back burner for at least a decade. Each space observatory, be it optical, infrared or X-ray, now costs more than US$2 billion. The worldwide space astronomy budget of about $5 billion a year is not enough to sustain a comprehensive programme.

NASA's current 'Great Observatories' — Hubble, Chandra and Spitzer — span the infrared to X-ray bands. They will probably have just one successor: the James Webb Space Telescope (JWST), which was designed to study cosmology and galaxy evolution in the distant Universe, and so works primarily in the infrared. Over the rest of the spectrum, astronomers will be blind. A US successor to Chandra is unlikely until 2030. By then, the JWST will probably be dead, preventing cross-fertilization of ideas among different wavelengths.

Ingenious, but more specialized, missions in X-ray astronomy can have more modest price tags of around $200 million. In the next few years, several such missions will be launched — one in pursuit of higher-energy rays (NuSTAR, part of NASA's Explorer class), one as an updated ROSAT (Germany's eROSITA) and one with relatively high spectral resolution (Japan's ASTRO-H). But none of these even approaches Chandra's spatial resolution.

Since 2010, some ideas for achieving more powerful missions on a tight budget have emerged, including combining the fine imaging of Chandra with a light-gathering surface 30 times more powerful. This would require the development of mirrors that can be actively corrected to ensure a sharp image, which seems feasible. But such a project would cost at least $2 billion. Prospects for a set of 'Greater Observatories' that span the electromagnetic spectrum look bleak.

A profitable solution

The only viable solution is to lower the cost of getting equipment into orbit. Launch costs have held steady at some $10,000 per kilogram of payload for more than 50 years. Spacecraft must have ingenious designs to keep their masses low, which makes the cost of building and launching high: $100,000 or more per kilogram of craft. Overall mission costs could plummet if launch costs were to decline.

The best — perhaps only — way to lower launch costs is to allow private enterprises to profit from space ventures. Profit is the counterweight to caution, and competition will drive down costs. Several companies have already developed operational launchers for government payloads, including Orbital Sciences Corporation of Dulles, Virginia, whose Pegasus rocket was, as Nature went to press, scheduled to launch NuSTAR on 13 June, and SpaceX of Hawthorne, California, which last month launched the first private mission to resupply the International Space Station.

Ultimately, a healthy market that spurs lower costs will require customers other than the government, such as those seeking minable resources more plentiful in space than on Earth. One new company, Planetary Resources of Bellevue, Washington, announced in April its aim to mine asteroids.

Such profit-seeking could decrease launch costs by more than an order of magnitude, such that fuel costs dominate the overall price. Then, and only then, will new generations of greater observatories, in X-rays and across the electromagnetic spectrum, become affordable for scientists.