Electrons are accelerated before entering an X-ray free electron laser. Credit: DESY

In the foothills above Palo Alto, California, physicists have set up an extreme obstacle course for some of the world's fastest electrons. First the particles are accelerated through a 3-kilometre vacuum pipe to almost the speed of light. Then they slam through a gauntlet of magnets that forces them into a violent zigzag. They respond with a blast of X-rays so fierce it could punch through steel.

Originality takes second place for young scientists with extreme workloads Why flawed diamonds are advancing quantum computing How plant-killing fungi protect forest diversity

But the scientists at the SLAC National Accelerator Laboratory have no interest in weaponry. Their machine, one of the world's most powerful X-ray free-electron lasers (XFELs), is a tool for studying challenging forms of matter, whether compressed to the kind of pressures and temperatures found deep inside a star, or folded into the complex tangle of a protein molecule.

Structural biologists, in particular, stand to benefit greatly from XFELs. With X-ray pulses short enough to capture strobe-like images of molecular motions, and intense enough to image the multitude of biomolecules that have defied conventional methods, XFELs are giving biologists new ways to scan for potential drug targets, to probe the mechanics of photosynthetic molecules, and more.

“XFELs, without any doubt, are disruptive technology,” says Keith Moffat, a crystallographer at the University of Chicago in Illinois who has served on the SLAC machine's scientific advisory board — “an advance that is so far beyond what has gone before that it alters the way you do things”.

But XFELs have also been controversial technology — especially the SLAC machine, known as the Linac Coherent Light Source (LCLS), which was one of the first and biggest. It was given the go-ahead by the US Department of Energy (DOE) in 2002 in the face of frequent criticism from researchers, many of whom doubted whether its scientific benefits would ever be worth its US$414-million cost — assuming that the unproven technology worked at all.

Nature Podcast: Three experts pick their favourite crystal structures

Those concerns have ebbed since the LCLS began operation in 2009, says Moffat: “This thing worked, pretty much as advertised, pretty much right out of the box, on schedule, on budget.” In its wake, Japan has built its own XFEL; Europe is following with an even more capable version set to open in 2015; and others are being planned for Switzerland and South Korea. Global investments in XFELs over the next few years will total billions of dollars. But to reach their full potential, these machines will have to surmount many more technical hurdles, from boosting their power and brightness to handling the deluge of data they produce.

“Physicists, biologists, laser scientists, high-energy-density scientists — a completely new community is being formed, because you have to understand all the processes involved,” says Janos Hajdu, a molecular biophysicist at Uppsala University in Sweden. “There are lots of developments that have to come together to make this work.”

Corralling X-rays

The path towards XFELs began just over 100 years ago, when pioneering physicists including Max von Laue recognized the power of X-rays for studying matter (see page 602). Only photons with extremely short wavelengths can image molecules or materials at the atomic scale — roughly 0.1 nanometres, or 1 ångström.

But getting images from X-rays is tricky. There is no X-ray equivalent of a visible-light microscope, mainly because there are no good lenses for focusing the rays. So for the past century, physicists have relied on X-ray crystallography, in which they fire a beam of X-rays through a crystal lattice of identical molecules and record the resulting 'diffraction pattern' of scattered X-rays. They then work backwards from the pattern to mathematically reconstruct the original structure.

Nature special: Crystallography at 100

In recent decades, this has been done mostly at synchrotrons: accelerators that generate X-rays by whipping electrons around in a circle. Dozens of these light sources have grown up around the world, and they have been a boon to structural biology: the international Protein Data Bank repository currently has nearly 100,000 structures on file, most obtained from synchrotrons.

Unfortunately, many of the most scientifically interesting biomolecules, such as some membrane-bound protein complexes that mediate molecular traffic in and out of the cell, are still out of the reach of synchrotrons because they do not grow into crystals that are large enough and perfect enough to produce a usable diffraction pattern.

Yet even the most crystallization-resistant macromolecules will often form nanocrystals a few dozen molecules across. Because the beams from synchrotrons are not bright enough to get usable diffraction patterns from such structures, researchers have turned to XFELs, which are at least a billion times brighter than synchrotrons.

The basic principles of XFELs were worked out in the 1980s, building on an earlier generation of free-electron lasers that produced photons much less energetic than X-rays. In both types of laser, a beam of unconfined electrons passes through magnets that force it into an undulating path, and the beam emits photons along its line of flight. But at X-ray energies, the photons interact with the electrons in a manner that produces ferociously bright X-ray laser pulses lasting only a few femtoseconds (10−15 seconds) each — short enough to essentially freeze the motion of molecules in the target (see 'X-ray vision').

In 1992, Claudio Pellegrini, a physicist at the University of California, Los Angeles, and the idea's leading champion, proposed to build one of these machines at SLAC, arguing that the facility's soon-to-retire 50-GeV electron beam could be adapted to make an XFEL operating at wavelengths of 1–40 ångströms.

To the idea's many sceptics, Pellegrini admits, this was a fool's errand: no one had ever demonstrated a free-electron laser at these energies. “There was a lot of scepticism that you could really reach 1 ångström,” he says.

Still, says Pellegrini, there were also many physicists around the world who thought that the idea was worth pursuing. And through experiments and simulations during the 1990s, advocates systematically built a persuasive argument that XFELs would work.

By the early 2000s, that case was solid enough for the DOE to commit to building the SLAC machine. Germany had already started to build the Free-Electron Laser in Hamburg (FLASH), a lower-energy 'soft' XFEL at the German Electron Synchrotron (DESY); and Japan and a group of European countries were initiating studies that would, a decade later, lead to their own machines.

Before the Explosion

Even as the first XFELs were taking shape, however, would-be users were grappling with a seemingly intractable problem — such bright beams would destroy any sample in their path. Only in 2000 did Hajdu and his team demonstrate an escape1: on a femtosecond timescale, even molecular explosions unfold slowly. It takes roughly 10 femtoseconds for photons to be absorbed, molecular bonds to break and atoms to start moving from their original positions. But all the while, the photons that are not absorbed — the ones that scatter off the individual atoms and produce the diffraction pattern — are racing through the crystal at the speed of light.

The team's simulations confirming this idea, called diffract-before-destruction, were published just in time to help the DOE to make the science case for the LCLS. But that left the question of how to implement it. Unlike at synchrotrons, where large crystals of a sample can be mounted at a precise angle and measurements taken at leisure, repeatedly, at the LCLS researchers would somehow have to take nanocrystals too small to see or touch, and position them in front of X-ray pulses that would make them explode — with the machine firing 120 pulses per second.

Researchers had never contemplated a computational challenge of this magnitude

John Spence, a physicist at Arizona State University in Tempe, took up this challenge in collaboration with Henry Chapman, a physicist now at the University of Hamburg. “Because every sample is destroyed, you have to provide new ones,” says Spence. The team's solution was a device that functioned much like an ink-jet printer: it would fire tiny droplets of water across the beam in a continuous stream with the nanocrystals in suspension.

Furthermore, says Spence, because the beam would be zapping those drops and producing new diffraction patterns so often, “a few days would give you 100 terabytes of data”. And each pulse would catch its nanocrystal in some unknown, random orientation, he says, so you would need to process every terabyte to reconstruct the original molecule. “This was a shocking thing to the crystallography community,” says Spence: such researchers had never contemplated a computational challenge of this magnitude. Only in 2008 did Spence's student Richard Kirian work out the algorithms required to do it2.

In late 2005, Chapman had led a team that demonstrated the technique using FLASH's longer-wavelength soft X-rays3. But that did not convince sceptics that it would work in a 'hard' XFEL, says Petra Fromme, a biochemist at Arizona State who was contributing her expertise in nanocrystals to the effort. “By this time, we had submitted ten different grant proposals to investigate big membrane complexes in XFELs,” she says — and had received ten rejections.

So the group, with SLAC and the DOE, had a lot of credibility at stake in December 2009, when XFEL technology, the injector and diffract-before-destruction all came together: their membrane-complex experiment would be one of the first at the newly operational LCLS. And when the computer monitors lining the walls of the tiny, underground LCLS control room suddenly started flashing twice per second with diffraction patterns, the dozens of scientists and technicians crowded inside erupted into cheers, applause and hugs. “There is extraordinary excitement that is building up around this,” Chapman wrote in an e-mail that evening.

Bigger and better

With this experiment4 and the many that have followed, says Moffat, “the gamble was absolutely validated.” Indeed, “thousands of people have been coming out of the woodwork, salivating to use this machine”. In 2013 alone, the published output of the LCLS ranged from a femtosecond-scale study of how matter is affected by an intense shock wave5 to the previously unknown structure of cathepsin B, an enzyme (and potential drug target) found in the sleeping-sickness parasite Trypanosoma brucei6. Demand for time on the machine is so high that the DOE is planning an upgrade dubbed LCLS-II, which would triple the number of simultaneously operating experimental stations by 2018.

Last November, the US National Science Foundation committed $25 million over the next five years to fund a centre for Biology with X-Ray Free Electron Lasers (BioXFEL) at the University at Buffalo in New York. With Spence as scientific director, the centre will push the technology on multiple fronts, from improving the preparation of nanocrystals to watching proteins in action as they react with other compounds.

Still, says John Tainer, a structural biologist at the Lawrence Berkeley National Laboratory in Berkeley, California, “we haven't yet shown XFEL's full potential”. For example, biologists are interested in exploring structures including protein–RNA complexes, proteins that can take on many different shapes, and highly flexible functional regions that allow one molecule to interact with others. “We haven't figured out how to use XFELs to solve those problems,” he says.

The good news is that the LCLS-II and a flurry of other new machines will give researchers plenty of opportunities. Since 2011, for example, Japan has been operating its SACLA XFEL in Harima. Utilizing a specially built compact accelerator, SACLA is six times brighter and slightly higher in energy than the SLAC machine. In 2015, a consortium of European research institutions expects to finish construction of the €1.15-billion (US$1.6-billion) European XFEL in Hamburg, which will be just as bright as SACLA, and a little more energetic still.

Fromme is particularly excited about the European machine's pulse rate. The LCLS's 120 pulses per second sound like a lot, she says. But the machine struggles to keep up with the nanocrystal injector, which spits out 10,000 drops per second. The European XFEL will produce 27,000 pulses per second. Not only will this allow researchers to avoid wasting more than 99% of the expensive, hard-to-make nanocrystals, but it will also allow the machine to accommodate many more users. “You could get millions of diffraction patterns in five or ten minutes, instead of five or ten hours,” says Fromme.

That would allow researchers to make movies of molecular motion; in a day, they could capture images of 10,000 time steps. Right now, she says, because each frame would require looking at thousands of nanocrystals to get a full structural determination, “you'd have to go all day for each time step”.

But the increase in pulse speed will work only if the system can capture and process the tsunami of data, says Fromme. The current top speed for detectors is about 3,000 diffraction patterns per second; that will have to be improved. And so will the computers, says Hajdu. “Currently, in a single experiment, one comes home with 100 terabytes of data,” he says. With the European XFEL, which will produce about 2 billion pulses per day, it will be 1,000 times that. “We'll have to develop methods to reduce data on the fly to allow us to deal with it,” he says.

Eventually, researchers hope to be able to get diffraction patterns from individual molecules, allowing them to watch biomolecules moving and interacting in a completely natural setting, surrounded by water, instead of trapped in the artificial environment of a crystal. “That's my future vision for crystallography,” says Fromme. “Get away from being a coroner imaging dead molecules, and instead get molecular movies.”

What makes this hard is that an isolated molecule does not have a host of identical twins to help it to scatter the incoming photons, as happens in a crystal. The only way to compensate is to hit it with a lot more photons to produce a stronger diffraction pattern — a flux between 1,000 and 10,000 times brighter than the current LCLS.

The European XFEL will be only about a factor of ten brighter, says Fromme. “So there are new challenges on the physics side to increase beam brightness.” Still, the LCLS upgrade is intended to get close, boosting brightness by a factor of 1,000. Fromme sees the goal in sight: “I'm optimistic that we could get there in ten years.”