Once upon a time, the fastest supercomputers were top-secret. In the days of the cold war, the best US machines were reserved primarily for the use of spy agencies and designers of nuclear weapons.

IBM's Blue Gene/L uses 130,000 processors. Credit: IBM

But now, the Pentagon's crack research agency is funding the development of machines that, it hopes, will be shared by industry and university scientists, as well as by spies and weapons designers. That way, it figures, more people will write code to harness the computers' massive power — and the cost and effort of developing applications will tumble.

Late last month, the Defense Advanced Research Projects Agency (DARPA) said it would grant almost $500 million to Cray and IBM to develop machines that will be about ten times faster than the most powerful existing supercomputers — and easier to program.

The award is the third and final phase of DARPA's High Productivity Computing Systems programme, which began in 2002 by supporting research teams at IBM, Cray, Sun Microsystems, Silicon Graphics and Hewlett-Packard. DARPA is managing the programme in partnership with other government agencies, including the Department of Energy, which runs the US nuclear-weapons programme, and the National Security Agency, which spies on radio signals, phone calls and e-mails.

Need for speed

The selection of IBM and Cray was based not only on project design, but also on the commercial viability of the proposed technologies. DARPA has been trying to whet the commercial sector's appetite for supercomputers, and has helped fund a programme within the nonprofit Council on Competitiveness to tout the usefulness of the machines.

The two companies will now build prototype machines, complete with operating systems and software tools, by 2010. The first customers are likely to be governmental agencies. But smaller versions of the machines will also be available to researchers.

Cray has a distinguished history in the field. The world's first genuine supercomputer — the Cray-1 — was released in 1976. It cost $8.8 million and contained a single central processor, performing up to 80 million floating point operations per second (flops). Today's fastest computer is IBM's Blue Gene/L at the Lawrence Livermore National Laboratory in California, which can perform up to 280 trillion flops and contains more than 130,000 processors.

Processor power

Using vast numbers of processors makes supercomputers tough to program.

But using vast numbers of processors makes supercomputers tough to program. The more processors you have, the more time they waste just communicating with each other. And although processing speed itself has multiplied over the years, the speed at which they can access computer memory hasn't kept up.

Programmers sometimes try to work around these speed bumps, says Jeffrey Gardner, an astrophysicist and computing specialist at the Pittsburgh Supercomputing Center in Pennsylvania. But he thinks that as more and more processors are added, programming complications will reach the point where scientists simply can't make effective use of the machines. To tackle this problem, both Cray and IBM intend to improve their programming languages so that the computers work more efficiently during lags in communication.

Rick Stevens, a supercomputer specialist at the Argonne National Laboratory in Illinois, says IBM will focus on optimizing its POWER processors. IBM makes its own chips, he says, giving it more control over their properties.

Cray, on the other hand, buys most of its processors from Advanced Micro Devices. According to Jan Silverman, Cray's vice-president for corporate strategy, it will continue to use these standard processors, which handle one calculation at a time, but will combine them with Cray-made versions. These will include vector processors, which can perform calculations on multiple pieces of data at once, and multithreaded processors, suitable for database mining. Cray also plans to optimize performance by using two distinct operating systems — one to perform basic chores such as accessing memory and regulating input and output, and the other to handle data.

Peter Ungaro, Cray's chief executive, hailed the DARPA award as a sign that Cray will return to market leadership. The company has struggled since the early 1990s — Silicon Graphics bought the firm for $767 million in 1996, only to sell it off again in 2000 for just $50 million. The DARPA award sent stock prices up by $2 to almost $12 — still sharply down from a September 2003 high of more than $50. Nevertheless, Stevens is optimistic about Cray's outlook, saying that several recent contracts bode well.

But Cray still derives most of its research and development money from government grants, as the market for supercomputers isn't large enough to support their high development costs. “The military has always been at the forefront of supercomputing,” says Robert Deupree, a physicist at St Mary's University in Nova Scotia, Canada, “in part because they needed it, but also because they had the money to be able to invest in it.”