Everyone always thought DARPA was cool. Last month, in a major study on US competitiveness, the National Academies suggested that the federal government build a new one — ARPA-E — to address energy research.

But what exactly is DARPA — and is it really that special? And if it has a magic all of its own, could it be replicated in a different time and place, when confronting different challenges?

The answer is complex. There's more to the venerable Pentagon research agency than meets the eye. Notwithstanding various Hollywood depictions of DARPA, it has never had any labs or opulent premises of its own. What marks it out, instead, are subtle structural touches that any successful imitator would need to recreate.

Three or four facets of DARPA set it apart. One is the loyal patronage of a leader — President Dwight D. Eisenhower when it was set up in response to Sputnik, and presidents and defence secretaries ever since. Without this support from the top, DARPA would have been extinguished by suspicious rivals in the army, navy and air force.

Second, at least in its hey-day, DARPA was not ‘mission-driven’ in the manner of, say, the National Institutes of Health. Most people probably think DARPA's role was to meet the needs of the army, navy and air force, but nothing could be further from the truth. The armed forces had their own labs and programmes to do that. DARPA spun out ideas that the forces said they didn't want, or hadn't even thought of. Defence secretaries used it as an agent for the type of sea change that the rest of the Pentagon could be relied on to resist.

Third, the agency has no bureaucracy or infrastructure to speak of. Its annual budget of $3 billion is handled by a director, a deputy director, a handful of office chiefs and a few dozen programme directors, most of them on short tenure.

It does, however, operate an effective congressional liaison office. It is true that some of its work goes ahead without the usual scrutiny because it is secret, but most of it is open and subject to the usual oversight. The committee structure that oversees the Department of Defense is relatively simple, however, and a few champions on Capitol Hill can protect DARPA from meddling. Even then, some observers see a steady grinding down of the agency's soul. They say it is getting more like the National Science Foundation, more “fair”. Everyone likes to be fair. But for DARPA, what counts is being agile.

ARPA-E is unlikely to fly in the way the National Academies suggests, unless the energy department is rebuilt from top to bottom.

That agility has brought remarkable success over half a century. DARPA concepts led directly to military innovations such as stealth materials and pilotless aircraft, helping to win the cold war. At the same time, it openly conducted pioneering public projects such as Arpanet, which grew into the Internet (apologies to CERN).

Some dissenters — who are given space in the academies' report, Rising Above the Gathering Storm — complain about government picking winners, and some even claim that the Department of Energy's sprawling network of laboratories has done just as well as DARPA, dollar for dollar.

But DARPA's track record of success fully justifies the National Academies' call for an ARPA-E. Unfortunately, the academies' report is silent on the obstacles that would need to be surmounted for such a body to work.

The inside of ARPA-E would be the easy part — smart people recruited at high wages for short periods, backing whatever horse they fancy and cajoling their grantees to push the envelope harder while collaborating intensively. It's the exterior linkages that make the project hard.

The original DARPA had the iron-clad commitment of the defence secretary, the president and Congress. But energy secretaries are marginal figures in the federal government, and presidents may or may not find the time to pay attention. Congressional oversight of the Department of Energy, meanwhile, is a basket-case of greedy and conflicting interests.

ARPA-E is unlikely to fly in the way the academies suggests, unless the energy department is rebuilt from top to bottom. But in different contexts — other nations, for example, facing other challenges — the lessons of DARPA's success are there to be learnt. Their resonance can only grow as research agencies around the world get larger, more comfortable, more audited and more risk-averse.