For the past 50 years the small organization now named the Defense Advanced Research Projects Agency (DARPA) has sought to influence the behemoth that is the US Department of Defense. The services generally walk the evolutionary road; DARPA aims for revolution. It has often succeeded — the $150 billion, in today's dollars, that the United States has spent on the agency's programmes has brought us the Internet, stealth aircraft, precision warfare, seismic systems, lasers and much more.

Being the agency's director was the most challenging and exciting job of my career. I did my best work, with the most important consequences. It was the only post that demanded all my energy, ability and determination, and made it worthwhile to give them. A shining star over my life, the agency also cast deep shadows. With power, responsibility and action came criticism and opposition — some helpful and well-meaning, some not. Lasting friendships and enmities grew. Looking back, I am satisfied with what we achieved, and with the credit that we got — although that was often slow in coming.

Now, plans are afoot to emulate DARPA's approach in fields such as energy, homeland security and intelligence. The time, then, is ripe to reflect on its ups and downs since February 1958.

The author (right) in 1963 with Robert Wienecke (centre) and Lt Col Reuter before an inspection trip in South Vietnam. Credit: US ARMY, 1963

Big problems

In October 1957, Sputnik surprised the United States. Its launch made President Dwight Eisenhower anxious about US military capability and high-tech research, and eager to reorganize the lumbering and diffuse US missile and space effort. Early in 1958, working with the President's Science Advisory Committee (PSAC) and his science adviser Jim Killian, Eisenhower created the Advanced Research Projects Agency (ARPA; the 'D' was added in 1972). For maximum flexibility and effectiveness, the president gave the agency a small number of big jobs, independent leadership and new funds. It was run from the Office of the Secretary of Defense, separate from the military services, under the directorship of the industrialist Roy Johnson.

ARPA's initial 100 or so staff had three tasks. First the agency was to restructure the missile and space programme, accelerating its most significant parts. Second, ARPA was to drive a new initiative in ballistic-missile defence. Third, it was to launch a programme to detect nuclear explosions in the atmosphere, the ocean, in space and underground.

This was just the beginning. Over the next 5 years, ARPA's other responsibilities and achievements would include reforming research and teaching in materials sciences and engineering — physics, chemistry, metallurgy, ceramics and others — by pursuing important integrations and restructuring. The agency led the development of advanced computers and communications for the defence department's command and control systems. Finally, and less successfully, the agency was tasked with helping the South Vietnamese defence against the North and the Viet Cong guerillas.

All these programmes attacked large problems that mixed technical, managerial and conceptual challenges. They sometimes required serious work outside physics and engineering, as in the implications for strategy and arms control of anti-ballistic missile systems and nuclear-test detection. They also necessitated an understanding of unconventional warfare, such as guerilla action and the strategic, linguistic, cultural and historical aspects of counter-insurgency. The army and navy were generally averse to such projects and, more comfortable with well-established techniques, concentrated on marginal improvements.

ARPA pursued a whole-lifecycle approach to invention, originating most of the technical ideas behind its programmes, using the best scientific advice to craft their structure, and working with government organizations and outside contractors to execute and apply the results. This holistic management style is the defining feature of the 'ARPA way', and it continues today.

Big solutions

Before its first year was out, ARPA had reorganized the US missile and space programme. Civilian and scientific space programmes went to a new agency, NASA, that absorbed the old National Advisory Committee for Aeronautics. Control of military missile programmes went to the services that used the weapons, and intelligence programmes went to intelligence agencies. A pattern of 'action before paper' was set by the launch of the US answer to Sputnik, Explorer 1, in January 1958 — a month before ARPA's official start.

The chief architect of ARPA's enduring atmosphere and approach was its third director, Jack Ruina, who ran the agency from 1961 to 1963. Ruina, a brilliant young professor of electrical engineering from the Massachusetts Institute of Technology, was creative, relaxed and witty, yet serious and stimulating. He managed through discussion and suggestion, not memos; he devoted his and our attention to the top issues and the hardest problems; he got the best technical advice; he stressed the importance of leadership within and by ARPA; and he mandated his staff to act quickly.

Specifically, Ruina restructured ARPA's advanced anti-ballistic missiles programme, which he hired me to take over late in 1961. This later provided most of the technology for the Safeguard System, the Strategic Defense Initiative, and the National Ballistic Missile Defense System currently being deployed.

And in 1962, Ruina hired J. C. R. 'Lick' Licklider, the inventor and inspiration of modern universal computing. Arpanet, the precursor to the Internet, was developed during Licklider's two stints as director of the Information Processing Technologies Office, switching on in 1969. Also during the late 1960s, a programme on supercomputers began, starting with the first highly parallel machine, the ILLIAC IV. My directorship from 1965 to 1967 was a time of consolidation and bridge-building, with Vietnam top of the agenda. I tried to maintain the heady entrepreneurial environment that Ruina created, while pushing for more collaboration with policy-makers, to keep ARPA's funding and its influence healthy.

The effort to support South Vietnam, Project Agile, did not save the country. But in 1963, ARPA gave South Vietnamese troops the lighter and more practical AR-15 rifle. This became the M-16, now the standard US military firearm. A few years later precision warfare began, when the first laser-guided bombs were developed and used in combat. Project Agile also founded the study of counter-insurgency strategy, tactics and philosophy. This helped in northeast Thailand in the 1970s and, unhappily, is relevant once more.

Also in 1963, ARPA launched what came to be called the Interdisciplinary Centers for Materials Science — 12 universities as part of the ARPA programme and 2 universities as part of the Atomic Energy Commission's programme. These interdisciplinary centres, now run by the National Science Foundation, invented the modern approach to materials science, producing much first-rate work. Their many thousands of PhDs and postdocs drove the electronics revolution.

In the 1970s and 1980s I was a member of the Defense Science Board, which advises the defence secretary on top-level matters of military technology; I chaired committees on, among other things, cruise missiles and ocean surveillance, and helped review some DARPA programmes. The 1990s, especially the first Iraq war, was a proving ground for much of the agency's work of previous decades — such as stealth aircraft and precision warfare. Ironically, the same era saw research at the defence department become much narrower in its purview.

Throughout this time, guidance from senior national security figures — especially Johnny Foster (director of defence research and engineering, 1965–72), and several defence secretaries, particularly Bill Perry (1994–97) — helped to provide programmatic continuity.

The art of ARPA

An exemplar of ARPA in action was the programme to detect nuclear tests, one of the agency's great successes of the 1960s and '70s. The schedule was tight, the range of issues enormous, the collaborations numerous, the risks daunting and the pay-offs huge. Thrilling stuff.

“Jack Ruina created a heady entrepreneurial environment at ARPA” Credit: DARPA

In 1958, the president asked ARPA to explore whether nuclear explosions could be detected so reliably that the United States might make arms-control agreements, assured that cheats would be caught. Crucially this assignment had the defence secretary's blessing, silencing the inevitable chorus of dissent. ARPA defined the programme's purpose: to build a system that could reliably distinguish natural events, such as earthquakes, solar flares and storms, from nuclear explosions let off underground, in the atmosphere, in space and underwater. These initial steps were primarily about policy, not science, engineering or implementation.

ARPA developed technical concepts for implementing the directive, working with PSAC, the Atomic Energy Commission and others. The work on underground detection alone is an example of the programme's scale. In 1958–59 ARPA began studying underground explosions and earthquakes using 200–300 seismic stations worldwide. By 1963 this showed promise, but more detail was needed. ARPA decided to build an array of connected, sensitive seismometers, and to analyse the data using the beamforming algorithms it had just developed for radar.

Next, ARPA had to come up with one system that cracked all the following problems: improve sensors, learn how best to bury and connect them, collect and clean the signals of noise, develop new computers and computing techniques for beamforming, design a user interface for the output, and explain the results to non-specialists.

The air force took the lead in building and installing the resulting system, with ARPA's help and occasional supervision. To calibrate it, ARPA coordinated two tailored nuclear explosions. Finally the agency oversaw the data analysis and contributed to the policy conclusions. After years of debate on whether the system could distinguish natural and nuclear events, what to improve and how to pay for it, it became apparent that the system gave reliable information on almost all foreign nuclear tests. And the seismic arrays transformed our understanding of Earth's interior.

To be at the heart of this integrative invention, and enjoy it, was truly 'doing ARPA'. An artform.

Lessons learned

In the five decades since ARPA's founding, and the four since my stint at the top, there have, of course, been budget cuts, programme halts, and losses of drive. Relations with the rest of the defence department and services have fluctuated, occasionally approaching crises. There is more oversight and more bureaucracy, and the press have become less friendly.

But one fundamental change to DARPA is more important than all these vicissitudes. In 1958, the body was designed to be an agent for change in the Department of Defense, located in the Office of the Secretary of Defense. In the 1960s, it became stronger and more effective in this role. Sometime in the 1970s or '80s, the agency shrank to being a agent for change in the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, which focuses on building and buying weapons. This marginalization went with a downgrading of the remit of the Director of Defense and Engineering, and the shrinking influence of science on US policy generally.

Yet DARPA is still the most vibrant, impactful organization in defence science and technology in the United States and probably the world. Current director Anthony Tether still leads the way in space technology, the applications of quantum physics and precision warfare. He has spearheaded several new fields, notably the application of neuroscience and electronics to war-zone prosthetics, which should have enormous civilian consequences. Tether has also returned to ARPA's roots with projects on supercomputing, and autonomous ground-vehicle navigation, again with many potential applications.

There are lessons from all this for today's DARPA and those trying to replicate it. Some previous imitators have failed because their goals were too small or leash too short; some had weak leadership or no new money. If you want to build a real agent for change, build yourself a real DARPA. Give it very hard, broad problems to work on. Hire the brightest leaders, problem-solvers and risk-takers, and give them significant resources, guidance, encouragement and freedom. Establish an open, demanding and easy style, with adequate turnover. Hire staff who will build, find more good people, and move on. Then let them work, let them succeed or fail, and give lots of praise. Finally, defend them against friendly fire, for there will be lots.