Scientists and policy-makers often insist that 'more research' is needed to solve a wide range of social challenges. But what is the most effective way to deploy our formidable knowledge-creating enterprise in the shadow of looming threats to human welfare, such as AIDS or climate change?

Last September, scientists and the media trumpeted a 'breakthrough' in AIDS vaccines after a 6-year clinical trial in Thailand seemed to show success in preventing HIV infection. Yet the results were almost immediately contested (see Nature 461, 1187; 2009), and the trial's importance was soon being spun as a scientific advance rather than a practical step toward a successful vaccine. This latest frustration in the 25-year quest for an AIDS vaccine highlights a difficult but essential lesson for science policy: rapid and short-term progress in addressing urgent problems is much more likely to come from improvements on existing capabilities than from radical breakthroughs that emerge from basic science.

Major efforts to apply science and technology to AIDS have followed two main paths — the search for drugs to treat those already infected, and the pursuit of a vaccine to protect against infection in the first place. The differences in outcome of these two paths could not be more stark. People with AIDS who have access to combination drug therapies are experiencing radically improved health outcomes. Vaccines have failed to demonstrate any benefit.

The reason the drug path has succeeded is startlingly simple: when the AIDS crisis began in the early 1980s, chemicals already existed that could slow the disease, and the state of the science was sufficient to identify them.

Better to be lucky

The compound azidothymidine (AZT) was first described in 1964; it was supposed to fight cancer, but didn't. A decade later, it was found to inhibit the growth of retroviruses in mice. By the early 1980s, the drug firm Burroughs Wellcome (now GlaxoSmithKline) was exploring the drug's potential as an antibiotic. When scientists determined in 1984 that AIDS was caused by the HIV retrovirus, the US National Institutes of Health (NIH) in Bethesda, Maryland, began working with drug companies to identify existing chemicals that could act against the infection. AZT was tested against live HIV by the NIH National Cancer Institute, where researchers found that it suppressed the virus. Clinical trials quickly followed, and the drug was approved by the US Food and Drug Administration in March 1987.

Innovation is a social process, and the discovery of a drug that could act against a devastating and mysterious disease galvanized the diverse social actors necessary for rapid progress. Government science administrators recognized the need for a coordinated effort; scientists saw promising opportunities to confront an emerging epidemic; drug companies smelled big profits; and people with AIDS, willing to try almost anything that might help, pushed the regulatory system to bring new drugs to patients faster than ever before.

In the next 25 years, 25 drugs were approved for use in treating the disease (E. De Clercq Rev. Med. Virol. 19, 287–299; 2009). Science didn't need to be driven by theory alone; rather, it could focus on the empirical problem of making things that worked better than AZT. By the mid-1990s, there had been a sudden and radical decrease in deaths from AIDS, at least in the United States and Europe. This inflection point was a true breakthrough, and it emerged not from new basic understanding but from the empirically discovered synergies achieved by combining different types of anti-HIV drugs into highly active antiretroviral therapies. In 1994, the 2-year survival rate for people diagnosed with AIDS was less than 50% in the United States; 3 years later it was nearly 75%. It continues to improve to this day.

Chances of success in basic research should be understood as both unpredictable and long-term. ,

The vaccine story, compellingly told by journalist Jon Cohen in his 2001 book Shots in the Dark (W. W. Norton & Company), couldn't be more different. In the absence of an even modestly effective way to stimulate immunity against HIV, vaccine research has lacked the focus and baseline for demonstrating the progress that is necessary to drive rapid innovation. The same groups whose disparate interests converged around the quest for antiretrovirals contribute, instead, to disorder. Research has been carried out mostly by academics whose main motive is to advance knowledge and careers. Private-sector involvement has generally been limited to biotech firms looking for a game-changing new product; most of the big drug companies have stayed away, seeing little opportunity for profit and lots of risk of liability. In the face of technical uncertainties, governments have faced excruciating dilemmas about how best to allocate scarce resources. In the absence of evidence of efficacy, patient groups have been ambivalent even about participating in trials. These are not reasons to stop trying to find a good vaccine, but they are reasons not to expect significant progress any time soon.

Downstream policy

If a problem demands short-term technical progress, then innovation policies need to focus on accelerating the improvement of approaches that already show clear evidence of efficacy. Research is an important aspect of this process, but mostly insofar as it supports the knowledge needs of private-sector innovators seeking to develop new and better products. Much more important are the downstream aspects of innovation policy, especially those that encourage private-sector participation, and the cultivation of markets.

The lesson applies in domains as disparate as human health and energy supply. For example, a key role for the NIH in the development of antiretrovirals was running drug trials to demonstrate safety and efficacy. An analogous role for governments in the energy domain is running demonstration projects for unproven technologies such as carbon capture and storage, or modular nuclear reactors, to create confidence in technical performance and cost estimates.

Basic research — on AIDS vaccines, on new energy technologies — should be aggressively pursued as part of a portfolio of public investment in innovation, but the chances of success should be understood as both unpredictable and long-term. The AIDS story provides a clear message for policy-makers seeking to address urgent problems with new technologies: look for what already works, and make it work better.