When a study produces unexpected findings, there can be a cloud of doubt cast over the results. In this sense, the data from the recent AIDS vaccine trial in Thailand created a perfect storm.

The trial aimed to test a combination of two products, each containing the HIV gp120 protein—the ALVAC canarypox-HIV vaccine and AIDSVAX. As early as 2004, some scientists had expressed concerns about the trial because the AIDSVAX and ALVAC formulations did not seem to sufficiently stimulate the immune system when used separately (Science 303, 316; 2004). And, in 2007, disappointing results from the Merck HIV vaccine trial, called STEP, did nothing to allay skepticism about a vaccine approach to thwarting the virus.

On 24 September, the researchers behind the Thai trial announced key findings ahead of a full release of the data one month later at an AIDS vaccine meeting in Paris and in The New England Journal of Medicine (N. Engl. J. Med., doi:10.1056/nejmoa0908492; 2009). Much to the surprise of many AIDS researchers, the findings indicated that the vaccine offered some protection against HIV infection.

How much protection, though, became a sticking point. Critics, quoted anonymously in the news media, said that the method used to establish efficacy—the so-called 'intention-to-treat' analysis—was not the most relevant analysis. This analysis took into account all 16,395 individuals from the general population who enrolled in the study and were HIV negative at the start of the trial. Of those who received the vaccine, 51 became infected with the virus over the course of the three-year trial, compared with 74 of those in the similarly-sized placebo group—yielding a 31.2 % efficacy rate.

What the study researchers did not release upfront was the 'per-protocol' analysis. This considers vaccine efficacy on the basis of the subset of subjects who closely adhered to the vaccine regimen. The involved nature of the study protocol, which consisted of four precisely timed vaccination visits over a six-month period, meant that only 12,542 participants fell into this category—an unusually large drop. As it turned out, the per-protocol analysis suggested a slightly lower efficacy rate, 26.2%. What's more, the calculations suggested that this perceived efficacy had a higher likelihood of being result of statistical chance than the intention-to-treat analysis (based on P values of 0.16 and 0.04, respectively). So when the per-protocol details emerged, a few scientists voiced doubt over the scale of the vaccine trial's success.

Nelson Michael, a researcher with the US Army who helped lead the Thai trial, defends the decision to emphasize the intention-to-treat analysis. “This was considered the most relevant [analysis],” he says. According to Michael, the per-protocol calculation becomes important at a later stage, when one is aiming to license a specific vaccine regimen for marketing purposes.

Others emphasize that the trial functions to show that creating some immunity to HIV is possible and worth exploring further. “It's important for people to understand that the point of the study, and the manner in which it was powered, was to prove a concept,” says Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases at the US National Institutes of Health, which helped to fund the study.