The day after concerns about Avandia hit the news, the emergency room at New York City's Bellevue Hospital was even more hectic than usual. Patients with diabetes filed in and asked to discontinue their medication while the phone rang with calls from other Avandia users. Harried doctors rushed to handle the extra flow.

Cardiologist James Underberg found the scene disturbing. Although he insists that he is a clinician, not a statistician, Underberg could see that the analysis that suggested Avandia might be causing heart attacks was plagued by limitations — the authors of the paper said so themselves. But he could now be sued the next time he writes a prescription for Avandia, he says, “all on the basis of one article that may or may not have significant flaws in the analysis”.

The article, published last week (S. E. Nissen and K. Wolski N. Engl. J. Med. doi:10.1056/NEJMoa072761; 2007), is a meta-analysis of 42 clinical trials of rosiglitazone, a drug used to lower blood sugar in patients with diabetes and marketed under the name Avandia by London-based pharmaceutical giant GlaxoSmithKline (GSK). The authors, Steven Nissen and Kathy Wolski of the Cleveland Clinic in Ohio, found that the chance of having a heart attack was 43% higher in patients who were taking the drug than in those who weren't, although precisely why is unclear (see 'How Avandia works').

An analysis of Avandia trials has highlighted potential side effects. Credit: J. SULLIVAN/GETTY

These data do not differ strikingly from others that have come before; in Europe, Avandia comes with a warning label that the drug increases the risk of heart attack by 31%, on the basis of a previous meta-analysis, by GSK. But the new paper has caught a lot of attention. The Lancet, which had earlier published the promising results of one of GSK's clinical trials of Avandia, responded with an editorial warning against alarmist responses to Nissen and Wolski's paper and its tone of “urgency” (Lancet doi:10.1016/S0140-6736(07)60787-9; 2007). But by then, GSK's stock price had plummeted and congressional hearings had been scheduled to review the handling of the drug by the US Food and Drug Administration (FDA).

Meta-analyses, which crunch together the results of many clinical trials, are powerful statistical tools to uncover the side effects of drugs. They were key in bringing to light the increased chance of heart attack associated with the painkiller Vioxx (also highlighted by Nissen, see 'Man on a mission'), and the added risk of suicidal thinking and behaviour in children taking some antidepressants. Few doubt that meta-analyses have their uses, but researchers caution that they should be read in the right light and, if possible, made more robust.

The studies crunched together in this case were not designed to look for cardiovascular side effects, but rather to assess the drug's main activity. Reports of heart incidents may not have followed the same rigorous criteria in all studies, nor were they reviewed by an outside panel. And Nissen and Wolski had access only to the trials' summaries, and not to patient-specific data (GSK has posted such summaries online since an unrelated court case in 2004 prompted it to do so), so any confounding effects could not be taken into account. If the cardiovascular risks are only for a specific subset of patients, this meta-analysis won't be able to tell.

To make such studies more robust, more data are needed. John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece who has examined numerous meta-analyses notes that journal articles reporting clinical trials tend to dedicate more space to listing the authors' names than to listing possible side effects associated with the drug. And studies that produce negative results are often not published at all, giving researchers an incomplete picture of how often trials fail. Ioannidis argues that use of complete data sets would strengthen the analysis and could weed out potential false alarms. “I think that companies would actually gain from such transparencies,” he says.

If I saw a meta-analysis done by GSK on Avandia, I don't think I would believe it.

He adds that meta-analyses should be done by independent groups. At the moment, they are frequently conducted as part of the drug approval process. But in many such cases, the analysis is being done by the pharmaceutical companies that manufacture the drug in question — something that most agree is not a good idea. “If I saw a meta-analysis done by GSK on Avandia, I don't think I would believe it,” says Ioannidis. “Someone who is independent should do this.”

But GSK doesn't think that crunching studies together is in itself the best way to find side-effects. “We don't think meta-analysis is the best way to address this,” says GSK spokeswoman Alice Hunt. Clinical trials, she argues, are the only way to achieve a definitive answer. GSK is in the middle of a clinical trial of more than 4,000 patients that is aimed specifically at assessing the cardiovascular effects of Avandia (compared with more than 10,000 patients in Nissen's number crunch). The Lancet editorial also recommends waiting for these results — although the news of the new findings has reportedly led to volunteers dropping out of the trial.

Preliminary results of the trial, scheduled to finish in 2008, have been shared with the FDA, says Hunt. “They could potentially be released, but that's not being looked at, at the moment,” she adds.

The FDA will not as yet comment on the ongoing trial, to “preserve the study integrity”, but it does have additional data on the drug. Without seeing the results for themselves, some doctors find it hard to know how to judge FDA assurances that the additional data it has do not point to a significant risk of heart attack. “They are unpublished, uncited data that are not available to the public. I've heard the same story before: 'We know there are weapons of mass destruction in Iraq. We can't tell you how we know, but we know',” says Nissen.

For some, even a preliminary analysis is reason enough to avoid the drug, if only because there is an alternative medication — pioglitazone, marketed under the name Actos in the United States — that has not yet been associated with cardiovascular risks. Nissen's analysis was enough to convince Harlan Krumholz, a cardiologist at Yale University: “When you're talking about safety and you're talking about a drug for which there are alternatives, you have to ask yourself, who has the burden of proof here?”

Although Avandia's fate remains unclear, Nissen's study has served to raise the profile of meta-analyses. Roger Chou, a clinician at the Oregon Evidence-based Practice Center in Portland notes that at the moment, independent analyses such as this are done on an ad-hoc basis. “Right now, it's really just when somebody's interested in it, or when there's something that's making people concerned,” he says. But there is a growing interest in independent groups monitoring drug safety in this way — and, importantly, signs of increased funding from organizations such as the US Agency for Healthcare Research and Quality to support meta-analyses.