It is one of the most politically charged questions that any researcher can tackle: how many people have died in Iraq since the US-led invasion?

Four public-health experts provided an answer last week. Their result — a death rate that has risen from 5.5 per thousand per year to 13.3 — implies that since the invasion there have been 650,000 'excess' deaths, 2.5% of the population. Predictably, the finding has met with criticism from supporters of the war from the US president downwards.

If it holds, the study will be a key publication in the growing field of conflict epidemiology. Understanding who is dying, how and where in Iraq is vital for efforts to rebuild the country. So once the political criticisms are set aside, how does the study stack up scientifically?

Estimates of deaths in Iraq all suggest that the death rate has risen since the invasion. But they have been much lower than the latest figure. Iraq Body Count, a left-leaning website that compiles deaths from media reports, quotes a maximum of just under 49,000. US government officials have given figures of 30,000–50,000. And a household survey, conducted in May 2004 by the United Nations and published last year, concluded that the war had caused at least 18,000–29,000 deaths, mostly from violent causes. The only previous estimate of the same order of magnitude is a figure of around 100,000 excess deaths in the first 18 months of the conflict, published in 2004 by Gilbert Burnham and his colleagues at Johns Hopkins University in Baltimore (L. Roberts et al. Lancet 364, 1857–1864; 2004), the same researchers who are behind the new work.

The uncertainty of figures relating to before the war, and the present security situation, hamper efforts to estimate the Iraqi death rate. Credit: ALI YUSSEF/AFP/GETTY

The authors say their estimate is so much higher because their methodology is more comprehensive. They organized detailed interviews, carried out by Iraqi researchers from the Al Mustansiriya University in Baghdad, in more than 1,800 households in 16 of Iraq's 18 administrative regions. Using random sampling similar to that used in political opinion polls, they documented 82 deaths in the period before the invasion, and 547 during the conflict. More than 90% of these were confirmed by death certificates. They show a 95% probability that the death toll has been between 390,000 and 940,000, most of which were due to violence such as gunshots (G. Burnham et al. Lancet doi:10.1016/S0140-6736(06)69491-9; 2006).

Random counts force you to go to places that aren't convenient. The media don't wander off to distant locations.

Data from other conflicts show that such sampling is much more accurate than media reports, which usually account for no more than 20% of deaths. “Random counts force you to go to places that aren't convenient,” says Jana Asher, a researcher with the Science and Human Rights Program of the American Association for the Advancement of Science in Washington DC. “The media don't wander off to distant locations. It's a very different type of data collection.”

Death tolls from Iraqi health officials, the source of US government figures, are also suspect. The Johns Hopkins team says that the process for issuing death certificates still works well in Iraq, but the system for monitoring the number of certificates issued does not. Even before the war, note the researchers, the government's surveillance system captured only one-third of all deaths.

Yet despite the weakness of other measurements, the new figure has still surprised researchers. Perhaps the most significant concern is the baseline rate for pre-invasion death rates used in the new study. The latest survey, which included questions about the situation before the invasion, put this at 5.5 deaths per 1,000 people per year, in line with figures on Iraq from the US Census Bureau. Iran, which has a well-run health system, has a similar rate, but Iraq was at the time suffering from years of sanctions. Some sources, including the United Nations Population Division, list a pre-invasion figure of 9.7.

The discrepancy does not invalidate the new result, and if the researchers underestimated the pre-war death rate, it's possible that they may have also underestimated the post-war rate. But some researchers say the paper should have addressed the issue. “There should have been more introspection,” says Beth Osborne Daponte, a demographer at Carnegie Mellon University in Pittsburgh, Pennsylvania. “That increased my discomfort.”

Other researchers share that discomfort. Debarati Guha-Sapir is director of the Centre for Research on the Epidemiology of Disasters in Brussels. She has some methodological concerns about the paper, including the use of local people — who might have opposed the occupation — as interviewers. She also points out that the result does not fit with any she has recorded in 15 years of studying conflict zones. Even in Darfur, where armed groups have wiped out whole villages, she says that researchers have not recorded the 500 predominately violent deaths per day that the Johns Hopkins team estimates are occurring in Iraq.

But overall Guha-Sapir says the paper contains the best data yet on the mortality rate in Iraq. And none of the experts contacted by Nature said that their doubts fatally undermined the study. Some, such as Daponte, would have liked the authors to have better assessed their method's shortcomings before releasing a result with such political impact. But most say the result is a welcome addition to conflict epidemiology, which is now seen as playing a central role in assessing the severity of wars, and in helping states recover from them (see 'Counting the cost of war').

Burnham says he would now like to study patterns of migration in Iraq and the state of the health system. He would also like to estimate deaths based on a sample 4–5 times bigger than that used so far. But survey teams are in danger on the streets of Iraqi towns, and Burnham doubts whether the need for more detailed data justifies the risk.