Poverty project opens to scrutiny

Panel set to reshape evaluation of Millennium Villages research after partial retraction of health claims.

Health interventions may have reduced child death in African villages, but it isn’t clear by how much. Credit: WpN/Photoshot

The Millennium Villages Project (MVP) aims to chart a course out of poverty for the most deprived people in Africa. But after being forced to partially retract the findings of a paper that included the claim that its interventions had slashed child mortality, the project is revamping its oversight to make its science more robust and determine whether its approach works.

The MVP, a joint venture between Columbia University and the non-profit organization Millennium Promise, both in New York, is simultaneously applying a raft of aid interventions — from bed nets for malaria control to fertilizers — to help impoverished villages to meet the United Nations’ eight Millennium Development Goals (MDGs), including reducing hunger, extreme poverty and the burden of major diseases, and increasing levels of primary education. The first Millennium Villages were launched in Ethiopia and Kenya in 2004 and 2005, and the project now operates in 10 African countries, reaching about 500,000 people in 80 villages over 14 sites.

In a paper published online in The Lancet last month¹, the project claimed a significant milestone. It reported that after three years of interventions, child mortality was decreasing three times faster in the project’s villages than in the host nations in general. But the analysis was criticized for underestimating nationwide improvements in child mortality, and over­estimating those in the Millennium Villages². The paper’s lead author, Paul Pronyk, later accepted that this claim had put the Millennium Village mortality reduction “in an excessively positive light”³, and retracted it. Pronyk then resigned as director of Monitoring and Evaluation at Columbia University’s Center for Global Health and Economic Development.

The MVP’s founder, Jeffrey Sachs, head of the Earth Institute at Columbia and a co-author of the partially retracted paper, says that the MVP research teams were too autonomous, and he regrets not having brought in external advisers earlier. “I don’t want such mistakes to occur again,” he says. Sachs has now created a faculty committee to oversee MVP research and increase interactions with outside researchers; he will co-chair the committee, along with Cheryl Palm, a scientist at the Earth Institute’s tropical agriculture programme.

An external advisory board called the Inter­national Scientific Expert Advisory Group will oversee how the MVP assesses the impact of its work. The panel is still being assembled, but will be chaired by Robert Black, head of the department of international health at Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. Black says that Sachs has given him a remit “to open the study to more scrutiny, to be more transparent, to be very explicit about what are the expected outcomes of the study, and to be sure that evaluation is as well done, as rigorously, as possible”. Black also wants to respond to criticisms by giving external researchers access to as much of the MVP data as he can, as well as to detailed information on the project’s costs, so that outsiders can have greater confidence in the project’s results and cost–benefit analyses.

It is too soon to say what sort of recommendations the external panel will make on the MVP’s future research in the study villages. But given that the project is already almost seven years into its planned ten-year pilot phase, the scope for change is limited, says Black. “It is impossible to completely redesign the study, though many of us would have designed it differently,” he says.

Michael Clemens, a migration and develop­ment researcher at the Center for Global Development, an independent research institution in Washington DC, is unimpressed with the creation of the advisory panel, and argues that the MVP needs rigorous and transparent evaluation from completely independent groups. Clemens, who co-authored a letter⁴ to The Lancet that led to the partial retraction of the MVP’s findings, believes that the paper still has problems — in particular, a claim that the child mortality rate in Millennium Villages fell by one-third more than that in matched comparison villages over the course of the study. “The whole study should have been retracted,” he says, adding that he has submitted a new letter to The Lancet to this effect.

Black says that whether the MVP ultimately proves a success or failure, it will still provide useful information, so it is important to improve its evaluation mechanisms. “I don’t think it should be dismissed because it has flaws,” he says.

The UK government’s Department for International Development last year launched its own £3.8-million (US$5.9-million) independent evaluation to accompany an £11.5-million ten-year grant to create Millennium Villages covering 30,000 people in Ghana. “A massive effort is needed to raise millions of people in Africa out of poverty. Millennium Villages represent one approach, and they could make a major contribution,” the department notes. “Such an opportunity should not be missed merely for want of evidence. But nor should scaled-up resources be committed to a model without assessing whether it is indeed a cost-effective approach.”