Building water-delivery systems is one measure used by the Millennium Villages Project in Africa. Credit: Guillaume Bonn/Corbis

The Millennium Villages Project (MVP) stands out among development efforts in Africa. Since its launch in 2004, it has attracted generous donations and high-wattage supporters — including Hollywood actor Angelina Jolie and United Nations secretary-general Ban Ki-moon — for its work on alleviating poverty in rural Africa. The programme has delivered aid to at least 500,000 people in 10 countries, and has been emulated in others.

But its effectiveness has never been thoroughly tested. With the publication of a research plan in The Lancet last month (see, the MVP has now embarked on its first comprehensive evaluation in the hope of addressing long-standing questions about its impact.

Led jointly by the Earth Institute at Columbia University and the non-profit organization Millennium Promise, both in New York, the MVP aims to lift clusters of villages out of poverty through interventions ranging from building health centres, roads and schools to improving agriculture and sanitation. It began in Kenya in 2004, and as of 2013 had an annual budget of about US$25 million.

“We will, I believe, be able to gain very useful insights about costs, processes, technologies, information systems, local and national governance, among other issues,” says Jeffrey Sachs, who runs the Earth Institute and who conceived the programme.

As economists increasingly advocate randomized controlled trials of international aid programmes (see page 150), Sachs has faced criticism for not setting up the MVP as a rigorous experiment. MVP researchers are now trying retroactively to compare villages that received the full intervention with similar ones that did not, but the research protocol readily acknowledges challenges in collecting data and producing statistically significant results.

“I expect that the authors will conclude that, although we cannot prove that MVP works, we also cannot rule out that it works,” says Annette Brown, who heads the Washington DC office of the International Initiative for Impact Evaluation, a non-profit organization that funds and analyses such evaluations.

In March, preliminary results from a study commissioned by the UK Department for International Development (DFID) in 2011 found little benefit from an £11.5-million ($18-million) expansion of the Millennium Villages project in northern Ghana. Sachs asserts that effects are hard to see at the Ghana site because it is in its early stages; his critics see the analysis as further evidence that the Millennium Villages approach may not work as advertised. “The trumpeting of the project as a model is just indescribably disproportionate to the deafening silence about its actual results,” says Michael Clemens, a senior fellow at the Center for Global Development, a non-profit think tank in Washington DC.

Clemens has long been an outspoken critic of the MVP and was among researchers who challenged1 a 2012 study in The Lancet2 that reported that child mortality had dropped in Millennium Villages three times faster than elsewhere in the host nations. The challenge ultimately led to a retraction of that claim by the paper’s lead author. Clemens argues that aid money should be spent either on projects that generate useful knowledge or on things that have been shown to work, noting that malaria bednets, which have a demonstrated benefit and are part of the MVP’s suite of interventions, cost $15–20 per household.

The MVP typically budgets $120 per capita annually, according to its website, although Sachs says that outside contributions can reduce MVP’s investment to half that. At the Ghana site funded by DFID, the total investment by all parties was projected to be $27.1 million over 5 years for 30,000 people. That is $181 per person annually, or about $4,500 per household over the course of the project — less than the $5,408 per household calculated by a randomized controlled trial in Ghana testing a two-year package of interventions that included food, cash, health services and training3.

The MVP hopes to release its analysis by the end of 2016, and Sachs says that his team will be in a better position to talk about cost-effectiveness and other considerations once the analysis is out.

Dean Karlan, an economist at Yale University in New Haven, Connecticut, says that it is probably too late for the project itself to advance the science of global development in a significant way, but he credits Sachs with raising awareness about global poverty issues. “I do see it as a missed opportunity,” Karlan says, “but in the grand scheme of things there are tons of missed opportunities.”