There is an intuitive appeal to the Millennium Villages international development project — the brainchild of economist Jeffrey Sachs from Columbia University in New York that aims to help lift villages in 14 sites across Africa from poverty. The initiative takes a broad approach and aims to tackle the root causes of poverty and ill health together, unlike most aid projects, which focus on just one area.

Improvements on the ground seem impressive. In Mwandama, Malawi, crop yields shot up after the project gave farmers free fertilizer and improved varieties of maize seeds. Any food not used immediately is stored in a purpose-built warehouse and sold later in the year, earning the villagers an income. With the project's support, many villagers are now able to grow mango trees and sweet potatoes to improve food security, diversify their diet and earn extra money.

Freed from the daily struggle to fill their bellies and armed with better access to health care and education, the villagers are now setting up cooperative business ventures and investments so that they can support themselves without the project's assistance.

The villagers of Mwandama, and probably the inhabitants of the other selected villages, are clearly better off than they were six years ago, when the project started. And given the economic crisis, the project has done well to attract the funds that it has.

But prominent international development researchers and experts have taken issue with some of the project's claims of progress (see page 158) — most recently for declines in child mortality (P. M. Pronyk et al. Lancet http://dx.doi.org/10.1016/s0140-6736(12)60207-4; 2012). Their main concerns, they say, are weaknesses in the project's design and data analysis, as well as a lack of transparency over the raw data and project costs.

Michael Clemens, a development researcher at the Center for Global Development in Washington DC, has done some of the most detailed independent analyses of the project. He questions some of the key findings in the latest paper, in part because of inadequate baseline data for child mortality in the control villages.

The project started to monitor control villages only after three years had passed, when independent development researchers argued that it was necessary. So, in the absence of actual data for that period, the study asked female villagers to recall child deaths over the previous three years.

The project's approach has potential, but little can be said for sure about its true impact.

The paper does acknowledge that this method is unreliable and can underestimate child deaths. So how reliable are comparisons made against such data? Clemens says that alarm bells should have started ringing when the verbal reports suggested that the child death rate in the control villages rose over the study period, contradicting continent-wide trends of falling child mortality. This would have resulted in an overestimation of the difference in child mortality between the two sites, Clemens argues.

Furthermore, he says that the study's statistical analysis fails to show that the annual rate of decline in the project villages is triple that of national rural trends, as claimed. And, he says, the data set used for national child mortality was a misleading comparison because it includes the period 2000–06, before the villages project started, when the national child death rate was falling more slowly than in 2006–10. This, he says, made national rural trends seem lower than they actually are, thereby inflating the improvements in the project villages.

Nature put these criticisms to the organizers of the Millennium Villages Project and received lengthy written and verbal responses. Not all of these helped to clarify the situation. When asked, for example, why the claimed improvements in child mortality were placed in such a prominent position in the paper despite being statistically insignificant, the organizers replied that they were there for “illustrative” purposes.

The Millennium Villages Project has problems beyond the analysis of data. The organizers have been reluctant to publish a full breakdown of costs — making it impossible for those not privy to the information to verify their cost–benefit analysis, which is crucial in development policy because spending is under great scrutiny. The project also seems to lack a coherent policy on when and how it will make data available to independent researchers.

Clemens and others are right to ask that the project make this information available. Greater transparency is essential to build trust and credibility. The project's approach has potential, but little can be said for sure yet about its true impact. The latest initiative of the Millennium Villages Project, in Ghana and funded by the UK government, is a welcome step in the right direction. It builds in independent scrutiny from the start, and has been open and transparent about its costs. All future projects should follow this model.