“What if someone had already figured out the answers to the world’s most pressing policy problems, but those solutions were buried deep in a PDF, somewhere nobody will ever read them?” asked a Washington Post blog last year.
It was on to something. Many of the tens of thousands of documents that are produced every year to assess the impacts of sustainability policies and programmes are never read1. In 2014, the World Bank2 found that almost one-third of its archived policy reports — documenting the impacts of its numerous projects, from dam construction to microcrediting — has never been downloaded.
It doesn’t have to be this way. Experts in evidence synthesis, a field that involves the use of various tools and methods to locate and combine many sources of data, are starting to produce evidence maps for wayfaring researchers and policymakers. These pull together and categorize systematic reviews, impact evaluations and other primary-research studies in a particular area (such as agriculture or education), and visually distil the scope and effects of interventions that have been implemented3.
Evidence maps can show at a glance which areas or relationships have been studied most — whether it be the impact of ecotourism on local economies or of education on reducing harmful fishing practices. They can also highlight key gaps in the evidence base, and so guide the prioritization of research.
We are an interdisciplinary working group supported by the Science for Nature and People (SNAP) Partnership that aims to understand how protecting nature can enhance human well-being (see go.nature.com/fdsj4v). We contend that evidence mapping should be applied to sustainable development much more broadly. To aid decision-making at the intersection of development and conservation, we have produced an interactive map that categorizes more than 1,000 studies documenting links between conservation efforts and human well-being. We urge policymakers and researchers working in sustainability to develop similar tools to enable researchers, donors and practitioners to rapidly find and assess the information relevant to them.
Mind the gaps
Failure to evaluate the existing evidence can result in unnecessary harm. For example, conducting a systematic review in the 1960s of the available data on cot death (instead of waiting until the 1990s) could have led to earlier recognition of risk factors associated with infant sleeping positions and prevented an estimated 10,000 infant deaths in the United Kingdom alone4.
Another consequence may be overlooking the relative costs and benefits of different interventions. Development projects designed to prevent diarrhoeal disease, for instance, continue to emphasize the importance of providing people with access to clean water. Yet a 2012 synthesis5 of different sanitation interventions indicates that prioritizing certain behavioural changes, such as hand-washing, can bring equivalent health and other benefits, and cost much less than introducing a new water supply.
“Since 1970, almost half of all rigorous impact evaluations have been conducted in only six countries.”
Often people assume that there is evidence to justify a particular intervention, or they act on the basis of what is familiar rather than what is proven. Establishing and maintaining national parks and other protected areas is one of the more prevalent conservation approaches used by governments and non-governmental organizations. Yet a 2013 systematic review6 of qualitative and quantitative assessments of protected areas worldwide shows that many of the broad assumptions that underlie their creation — that protected areas have positive social impacts, say, or provide economic benefits through tourism — are not reliably supported.
Systematic reviews of multiple studies, such as the three just described, enable researchers to assess the quality of the available evidence and to make predictions about the effectiveness of a programme more broadly. Yet because documents are often inaccessible — hidden behind paywalls, or buried in hard drives and filing cabinets in field offices — obtaining the relevant publications and reports to conduct such reviews is costly and time consuming.
The resources currently allocated to monitoring and evaluation — on average, less than 5% of a conservation project’s budget — do not come close to what is needed to satisfy the increasing demand from policymakers for more and higher-quality evidence on the impacts of conservation and development interventions1.
Researchers and others need new ways to prioritize efforts, to ensure that funded evaluations and systematic reviews address relevant questions and to track progress in the generation of evidence.
Encouragingly, various organizations have begun to produce and fund systematic reviews and high-quality evaluations to assess the effects of conservation interventions. Among them is the International Initiative for Impact Evaluation (3ie), a non-governmental organization that promotes evidence-informed policies and programmes for development.
Source: M. Bottrill et al. Environ. Evid. 3, 16 (2014)
Standards, tools and practical guidance — for instance, on steps to reduce bias — have also begun to emerge. For instance, international research networks, such as the Collaboration for Environmental Evidence, are promoting the use of rigorous methodologies to assess the environmental, social and health impacts of specific interventions7, such as planting hedgerows or the use of genetically modified crops. The Evidence for Policy and Practice Information and Co-ordinating Centre at the UCL Institute of Education is developing various tools to help researchers to retrieve, characterize and extract data when doing evidence synthesis. We think that evidence maps should be added to this growing stock of synthesis tools.
Over the past five years, 3ie, and researchers funded by the organization, have generated five maps for a broad range of topics, from education to sanitation and hygiene. These maps identify well-studied areas, such as the effects of hand-washing and other sanitation measures on the incidence of diarrhoea. The maps also flag gaps in research. For instance, little is known about the impact of education-related interventions on household expenditure or child labour. As far as we know, no other evidence maps relevant to sustainable development have been created.
We have extended 3ie’s effort by creating a map of the scope and amount of existing evidence on the impact of nature conservation on human well-being globally8. To do this, we mined online databases and websites and asked individuals who were well-positioned to alert us to studies. We narrowed our search to studies published since 1970 that had been conducted in developing countries. We then took systematic steps — based on factors such as study design and type of conservation intervention — to winnow our collection. Following 3ie, we included systematic reviews and impact evaluations. We also categorized other primary research studies, such as non-comparative studies in which no controlsituation was available to compare the causal effect of the intervention.
Our map reveals that around 25% of the 1,014 studies we categorized measured the economic impacts of establishing and maintaining protected areas, whereas only 2% considered equally important measures of well-being, such as the health of local communities (unpublished work). More data, for instance, on the country and biome in which the study was conducted and on the study design, enable users to explore in more detail the distribution and robustness of the evidence base (see ‘Navigating what’s known’). For instance, we were surprised to learn that since 1970, almost half of all rigorous impact evaluations have been conducted in only six countries — including Tanzania, China and Thailand. Eight of the 90 such studies were carried out in Costa Rica, probably in part thanks to the country’s strong governance in environmental issues and its well-established research community.
Our map also exposes areas where more data are needed. In Indonesia, for example, little is known about the relationship between conserving marine ecosystems and the sense of empowerment felt by local communities. And it illuminates which relationships are well-studied and ripe for analysts to explore trends and examine causal pathways through full systematic reviews — such as those between fisheries resource management and economic and material well-being.
Evidence maps are needed for a broad range of topics that are central to sustainable development, such as renewable energy, food security and disaster-risk management. As new evidence becomes available, such maps will need to be updated, ideally by a central curator, to enable researchers to track progress in plugging knowledge gaps. Publishers are well placed to catalyse the development of such shared map resources by enabling broader access to their journals and encouraging the use of standardized search terms and approaches in individual studies. Donors should make the placement of a study in an open evidence repository a condition of funding — much as public funding agencies, such as the US National Institutes of Health, require grant recipients to make their research openly available to the public through PubMed Central.
The Sustainable Development Goals (or Global Goals) were launched in September by the United Nations. Among the 17 priorities for governments, businesses and others are addressing climate change, alleviating poverty and malnutrition and protecting the planet’s natural resources. Information on what kinds of interventions are effective in helping the world to meet these and other challenges, and under what conditions, will be crucial to guiding policy and tracking progress9.
An evidence atlas for sustainable development, progressively updated and improved, could help to transform the work of addressing global challenges into a rigorous science.
- Journal name:
- Date published: