In September, a United Nations summit of heads of state will adopt the Sustainable Development Goals (SDGs) — a set of 17 goals and 169 targets to guide international development. A diverse range of indicators and monitoring strategies is being proposed, covering every dimension of development, from human well-being to the environment1.
Next week, high-level political representatives meeting in Addis Ababa for the International Conference on Financing for Development will discuss how to fund the SDGs. The participating governments, development institutions, non-governmental organizations (NGOs) and business stakeholders will negotiate an agreement on domestic commitments and international action around financing initiatives.
The SDG monitoring framework makes great demands on nations — it must help countries to implement strategies and allocate resources, measure progress towards sustainability and hold stakeholders to account1. A country found to be failing in sustainable forestry, for example, may choose to invest more in forestry or receive penalties and lose aid. Target-setting is trendy among aid and development organizations as well as in multilateral agreements for accountability, impact and value for money.
We contend that target-setting is flawed, costly and could have little — or even negative — impact.
First, targets may have unintended consequences. For example, education quality as a whole suffered in some countries that diverted resources to early schooling to meet the target of the Millennium Development Goal (MDG) of achieving universal primary education2.
Second, target-setting inhibits learning by focusing efforts on meeting the target rather than solving the problem3. The milestones are easily manipulated — aims such as halving deaths from road-traffic accidents can trigger misreporting if the performance falls short or encourage underperformance if the goal can be exceeded.
Third, it is costly: development partners will have to reallocate scant resources for a 'data revolution' that will cost an estimated US$1 billion a year4.
We advocate a different approach. Governments and the development community need to embrace decision-analysis concepts and tools that have been used for decades in mining, oil, cybersecurity, insurance, environmental policy and drug development5, 6. Our call to adopt this approach is based on five principles.
Replace targets with measures of investment return. The SDGs should state a few broad strategic goals and assess how to achieve them by measuring each project in terms of a return on investment: how well the goals are met given the resources used. For example, were the environmental benefits and reduction of poverty enough to justify the allocation of limited resources?
Decision-makers would use economic models that project long-term costs, benefits and risks of intervention options. They would seek to maximize the risk–return position of a portfolio of options towards achieving the development objectives5. This will require the relative value of different aims to be stated in monetary terms. A government could assess, for instance, whether its objective would best be achieved by spending $50 million on training farmers, building roads, improving education or some combination of them.
Model intervention decisions. Enabling decision-making must be at the heart of SDG monitoring strategies. It is difficult, however, to pinpoint which data are required to support better decision-making without formal decision analysis.
For example, public-health scoring systems — such as the Framingham Risk Score for cardiovascular disease — that assess and prioritize patients according to factors such as age, blood pressure and cholesterol level do not account for people with the most susceptibility who have received treatment. The scoring system underestimates the risk factors if treatment is not recorded, no matter how many other data are collected6.
In 2013, we conducted a survey7 of 110 stakeholders in African agriculture (including scientists, universities, donors, government ministries, NGOs, the private sector and farmer associations). Most (54%) could not identify a policy or management decision that would be supported by further data. They might say, for example, that better soil data would help them to manage erosion-control policies better, but they could not name a particular decision, investment, intervention or policy that would be different if they knew more about the soil. Only 15% of respondents were able to articulate how acquiring data would reduce a crucial uncertainty to enable a decision.
The survey showed that there was a tendency, especially among scientists, to seek data for the sake of having them. For example, biodiversity and poverty data were frequently cited as a focus of effort but infrequently as a perceived need or uncertainty. Climate data were needed and satisfied an uncertainty, but were infrequently collected.
The SDG community must define the actions, policies, programmes or projects that the indicators are expected to inform. These should reflect the practical choices that development planners on the ground will face, such as whether to build one large dam or many small ones to secure water and energy needs, or which of several child-nutrition programmes should be implemented in a region.
The impact of interventions on different groups of people should be factored in: for example, upstream and downstream water users, male and female farmers, or rural and urban populations may be affected differently by a given policy. Such a user-centred approach8 to deciding the best actions would make decision-makers' assumptions and preferences transparent — for instance, the degree of risk they are willing to accept.
Integrate expert knowledge. It is a common mistake to assume that 'evidence' is the same as 'data' or that 'subjective' means 'uninformative'. Decision-making should draw on all appropriate sources of evidence. In developing countries where data are sparse, expert knowledge can fill the gaps. For instance, in our assessment of the viability of agroforestry projects in Africa, we used our experience to set ranges on tree-survival rates, costs of raising tree seedlings and farm prices of tree products. Decision theorists and local experts will have to work together to identify relevant variables, causal associations and uncertainties.
“Decision-makers who are implementing and tracking the SDGs should employ probabilistic decision analysis.”
There are well-established procedures for 'calibrating' experts when using subjective probabilities to quantify uncertainty about estimates5, 6. For example, the World Agroforestry Centre assessed the relative benefits of agricultural interventions for developing regions by calibrating experts for how well they estimated probabilities and by holding workshops to define a probabilistic model5.
The most widely accepted method of incorporating knowledge for probability assessment is Bayes' theorem. This updates the likelihood of a belief in some event (such as whether an intervention will reduce poverty) when observing new evidence about the event (such as the occurrence of drought)6. Bayesian analyses — incorporating historical data and expert judgement — are used in transport and systems-safety assessments, medical diagnosis, operational risk assessment in finance and in forensics6, but seldom in development. They should be used, for example, to evaluate the relative risks of competing development interventions.
Include uncertainty in predictive models. Scientists often use simulations of climate, hydrology, crop growth or disease spread to guide policy or management decisions. Such models of physical systems have two limitations for allocating resources. First, they usually omit behavioural and economic factors; and second, they commonly fail to represent uncertainty in input data, model parameters and outputs.
Decision-makers who are implementing and tracking the SDGs should employ probabilistic decision analysis, for example Monte Carlo simulations5 or Bayesian network models6. Provided that such models are developed using properly calibrated expert judgement and decision-focused data, they can incorporate the key factors and outcomes and the causal relationships between them. For instance, simulations for evaluating options for building a water pipeline could take into account rare 'what-if' scenarios, such as a hurricane during development, and predict (with probabilities) the time and cost of implementation and the benefits of improved water supply.
Measure the most informative variables. An analysis of more than 80 models from a variety of decisions and industries reveals that managers tend to choose to measure variables that are unlikely to improve decisions while ignoring more useful ones5. For example, the adoption rate of a method by farmers is easy to measure, but its effect on yields may be more relevant for making choices. Quantities for which there is already a great deal of information, such as financial costs, are more likely to be tracked but cannot influence decisions because there is little left to learn about them. Less common variables such as social and long-term benefits (such as on mental health) and environmental impacts (such as water pollution from soil erosion) may be of greater value.
Reducing decision uncertainty should be the purpose of measurement5. Only a few variables may be relevant, and data collection should focus on those that narrow choices the most5. For example, a US Environmental Protection Agency analysis of alternative information systems for water quality found that only one variable dominated the uncertainty around investment in the information system: the average health effects of safe-drinking-water policies. Uncertainties about adoption rates of the technology, efficiency improvements and improved reporting rates turned out to have no information value for the agency5.
“Reducing decision uncertainty should be the purpose of measurement.”
In decision theory, the value of information is the amount that a rational decision-maker would be willing to pay for that knowledge before making a decision — the value of clairvoyance9. This can be estimated only by analysing the uncertainties in all the variables that have a bearing on a decision. Such value-of-information analysis is not used in development but is in, say, health economics10. The UK National Institute for Health and Care Excellence uses it in deciding whether a drug or intervention should be approved for widespread use10.
Some proposed SDG indicators will be difficult and expensive for low-income countries to collect, for example the “percentage of women, men, indigenous peoples, and local communities with secure rights to land, property, and natural resources”, and “nitrogen use efficiency in food systems”. Limited resources would be better spent on gathering data with high decision-making value. Those data can be identified only by analysing the specific decisions to be made, and will change as new decision nodes emerge.
Value-of-information analysis helps to identify metrics for monitoring performance. These are often not intuitive and therefore missed. For example, we did a study of natural-resource management interventions, such as integrated watershed projects and seed improvements for maintaining agro-biodiversity. We found that the most useful factors to know were rural-to-urban migration rates, market prices, project failure risks, negative consequences (such as disadvantaging poorer sectors of the community) and adoption rates5.
A new direction
Decision analysts should be embedded in all government and UN policy-development and management units, through a capacity-development programme paid for by governments and international donors, including from the private sector. The UN should establish a forum of decision-analysis experts to steer this initiative.
These analysts would work with decision-makers and subject experts to clarify key intervention decisions and develop probabilistic models of alternative actions. They would build models in a participatory way, involving key stakeholder groups and training experts in subjective probability estimation.
Value-of-information analysis should guide data-collection efforts and define high-value metrics that have the potential to improve decisions and performance. Some of the proposed SDG indicators might be among them, but would be rationally justified, and may change as new priorities emerge.
For commonly occurring variables, such as carbon and commodity prices and risks of extreme climate events, governments and the UN should establish open-access libraries of probability distributions for running simulations5. Monitoring real change against decision models provides a realistic alternative in circumstances in which it is difficult to conduct randomized control trials, such as when considering major new environmental interventions.
We call on the delegates of the Financing for Development conference in Addis Ababa to establish a task force to explore our approach. We recommend that some of the aid money earmarked for improved monitoring of the SDGs be directed to establishing this initiative. Forward-looking governments, especially in data-sparse countries, should consider pioneering decision-analysis approaches.
The principles that we have outlined are applicable to the improvement of any policy or management process, from international policy (such as climate-change negotiations) down to the individual project level (such as whether a village should install a new water storage system). Training a generation of decision analysts to work with policy-makers could do more for development than any other single intervention.
- Journal name:
- Date published: