Global assessments need to adopt more rigorous and focused processes for collation and review, says William J. Sutherland.
Society increasingly demands that policies are based on scientific evidence, as it should. Yet policy-makers rarely refer to journal papers — the actual evidence — instead preferring to base decisions on summaries, reviews and assessments.
Such assessments can be deeply influential but fallible: the Intergovernmental Panel on Climate Change (IPCC) has effectively underpinned climate-change policies worldwide, yet the infamous Himalayan glacier blunder in the 2007 report is still regularly unearthed to taunt the IPCC’s authority.
Less likely to make international headlines, but still instructive, are problems my colleague Lynn Dicks and I discovered when we reviewed the pollinator section of the United Kingdom’s National Ecosystem Assessment, a similarly sprawling and influential report that fed into the British government’s white paper on the environment. It states, for example, that “since 1980, wild bee diversity has declined in most landscapes”. This exaggerates the findings of the cited paper, which reports only that bee diversity is lower post-1980 than pre-1980 in 52% of 81 10 × 10 kilometre grid squares. The report asserts that “decreases in pollination services would, therefore, result in short-term economic losses for farmers”, but it fails to mention that this is based on indirect field studies of insect visitors to crops. In fact, no economic effects of pollinator decline have been detected on a national or sub-national scale. A crucial table in the assessment report that estimates the value of insect-pollinated crops in the United Kingdom comes from a PhD thesis, and the analysis behind it has not yet been published in the peer-reviewed scientific literature; this is not made clear. The same quality-control issues appear elsewhere in the report and in the Millennium Ecosystem Assessment.
The problem of quality control will continue unless we change the way in which such assessments are carried out. A good place to start is with the new Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES), which will meet early next month in Turkey to establish its assessment programme.
In broad terms, such assessments must be less ambitious. The headline conclusions of these reports are subjected to extensive scrutiny, yet they are linked to a vast expanse of text that is impossible to check thoroughly. The greater the intended scope of the assessment, and the more material that needs to be included for the sake of completeness, the higher the risk of error becomes. The pleading from climate scientists that the glacier mistake was just one paragraph among 938 pages in one of four reports, and was not even noticed for two years, has not quietened critics of the IPCC.
Environmental assessments should use formal consensus methods.
Although there might be a desire among some involved in the forthcoming IPBES assessment to write freely about many aspects of pollination, they should instead concentrate on collating and synthesizing the global evidence on just the key issues relating to the main questions. Which pollinators are most economically important? What have been the changes to their status? What are the main threats? And what are the most economically effective means of maintaining and restoring these pollinators? To support the answers to these questions, the assessment must first produce a systematic and publicly available review of the literature it will be based on, ideally with accessible summaries of papers.
For such reviews, the vision is often the meta-analyses of evidence-based medicine, which pool and offer an overview of all studies of appropriate quality. This approach is less suited to environmental reviews, which typically use a complex mix of experiments, correlative studies and models with a range of questions, response variables and locations that have differing relevance to the issue of interest.
To better synthesize the findings of such diverse research, environmental assessments should use formal consensus methods such as the Delphi technique, which is an iterative process of consensus-seeking that is based on sequential rounds of confidential scoring and commenting on anonymized results. In addition, all of the material should be publicly available — possibly even including the individual final scores of the experts, as happens for the members of the UK Monetary Policy Committee who meet monthly to decide the official interest rate. It would thus be relatively straightforward to trace a conclusion to its assessment of the evidence and then to its evidence base.
This approach seems to have a range of advantages. It concentrates on the few crucial issues but presents them in a more transparent and rigorous manner that is likely to provide greater confidence and reduce the likelihood of errors. After the evidence is collated, it can be updated regularly to allow for quick reassessment (conversely, the IPCC assessment is repeated about every six years and is hugely expensive). With the bedrock of the evidence assembled and presented in a user-friendly way, the evidence on key issues can be continually collated and regularly assessed. An equivalent example in medicine is Clinical Evidence, which reassesses interventions every six months.
This approach should be adopted not just by the IPBES and other future national or international assessments, but, where possible, by the innumerable reports that are written for decision makers. This should increase quality, transparency and authority.
Related links in Nature Research
Related external links
About this article
Cite this article
Sutherland, W. Review by quality not quantity for better policy. Nature 503, 167 (2013). https://doi.org/10.1038/503167a
This article is cited by
Collation of Indigenous and Local Knowledge as Evidence Base for Herpetofauna Conservation Outside Protected Areas: Case Study from an Agricultural Landscape in Eastern India
Proceedings of the Zoological Society (2022)
Environmental Evidence (2015)