Individual decision-makers need communications that succinctly describe potential harms and benefits of different options, but policymakers or citizens evaluating a policy are rarely given a balanced and easily understood summary of the potential outcomes of their decision. We review current policy option communication across diverse domains such as taxes, health, climate change, and international trade, followed by reviews of guidance and evidence for communication effectiveness. Our conceptual synthesis identifies four characteristics of policy options that make their communication particularly difficult: heterogeneous impacts on different segments of the population, multiple outcomes, long timescales, and large uncertainties. For communicators that are trying to inform rather than persuade, these complexities reveal a core tension between issue coverage and comprehensibility. We find little empirical evidence for how to communicate policy options effectively. We identify promising current communications, analyze them based on the above synthesis, and suggest priorities for future research. Recognizing the particular challenges of balanced, effective policy option communications could lead to better guidelines and support for policy decision-making.
To make an informed decision, individuals need to weigh up the potential impacts of different options. Citizens or policy-makers evaluating a policy are rarely given a balanced and clear summary of the potential outcomes of their decision. Providing such summaries requires: (1) learning what impacts to consider and identifying the most important effects, (2) gathering the evidence, and (3) communicating that evidence such that it is understood. The first two stages are well-defined by processes within governments and organizations (e.g., HM Treasury, 2011), but we review the third step and find little evidence for how to effectively communicate multiple policy options.
For individual decisions about personal health or finances, much is known about how to present potential harms and benefits to effectively inform rather than persuade (e.g., Fischhoff et al., 2011; Spiegelhalter, 2017). However, policy decisions including tax and economic policies, corporate strategy, environmental and educational policies, etc. are particularly difficult. Far more than individual decisions, policy decisions often affect many people and even other species, impact different groups in different ways, and cause multi-generational outcomes. Because weighing up options for an informed policy decision is difficult, it is particularly important to assist policy decision-makers with concise, clear, and comprehensive descriptions of the potential outcomes of policy options. There can be profound costs when communications fail to inform decision makers. For example, lack of clarity about how a tax policy will affect different groups can lead to the adoption of policies that worsen inequality.
Policy decision-makers are a broad group, including corporate board members deciding on corporate strategy options, government ministers appraising policy options, and voters considering options in a referendum. An equally broad set of groups attempt to provide these decision-makers with clear, non-partisan communications, including civil servants, corporate consultants, public service broadcasters, and non-governmental organizations (NGOs). However, despite the importance and ubiquity of policy option communication, nearly all guidelines for designing effective communications appear based on the research into individual-level decision-making, e.g., deciding on a medical treatment.
How best to summarize the potential outcomes of policy-level decisions in simple and balanced formats is currently unknown. We review the current communication guidance below, and find it vague and insufficient to support policy decisions. Because decision makers have finite attention and cognitive resources, presenting too much detail may cause them to disengage or misunderstand. However, too little detail can leave individuals less able to identify the options most likely to achieve their goals. Excellent reviews describe how to communicate risks (Fischhoff et al., 2011; Pidgeon and Fischhoff, 2011) and communicate research findings (Wilson et al., 2010), but there is little general advice for explaining the expected consequences of policy options.
We propose a new research area to support policy option decision-making. This paper focuses on balanced communications: messages that allow recipients to weigh up potential benefits and harms of different policy options to different groups. Overtly persuasive communications such as advertising or lobbying materials are beyond our scope as their primary goal is to make some outcomes appear more attractive. These persuasive materials avoid the key challenges of balanced communication by selectively highlighting and omitting expected harms and benefits.
This scoping review (Gough et al., 2012) aims to assess how the potential harms and benefits of policy options are being communicated and to synthesize the advice and evidence for designing such communications. First, we highlight lessons from existing work on individual-level communication. Then, we describe three new reviews of: (1) current policy-level communications, (2) communications guidelines, and (3) evaluations of policy-level communication. Last, we critique examples of current policy-level communications based on our findings. We end by calling for a new area of empirical research on policy option communications and by suggesting research priorities. To introduce the type of policy option communications considered in this paper, Fig. 1 shows three current examples.
Lessons from individual-level communication
The robust research into individual-level communication is likely to apply also to policy-level communication. Individual- and policy-level communication overlap rather than form separate categories. For example, when explaining the potential harms and benefits of an ostensibly individual-level decision such as vaccination, communications may also describe potential benefits to others through herd immunity (Betsch et al., 2017). On the other hand, the expected impacts of large-scale flood protection programs may be communicated at a societal level through maps that can then be used for individual decision-making by focusing on specific locations.
In addition, communications designed to support individual decisions often contain illustrations that actually focus on groups. The standard recommendation for individual risk communication (Gigerenzer et al., 2007) is to avoid terms such as the “probability” or “chance” of events happening to a specific person (Trevena et al., 2013), and instead to embed an individual in an imaginary group of similar people, e.g., “of 100 people like you, we would expect 12 people to have a heart attack or stroke in the next 10 years.” In policy decisions, the groups of people are real rather than hypothetical.
In Table 1, we summarize the literature on effective communication of individual-level outcomes, which has been excellently reviewed elsewhere (McInerny et al., 2014; Trevena et al., 2013). In the absence of evidence to the contrary, we suggest that these guidelines are likely to apply to policy-level communication. However, the communication of potential policy outcomes introduces additional challenges which these guidelines do not address.
Policy-level communication, guidance, and evaluation
In this section, we describe three reviews on communications that contain policy options and potential outcomes: conceptual syntheses of existing communications (Review 1) and of current guidance (Review 2), followed by an aggregative review of communication effectiveness (Review 3) (Fig. 2) (Gough et al., 2012).
Review 1: Current communications
First, we collected examples of print and online communications that attempted to communicate the outcomes of different policy choices. These were produced by media organizations, governments, corporations, NGOs, and scientists for a range of audiences from the lay public to professional policy-makers (Figs. 1 and 3–6). We focused on examples where the intention appeared to be to inform rather than to persuade by excluding examples from charities, lobby groups, and overtly partisan political campaigns. The majority of these examples were from the grey literature, i.e., produced by governments, academics, corporations, and the media, and unmediated by academic publishers or peer review (for another review of the grey literature, see Davidson, 2017). Therefore, this was a configuring review that used iterative methods to reveal emerging concepts (Gough et al., 2012).
Second, we used online tools including Google image and Google text searches using the term communicat* combined with: evidence, policy/ies, graph, option, impact, assessment, report, or scenario. We also searched the government websites of provinces/states and countries where referendums are common, because these governments regularly need to communicate policy options in a balanced way to citizens. We additionally searched the websites of NGOs involved in evidence communication (e.g., Cochrane and Campbell Collaborations, What Works Foundations, American Clearinghouse, U.S. Congressional Budget Office, Intergovernmental Panel on Climate Change [IPCC]), and solicited examples from our personal networks of scholars and industry professionals focused on risk, communication, and/or policy.
The examples that we found mostly concerned health, finance, business, transport, education, ecology, migration, war, and terrorism. They spanned multiple formats from single graphics to lengthy policy documents, and included numeric tables, static representations such as bar graphs, interactive graphs and video visualizations, and narratives (text) that described expected outcomes from policy decisions.
Review 2: Current guidelines
In the second review, we gathered organizational guidelines on communicating policy options. The inclusion criteria were guidelines that specifically address how to design or present communications of evidence to inform a policy-level decision. Therefore, we excluded advice on communicating individual-level options such as personal medical treatments. The second review was also configurative and designed to reveal emerging concepts.
We searched for and requested documents from a range of governments and NGOs. As in Review 1, we included governments and constituencies with referendums (Austria, California, Canada, EU, Germany, Switzerland, U.K., & U.S.). We requested guidelines from NGOs including international accountancy firms and management consultancies (PwC, Deloitte, EY, KPMG, McKinsey, Boston, Accenture, & Cap Gemini). We also searched for guidelines produced by libraries of systematic reviews including the Cochrane and Campbell Collaborations (Campbell Policies, 2016; Cochrane Collaboration, 2011), the International Association for Impact Assessment (Vanclay, 2015), and the International Initiative for Impact Evaluation (3ie). We additionally reviewed guidelines on presenting future scenarios where policy options may not be explicitly communicated, such as the U.K.‘s Financial Reporting Council guidelines on writing corporate strategic reports, which summarize past and future potential performance of a company (Financial Reporting Council, 2014).
As an example of our discovery process for policy option communications, within the U.K. we identified governmental bodies and reports involved with evidence communication (e.g., Department for Environment, Food & Rural Affairs; Parliamentary Office of Science & Technology; National Risk Register), and the groups authoring governmental workflow documents such as the Magenta Book (HM Treasury, 2011) and the U.K. Evidence for Policy and Practice Information (EPPI, 2017). We then searched their websites for documents and reports containing instructions or guidance on communicating policy options, including advice on audience, report length, and communication format (e.g., visual aids).
Review 3: Empirical evidence
Third, we reviewed the academic literature for empirical evaluations on comprehension or knowledge of policy-level communications (definition below). Because policy option communication occurs across such diverse fields, we began with a configural review to identify the key terms, sub-literatures, and concepts, which we then used in an aggregative review to summarize the state of evidence. We searched Web of Knowledge and Google Scholar separately for all peer-reviewed research published by December 2017 using combinations of these terms: communication policy option outcome risk evidence decision evaluation. At Google Scholar we used the full-text search, and at Web of Knowledge we searched under “topics”.
The titles and abstracts from both searches were then reviewed for whether articles described the review or evaluation of the effectiveness of policy option communication on comprehension, knowledge, or similar outcomes (whether subjective or objective). We then reviewed the titles and abstracts of the cited and citing papers from that set, and repeated that process for any new inclusions until no new papers were identified, which left 370 articles. We also presented this project at international conferences and discussed widely with colleagues to reveal other literature. No papers were uniquely identified by this last step. We then excluded papers that did not evaluate nor review the effectiveness of policy option communications or evidence summary formats. We also excluded empirical articles evaluating communications that were narrowly designed for individual decisions, e.g., one’s own medical care. After all exclusions, we identified 72 articles. See Review 3: Results and Discussion for a summary of this literature, and the Supplement for the 72 articles in a table categorizing each as empirical, review, methodological advance, or organizational report.
Results and discussion
Review 1: Current communications
Based on the first review, we identified four challenges that make policy option communication particularly difficult compared to individual-level communication:
Heterogeneity of impacts
Policy options generally have different impacts on different groups (e.g., geographical, sociocultural, demographical, or political). For instance, tax policy reform often benefits some income classes over others and communicating these trade-offs between groups is difficult (Fig. 1c).
The first challenge is deciding which populations and potential outcomes to describe. For example, banning the sales of new petrol or diesel cars has local environmental benefits (DEFRA, 2017; Le Monde, 2017). However, one consequence of such bans is likely to be an increase in demand for lithium-ion batteries containing cobalt, of which around half currently comes from mines in the Democratic Republic of Congo (Yager, 2017). Amnesty International and UNICEF reported widespread health risks to tens of thousands of miners in the DRC, including many children (Amnesty International, 2016), and the UN reported major environmental damage (UNEP, 2011). Citizens and policymakers may be unaware of the health and environmental risks in the DRC and it is unclear whether these risks should be communicated as a potential outcome of a policy in Europe, India, or China. There is currently no specific guidance on how to decide what outcomes to include when considering distant impacts.
Detailing all outcomes for sub-populations may reduce comprehensibility (Fig. 3a) (e.g., Rosenbaum et al., 2011), but using a population-wide average can obscure different impacts in different groups (Fig. 3b). Three common but incomplete approaches are to: (1) mention only the potential benefits or potential harms (rather than both), which may mislead recipients and constitute a persuasive message, (2) mention only one type of outcome, such as financial but not environmental impacts, or (3) describe only the main potential benefit and main potential harm. These approaches can be pragmatic but can also reduce the usefulness of communications to decision-makers. It is often undisclosed that options or outcomes were omitted, which may undermine trust. Transparency about which policy effects are considered during the decision-making process may boost the perceived fairness and objectivity of the decision process, although this claim is contested (Grimmelikhuijsen et al., 2013; O’Neill, 2002).
Multiple outcome scales
Policies and interventions have diverse outcomes that can be measured in different ways. For instance, a policy may have financial, health, environmental, and employment consequences, each with a separate metric, making it difficult to compare the outcomes. Even combining data about the same outcome is difficult because continuous outcome measures are often separated at different scale points (e.g., combining data from overlapping income groups; also see Guyatt et al., 2013).
Communicators sometimes attempt to express all the possible impacts of a policy with a single metric or common currency (e.g., Fig. 3b) that enables a direct ranking of options. For example, the UK Government’s Green Book (HM Treasury, 2003) recommends a cost-benefit analysis of options in which explicit monetary values are given to each outcome. However, conclusions based on a single metric may not be supported by public audiences nor policymakers.
For example, the Safe and Sustainable initiative was set up in 2008 by the National Health Service Commissioning Groups in the U.K. to make recommendations for concentrating services for surgery for congenital heart disease in England, at that time provided by ten centers. A multi-criteria decision analysis compared a shortlist of six options for patterns of center closures on four criteria: access and travel times for patients, quality, deliverability and sustainability. The individual scores and criterion weights were obtained by a deliberative process intended to be transparent (see Table 2).
The analysis contained no options without closures, was widely criticized, and the final recommendation (Option 1) was opposed through public demonstrations, political lobbying, and legal action. As of 2017, there were still ten NHS centers in England carrying out children’s heart surgery, including all three recommended for closure under Option 1.
Many individual-level decisions primarily affect the decision-maker or their family over single generations. However, many policy options such as waging war or establishing environmental protections have effects across longer time periods. The impacts of policy-level decisions may also vary through time, requiring communications that describe impacts at several time points (see Fig. 4 for climate change impacts over time).
Outcome predictions are more uncertain for policies than for individual-level decisions due to long timescales, complex economic, environmental and social systems that influence policy outcomes, and the lack of well-controlled policy experiments. Fully quantified probability distributions are precise but require the specification of all potential outcomes and are therefore rarely attainable. One feasible alternative is to present a high/low/best estimate of impact. However, our review showed that all types of uncertainty (aleatory, epistemic, and ontological; Spiegelhalter, 2017) are usually absent in current policy impact summaries (e.g., in Figs. 1, 3, 5, and 6).
These four challenges—heterogeneous impacts, multiple outcome scales, longer timescales, and uncertainty—make balanced communication of the potential impacts of policy options more difficult than communicating individual-level outcomes. It is not possible to communicate unlimited information (Cairney and Kwiatkowski, 2017), but omitting information often creates bias. Partisan communicators exploit this tension by limiting the information to highlight their preferred option, and they create very simple and easily understood messages. For the communicator attempting to be non-partisan, there is a temptation to avoid bias by including as much information as possible.
A typical example is the impact assessment for additional airport runways in South East England (UK Department for Transport, 2017). This report includes comprehensive descriptions of multiple policy options, and outcomes including heterogeneities, uncertainties, and long timescales. Comprehending this detail requires substantial literacy, numeracy, motivation, and time, and therefore excludes most audiences. Therefore, this and many reports include an overview and figures to communicate the key findings. In reducing the comprehensive report to an easily comprehended overview, potentially important details are again inevitably omitted (Fig. 3b).
A central tension in policy option communication
As shown in the above discussion, in policy option communication there is a particular tension between two factors:
The extent to which the communication describes the most important options and their potential outcomes, e.g., who is affected, which outcomes are included, short- and long-term benefits and harms, and uncertainties.
Whether the communication can be easily understood by the intended audiences in their different contexts.
Greater issue coverage usually reduces message comprehensibility because it places greater demands on limited cognitive resources such as attention and memory (Fig. 3a) (Cowan, 2010). On the other hand, reducing coverage may improve comprehensibility at the expense of misunderstanding the decision environment (Fig. 3c). Moreover, if the information that a decision-maker feels to be most relevant is not provided, this may damage trust in the message and the sender (Boulton et al., 2012).
Review 2: Current guidelines
Research units, NGOs, businesses, regulators, and governments produce guidelines on how to collect and present evidence for policymakers. The second review collected these guidelines. We focused on specific practical advice on how to communicate potential policy outcomes from different scenarios or options.
Our review uncovered a range of similar evidence synthesis or research summary formats recommended by various guidelines, such as: evidence summary (Rosenbaum et al., 2011; Synnot et al., 2017); Cochrane summary of findings (SUPPORT Collaboration; Conway et al., 2017); plain language summary or significance statement (Cochrane; Langendam et al., 2013; Shailes et al., 2017); evidence profile (Guyatt et al., 2013); policy brief (Rajabi, 2012; cf. Beynon et al., 2012), and similar terms (Haynes, 2007; Wilson et al., 2010; cf. Glenton et al., 2010). While these formats are often co-designed with policy decision-makers to support informed decision-making and/or behavior, their content is driven by the research—e.g., the results of a systematic review or the current state of evidence in a field—not by the information needs of a decision-maker. Although these summaries grapple with some of the same challenges of policy option communication such as brevity, uncertainty, and clarity, they do not attempt to make overt predictions of the potential outcomes of specified policy options. Therefore, these summaries avoid the problem of balanced comprehensiveness in considering all potential outcomes and aggregating their different metrics. Policy briefs, although they can include policy options, are also often overtly used for partisan, lobbying, communication which again means that they avoid the challenges of balanced communication. Fact boxes (McDowell et al., 2016; Woloshin and Schwartz, 2011) are a format that do present balanced options (e.g., harms and benefits of selecting a medical treatment), but they are currently limited to individual decisions and have not been used to present outcomes for different subpopulations to support policy decision-making. The lessons learned from research into fact boxes were therefore included in the previous section on individual-level communication.
The guidelines that we found often specified what to include in a report about possible futures, e.g., the findings from all primary outcomes (Peters et al., 2007), but did not address how to balance coverage with comprehensibility. Our review showed that most guidelines stated that communications should be high-quality and clear without specifying how those goals were to be accomplished.
Some guidelines (e.g., HM Treasury, 2003) provided advice aimed at evidence collection, including the quantification of uncertainty and use of a single metric. Collapsing multiple types of outcomes such as health or environmental into a common metric, e.g., financial cost, can simplify communications and possibly improve comprehension (cf. Review 3: Results). In the rare guidelines where specific writing advice was provided to increase comprehensibility, the supporting evidence for this advice appeared missing. For example, a report from the Canadian government on how to write policy summaries (Canadian Health Services Foundation, 2017) specified the optimal length of types of summary reports but cited no evaluation evidence. A notable contrast is given by the U.S. Food and Drug Administration guide to communicating risks and benefits (Fischhoff et al., 2011), which did not claim universal advice and instead recommended user-centered design and evaluation for comprehension for all communications. Unfortunately, such evaluation in policy-level communication is hard to find, as revealed in Review 3 below.
Overall, existing guidelines on producing policy-level communications provide little advice that addresses the specific challenges of communicating policy outcomes, and rarely reference evidence for their guidance.
Review 3: Empirical evidence
This review identified empirical evaluations of policy outcome communications, and excluded studies on benefits and harms for individual decision-making such as for personal medical treatment or literature evaluating overtly partisan material used for marketing or lobbying.
Much of the evaluative work we found concerned evidence summary formats, whose content is driven by existing research rather than policymaker needs. Health policy summaries were relatively well studied, including work on the Cochrane summary-of-findings table (e.g., Glenton et al., 2010; Rosenbaum et al., 2010) and GRADE evidence tables (e.g., Carrasco-Labra et al., 2016; Treweek et al., 2013). Most studies used an audience of members of the public, but some used policymakers (e.g., Rosenbaum et al., 2011; Vogel et al., 2013). Also in this vein, a recent systematic review examined health policymakers and health system managers’ comprehension of an issue with or without evidence summaries (Petkovic et al., 2016). Six studies met their criteria for inclusion. That review concluded that summaries likely aided comprehension but was unable to conclude whether summaries affected decisions.
However, because these summaries were designed to summarize academic knowledge rather than answer the specific needs of those making policy decisions, the summaries often left out information that policymakers likely want such as financial cost. This limitation makes the summaries easier to generate and evaluate, but they do not address the full challenges of comparing policy options to support informed decisions.
One study of healthcare policy communication was notable because it evaluated four methods of communicating the potential outcomes of different mammography policies to policymakers (Brownson et al., 2011). However, the policy briefs were designed to be partisan and favor one policy option. The study tested whether the formats differed in being understandable, credible, likely to be used, and likely to be shared based on self-report (Brownson et al., 2011). Objective comprehension was not assessed, as was typical in the small evaluation literature we found. All communications were provided in booklet format and included heterogeneous impacts across the population, but they varied between data-driven or story-driven, and whether the data was state-wide or only local. Different groups of policymakers had different preferences, e.g., legislators preferred the data over narrative formats.
Outside of healthcare, climate change is the main field in which policy-level communications have been evaluated. In this domain, we consider emissions scenarios as policy options even though the scenarios represent many processes and multiple policies. Most of this work evaluated whether IPCC communications of uncertainty were understood (e.g., Kandlikar et al., 2005; Risbey and Kandlikar, 2007; Budescu et al., 2012; Budescu et al., 2014; McMahon et al., 2015). A key finding was that subtle displays of uncertainty, e.g., map stippling, are frequently missed or misunderstood (McMahon et al., 2015). A recent study suggested that the IPCC’s summaries for policymakers were too linguistically complex (Barkemeyer et al., 2015). Another study compared different visual representations for land use in response to climate change across multiple scenarios, but did not identify a single best format (Bishop et al., 2013). Others authors have called for more empirical research into climate policy communications (Pidgeon and Fischhoff, 2011).
Another area where we found multiple evaluation studies was economics. One systematic review of guidance for presenting economic evidence to policymakers found 31 sets of economic reporting guidelines and described them as generally low-quality because of a lack of systematic evidence (Sullivan et al., 2015). Efforts were made to determine the needs of policymakers and use that information to design the presentation format, but they identified a particular lack of research on communicating economic principles to non-specialist audiences.
In a different domain, one study presented the safety risks of a building policy across multiple locations. They tested eight risk communication formats and found some versions led to more comprehension but less policy support (Bruine de Bruin et al., 2013; cf. Hildon et al., 2012). This finding is common in the communication literature and bears repeating: when seeking ideal message formats, designs that lead to the most comprehension may not change beliefs or behavior most effectively (Akl et al., 2011; Stone et al., 2017). For a communicator seeking to inform rather than persuade, objective comprehension may be the most appropriate outcome measure, but how maximizing comprehension affects the decision-making process and behavior is poorly understood.
In summary, the third review uncovered some evidence for communication design features such as graphical complexity and information omission, but most of the literature on evaluating messages was on science communication rather than policy option communication, and therefore did not directly address trade-offs between coverage and comprehensibility. The existing evaluations mostly focus on healthcare policy, climate change, and economic forecasts, particularly the economics of healthcare policy. Additionally, individual-level communication work has evaluated graphical (McInerny et al., 2014) and numerical (Heller et al., 2003) methods of presenting evidence that can be applied to policy option presentations. Overall, there is little evidence on the best formats for communicating potential policy impacts.
Synthesis: balancing coverage and comprehensibility
All communicators must select what information to include and omit, but there is currently little guidance for how to select policy options and their potential outcomes while minimizing bias. The existing guidance for communicators is shallow, and when evidence-based relies on the literature on individual-level communication. Finally, there is a critical need for evaluation of communication methods to describe the potential harms and benefits of policy options.
Based on the first review, current attempts at balanced policy outcome communications typically resolve the core tension of coverage and comprehensibility by reducing coverage: omitting information that could nonetheless be important to some decision-makers. Other communications achieve high coverage only through long and complex presentations, e.g., a 67-page report on airport runway capacity options in southeast England (UK Department for Transport, 2017). Only a few policy communications appear to balance coverage with comprehensibility well in a brief summary. In Table 3, we analyze three such examples based on the conceptual framework of four challenges identified in Review 1.
The first example comes from an IPCC report on climate change (Fig. 4) (IPCC, 2014). It illustrates the expected changes in four global measures based on two future emissions scenarios, RCP 2.6 and RCP 8.5, which we consider two macro policy options.
The second example is an online Teaching and Learning Toolkit (EFF, 2017a) by the Education Endowment Foundation in the U.K., who produce guidance on specific interventions (Fig. 5) (EFF, 2017b). These communications are designed for policymakers: in this case, senior teachers or principals who implement education policies in schools.
The final example comes from health care. The MAGIC group produces summaries of the potential impacts of health interventions, and these are presented as interactive summaries online and published for use by policymakers and individual clinicians (Fig. 6) (Vandvik et al., 2016).
All three figures approached the challenges in different ways. Without user testing, it is unknown whether they were successful in communicating the information required by the target audiences. However, these figures appear to achieve high coverage while remaining concise and comprehensible.
Individuals faced with personal choices that will affect their lives are increasingly provided with balanced, clear, and concise summaries of the likely impacts of their decisions. In contrast, such summaries are rarely available to policymakers despite the large and wide-ranging impacts of policy decisions. This gap likely stems from the difficulties in representing heterogeneous impacts, diverse outcomes with different metrics, outcomes over long time scales, and large uncertainties. However, the advances in high-quality and ethical decision support for individuals came from steady research, practical implementation, and evaluation. We hope that recognition of the particular challenges of policy-level communication will spur future research and guidelines to better support those making policy-level decisions.
Providing balanced and comprehensible summaries of detailed policy outcomes is difficult, and one unresolved challenge is how best to approach the necessary omission of information that might be important to some decision makers to keep the summary concise and comprehensible. Best practices for communication design will likely include a dialog with intended audiences to learn their interests and responses to communications within their domain (Bruine de Bruin and Bostrom, 2013), including interactive and/or layered figures and resources to achieve high coverage and personalization without overwhelming recipients (McInerny et al., 2014; Sivanathan and Kakkar, 2017; Stephens et al., 2017). However, measuring only message comprehension may not capture reactions to missing information, whose absence might not be noticed.
We found very few studies that experimentally evaluated the effectiveness of communications about policy options. In addition, these studies typically aggregated across recipient characteristics that can affect comprehension, such as motivations, numeracy, graph literacy, and demographics (see Paling, 2003; Rovner et al., 2004; Garcia-Retamero and Cokely, 2013; Okan et al., 2015). As the evidence base expands, we expect further nuance rather than the discovery of universally effective communication techniques.
The three reviews in this paper combine configural, conceptual, and aggregative methods (Gough et al., 2012) for a novel synthesis of an emerging area with great public need. The review methods nonetheless introduced limitations. Conducting the first and second reviews was difficult because much of the relevant material was in the grey literature, which is hard to scope. It was unrealistic to review every form of policy option communication used in governments and organizations around the world. A related limitation is that all of the reviews were based on English-language results due to a lack of translation capability. This limited the representativeness and generalizability of the reviews of existing communication examples and organizational guidelines. The limited scope was less likely to affect the empirical evaluation review as most international scientific journals are currently in English. However, given that the field of policy-outcome communication is not currently well-defined, search terms that would locate academic papers were difficult to exhaustively identify, meaning that the evaluation literature would likely benefit from additional reviews.
Overall, we suggest that the ideal balanced communication would provide appropriate detail in a quickly and easily understood format to help citizens and policy decision-makers apply their own values and priorities to decisions. Whether or not this ideal is achievable in a particular domain, specifying the goal can reveal the empirical questions that guide future progress. We identified four key challenges affecting the communication about impacts of policy options: broad and heterogeneous effects, outcomes with different metrics, potentially long timescales, and large uncertainties. These challenges increase the inherent tension between coverage and comprehensibility in messages. The current evidence on resolving this tension is insufficient to deliver confident recommendations to policy communicators (see Alonso-Coello et al., 2016). We hope that identifying these challenges will stimulate the development of effective, non-partisan communications and their evaluation, and thereby support citizens and diverse policy decision-makers in understanding the possible impacts of policies.
All data analysed in this study is available in the paper.
Akl EA, Oxman AD, Herrin J, Vist GE, Terrenato I, Sperati F, Costiniuk C, Blank D, Schünemann H (2011) Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD006776.pub2
Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Vandvik PO, Meerpohl J, Guyatt GH, Schünemann HJ (2016) GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ 353:i2089. https://doi.org/10.1136/bmj.i2089
Amnesty International (2016) This is what we die for: Human rights abuses in the Democratic Republic of the Congo power the global trade in cobalt https://www.amnesty.org/en/documents/afr62/3183/2016/en/
Barkemeyer R, Dessai S, Monge-Sanz B, Renzi BG, Napolitano G (2015) Linguistic analysis of IPCC summaries for policymakers and associated coverage. Nat Clim Change 6:311–316. https://doi.org/10.1038/nclimate2824
Betsch C, Böhm R, Korn L, Holtmann C (2017) On the benefits of explaining herd immunity in vaccine advocacy. Nat Human Behav 1:56. https://doi.org/10.1038/s41562-017-0056
Beynon P, Gaarder M, Chapoy C, Masset E (2012) Passing on the hot potato: lessons from a policy brief experiment. Inst Dev Stud Bull 43:68–75. https://doi.org/10.1111/j.1759-5436.2012.00365.x
Bishop ID, Pettit CJ, Sheth F, Sharma S (2013) Evaluation of data visualisation options for land-use policy and decision making in response to climate change. Environ Plan B Plan Des 40:213–233. https://doi.org/10.1068/b38159
Boulton G, Campbell P, Collins B, Elias P, Hall W, Laurie G, O’Neill O, Rawlins M, Thornton J, Vallance P, Walport M (2012) Science as an open enterprise. The Royal Society, London, UK
Brownson RC, Dodson EA, Stamatakis KA, Casey CM, Elliott MB, Luke DA, Wintrode CG, Kreuter MW (2011) Communicating evidence-based information on cancer prevention to state-level policy makers. J Natl Cancer Inst 103:306–316. https://doi.org/10.1093/jnci/djq529
Bruine de Bruin W, Bostrom A (2013) Assessing what to address in science communication. Proc Natl Acad Sci USA 110: 14062–14068. https://doi.org/10.1073/pnas.1212729110
Bruine de Bruin W, Stone ER, Gibson JM, Fischbeck PS, Shoraka MB (2013) The effect of communication design and recipients’ numeracy on responses to UXO risk. J Risk Res 16:981–1004. https://doi.org/10.1080/13669877.2013.788055
Budescu DV, Por H-H, Broomell SB (2012) Effective communication of uncertainty in the IPCC reports. Clim Change 113:181–200. https://doi.org/10.1007/s10584-011-0330-3
Budescu DV, Por H-H, Broomell SB, Smithson M (2014) The interpretation of IPCC probabilistic statements around the world. Nat Clim Change 4:508–512. https://doi.org/10.1038/nclimate2194
Cairney P, Kwiatkowski R (2017) How to communicate effectively with policymakers: combine insights from psychology and policy studies. Palgrave Commun 3:37. https://doi.org/10.1057/s41599-017-0046-8
Campbell Policies (2016) Methodological expectations of Campbell Collaboration intervention reviews: reporting standards. https://doi.org/10.4073/cpg.2016.4
Canadian Health Services Foundation (2017) Communication notes: reader-friendly writing 1:3:25, https://www.cfhifcass.ca/Migrated/PDF/CommunicationNotes/cn-1325_e.pdf
Carr RH, Montz B, Maxfield K, Hoekskstra S, Semmmmens K, Goldman E (2016) Effectively communicating risk and uncertainty to the public: assessing the national weather service’s flood forecast and warning tools. Bull Am Meteorol Soc 97:1649–1665. https://doi.org/10.1175/BAMS-D-14-00248.1
Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P, Meerpohl JJ, Vandvik PO, Brozek JL, Akl EA, Bossuyt P, Churchill R, Glenton C, Rosenbaum S, Tugwell P, Welch V, Garner P, Guyatt G, Schünemann HJ (2016) Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format. J Clin Epidemiol 74:7–18. https://doi.org/10.1016/j.jclinepi.2015.12.007
CBO (2014) The effects of a minimum-wage increase on employment and family income | Congressional Budget Office, Washington, DC, USA
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.
Conway A, Clarke MJ, Treweek S, Schünemann H, Santesso N, Morgan RL, Darragh M, Maguire LK, Devane D (2017) Summary of findings tables for communicating key findings of systematic reviews. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.MR000044
Cowan N (2010) The magical mystery four: how is working memory capacity limited, and why? Curr Dir Psychol Sci 19:51–57. https://doi.org/10.1177/0963721409359277
Davidson B (2017) Storytelling and evidence-based policy: lessons from the grey literature. Palgrave Commun 3:170–93. https://doi.org/10.1057/palcomms.2017.93
DEFRA (2017) UK plan for tackling roadside nitrogen dioxide concentrations: an overview https://www.gov.uk/government/publications/air-quality-plan-for-nitrogen-dioxide-no2-in-uk-2017
EEF (2016) Classifying the security of EEF findings, https://educationendowmentfoundation.org.uk/help/projects/the-eef-security-ratinghttps://educationendowmentfoundation.org.uk/help/projects/the-eef-security-rating
EFF (2017a) Teaching and learning toolkit. Education Endowment Foundation. https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit. Accessed 12 Dec 2017
EFF (2017b) Funded projects. Education Endowment Foundation. https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/. Accessed 12 Dec 2017
Times of India (2017) Electric vehicles: India aiming for all-electric car fleet by 2030, petrol and diesel to be tanked
EPPI (2017) EPPI resources. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=88. Accessed 17 Dec 2017
Financial Reporting Council (2014) Guidance on the strategic report https://www.frc.org.uk/accountants/accounting-and-reportingpolicy/clear-and-concise-and-wider-corporate-reporting/narrative-reporting/guidance-on-the-strategic-report
Fischhoff B, Brewer NT, Downs J (2011) Communicating risks and benefits: an evidence-based user’s guide. FDA, Silver Spring, MD, USA
Garcia-Retamero R, Cokely ET (2013) Communicating health risks with visual aids. Curr Dir Psychol Sci 22:392–399. https://doi.org/10.1177/0963721413491570
Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S (2007) Helping doctors and patients make sense of health statistics: toward an evidence-based society. Psychol Sci Public Interest 8:53–96. https://doi.org/10.1111/j.1539-6053.2008.00033.x
Glenton C, Santesso N, Rosenbaum S, Nilsen ES, Rader T, Ciapponi A, Dilkes H (2010) Presenting the results of cochrane systematic reviews to a consumer audience: a qualitative study. Med Decis Mak 30:566–577. https://doi.org/10.1177/0272989X10375853
Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Syst Rev 1:28. https://doi.org/10.1186/2046-4053-1-28
Grimmelikhuijsen S, Porumbescu G, Hong B, Im T (2013) The effect of transparency on trust in government: a cross-national comparative experiment. Public Adm Rev 73:575–586. https://doi.org/10.1111/puar.12047
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ (2009) GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Chin J Evid Based Med 9:8–11. https://doi.org/10.1136/bmj.39489.470347.AD
Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, Johnston BC, Karanicolas P, Akl EA, Vist G, Kunz R, Brozek J, Kupper LL, Martin SL, Meerpohl JJ, Alonso-Coello P, Christensen R, Schunemann HJ (2013) GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles—continuous outcomes. J Clin Epidemiol 66:173–183. https://doi.org/10.1016/j.jclinepi.2012.08.001
Haynes B (2007) Of studies, syntheses, synopses, summaries, and systems: The “5S” evolution of information services for evidence-based healthcare decisions. Evid Based Nurs 10:6–7. https://doi.org/10.1136/ebn.10.1.6
Heller RF, Buchan I, Edwards R, Lyratzopoulos G, McElduff P, Leger SS (2003) Communicating risks at the population level: application of population impact numbers. BMJ 327:1162–1165. https://doi.org/10.1136/bmj.327.7424.1162
Hildon Z, Allwood D, Black N (2012) Impact of format and content of visual display of data on comprehension, choice and preference: a systematic review. Int J Qual Health Care 24:55–64. https://doi.org/10.1093/intqhc/mzr072
HM Government (2016) HM treasury analysis: the long-term economic impact of EU membership and the alternatives
HM Treasury (2011) The magenta book: guidance for evaluation, London, UK
HM Treasury (2003) The green book: appraisal and evaluation in central government: treasury guidance, London, UK
Hood A and Waters T (2015) Living standards, poverty and inequality in the UK: 2015. Institute for Fiscal Studies, London, UK
IPCC (2014) Climate change 2014: synthesis report. In: Core Writing Team, Pachauri RK, Meyer LA (eds) Contribution of Working Groups I, II and III to the fifth assessment report of the intergovernmental panel on climate change, Geneva, Switzerland
IPCC (2007) Fourth assessment report: Climate Change 2007 (AR4). IPCC, Cambridge
Kandlikar M, Risbey J, Dessai S (2005) Representing and communicating deep uncertainty in climate-change assessments. Comptes Rendus - Geosci 337:443–455. https://doi.org/10.1016/j.crte.2004.10.010
Langendam MW, Akl EA, Dahm P, Glasziou P, Guyatt G, Schünemann HJ (2013) Assessing and presenting summaries of evidence in Cochrane reviews. Syst Rev 2:81. https://doi.org/10.1186/2046-4053-2-81
Le Monde (2017) Nicolas Hulot: “Nous visons la fin de la vente des voitures à essence et diesel d’ici à 2040”. Le Monde. http://www.lemonde.fr/planete/article/2017/07/06/nicolas-hulot-annonce-une-prime-pour-remplacer-les-vehicules-les-plus-polluants_5156706_3244.html. Accessed 15 Dec 2017
MAGIC (2017) MAGIC project. http://magicproject.org/ Accessed 12 Dec 2017
McDowell M, Rebitschek FG, Gigerenzer G, Wegwarth O (2016) A simple tool for communicating the benefits and harms of health interventions. Med Decis Mak Policy Pract 1:1–10. https://doi.org/10.1177/2381468316665365
McInerny GJ, Chen M, Freeman R, Gavaghan D, Meyer M, Rowland F, Spiegelhalter DJ, Stefaner M, Tessarolo G, Hortal J (2014) Information visualisation for science and policy: Engaging users and avoiding bias. Trends Ecol Evol 29:148–157. https://doi.org/10.1016/j.tree.2014.01.003
McMahon R, Stauffacher M, Knutti R (2015) The unseen uncertainties in climate change: reviewing comprehension of an IPCC scenario graph. Clim Change 133:141–154. https://doi.org/10.1007/s10584-015-1473-4
NFU (2016) British agriculture: the implications of a UK exit from the EU, https://www.nfuonline.com/assets/61142
O’Neill O (2002) Reith Lectures 2002: a question of trust, Lecture 4: trust & transparency. BBC Reith Lectures, London, UK
Okan Y, Garcia-Retamero R, Cokely ET, Maldonado A (2015) Improving risk understanding across ability levels: encouraging active processing with dynamic icon arrays. J Exp Psychol Appl 21:178–94. https://doi.org/10.1037/xap0000045
Paling J (2003) Strategies to help patients understand risks. BMJ 327:745–748. https://doi.org/10.1136/bmj.327.7417.745
Peters E, Dieckmann N, Dixon A, Hibbard JH, Mertz CK (2007) Less is more in presenting to consumers. Med Care Res Rev 64:169–190. https://doi.org/10.1177/10775587070640020301
Petkovic J, Welch V, Jacob MH, Yoganathan M, Ayala AP, Cunningham H, Tugwell P (2016) The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci 11:162. https://doi.org/10.1186/s13012-016-0530-3
Pidgeon N, Fischhoff B (2011) The role of social and decision sciences in communicating uncertain climate risks. Nat Clim Change 1:35–41. https://doi.org/10.1038/nclimate1080
Rajabi F (2012) Evidence-informed health policy making: the role of policy brief. Int J Prev Med 3:596–598
Risbey JS, Kandlikar M (2007) Expressions of likelihood and confidence in the IPCC uncertainty assessment process. Clim Change 85:19–31. https://doi.org/10.1007/s10584-007-9315-7
Rosenbaum SE, Glenton C, Nylund HK, Oxman AD (2010) User testing and stakeholder feedback contributed to the development of understandable and useful summary of findings tables for Cochrane reviews. J Clin Epidemiol 63:607–619. https://doi.org/10.1016/j.jclinepi.2009.12.013
Rosenbaum SE, Glenton C, Wiysonge CS, Abalos E, Mignini L, Young T, Althabe F, Ciapponi A, Marti SG, Meng Q, Wang J, De la Hoz Bradford AM, Kiwanuka SN, Rutebemberwa E, Pariyo GW, Flottorp S, Oxman AD (2011) Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ 89:54–61. https://doi.org/10.2471/BLT.10.075481
Rovner DR,Wills CE,Bonham V,Williams G,Lillie J,Kelly-Blake K,Williams MV,Holmes-Rovner M,(2004) Decision aids for benign prostatic hyperplasia: applicability across race and education Med Decis Mak 24:359–366. https://doi.org/10.1177/0272989X04267010
Shailes S, Fidahic M, Vujcic M, Saric F, Propadalo I, Marelja I, Dosenovic S, Puljak L (2017) Something for everyone: journals and other scientific organizations produce a diverse variety of plain-language summaries. eLife 6:13469. https://doi.org/10.7554/eLife.25411
Sivanathan N, Kakkar H (2017) The unintended consequences of argument dilution in direct-to-consumer drug advertisements. Nat Human Behav 1:797–802. https://doi.org/10.1038/s41562-017-0223-1
Spiegelhalter DJ (2017) Risk and uncertainty communication. Annu Rev Stat Appl 4:31–60. https://doi.org/10.1146/annurev-statistics-010814-020148
Spiegelhalter DJ, Pearson M, Short I (2011) Visualizing uncertainty about the future. Science 333:1393–1400. https://doi.org/10.1126/science.1191181
Stephens SH, DeLorme DE, Hagen SC (2017) Evaluation of the design features of interactive sea-level rise viewers for risk communication. Environ Commun 11:248–262. https://doi.org/10.1080/17524032.2016.1167758
Stone ER, Bruine de Bruin W, Wilkins AM, Boker EM, MacDonald Gibson J (2017) Designing graphs to communicate risks: understanding how the choice of graphical format influences decision making. Risk Anal 37:612–628. https://doi.org/10.1111/risa.12660
Sullivan SM, Wells G, Coyle D (2015) What guidance are economists given on how to present economic evaluations for policymakers? A systematic review. Value Health 18:915–924. https://doi.org/10.1016/j.jval.2015.06.007
Synnot AJ, Lowe D, Merner B, Hill S (2017) The evolution of Cochrane evidence summaries in health communication and participation: seeking and responding to stakeholder feedback. Evid Policy. https://doi.org/10.1332/174426417X14942367785260
Trevena LJ, Zikmund-Fisher BJ, Edwards A, Gaissmaier W, Galesic M, Han PKJ, King J, Lawson ML, Linder SK, Lipkus IM, Ozanne E, Peters E, Timmermans DRM, Woloshin S (2013) Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak 13(Suppl 2):S7. https://doi.org/10.1186/1472-6947-13-S2-S7
Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brożek J, Davoli M, Flottorp S, Harbour R, Hill S, Liberati A, Liira H, Schünemann HJ, Rosenbaum S, Thornton J, Vandvik PO, Alonso-Coello P (2013) Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci 8:6. https://doi.org/10.1186/1748-5908-8-6
UK Department for Transport (2017) Updated appraisal report: airport capacity in the South East: moving Britain ahead, https://www.gov.uk/government/publications/airport-expansionupdated-cost-and-benefits-appraisal
UNEP (2011) The Democratic Republic of the Congo: Post-conflict environmental assessment—synthesis for policy makers, http://wedocs.unep.org/handle/20.500.11822/22069
Vanclay F (2015) Social impact assessment: guidance for assessing and managing the social impacts of projects. Purpose and intended readership. International Association for Impact Assessment, Fargo, ND, USA
Vandvik PO, Otto CM, Siemieniuk RA, Bagur R, Guyatt GH, Lytvyn L, Whitlock R, Vartdal T, Brieger D, Aertgeerts B, Price S, Foroutan F, Shapiro M, Mertz R, Spencer FA (2016) Transcatheter or surgical aortic valve replacement for patients with severe, symptomatic, aortic stenosis at low to intermediate surgical risk: a clinical practice guideline. BMJ 5085:i5085. 10.1136/bmj.i5085
Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gülmezoglu AM, Souza JP (2013) Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst 11:19. 10.1186/1478-4505-11-19
Wilson PM, Petticrew M, Calnan MW, Nazareth I (2010) Disseminating research findings: what should researchers do? A systematic scoping review of conceptual frameworks. Implement Sci 5:91. https://doi.org/10.1186/1748-5908-5-91
Woloshin S, Schwartz LM (2011) Communicating data about the benefits and harms of treatment. Ann Intern Med 155:87. https://doi.org/10.7326/0003-4819-155-2-201107190-00004
Yager TR (2017) 2014 Minerals yearbook. Kinshasa, Congo, U.S. Geological Survey
Thanks to Jessica Gasiorek and other colleagues. Funding was provided by the David and Claudia Harding Foundation and the Winton Charitable Foundation.
The authors declare no competing interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
About this article
Cite this article
Brick, C., Freeman, A.L.J., Wooding, S. et al. Winners and losers: communicating the potential impacts of policies. Palgrave Commun 4, 69 (2018). https://doi.org/10.1057/s41599-018-0121-9
This article is cited by
Nature Food (2020)