Introduction

To make an informed decision, individuals need to weigh up the potential impacts of different options. Citizens or policy-makers evaluating a policy are rarely given a balanced and clear summary of the potential outcomes of their decision. Providing such summaries requires: (1) learning what impacts to consider and identifying the most important effects, (2) gathering the evidence, and (3) communicating that evidence such that it is understood. The first two stages are well-defined by processes within governments and organizations (e.g., HM Treasury, 2011), but we review the third step and find little evidence for how to effectively communicate multiple policy options.

For individual decisions about personal health or finances, much is known about how to present potential harms and benefits to effectively inform rather than persuade (e.g., Fischhoff et al., 2011; Spiegelhalter, 2017). However, policy decisions including tax and economic policies, corporate strategy, environmental and educational policies, etc. are particularly difficult. Far more than individual decisions, policy decisions often affect many people and even other species, impact different groups in different ways, and cause multi-generational outcomes. Because weighing up options for an informed policy decision is difficult, it is particularly important to assist policy decision-makers with concise, clear, and comprehensive descriptions of the potential outcomes of policy options. There can be profound costs when communications fail to inform decision makers. For example, lack of clarity about how a tax policy will affect different groups can lead to the adoption of policies that worsen inequality.

Policy decision-makers are a broad group, including corporate board members deciding on corporate strategy options, government ministers appraising policy options, and voters considering options in a referendum. An equally broad set of groups attempt to provide these decision-makers with clear, non-partisan communications, including civil servants, corporate consultants, public service broadcasters, and non-governmental organizations (NGOs). However, despite the importance and ubiquity of policy option communication, nearly all guidelines for designing effective communications appear based on the research into individual-level decision-making, e.g., deciding on a medical treatment.

How best to summarize the potential outcomes of policy-level decisions in simple and balanced formats is currently unknown. We review the current communication guidance below, and find it vague and insufficient to support policy decisions. Because decision makers have finite attention and cognitive resources, presenting too much detail may cause them to disengage or misunderstand. However, too little detail can leave individuals less able to identify the options most likely to achieve their goals. Excellent reviews describe how to communicate risks (Fischhoff et al., 2011; Pidgeon and Fischhoff, 2011) and communicate research findings (Wilson et al., 2010), but there is little general advice for explaining the expected consequences of policy options.

We propose a new research area to support policy option decision-making. This paper focuses on balanced communications: messages that allow recipients to weigh up potential benefits and harms of different policy options to different groups. Overtly persuasive communications such as advertising or lobbying materials are beyond our scope as their primary goal is to make some outcomes appear more attractive. These persuasive materials avoid the key challenges of balanced communication by selectively highlighting and omitting expected harms and benefits.

Overview

This scoping review (Gough et al., 2012) aims to assess how the potential harms and benefits of policy options are being communicated and to synthesize the advice and evidence for designing such communications. First, we highlight lessons from existing work on individual-level communication. Then, we describe three new reviews of: (1) current policy-level communications, (2) communications guidelines, and (3) evaluations of policy-level communication. Last, we critique examples of current policy-level communications based on our findings. We end by calling for a new area of empirical research on policy option communications and by suggesting research priorities. To introduce the type of policy option communications considered in this paper, Fig. 1 shows three current examples.

Fig. 1
figure 1

ac Example communications of potential policy impacts. a This U.K. National Farmers Union Fig. (NFU, 2016) shows the impacts of three scenarios (policy options) following Britain’s exit from the EU and their expected effects on different sectors of the agricultural industry. This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of U.K. National Farmers Union; copyright © U.K. National Farmers Union, all rights reserved. b This table was adapted from the U.S. Congressional Budget Office (CBO, 2014) and illustrates the effects of two possible minimum wage policies on employment and workers at different income levels. Note the use of a table for ease of comparison across options, and the dilemma of comparing different metrics (earnings vs. counts of workers). This figure is covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of U.S. Congressional Budget Office; copyright © U.S. Congressional Budget Office, all rights reserved. c The U.K. Institute for Fiscal Studies communicates the effects of two policy options (to reform or not) on poverty across geographical regions and overall (Hood and Waters, 2015). Visual grouping is by region rather than by policy as in (a) and comparisons between policies is facilitated by an additional scale (the black dot). This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of U.K. Institute for Fiscal Studies; copyright © U.K. Institute for Fiscal Studies, all rights reserved

Lessons from individual-level communication

The robust research into individual-level communication is likely to apply also to policy-level communication. Individual- and policy-level communication overlap rather than form separate categories. For example, when explaining the potential harms and benefits of an ostensibly individual-level decision such as vaccination, communications may also describe potential benefits to others through herd immunity (Betsch et al., 2017). On the other hand, the expected impacts of large-scale flood protection programs may be communicated at a societal level through maps that can then be used for individual decision-making by focusing on specific locations.

In addition, communications designed to support individual decisions often contain illustrations that actually focus on groups. The standard recommendation for individual risk communication (Gigerenzer et al., 2007) is to avoid terms such as the “probability” or “chance” of events happening to a specific person (Trevena et al., 2013), and instead to embed an individual in an imaginary group of similar people, e.g., “of 100 people like you, we would expect 12 people to have a heart attack or stroke in the next 10 years.” In policy decisions, the groups of people are real rather than hypothetical.

In Table 1, we summarize the literature on effective communication of individual-level outcomes, which has been excellently reviewed elsewhere (McInerny et al., 2014; Trevena et al., 2013). In the absence of evidence to the contrary, we suggest that these guidelines are likely to apply to policy-level communication. However, the communication of potential policy outcomes introduces additional challenges which these guidelines do not address.

Table 1 Summary of the literature on individual-level communication

Policy-level communication, guidance, and evaluation

In this section, we describe three reviews on communications that contain policy options and potential outcomes: conceptual syntheses of existing communications (Review 1) and of current guidance (Review 2), followed by an aggregative review of communication effectiveness (Review 3) (Fig. 2) (Gough et al., 2012).

Fig. 2
figure 2

Structure of the current paper

Method

Review 1: Current communications

First, we collected examples of print and online communications that attempted to communicate the outcomes of different policy choices. These were produced by media organizations, governments, corporations, NGOs, and scientists for a range of audiences from the lay public to professional policy-makers (Figs. 1 and 36). We focused on examples where the intention appeared to be to inform rather than to persuade by excluding examples from charities, lobby groups, and overtly partisan political campaigns. The majority of these examples were from the grey literature, i.e., produced by governments, academics, corporations, and the media, and unmediated by academic publishers or peer review (for another review of the grey literature, see Davidson, 2017). Therefore, this was a configuring review that used iterative methods to reveal emerging concepts (Gough et al., 2012).

Fig. 3
figure 3

ac Trade-offs between coverage and comprehensibility. a (IPCC, 2007, p 805) illustrates an attempt to present multiple emissions scenarios (columns) and multiple outcomes (rows). This detail may lower comprehensibility (NFU, 2016). The inclusion of additional, less-diagnostic information may reduce the impact of more important data (Sivanathan and Kakkar, 2017). This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of IPCC; copyright © IPCC, all rights reserved. b (UK Department for Transport, 2017) converts multiple outcomes related to airport expansion to a common metric to allow comparison in a single table. This is attractively simple but obscures heterogeneity. For example, individuals living near vs. distant the proposed runways are likely to want information about their specific neighborhoods. This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of UK Department for Transport, 2017; copyright © UK Department for Transport 2017, all rights reserved. c (HM Government, 2016) is a U.K. Brexit communication from a Government department (not from the specific campaign organizations), in which simplicity comes at the expense of coverage of the issue. This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of HM Government; copyright © HM Government, all rights reserved

Fig. 4
figure 4

The IPCC communication of complex climate impacts of different emissions scenarios to policymakers (IPCC, 2014, p 29). This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of IPCC; copyright © IPCC, all rights reserved

Fig. 5
figure 5

a, b Summary of the evidence for potential benefits (a) and costs (b) of introducing Dialogic Teaching to a school, from the Educational Endowment Foundation (EFF, 2017a). This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of the Education Endowment Foundation; copyright © Education Endowment Foundation, all rights reserved

Fig. 6
figure 6

a–c The MAGIC group communication of two options for a healthcare intervention (treatments for patients with severe aortic stenosis) (MAGIC, 2017). The database allows presentation of the information as: a a summary for policymakers, b in detail for a particular group, and c further information to support an individual decision. These images are extracts from larger visualizations (Vandvik et al., 2016). This figure is not covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of MAGIC; copyright © MAGIC, all rights reserved

Second, we used online tools including Google image and Google text searches using the term communicat* combined with: evidence, policy/ies, graph, option, impact, assessment, report, or scenario. We also searched the government websites of provinces/states and countries where referendums are common, because these governments regularly need to communicate policy options in a balanced way to citizens. We additionally searched the websites of NGOs involved in evidence communication (e.g., Cochrane and Campbell Collaborations, What Works Foundations, American Clearinghouse, U.S. Congressional Budget Office, Intergovernmental Panel on Climate Change [IPCC]), and solicited examples from our personal networks of scholars and industry professionals focused on risk, communication, and/or policy.

The examples that we found mostly concerned health, finance, business, transport, education, ecology, migration, war, and terrorism. They spanned multiple formats from single graphics to lengthy policy documents, and included numeric tables, static representations such as bar graphs, interactive graphs and video visualizations, and narratives (text) that described expected outcomes from policy decisions.

Review 2: Current guidelines

In the second review, we gathered organizational guidelines on communicating policy options. The inclusion criteria were guidelines that specifically address how to design or present communications of evidence to inform a policy-level decision. Therefore, we excluded advice on communicating individual-level options such as personal medical treatments. The second review was also configurative and designed to reveal emerging concepts.

We searched for and requested documents from a range of governments and NGOs. As in Review 1, we included governments and constituencies with referendums (Austria, California, Canada, EU, Germany, Switzerland, U.K., & U.S.). We requested guidelines from NGOs including international accountancy firms and management consultancies (PwC, Deloitte, EY, KPMG, McKinsey, Boston, Accenture, & Cap Gemini). We also searched for guidelines produced by libraries of systematic reviews including the Cochrane and Campbell Collaborations (Campbell Policies, 2016; Cochrane Collaboration, 2011), the International Association for Impact Assessment (Vanclay, 2015), and the International Initiative for Impact Evaluation (3ie). We additionally reviewed guidelines on presenting future scenarios where policy options may not be explicitly communicated, such as the U.K.‘s Financial Reporting Council guidelines on writing corporate strategic reports, which summarize past and future potential performance of a company (Financial Reporting Council, 2014).

As an example of our discovery process for policy option communications, within the U.K. we identified governmental bodies and reports involved with evidence communication (e.g., Department for Environment, Food & Rural Affairs; Parliamentary Office of Science & Technology; National Risk Register), and the groups authoring governmental workflow documents such as the Magenta Book (HM Treasury, 2011) and the U.K. Evidence for Policy and Practice Information (EPPI, 2017). We then searched their websites for documents and reports containing instructions or guidance on communicating policy options, including advice on audience, report length, and communication format (e.g., visual aids).

Review 3: Empirical evidence

Third, we reviewed the academic literature for empirical evaluations on comprehension or knowledge of policy-level communications (definition below). Because policy option communication occurs across such diverse fields, we began with a configural review to identify the key terms, sub-literatures, and concepts, which we then used in an aggregative review to summarize the state of evidence. We searched Web of Knowledge and Google Scholar separately for all peer-reviewed research published by December 2017 using combinations of these terms: communication policy option outcome risk evidence decision evaluation. At Google Scholar we used the full-text search, and at Web of Knowledge we searched under “topics”.

The titles and abstracts from both searches were then reviewed for whether articles described the review or evaluation of the effectiveness of policy option communication on comprehension, knowledge, or similar outcomes (whether subjective or objective). We then reviewed the titles and abstracts of the cited and citing papers from that set, and repeated that process for any new inclusions until no new papers were identified, which left 370 articles. We also presented this project at international conferences and discussed widely with colleagues to reveal other literature. No papers were uniquely identified by this last step. We then excluded papers that did not evaluate nor review the effectiveness of policy option communications or evidence summary formats. We also excluded empirical articles evaluating communications that were narrowly designed for individual decisions, e.g., one’s own medical care. After all exclusions, we identified 72 articles. See Review 3: Results and Discussion for a summary of this literature, and the Supplement for the 72 articles in a table categorizing each as empirical, review, methodological advance, or organizational report.

Results and discussion

Review 1: Current communications

Based on the first review, we identified four challenges that make policy option communication particularly difficult compared to individual-level communication:

Heterogeneity of impacts

Policy options generally have different impacts on different groups (e.g., geographical, sociocultural, demographical, or political). For instance, tax policy reform often benefits some income classes over others and communicating these trade-offs between groups is difficult (Fig. 1c).

The first challenge is deciding which populations and potential outcomes to describe. For example, banning the sales of new petrol or diesel cars has local environmental benefits (DEFRA, 2017; Le Monde, 2017). However, one consequence of such bans is likely to be an increase in demand for lithium-ion batteries containing cobalt, of which around half currently comes from mines in the Democratic Republic of Congo (Yager, 2017). Amnesty International and UNICEF reported widespread health risks to tens of thousands of miners in the DRC, including many children (Amnesty International, 2016), and the UN reported major environmental damage (UNEP, 2011). Citizens and policymakers may be unaware of the health and environmental risks in the DRC and it is unclear whether these risks should be communicated as a potential outcome of a policy in Europe, India, or China. There is currently no specific guidance on how to decide what outcomes to include when considering distant impacts.

Detailing all outcomes for sub-populations may reduce comprehensibility (Fig. 3a) (e.g., Rosenbaum et al., 2011), but using a population-wide average can obscure different impacts in different groups (Fig. 3b). Three common but incomplete approaches are to: (1) mention only the potential benefits or potential harms (rather than both), which may mislead recipients and constitute a persuasive message, (2) mention only one type of outcome, such as financial but not environmental impacts, or (3) describe only the main potential benefit and main potential harm. These approaches can be pragmatic but can also reduce the usefulness of communications to decision-makers. It is often undisclosed that options or outcomes were omitted, which may undermine trust. Transparency about which policy effects are considered during the decision-making process may boost the perceived fairness and objectivity of the decision process, although this claim is contested (Grimmelikhuijsen et al., 2013; O’Neill, 2002).

Multiple outcome scales

Policies and interventions have diverse outcomes that can be measured in different ways. For instance, a policy may have financial, health, environmental, and employment consequences, each with a separate metric, making it difficult to compare the outcomes. Even combining data about the same outcome is difficult because continuous outcome measures are often separated at different scale points (e.g., combining data from overlapping income groups; also see Guyatt et al., 2013).

Communicators sometimes attempt to express all the possible impacts of a policy with a single metric or common currency (e.g., Fig. 3b) that enables a direct ranking of options. For example, the UK Government’s Green Book (HM Treasury, 2003) recommends a cost-benefit analysis of options in which explicit monetary values are given to each outcome. However, conclusions based on a single metric may not be supported by public audiences nor policymakers.

For example, the Safe and Sustainable initiative was set up in 2008 by the National Health Service Commissioning Groups in the U.K. to make recommendations for concentrating services for surgery for congenital heart disease in England, at that time provided by ten centers. A multi-criteria decision analysis compared a shortlist of six options for patterns of center closures on four criteria: access and travel times for patients, quality, deliverability and sustainability. The individual scores and criterion weights were obtained by a deliberative process intended to be transparent (see Table 2).

Table 2 Example of multi-criteria decision analysis

The analysis contained no options without closures, was widely criticized, and the final recommendation (Option 1) was opposed through public demonstrations, political lobbying, and legal action. As of 2017, there were still ten NHS centers in England carrying out children’s heart surgery, including all three recommended for closure under Option 1.

Long timescales

Many individual-level decisions primarily affect the decision-maker or their family over single generations. However, many policy options such as waging war or establishing environmental protections have effects across longer time periods. The impacts of policy-level decisions may also vary through time, requiring communications that describe impacts at several time points (see Fig. 4 for climate change impacts over time).

Large uncertainties

Outcome predictions are more uncertain for policies than for individual-level decisions due to long timescales, complex economic, environmental and social systems that influence policy outcomes, and the lack of well-controlled policy experiments. Fully quantified probability distributions are precise but require the specification of all potential outcomes and are therefore rarely attainable. One feasible alternative is to present a high/low/best estimate of impact. However, our review showed that all types of uncertainty (aleatory, epistemic, and ontological; Spiegelhalter, 2017) are usually absent in current policy impact summaries (e.g., in Figs. 1, 3, 5, and 6).

These four challenges—heterogeneous impacts, multiple outcome scales, longer timescales, and uncertainty—make balanced communication of the potential impacts of policy options more difficult than communicating individual-level outcomes. It is not possible to communicate unlimited information (Cairney and Kwiatkowski, 2017), but omitting information often creates bias. Partisan communicators exploit this tension by limiting the information to highlight their preferred option, and they create very simple and easily understood messages. For the communicator attempting to be non-partisan, there is a temptation to avoid bias by including as much information as possible.

A typical example is the impact assessment for additional airport runways in South East England (UK Department for Transport, 2017). This report includes comprehensive descriptions of multiple policy options, and outcomes including heterogeneities, uncertainties, and long timescales. Comprehending this detail requires substantial literacy, numeracy, motivation, and time, and therefore excludes most audiences. Therefore, this and many reports include an overview and figures to communicate the key findings. In reducing the comprehensive report to an easily comprehended overview, potentially important details are again inevitably omitted (Fig. 3b).

A central tension in policy option communication

As shown in the above discussion, in policy option communication there is a particular tension between two factors:

Coverage

The extent to which the communication describes the most important options and their potential outcomes, e.g., who is affected, which outcomes are included, short- and long-term benefits and harms, and uncertainties.

Comprehensibility

Whether the communication can be easily understood by the intended audiences in their different contexts.

Greater issue coverage usually reduces message comprehensibility because it places greater demands on limited cognitive resources such as attention and memory (Fig. 3a) (Cowan, 2010). On the other hand, reducing coverage may improve comprehensibility at the expense of misunderstanding the decision environment (Fig. 3c). Moreover, if the information that a decision-maker feels to be most relevant is not provided, this may damage trust in the message and the sender (Boulton et al., 2012).

Review 2: Current guidelines

Research units, NGOs, businesses, regulators, and governments produce guidelines on how to collect and present evidence for policymakers. The second review collected these guidelines. We focused on specific practical advice on how to communicate potential policy outcomes from different scenarios or options.

Our review uncovered a range of similar evidence synthesis or research summary formats recommended by various guidelines, such as: evidence summary (Rosenbaum et al., 2011; Synnot et al., 2017); Cochrane summary of findings (SUPPORT Collaboration; Conway et al., 2017); plain language summary or significance statement (Cochrane; Langendam et al., 2013; Shailes et al., 2017); evidence profile (Guyatt et al., 2013); policy brief (Rajabi, 2012; cf. Beynon et al., 2012), and similar terms (Haynes, 2007; Wilson et al., 2010; cf. Glenton et al., 2010). While these formats are often co-designed with policy decision-makers to support informed decision-making and/or behavior, their content is driven by the research—e.g., the results of a systematic review or the current state of evidence in a field—not by the information needs of a decision-maker. Although these summaries grapple with some of the same challenges of policy option communication such as brevity, uncertainty, and clarity, they do not attempt to make overt predictions of the potential outcomes of specified policy options. Therefore, these summaries avoid the problem of balanced comprehensiveness in considering all potential outcomes and aggregating their different metrics. Policy briefs, although they can include policy options, are also often overtly used for partisan, lobbying, communication which again means that they avoid the challenges of balanced communication. Fact boxes (McDowell et al., 2016; Woloshin and Schwartz, 2011) are a format that do present balanced options (e.g., harms and benefits of selecting a medical treatment), but they are currently limited to individual decisions and have not been used to present outcomes for different subpopulations to support policy decision-making. The lessons learned from research into fact boxes were therefore included in the previous section on individual-level communication.

The guidelines that we found often specified what to include in a report about possible futures, e.g., the findings from all primary outcomes (Peters et al., 2007), but did not address how to balance coverage with comprehensibility. Our review showed that most guidelines stated that communications should be high-quality and clear without specifying how those goals were to be accomplished.

Some guidelines (e.g., HM Treasury, 2003) provided advice aimed at evidence collection, including the quantification of uncertainty and use of a single metric. Collapsing multiple types of outcomes such as health or environmental into a common metric, e.g., financial cost, can simplify communications and possibly improve comprehension (cf. Review 3: Results). In the rare guidelines where specific writing advice was provided to increase comprehensibility, the supporting evidence for this advice appeared missing. For example, a report from the Canadian government on how to write policy summaries (Canadian Health Services Foundation, 2017) specified the optimal length of types of summary reports but cited no evaluation evidence. A notable contrast is given by the U.S. Food and Drug Administration guide to communicating risks and benefits (Fischhoff et al., 2011), which did not claim universal advice and instead recommended user-centered design and evaluation for comprehension for all communications. Unfortunately, such evaluation in policy-level communication is hard to find, as revealed in Review 3 below.

Overall, existing guidelines on producing policy-level communications provide little advice that addresses the specific challenges of communicating policy outcomes, and rarely reference evidence for their guidance.

Review 3: Empirical evidence

This review identified empirical evaluations of policy outcome communications, and excluded studies on benefits and harms for individual decision-making such as for personal medical treatment or literature evaluating overtly partisan material used for marketing or lobbying.

Much of the evaluative work we found concerned evidence summary formats, whose content is driven by existing research rather than policymaker needs. Health policy summaries were relatively well studied, including work on the Cochrane summary-of-findings table (e.g., Glenton et al., 2010; Rosenbaum et al., 2010) and GRADE evidence tables (e.g., Carrasco-Labra et al., 2016; Treweek et al., 2013). Most studies used an audience of members of the public, but some used policymakers (e.g., Rosenbaum et al., 2011; Vogel et al., 2013). Also in this vein, a recent systematic review examined health policymakers and health system managers’ comprehension of an issue with or without evidence summaries (Petkovic et al., 2016). Six studies met their criteria for inclusion. That review concluded that summaries likely aided comprehension but was unable to conclude whether summaries affected decisions.

However, because these summaries were designed to summarize academic knowledge rather than answer the specific needs of those making policy decisions, the summaries often left out information that policymakers likely want such as financial cost. This limitation makes the summaries easier to generate and evaluate, but they do not address the full challenges of comparing policy options to support informed decisions.

One study of healthcare policy communication was notable because it evaluated four methods of communicating the potential outcomes of different mammography policies to policymakers (Brownson et al., 2011). However, the policy briefs were designed to be partisan and favor one policy option. The study tested whether the formats differed in being understandable, credible, likely to be used, and likely to be shared based on self-report (Brownson et al., 2011). Objective comprehension was not assessed, as was typical in the small evaluation literature we found. All communications were provided in booklet format and included heterogeneous impacts across the population, but they varied between data-driven or story-driven, and whether the data was state-wide or only local. Different groups of policymakers had different preferences, e.g., legislators preferred the data over narrative formats.

Outside of healthcare, climate change is the main field in which policy-level communications have been evaluated. In this domain, we consider emissions scenarios as policy options even though the scenarios represent many processes and multiple policies. Most of this work evaluated whether IPCC communications of uncertainty were understood (e.g., Kandlikar et al., 2005; Risbey and Kandlikar, 2007; Budescu et al., 2012; Budescu et al., 2014; McMahon et al., 2015). A key finding was that subtle displays of uncertainty, e.g., map stippling, are frequently missed or misunderstood (McMahon et al., 2015). A recent study suggested that the IPCC’s summaries for policymakers were too linguistically complex (Barkemeyer et al., 2015). Another study compared different visual representations for land use in response to climate change across multiple scenarios, but did not identify a single best format (Bishop et al., 2013). Others authors have called for more empirical research into climate policy communications (Pidgeon and Fischhoff, 2011).

Another area where we found multiple evaluation studies was economics. One systematic review of guidance for presenting economic evidence to policymakers found 31 sets of economic reporting guidelines and described them as generally low-quality because of a lack of systematic evidence (Sullivan et al., 2015). Efforts were made to determine the needs of policymakers and use that information to design the presentation format, but they identified a particular lack of research on communicating economic principles to non-specialist audiences.

In a different domain, one study presented the safety risks of a building policy across multiple locations. They tested eight risk communication formats and found some versions led to more comprehension but less policy support (Bruine de Bruin et al., 2013; cf. Hildon et al., 2012). This finding is common in the communication literature and bears repeating: when seeking ideal message formats, designs that lead to the most comprehension may not change beliefs or behavior most effectively (Akl et al., 2011; Stone et al., 2017). For a communicator seeking to inform rather than persuade, objective comprehension may be the most appropriate outcome measure, but how maximizing comprehension affects the decision-making process and behavior is poorly understood.

In summary, the third review uncovered some evidence for communication design features such as graphical complexity and information omission, but most of the literature on evaluating messages was on science communication rather than policy option communication, and therefore did not directly address trade-offs between coverage and comprehensibility. The existing evaluations mostly focus on healthcare policy, climate change, and economic forecasts, particularly the economics of healthcare policy. Additionally, individual-level communication work has evaluated graphical (McInerny et al., 2014) and numerical (Heller et al., 2003) methods of presenting evidence that can be applied to policy option presentations. Overall, there is little evidence on the best formats for communicating potential policy impacts.

Synthesis: balancing coverage and comprehensibility

All communicators must select what information to include and omit, but there is currently little guidance for how to select policy options and their potential outcomes while minimizing bias. The existing guidance for communicators is shallow, and when evidence-based relies on the literature on individual-level communication. Finally, there is a critical need for evaluation of communication methods to describe the potential harms and benefits of policy options.

Based on the first review, current attempts at balanced policy outcome communications typically resolve the core tension of coverage and comprehensibility by reducing coverage: omitting information that could nonetheless be important to some decision-makers. Other communications achieve high coverage only through long and complex presentations, e.g., a 67-page report on airport runway capacity options in southeast England (UK Department for Transport, 2017). Only a few policy communications appear to balance coverage with comprehensibility well in a brief summary. In Table 3, we analyze three such examples based on the conceptual framework of four challenges identified in Review 1.

Table 3 Evaluating graphical policy-level communications

The first example comes from an IPCC report on climate change (Fig. 4) (IPCC, 2014). It illustrates the expected changes in four global measures based on two future emissions scenarios, RCP 2.6 and RCP 8.5, which we consider two macro policy options.

The second example is an online Teaching and Learning Toolkit (EFF, 2017a) by the Education Endowment Foundation in the U.K., who produce guidance on specific interventions (Fig. 5) (EFF, 2017b). These communications are designed for policymakers: in this case, senior teachers or principals who implement education policies in schools.

The final example comes from health care. The MAGIC group produces summaries of the potential impacts of health interventions, and these are presented as interactive summaries online and published for use by policymakers and individual clinicians (Fig. 6) (Vandvik et al., 2016).

All three figures approached the challenges in different ways. Without user testing, it is unknown whether they were successful in communicating the information required by the target audiences. However, these figures appear to achieve high coverage while remaining concise and comprehensible.

Future directions

Individuals faced with personal choices that will affect their lives are increasingly provided with balanced, clear, and concise summaries of the likely impacts of their decisions. In contrast, such summaries are rarely available to policymakers despite the large and wide-ranging impacts of policy decisions. This gap likely stems from the difficulties in representing heterogeneous impacts, diverse outcomes with different metrics, outcomes over long time scales, and large uncertainties. However, the advances in high-quality and ethical decision support for individuals came from steady research, practical implementation, and evaluation. We hope that recognition of the particular challenges of policy-level communication will spur future research and guidelines to better support those making policy-level decisions.

Providing balanced and comprehensible summaries of detailed policy outcomes is difficult, and one unresolved challenge is how best to approach the necessary omission of information that might be important to some decision makers to keep the summary concise and comprehensible. Best practices for communication design will likely include a dialog with intended audiences to learn their interests and responses to communications within their domain (Bruine de Bruin and Bostrom, 2013), including interactive and/or layered figures and resources to achieve high coverage and personalization without overwhelming recipients (McInerny et al., 2014; Sivanathan and Kakkar, 2017; Stephens et al., 2017). However, measuring only message comprehension may not capture reactions to missing information, whose absence might not be noticed.

We found very few studies that experimentally evaluated the effectiveness of communications about policy options. In addition, these studies typically aggregated across recipient characteristics that can affect comprehension, such as motivations, numeracy, graph literacy, and demographics (see Paling, 2003; Rovner et al., 2004; Garcia-Retamero and Cokely, 2013; Okan et al., 2015). As the evidence base expands, we expect further nuance rather than the discovery of universally effective communication techniques.

The three reviews in this paper combine configural, conceptual, and aggregative methods (Gough et al., 2012) for a novel synthesis of an emerging area with great public need. The review methods nonetheless introduced limitations. Conducting the first and second reviews was difficult because much of the relevant material was in the grey literature, which is hard to scope. It was unrealistic to review every form of policy option communication used in governments and organizations around the world. A related limitation is that all of the reviews were based on English-language results due to a lack of translation capability. This limited the representativeness and generalizability of the reviews of existing communication examples and organizational guidelines. The limited scope was less likely to affect the empirical evaluation review as most international scientific journals are currently in English. However, given that the field of policy-outcome communication is not currently well-defined, search terms that would locate academic papers were difficult to exhaustively identify, meaning that the evaluation literature would likely benefit from additional reviews.

Overall, we suggest that the ideal balanced communication would provide appropriate detail in a quickly and easily understood format to help citizens and policy decision-makers apply their own values and priorities to decisions. Whether or not this ideal is achievable in a particular domain, specifying the goal can reveal the empirical questions that guide future progress. We identified four key challenges affecting the communication about impacts of policy options: broad and heterogeneous effects, outcomes with different metrics, potentially long timescales, and large uncertainties. These challenges increase the inherent tension between coverage and comprehensibility in messages. The current evidence on resolving this tension is insufficient to deliver confident recommendations to policy communicators (see Alonso-Coello et al., 2016). We hope that identifying these challenges will stimulate the development of effective, non-partisan communications and their evaluation, and thereby support citizens and diverse policy decision-makers in understanding the possible impacts of policies.

Data availability

All data analysed in this study is available in the paper.