Introduction

Publication of scientific literature has outstripped any individual’s capacity to read and absorb information1. Prior to the release of the Intergovernmental Panel on Climate Change (IPCC)’s sixth assessment report2, over 230,000 climate change papers (averaging 80 per day) were published (see Fig. 1), with multiple papers each day on specialised topics, such as wildfire (1.4), extreme rainfall (2.7), or drought (9.5). This “big scholarly data”3 is growing exponentially, meaning a ‘comprehensive’ assessment, one of the pillars of the IPCC, is no longer possible with the conventional review processes used in climate science4,5,6.

Fig. 1: Growing number of publications referring to climate change each year, plotted as annual average papers per day.
figure 1

All publications mentioning “climate change” or “global warming” (left), and those mentioning also “extreme rainfall” or “heavy precipitation” (right). Numbers above the bars show daily average (cumulative) papers published for each IPCC assessment cycle. FAR/SAR/TAR/AR4/AR5/AR6 = First/Second/Third/Fourth/Fifth/Sixth Assessment Report of the IPCC; Source: ISI Web of Knowledge.

Full size image

As the scale and speed of scientific publication has increased, more thorough methods for ongoing assessments of scientific developments have been adopted. Literature reviews are the most basic form of scientific synthesis, but they usually follow a narrative style and are not always comprehensive7,8. IPCC assessments are based on literature reviews, complemented by an expert’s assessment and a thorough multi-stage review process. In response to bias in literature reviews, more systematic review processes have been developed. Systematic reviews, pioneered by health researchers, are exhaustive, computer-assisted searches of all the literature relating to one research question with specific predetermined inclusion, exclusion, and reporting criteria9. Meta-analyses are a more statistical tool for summarising empirical evidence across many studies. A meta-analysis involves building consensus through integrating the findings from many studies that pose similar questions into one dataset to pool effect sizes10. However, these methods require painstaking work, are usually static analyses, are limited to specific research questions, and are not geared for policy communications.

To maintain relevance, the living systematic review approach was developed to continually integrate emerging evidence11. Systematic reviews are extremely resource intensive, even for narrowly defined research questions, therefore, machine assisted research can assist by combining human expertise and machine automation in complementary ways12. A field as diverse as climate change now requires machine automation to systematically synthesise evidence in a timely, transparent and unbiased manner4,5,6,13, though further development is still required12.

As reflected by the rapid rise in publications, research in climate change has grown and diversified. Assessment reports that engage a broader group of experts would enrich assessments by providing access to new and potentially more balanced information, for example, through more diverse case studies, or publications in different languages. New asynchronous connected technologies can help to increase equality in expert engagement with the IPCC process, by improving access from the global south and other under-represented regions, and among indigenous people and early career researchers, addressing known biases14,15. Technology platforms could be developed to automatically tackle self-citation bias for example, perhaps utilising artificial intelligence (AI) tools. This would complement and strengthen efforts by the IPCC to address biases, such as the introduction of the FAIR data principles during AR616; collaboration with international research programs, such as the World Climate Research Program (WCRP) review on climate sensitivity17; and early career review rounds by organisations such as the Association of Polar Early Career Scientists (APECS), Young Earth System Scientists (YESS) and others. New technology has the potential to augment, rather than replace, expert’s knowledge by distiling literature into easily managed groups of topics, maximising visual encoding. Asynchronous and connected technology allows the necessary scalability for large numbers of synchronous users to build an assessment collaboratively.

Furthermore, the IPCC process would benefit from the ability to make more frequent, and perhaps even continuous, assessments18, which could be in response to significant events, new scientific analysis, or socio-political discourse. This would potentially help to address concerns that some parts of IPCC reports escape thorough or consistent review19. Focussed updates on specific themes provide scientists with an authoritative voice to respond to misinformation and to reinforce key messages.

As the volume of scientific publications has grown exponentially, developments in technology have been tuned to aid scientific consensus-building. Digital libraries and academic search engines leverage state-of-the-art techniques in information retrieval, recommender systems, and natural language processing to identify tailored, high-quality publications to assist literature reviews3,20,21,22. Research assistants like elicit.org23 have enabled automated literature review, but in general, the literature regards AI tools used for generating paper interpretations such as text summarisation, automated fact-checking, and stance detection as not yet accurate enough for inclusion in high-stakes scientific public-facing texts or systems24,25,26. Large language modelling has significantly improved since this manuscript was first submitted, and the inclusion of technologies like ChatGPT27 in public-facing texts could soon be advantageous. However, these models are currently not suited to understanding complex scientific concepts, lack important context for climate change, and they do not guarantee veracity28. For example, tools such as chatclimate.ai29 and climateQ&A30 use ChatGPT to extract answers from IPCC reports, however they are not yet able to account for expert judgment by e.g. prioritising information from chapter executive summary or summary for policy makers above chapter contents. Being text-driven language models, they are unable to extract information from tables or figures. Thus, many tasks in the scientific literature review process still require expert input despite technological advances.

This paper proposes a new way in which a technological solution can assist with the problem of knowledge acquisition and creation, specifically building on the IPCC’s existing strengths and ambition to conduct its assessments on a comprehensive, objective, open and transparent basis. We introduce and discuss the experience of the ScienceBrief web platform as an example to illustrate that potential.

ScienceBrief is an asynchronous-connected (see Table 1) web application that went online in 2017. The aim of ScienceBrief was to help keep up with science, by providing scientists with the means to show (rather than tell) the state of scientific consensus on key topics using peer-reviewed publications (for a technical overview see supplementary information). The aspiration was for the platform to support major scientific assessments, including IPCC, by streamlining workflows and maximising collaboration. The platform was supported by public research funds and developed by a small team (three of whom are authors of this paper) composed of one scientist, one developer, and one content writer (from 2019), with ad-hoc (unpaid) support from several scientific and technical advisors. ScienceBrief was developed to assist scientists in their assessments of key scientific questions, by facilitating the step of reviewing a growing body of publications through expert crowdsourcing, and by providing a tool that enabled living updates as more publications arise. This experience is detailed here and shows how it could be adapted to support a revised process in IPCC AR7.

Table 1 Breakdown of working operation and scalability for differing modes of collaboration.
Full size table

In the introduction to this special collection of papers discussing the future role of the IPCC process in driving climate action, the authors propose four stylised reform agendas to enable the IPCC to maintain production of relevant and timely knowledge. We suggest that the introduction of a machine assisted technology platform could complement these reforms, to maintain comprehensive, open, and accessible reports that facilitate policy progress. For example, the technology platform could reduce the scale of prioritisation implied in the ‘Back to Basics’ mode; support dissemination of key messages in the ‘Tailored Broadcasting’ mode; or enable transparent knowledge exchange among potential partner agencies in either the ‘Orchestrating Broad Knowledge Generation’ or ‘Reflexive Learning’ modes. In all modes, the technology platform would enable the IPCC to respond more quickly with new assessments.

Role of technology to enhance scientific assessment processes

A key role of the IPCC is to provide a comprehensive assessment of the science of climate change, its impacts and future risks, and adaptation and mitigation options. Recent technological developments could be used to enhance these processes in multiple ways. ScienceBrief for example, leverages technology to help in two specific ways: firstly, by upscaling collaborations to reduce duplication and facilitate concurrent workflows, and secondly by visualising scientific consensus to help both integrate information and communicate more broadly.

Upscaling collaborations to enhance production

Research on scientific workflows has found that, when leveraged effectively, technology tools can be used to increase the speed and effectiveness of collaboration31,32,33. Two or more people working on the same task can collaborate either ‘synchronously’ (actions of one worker blocks those of another) or ‘asynchronously’ (non-blocking); and either ‘connected’ and interacting with others, or ‘disconnected’ and working alone. As shown in Table 1, there are consequences for scalability of work depending on which methods are employed.

Synchronous disconnected work occurs within a single document exchanged between collaborators i.e. only one person is active at any point in time. Synchronous connected work involves collaboration among multiple co-located contributors, with just the main contributor processing inputs. Asynchronous disconnected work enables multiple people to work concurrently, but there is a bottleneck collecting and distiling work products by e.g. the lead author. Asynchronous connected work is the most efficient mode as people work autonomously, simultaneously and independently. Each worker can see in real-time the actions of others but are free to work on different sections. Maximising use of asynchronous connected working and development of tools to facilitate such workflows increases efficiency. Historically, academic work has utilised the synchronous disconnected mode, working individually and communicating through a lead author. Recently there has been a rapid progression toward connected asynchronous methods using collaborative literature tools (e.g. Google Docs, Figma), but there are still productivity gains possible by developing connected workflows. Based on these concepts, the ScienceBrief platform set itself out to support asynchronous connected and disconnected collaboration workflows to be most efficient. The IPCC has historically worked in an asynchronous disconnected mode, with the introduction of asynchronous connected collaborations via the use of shared documents within teams of authors in its last cycle (Table 1). We will show below how the IPCC could go one step further and conduct part of its production process in an asynchronous connected mode open to the entire scientific community.

Visualising scientific consensus

Past work has demonstrated that effective data visualisations can help experts and the public absorb large amounts of evolving information34,35. When creating data visualisations for large data sets, presenting data using mental models (e.g. a mind-map) enables the efficient acquisition and distillation of knowledge by supporting cognitive functions36. The way people organise information is crucial to easily discerning patterns or recalling information, and spatially grouping related information can help us make associations more easily37. In particular, there are benefits of interactive visualisations for large datasets as people can filter data more easily and studies have suggested information recall may be higher when people can directly interact with a data source34,35,36,37,38. Visual analytics tools can aid expert assessment of big scholarly data by better representing connections between papers39,40,41. In the context of public communication, visualisations of stances (polarity) between scientific papers on key issues have been used to synthesise how well-supported are varying scientific claims42,43. Although communicating expert consensus on climate change is vital to improving public support for climate policy44,45, visualisations of scientific consensus on climate change are few.

Current organising software (e.g. Endnote, Mendeley, Zotero) permit easy searching and the addition of colour labels, but do not include data visualisation. Based on the literature, improvements could be made to assist with the organisation, visualisation, and accessibility of literature reviews. The ScienceBrief platform presents one approach to using stance-based data visualisation to communicate scientific consensus to experts and the public. Other technologies (e.g. connectedpapers.com or insightful.xyz) enable citation mapping, where lines of evidence can be traced visually by tracking papers that cite another. Specific technology developments could go further and enable linkages between IPCC assessments by the authors and the underlying literature.

Case study of ScienceBrief, 2017–2021

The first phase of ScienceBrief (2017–2019) used a review approach analogous to the IPCC, focussing on the natural carbon cycle topic. It detailed seventeen research sub-topics with a Brief (composed of a statement and summary paragraph) outlining the latest science for each sub-topic. Relevant evidence (published journal papers) was uploaded to the Brief by scientists, which were sifted for their expertise on first registration using a self-declaration coupled with an automated check of home institutions against a list of acceptable academic and research institutions. Scientists allocated a score to the evidence indicating the level of agreement with the Brief. Scores were aggregated to determine the level of scientific consensus (explained in the Supplementary Information). Additionally, experts could add their own interpretation of a piece of evidence. At this early stage, low levels of engagement by scientists meant the platform’s concept could not be demonstrated and impact was very low.

A user experience (UX) review in 2019 determined that the platform would benefit from re-design to be more streamlined, intuitive and user-friendly, to facilitate greater engagement by scientists. The UX review also highlighted the need for the publication of brief reviews to synthesise the peer-reviewed publications in a more accessible format. Platform development in 2020 optimised UX for enhanced visual cognitive efficiency. The explorer tool (Fig. 2) was developed, providing a living map of evidence that used motion and spatial reasoning through animation and physical simulation to graphically visualise the scientific consensus for each Brief. In addition, some features were added to take advantage of indirect collaboration, such as hashtags and geotags, which are topical or geographic keyword tags (metadata) applied to evidence to aid searching, categorising or grouping by keyword or location.

Fig. 2: Snapshot of ScienceBrief explorer tool that graphically maps evidence by level of agreement with the Brief (x axis) and publication date (y axis).
figure 2

Each octagon symbol is a paper labelled with publication year and lead author name. The consensus ranking (“Clear consensus”) for the Brief is shown at the top, as well as the number of pieces of evidence, the number of interpretations added and the number of contributors.

Full size image

The second phase (2020–2021) focused on adding more content of higher public interest. The “Critical issues in climate change science” topic focussed on climate change impacts, to help explain the links between climate change and extreme weather events, and improve the general understanding by the media, the public, and policymakers. To drive scientist’s engagement and media coverage, short briefing notes (ScienceBrief Reviews) were launched to summarise the evidence. Each review contained key-points in bullet form, providing clear messaging of the latest scientific understanding, followed by longer explanations summarising the evidence. Typically 2–4 subject experts were invited to co-author Reviews that were then checked by the ScienceBrief Advisory Board and peer-reviewed by an independent expert.

In January 2020, an international group of experts used ScienceBrief to quickly publish the 3-page Review “Climate change increases the risk of wildfires”46, during the 2019–2020 Australian bushfires. Quickly responding to wildfires in western North America, an updated ScienceBrief Review of the same title47 was published in September 2020, integrating newly published evidence. The rapid availability of these reviews, expedited by the technology platform, enabled a timely response to scientific questions during a period of intense global public interest. In another example, a ScienceBrief Review on the role of independent expert advisory bodies48 was developed by a PhD student and two colleagues in just 2 weeks, and published ahead of important negotiations about this topic in Europe.

ScienceBrief Reviews are short briefing summaries, clarifying the main findings among the evidence, so they are light-touch in comparison to IPCC assessments. Prior to independent review, the author team compared key findings with latest IPCC reports to ensure that, while being brief, findings were consistent (except when justified by new and more recent publications) and there were no significant gaps. A significant scaling-up would be required for ScienceBrief to support production of IPCC chapters, but our experience leads us to think this is possible, in close coordination with the IPCC secretariat and bureau. As an example, the ScienceBrief Review on wildfires47 published in September 2020, examined over 116 papers published since the fifth assessment report, working group I (AR5 WGI), finding a strengthening of the consensus that climate change promotes periods of extreme fire weather, leading to increased intensity and duration of fire seasons. The evidence acknowledged that land management practices can either compound or ameliorate climate-driven changes in fire risk, but that this alone does not explain recent increases in wildfire extent or severity. Similarly, the sixth assessment report (AR6 WGI)49 assessed with high confidence that future extreme fire weather will become more frequent at higher levels of warming, and with medium confidence that weather conditions promoting wildfires were more probable in some regions (southern Europe, northern Eurasia, western northern America, Australia)49,50. Confidence in this was limited due to the complexity of quantifying trends across different land cover and vegetation types49. AR6 also assessed that climate change was responsible for changing fire weather conditions in Western North America and Southeastern Australia50, as was noted in the ScienceBrief Review. Many of the papers cited by AR6 were also cited in the ScienceBrief Review on wildfires. The AR6 WGI and WGII assessments (see FAQ2.3 Is climate change increasing wildfire?51) built upon the Special Report on Global warming of 1.5 oC (SR15)52 and Special Report on Climate Change and Land (SRCCL)53.

ScienceBrief has focussed on summarising and communicating natural science research relevant to the IPCC’s WGI. However, the platform can also be applied to social sciences, climate adaptation and climate resilience/solutions research applicable to WGII and WGIII, or indeed, to any discipline with fast-moving, high-stakes research requiring broad consensus.

Impact

Although it is difficult to capture the direct impact of ScienceBrief, the correspondence between publications and media coverage provides an indication of plausible influence. The January 2020 wildfire review was covered globally by over 220 online articles54,55 as well as television news, driving significant traffic (4934 weekly visits) to the platform (Fig. 3). The September update was covered by over 135 articles56 including the World Meteorological Organization57. Our analysis suggests that these two reviews, together with other expert works, may have contributed to changing the media narrative for wildfire coverage. For example, BBC articles in late-2019–early-2020 began to mention a “hotter, drier climate” alongside “more frequent and intense fires”58, which by mid-2021, following publication of the AR6 WGI2, evolved into the almost systematic use of the sentence “Climate change increases the risk of the hot, dry weather that is likely to fuel wildfires”59 within BBC wildfire coverage (Fig. 3).

Fig. 3: Web site visits and mentions of related topics in the media.
figure 3

Time-series of the number of weekly visits to ScienceBrief.org (blue) and News.ScienceBrief.org (orange), Jan 2015 to Aug 2022. Note y-axis log scale. Key events annotated (red): 1 = Wildfires Review published, Jan 2020; 2 = Wildfires Update published and News.ScienceBrief.org launched Sep 2020; (yellow): 3 = Cyclones Review published, Mar 2021; 4 = Extreme rainfall Review published, Jun 2021; 5 = extreme rainfall & flooding in Central Europe, Jul 2021; 6 = IPCC AR6 WGI report published, Aug 2021; 7 = multiple Reviews published, Oct 2021; 8 = COP26, Nov 2021; 9 = European heatwave, Jun 2022. Source: Google Analytics. Time-series of the changing narrative of BBC wildfire coverage, 2015–2022. Blue bars show BBC articles mentioning: “wildfires” and “climate change”; yellow bars show articles mentioning key words: “wildfire” and “climate” and “risk”; orange bars show articles mentioning the sentence: “Climate change increases the risk of the hot, dry weather that is likely to fuel wildfires”. Key events 1, 2 and 6 as above. Grey shading represents major wildfire outbreaks: A Fort McMurray, Canada, 2016; B Mediterranean, 2017; C British Columbia, 2017; D California, 2018; E ‘Black Summer’, Australia, 2019–20; F Siberia, 2020; G Western North America, 2020; H Western North America, 2021; I Siberia 2021; J Greece, 2021; K Europe & Mediterranean, 2022.

Full size image

The timing of publication determined the strength of media coverage. Subsequent ScienceBrief Reviews, where publication did not coincide with a major event, achieved limited media coverage. However, once published, traffic is seen to peak (Fig. 3) after a major event (e.g. cyclone or extreme rainfall), with users searching for information.

Auto-evaluation

Experience with ScienceBrief highlights key potential benefits for assisting in IPCC-style assessment processes. The ScienceBrief platform included the ability to quickly visualise the scientific consensus for key subjects and highlight any controversies or research gaps. In contrast to a static report, ScienceBrief was updated in real-time to keep up with the science and enable a timely response to key events, emerging science or misinformation. The ScienceBrief platform allowed interactions at varying levels of detail, with the Brief, expert interpretations, and ScienceBrief Reviews offering incremental depths of information. Users could also link to the journal paper at source. ScienceBrief enabled participation from a broad body of experts from all countries and career stages, complementing efforts by the IPCC in AR6 to include early career reviews. Advisory and Editorial Boards were established to fulfil quality control tasks ensuring the evidence and interpretations were representative of the underlying research. Experience with ScienceBrief highlighted a generally low level of engagement by experts, unless specifically recruited to contribute to the process. The number of page visits suggests better uptake by users reading content, than by contributors. Barriers to engagement would need to be addressed by incentivising contributors in different ways, such as specific solicitations and community recognition for their input. A further challenge was the difficulty attracting co-authors to participate in ScienceBrief Reviews in a timely manner, while demonstrating that timeliness was critical to ensure broad exposure of the scientific insights.

Lessons learned: how technology can enhance the IPCC process

While working on a small scale compared to the IPCC process, the ScienceBrief experience illustrates that this type of collaborative technology platform could assist and enhance the assessment process but not replace it (and therefore not replicate it). This experience suggests potential future directions for developing and using technology to assist in IPCC and other scientific assessments. First, the design of such a platform would need to be engineered to meet the specific needs of the assessment, with a clear structure mapping directly to the chapters of each working group, to ensure evidence is uploaded at the relevant point. Additionally, current guidelines on effective public science communication and policy takeaways should be integrated in the platform.

Second, the platform would need to be accompanied by systemic incentive structures to successfully engage experts. Even in instances where technology platforms are novel and meet distinct needs, lack of contributions and few rewards for participation are known to hinder the formation of long-term online communities60,61. Previous examples show experts can struggle to adopt new technologies that require adjustments to current processes or are time-consuming to learn62. To sustain long-term expert contributions, technology platforms must embed incentive structures that are aligned with altruistic goals. These incentives can be grouped into formal processes, such as a call for evidence, or rewards for participation, including additional exposure. Though in the scientific cadre, incentive structures are known to pose significant problems to desirable outcomes like contributing to open science, completing replication studies, or providing quality reviews63,64,65. In order to ensure success, the IPCC would need to actively manage the risk of low participation. This could be done for example by establishing a new author status for those contributing through the platform or an expansion of the Contributing author status, acknowledging all (or large) contributors in the reports, publicising calls for participations, handing out prizes, and targeting, training and supporting specific groups to overcome gaps for scientific information specific to regions or topics.

Third, state-of-the-art machine learning and natural language processing techniques could be integrated to lower barriers to expert contribution, increase effectiveness of the platform, and make the platform more self-sustaining. Current capabilities could be enhanced to automatically detect and upload new evidence and trigger an invitation to experts to add their interpretations. As experts are engaging with the platform, further integration of AI technologies can help to cluster papers with common themes and perspectives, improving the usability of the platform. While ScienceBrief did not use extensive AI technologies, a workflow for including expert summaries and labels has been implemented. Finally, while the published literature concludes that automated text generation is not yet advanced enough to automatically and reliably create summaries or interpretations26, recently developed AI tools have made significant progress. Once integrated, the platform could suggest interpretations and even stances, which experts can minimally edit. In other high-stakes scenarios, suggest-and-edit models like these have been shown to save time while maximising contributions66.

The use of a collaborative technology platform would enable a more open and modern production process within IPCC that enhances transparency further (Fig. 4), with the successive production of intermediate chapter drafts produced by the authors and their review by experts and governments. The use of a technology platform could be introduced after the first iteration among authors, which would focus on identifying topical issues and associated uncertainties and preparing the call for expert’s contributions, which would be done openly on the technology platform. The same cycle of iterations as currently exists can then follow, with the difference that the author’s revisions of their main chapter texts would be public and openly scrutinised, and only the executive summaries would remain reviewed offline. Production of the summary for policy makers (SPM) would follow existing IPCC procedures, based on evidence reported within chapters. The content of the platform could then also be updated continuously after the assessment report has been published, therefore providing a means by which new findings can be contrasted with existing IPCC statements, and in time facilitating updates.

Fig. 4: Current and proposed process of production of an IPCC Chapter.
figure 4

The main attributes of the proposed process are described at the bottom. All processes in blue are publicly visible, whereas those in black are not, though results are subsequently made available.

Full size image

An intermediate and complementary development could be to set up the technology platform to serve the scoping stage of the reports, where new questions are considered, along with scientific advances and policy relevance. This stage would take place ahead of the assessment cycle and could assist in identifying, organising and clustering new and emerging findings. Such a step could serve as a test for the platform, which could then be adjusted as a function of the experience.

Such a system would have multiple advantages, including enabling a more inclusive process supporting the contributions from early career and scientists from across the world, the easier access of information presented within the core of the reports which is fragmented into small, self-contained documents, and the facilitation of consensus reaching (Fig. 4). There are also new risks, including the potential lack of participation, or at the other end the excessive participation of some experts potentially leading to biases. However these risks can be managed with modification of an established vetting system and operational and verification rules, which should be defined and agreed by the IPCC Bureau. A further challenge comes from the integration of grey literature, which cannot be identified by an automated process. Some provision of manual verification and vetting of the quality of the input evidence would need to be anticipated, but nothing above what is already done in current assessments. Finally, it is unclear if the use of a technology platform would overall reduce the burden to the authors, or simply shift their workload from time spent on assessing the literature, to time spent on assessing the community’s input. The benefits lie more in the help that the platform provides for managing a growing literature-base and engaging more broadly, than in the reduction of the burden.

Conclusion

There are numerous ways technology can assist with the synthesis and dissemination of scientific knowledge. As the pace and need for science output increases, scientists require tools and methods to increase the efficiency with which information can be processed. We suggest taking advantage of more efficient mental models as well as increasing the scalability of collaboration to do so. The ScienceBrief platform has been introduced as an asynchronous-connected tool with a focus on visual interfaces, in order to assist with the task of keeping up with science. ScienceBrief is an example of a collaborative technology platform that could benefit the IPCC by assisting knowledge extraction from a rapidly expanding and overwhelming scientific literature; engagement of expertise from diverse geographic locations and career stages; and both regular and rapid assessment of scientific consensus on specific topics. Further function development and adjustments targeted to assist in production of IPCC assessment reports, agreement of quality-control processes, as well as enhanced utilisation by the scientific community, would amplify these benefits.