Introduction

Assessments of long-term global mitigation pathways have taken a central role in the Intergovernmental Panel on Climate Change (IPCC) Working Group III (WGIII) reports on climate mitigation1,2,3,4. The focus is generally on the implications of long-term objectives (e.g., climate targets) for short- and medium-term system transformations and the associated greenhouse gas emissions and climate response1. These assessments are generally based on quantitative scenarios, which are integrated descriptions of a hypothetical future of the human–environment system, capturing interactions and processes such as population, economic activity, technology, and policy, and their consequences on energy use, land use, and emissions1. Quantitative mitigation scenarios assessed by the IPCC have been compiled into databases, aiding data storage and archival, and the assessment itself. Since the WGIII Fifth Assessment Report5, a scenario database has been curated by the scientific community, formally under the Integrated Assessment Modelling Consortium (IAMC), to facilitate the assessment process amid a growing number of global mitigation scenarios6. From the outset of the WGIII Sixth Assessment Report (AR6), a call for mitigation scenarios was issued jointly by the IAMC and the International Institute for Applied Systems Analysis (IIASA)7, and supported by IPCC WGIII8. The AR6 Scenarios Database facilitated the assessment in the chapter on Mitigation Pathways Compatible with Long-term Goals1 (Chapter 3), but was also used in other chapters. The AR6 scenarios database is downloadable via a scenarios explorer hosted at IIASA which provides easy access to the data and also facilitates interactive exploration of the data using tools designed by the community3.

The AR6 scenarios database clearly has advantages, but the method used to curate the database makes it subject to biases when analysed. Scenario data can come in a variety of formats and definitions, and compiling the underlying data and metadata in a central and harmonised database greatly simplifies the assessment process and is of immense value to the scientific and user community after publication9. However, the method used to curate and process the database may have important implications for analyses based on the database. The database is curated based on ad-hoc and voluntary submissions, and consists of scenarios from individual studies and a diverse set of Model Intercomparison Projects (MIPs), which generally apply the same scenario protocol across different models2. Most quantitative scenarios are generated by process-based Integrated Assessment Models (IAMs), with other model types only sparsely represented. Furthermore, it can not be claimed that the large number of scenarios submitted to the database cover the entire solution space2. Additional processing and metadata may further exacerbate biases. As a consequence, the AR6 scenarios database is not a random statistical sample representative of the hypothetical model-scenario space. Analyses based on such a non-representative database may be incomplete, misleading, and biased10,11,12. The purpose of this article is to identify the risks of using a scenario database based on non-random sampling and to outline ways where a community-driven approach incorporating community tools and peer review processes could help mitigate such risks.

The AR6 scenarios database and its application

The AR6 scenarios database3 was curated by the scientific community based on a joint request from the IAMC and IIASA7, and supported by IPCC WGIII8. The submitted scenarios data was managed and processed by members of IPCC AR6 WGIII Chapter 3 author team, but the database was closed to the public until the time of publication of the IPCC WGIII report. Authors in other chapters and expert reviewers had access to the database on request, and some scenarios data is already available via published literature. We briefly summarise five phases of the curation process: (1) submission, (2) pre-processing, (3) vetting, (4) climate assessment, and (5) metadata.

Submission

Submissions are voluntary and come from various MIPs and individual studies. The call for scenarios was published by the IAMC and IIASA and thereby naturally captures scenarios based on process-based IAMs, but may unintentionally exclude scenarios developed by researchers outside of that community13. Formally, submissions “should constitute an emissions trajectory over time with underlying socio-economic development until at least the year 2050 generated by a formal model…” and “integrated global scenarios that cover emissions from all sectors and regions with a time horizon between 2050 and 2100 is particularly encouraged, but submissions of emissions scenarios for individual regions and sectors are also welcomed”7. There were additional calls for national scenarios and sectoral scenarios. The submission occurs via a reporting template developed by the IAMC and is often used in IAM project consortia. Use of the template can be a burdensome process for new entrants or for those with models with variables that do not clearly fit into the template. The scenario needs to meet the IPCC requirements for academic peer reviewed literature or eligible grey literature.

Pre-processing

After basic quality control the scenario formally becomes a part of the database. While this step does not modify the scenarios, it is an important and time consuming step to ensure the quality of the database.

Vetting

A vetting process of all submitted scenarios was undertaken in AR6 to flag scenarios with emissions and energy data that were within reasonable historical ranges. If scenarios did not meet this historical vetting criteria, they were retained in the database, but were not assessed in AR62. Additional vetting on future trends, such as the level of carbon capture and storage in 2030, was applied to a subset of ‘Illustrative Mitigation Pathways’.

Climate assessment

Scenarios that had sufficient emissions quantifications until 2100 (natively reported CO2 from fossil fuel and industry, CO2 from agriculture, forestry and other land use, CH4, and N2O), and passed the vetting criteria, were harmonised with historical data, infilled for missing data if needed, and then received a climate assessment consisting of estimates quantities such as concentrations, radiative forcing, temperature, and ocean heat content10,14. It is the vetted mitigation scenarios with a climate assessment that form the basis of the chapter on long-term mitigation scenarios in the AR6 WGIII report1.

Metadata

Using information provided during scenario submission (model and scenario details), outcomes of the vetting and climate assessment, and summary statistics of some scenario data (such as year of net zero emissions), metadata is compiled.

After these five steps, the scenario database is complete. The database remains closed, only accessible by registered IPCC authors and upon requests received from IPCC expert reviewers, until the public release of the WGIII report.

The final database consists of thousands of scenarios, each with potentially hundreds of variables and tens of regions. Figure 1 shows a histogram of the numbers of scenarios submitted, passed vetting, and receiving a climate assessment, based on the model family (panel a) and the MIP (panel b). It is evident in these figures that there is a considerable model and MIP bias in the database. These biases are primarily introduced at the submissions stage, but are exacerbated in the vetting and climate assessment stage (Fig. 1). Most scenarios submitted came from modelling groups that are historically well versed in the IPCC process and the IAMC reporting templates4,13, noting that there has been a large increase in the number of contributing models in the latest assessment cycle. Nonetheless modelling communities, whether from academia, industry, or civil society, may not know about the call for scenarios, not feel the call is for them, not feel compelled to submit scenarios, or simply do not feel they have ownership to the process and its outcomes. Even if all published scenarios were hypothetically submitted, biases will remain, as the underlying scientific literature does not span all relevant models or scenario outcomes11. The database is therefore often referred to as an ‘ensemble of opportunity’, and is recognised as not being a meaningful statistical sample of the hypothetical scenario or models space1,5,9. Despite this, it is often used as a random statistical sample in IPCC Assessment Reports, such as providing median and percentiles of relative emission reductions or energy system changes (e.g., coal, oil, and gas).

Fig. 1: The distribution of scenarios across model family and project.
figure 1

The number of scenarios submitted, vetted, and vetted with a climate assessment for model family (panel a) and model intercomparison projects (MIP) (panel b). A total of 2304 scenarios were submitted to the global emissions database (total of each column), of these, 618 did not passing vetting for sufficiently consistency with historical energy and emissions data (yellow) and a further 484 did not have sufficient data to perform a climate assessment (blue), leaving a total of 1202 used in the primary assessment of scenarios1,2.

Full size image

The CMIP database and applications

The WG I and II communities similarly curate a major database to facilitate research and the assessment process. Here we draw analogies to the Coupled Model Intercomparison Project15 (CMIP) used by the physical climate science community, which is a project of the World Climate Research Programme (WCRP)’s, specifically its Working Group of Coupled Modelling (WGCM). The CMIP approach is based on coordinating climate model experiments involving multiple international modelling teams. While IAMs are more diverse in their structure than climate models, the CMIP process may offer valuable insights considering the larger ESM community and the extensive technical support available in the long-established climate modelling domain.

CMIP defines common experiment protocols, forcings and output applied through standardised, open, and peer-reviewed MIPs, as well as Diagnostic, Evaluation and Characterisation of Klima (DECK) experiments. A participating modelling group must contribute to the common set of DECK experiments to enable ensemble-wide validation. Beyond this, groups can participate in peer-reviewed MIPs, which explore areas of scientific or societal interest. Submitted scenarios are made public from the time of submission. This process has grown over time, with an ongoing DECK archive and 23 endorsed MIPs in CMIP6 (6th phase of CMIP), in addition to other MIPs that are not formally associated with CMIP6.

There are similarities between CMIP and the AR6 scenario database practices in that both have standardised data submission and preprocessing, consisting of common agreed data formats, and common analysis platforms (e.g. IIASA Scenario Explorer for the mitigation database or the Earth System Model Evaluation Tool16 (ESMValTool) for CMIP). Both include centralised efforts to ensure quality control in the submitted experiments17—version control, naming conventions and metadata requirements. Though, CMIP has performance metrics and tailored diagnostics that are used to enhance quality control.

CMIP takes an active role in directing the scientific community’s efforts to focus on specific scientific questions. These questions are grounded in WCRP’s so-called Grand Scientific Challenges that represent areas of emphasis in scientific research, modelling and analysis of high importance now and in the coming decade. For example, ScenarioMIP provides the framework for exploring a reduced set of future socioeconomic scenarios for Earth System Models applications18 and was used extensively in IPCC AR6 WGI, or Cloud Feedback MIP which conducts idealised experiments to understand processes controlling cloud feedbacks and climate sensitivity19. These MIPs can be endorsed by CMIP6 if they are supported by a minimum number of modelling centres, aim to address at least one of the specific scientific questions, meet predefined CMIP standards and conventions, and provide an analysis plan describing how it will address the science questions.

The preprocessing, vetting and participation requirements set within each MIP, but also by CMIP on the whole (DECK), provide a structured database and facilitates robust assessments. The CMIP process is not without problems. The database is still not a random statistical sample given that even large scenario ensembles can not fully explore the space of possibilities, and hence biases can not be avoided. The institutional and computational resources required for implementation are significant. The number of MIPs has grown with each generation of CMIP—creating resource challenges for new and existing groups to participate. Furthermore, MIPs can arise due to novel scientific questions20,21 or unexpected events22, and the rigid structure of approving MIPs may make it hard to get new initiatives moving sufficiently fast. There are clear trade-offs, with a big process requiring some structural organisation, but the organisation potentially limiting creativity and flexibility. However, some of these lessons can help guide recommendations for the evolution of mitigation scenarios databases.

A new generation mitigation scenarios database

Building on the AR6 scenarios database and CMIP process, we identify four key areas for improvement leading to the next generation of mitigation scenarios database: (1) Purpose, (2) Organisation, (3) Submission and processing, and (4) Participation and inclusivity.

Purpose

We propose that the next generation of the mitigation scenarios database should be based on a community effort with a more active role in curating the contents of the database. We suggest that an increasing effort should be used to curate the database around open and peer-reviewed MIPs that attempt to address policy-relevant areas of scientific inquiry. The MIPs would be peer reviewed before formal submissions to the MIP start, to ensure a more diverse feedback on the research questions, scenario protocols, and expected outcomes. This will incentivise the scientific community to focus on knowledge gaps and knowledge needs, with the community providing the checks and balances through peer review. The submission of individual studies should still be encouraged, to facilitate future research and assessments, and fill specific knowledge gaps. This means that the database itself will not be a random statistical sample of all hypothetical scenarios, and thereby more effort will be needed by the scientific community to ensure that biases in the database are addressed through community tools and peer review. Best practice guidelines may be needed to ensure, for example, that MIPs are not inappropriately weighted together to give misleading statistical results.

There are many examples of MIPs that would be relevant, and to illustrate our point, we give several examples. One of the sticking points during WGIII AR6 approval was how equity considerations were captured in mitigation scenarios23, issues which could be addressed by a diversity of models and approaches by an ‘Equity MIP’.There is an emerging interest to address both shorter (e.g., climate finance) and longer time horizons (e.g., beyond 2100 to capture Earth System feedbacks). A blind spot in existing scenarios databases are societal, geopolitical and climatic tipping points and disruptions, some of which may require novel techniques to analyse11. Around half of the scenarios submitted to the AR6 scenarios database came from one study24, which had a scenario design where the outcome was to push the year of net zero greenhouse gas emissions back in time25, but there has not been a MIP focussing on limiting carbon dioxide removal or focussing on demand-side measures. It is also possible to utilise standardised experiments and model diagnostics26, in a way similar to the CMIP DECKs. The WGIII scenario database has primarily served WGIII, but more consideration is needed on how the database can serve the needs of the wider IPCC process18,27 (WGI, II, and III)—an open and ‘living’ scenario database may indeed encourage more integration between the three IPCC Working Groups. The emerging research questions are potentially endless, and therefore require an organisational structure to ensure that the needs of the policy, civil society, and research communities are met.

Organisation

Our proposals would require an overarching entity that oversees and governs these activities, and consults with the wider scientific community. This entity should ensure the diversity of modelling approaches relevant for the identified research questions. While the IAMC largely fills a similar function today, since it is based around a modelling framework (IAMs), it may not be seen as sufficiently inclusive by the broader policy, civil society, or research community. In addition to overseeing a process to curate the scenarios database, such an entity needs to consider database maintenance issues including versioning, funding, and servicing, an activity largely done by IIASA for AR6, with funding provided in part by the ENGAGE project (EU’s Horizon 2020 programme).

Submission and processing

We propose the submission process continue to follow reporting conventions adopted by the IAMC for global scenarios, but to include a broader process of community feedback to ensure there are minimal barriers to new entrants and diverse model types. Continual improvements on database curation and processing could greatly enhance transparency.

We propose a more expansive metadata to help characterise each model and scenario, improve assessments, and facilitate the identification and potential correction of ensemble biases. In addition to model and scenario specific metadata, key assumptions should also be included, covering, but not limited to, discount rates, technology assumptions, constraints on deployment, etc. Currently, many of these sorts of assumptions are difficult to find, even if they are in the public sphere2,28,29 or encoded in open source software. We note that there are trade-offs between simplifying the registration process to lower barriers for model teams while maintaining the collection of necessary metadata for a robust assessment.

We propose that quality control needs more resources to cater to the diversity of models and variables that are submitted. The submission process and quality control may ultimately need to be centralised, as currently done at IIASA. Further processing steps such as vetting, diagnostics, or climate assessment may be better tailored to the scientific question of the underlying MIP or individual study, with decentralised alternative approaches encouraged to ensure robust outcomes. To ensure community feedback and encourage improvements, these processing aspects should be transparent and preferably open source.

Participation and inclusivity

We propose that scenarios submitted under a MIP, should have the MIP protocol peer reviewed. The MIP would need clearly defined research questions with an indication of how those research questions can be answered with the MIPs. Open and peer-reviewed MIPs would hopefully ensure greater community involvement in the development of MIPs, to ensure they are relevant for a diversity of models, modelling approaches, and geographies. The MIP protocol would cover topics such as research questions, experimental design, and requirements on input data (socioeconomic, technology, energy, and emissions). This may also encourage greater community sharing of input data, to ensure it is updated, consistent, reduce the importance of historical vetting, and reduce uncertainties in the climate assessment process14. Once the MIP is published, any modelling group should be able to submit a scenario under the MIP. We believe it is important to retain the ability to submit individual studies to a database, as this is key to ensure others can painlessly access the scenario data and encourage independent and more focused studies. This approach would further emphasise the need for community tools and efforts to address bias issues in analysis and assessments.

We propose that a scenarios database should be a live and open-access database: scenarios would be publicly available from the time of submission and after passing quality control. A license should be use that allows to share and adapt with attribution (e.g., CC BY 4.0), in contrast to the current license of the AR6 emissions scenarios database which has limits on sharing a substantial portion of the database. An open license significantly broadens the scientific scope and benefits of the database, allowing a scientific literature to build around the analysis of the scenarios in the database, which is then assessed in a future IPCC report. New scientific insights may therefore come before the assessment, not after10,11,25,30. A live database is not without challenges, and may need decisions on when scenarios are also excluded from the database due to age or other criteria. Since a live database would remain an ‘ensemble of opportunity’, the onus would be on the community via community tools and peer review to ensure the database is used correctly. We believe this live and open database approach would be a key step to shift the IPCC assessments away from an analysis of the scenario database to an assessment of the scenario literature.

Concluding remarks

Comparing and contrasting the strengths and weaknesses of the processes of curating the AR6 Scenarios Database with the approach used in CMIP6, we propose that the next generation of the mitigation scenarios database should be based on a community effort with a more active role in shaping the research directions and activities of the scientific community. There should be a much greater focus on openness and transparency, to ensure scientific diversity, enhanced participation, and inclusivity. Curating a database around peer reviewed, and therefore open, scenario protocols (MIPs), is one mechanism to encourage broader participation and inclusivity. A living database, with scenario data available upon submission, will encourage a broader community effort to analyse the scenarios database and make for a much stronger foundation to future scientific assessments. A strong institutional foundation will remain important, to ensure database curation and maintenance is sufficiently resourced, and to manage interactions with a much broader scientific community. The governing body needs a diversity of voices, broader than the representation of models and methodologies in the AR6 scenarios database, including users.