Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Uncertainty Quantification of Extratropical Forest Biomass in CMIP5 Models over the Northern Hemisphere

## Abstract

Simplified representations of processes influencing forest biomass in Earth system models (ESMs) contribute to large uncertainty in projections. We evaluate forest biomass from eight ESMs outputs archived in the Coupled Model Intercomparison Project Phase 5 (CMIP5) using the biomass data synthesized from radar remote sensing and ground-based observations across northern extratropical latitudes. ESMs exhibit large biases in the forest distribution, forest fraction, and mass of carbon pools that contribute to uncertainty in forest total biomass (biases range from −20 Pg C to 135 Pg C). Forest total biomass is primarily positively correlated with precipitation variations, with surface temperature becoming equally important at higher latitudes, in both simulations and observations. Relatively small differences in forest biomass between the pre-industrial period and the contemporary period indicate uncertainties in forest biomass were introduced in the pre-industrial model equilibration (spin-up), suggesting parametric or structural model differences are a larger source of uncertainty than differences in transient responses. Our findings emphasize the importance of improved (1) models of carbon allocation to biomass compartments, (2) distribution of vegetation types in models, and (3) reproduction of pre-industrial vegetation conditions, in order to reduce the uncertainty in forest biomass simulated by ESMs.

## Introduction

The amount of carbon stored in vegetation biomass in terrestrial ecosystems plays an important role in influencing Earth’s climate. Variations in productivity, respiration, carbon turnover and in carbon allocation to biomass compartments in response to anthropogenic and natural climate change influence the magnitude, spatial distribution and allocation of vegetation carbon stocks. At present, the land biosphere is a carbon sink since atmospheric carbon dioxide (CO2) is increasingly sequestered in terrestrial ecosystems1,2,3,4,5. The largest pools of terrestrial carbon are within soils and the live biomass of forests6,7,8,9. Previous studies found that forests presently store 47% of total global carbon on land around the world3, and land absorbs approximately one-third of global anthropogenic CO2 emissions each year5. A recent report showed that 11.5% of total greenhouse gas emissions over the contiguous United States in 2014 were offset by land use, land-use change, and forestry10. Therefore, investigations of forest biomass magnitude and carbon allocation are critical to reduce the uncertainty of future global and regional carbon stock estimates in the Earth system models (ESMs).

Differences in parametric and structural representations of vegetation and soil processes in ESMs, as well as uncertainties in simulated climate drivers, result in a wide range of carbon stock estimates. As we will show, biases induced by such parametric and structural differences in the initial state of vegetation biomass (i.e., during the pre-industrial (PI) period following model spin-up) often persist into future biomass predictions. In addition, poor representations of carbon allocation processes11,12,13, inconsistent definitions of biomass in wood and in roots across ESMs14, and uncertainties in allocation trade-offs in a changing climate15,16 also enlarge uncertainties of projected carbon allocation in the terrestrial biosphere, increasing the range of carbon–climate feedback estimates. Another challenge to minimize uncertainties in ESMs is accurately representing the types and fractional coverage of global forests17,18,19. Despite the creation of several observationally constrained estimates of biomass in above- and below-ground vegetation compartments over the past few decades3,18,20,21,22,23,24, the uncertainties of observation-based forest biomass due to different retrieval techniques25,26,27 and upscaling procedures in regions with sparse data availability28,29 remain large and poorly quantified. Confidence in the feedback of the vegetation carbon cycle in forest ecosystems to climate change thus requires accurate representation of carbon stocks, forest distribution, and allometric relationships between biomass compartments and the responses of underlying processes to environmental conditions in coupled ESMs.

In this study, we quantify the uncertainty of forest biomass, in terms of its magnitude and carbon allocation among vegetation components, from ESM outputs archived in the Coupled Model Intercomparison Project Phase 5 (CMIP5)30 by comparing the simulated results to a recent observation-based data set across the northern extratropical region (30°N–80°N)24. Spatial correlations of forest biomass and climatic conditions over this region are also assessed. Some of the ESMs analyzed here allowed for prognostic vegetation distributions using a Dynamic Global Vegetation Model (DGVM) and some did not. In addition, we compare the differences between area-weighted forest biomass at grid cell level and detailed information on forest biomass of individual plant functional types (PFTs) for each model realization. The former provides only the mean carbon density of all PFTs within a grid cell but no carbon density for individual PFTs, while the latter retains different carbon densities for each PFT within a grid cell. Moreover, we analyze the differences of biases in ESMs during the contemporary period and the PI period to investigate the uncertainty caused by parametric or structural model configurations. Hence, taking into account the uncertainty in observations, this study indicates sources of uncertainty for the magnitude and carbon allocation of extratropical forest biomass estimates that may facilitate improved representation of land processes in ESMs that will impact carbon–climate simulations such as those for the Coupled Model Intercomparison Project Phase 6 (CMIP6).

## Results

### Spatial differences of forest fractions and total carbon mass

Four ESMs outputs, one per modeling center, from the eight selected ESM realizations (see Supplementary Table S1) demonstrate substantial differences of spatial forest fractions and forest total biomass compared to observations (Figs 1 and 2). BNU-ESM overestimates forest fractions and total biomass at midlatitudes over Europe, the United States, and China while it underestimates forest fractions in Siberia. These discrepancies of forest fractions (min = −0.92; max = 0.95) and total carbon mass (min = −0.39 Pg C; max = 1.63 Pg C) at grid cell level are mainly due to broadleaf deciduous temperate trees in the regions of overestimated forest fractions and C3 arctic grass in the regions of underestimated forest fractions (see Supplementary Fig. S1). HadGEM2-ES, one of the two configurations from the Met Office Hadley Centre, noticeably underestimates forest fractions and biomass over Russia and Northern Canada with a magnitude up to 0.98 (min = −0.98; max = 0.96) and 0.14 Pg C (min = −0.14 Pg C; max = 0.14 Pg C), respectively. The uncertainty originates from the model identifying these regions as shrub-dominant (see Supplementary Fig. S1) whereas large portions of needleleaf forest are present in the observations. IPSL-CM5A-MR, one of the three ESM configurations from the Institut Pierre Simon Laplace (IPSL), shows biases in forest total biomass ranging from −0.19 Pg C to 0.41 Pg C with relatively lower biases in forest fractions (min = −0.58; max = 0.78) compared to the other three centers’ ESMs; nevertheless, they generally exhibit positive biases in forest fractions across northern high latitudes, especially between 45°N–60°N. This is a consequence of excessively large fractions of boreal needleleaf evergreen trees and boreal needleleaf summergreen trees apportioned in the three IPSL ESMs (see Supplementary Fig. S1). MIROC-ESM, one of the two model configurations from the Model for Interdisciplinary Research on Climate (MIROC), demonstrates that the simulated forest fractions deviate from the observations by −0.68 to 0.97 with overpredicted forest fractions in Northern Canada and Eurasia north of 60°N (boreal forests) but with underestimated forest fractions over the northeast coast of North America (dominated by C4 grasses) and over Eurasia south of 60°N (dominated by crops) (see Supplementary Fig. S1). The differences of global forest total biomass range from −0.37 Pg C to 0.62 Pg C. Spatial differences of forest fractions and total biomass of all eight ESMs are illustrated in Supplementary Figs S2 and S3.

A further investigation on the similarity of the spatial patterns of forest biomass between ESMs and obsESMs is illustrated by Taylor diagrams (Fig. 3a,b). The ESMs, while able to capture the spatial distributions of forest biomass at high Ff levels, have huge variations in terms of carbon mass magnitude owing to different land cover types and forest fractions (Fig. 3a). Correlations between the spatial patterns of simulated and observed forest carbon mass increase with Ff for all ESMs, while the variances of modeled forest biomass are generally greater than that of observed values since the normalized deviations are greater than one (except for MIROC-ESM at Ff = 0.8 and MIROC-ESM-CHEM at Ff = 0.9) (Fig. 3b). BNU-ESM exhibits the lowest skill scores (see Eq. (1) in Methods) (0.006 ± 0.005) because it has the largest variance among all ESMs at all Ff levels (Fig. 3a; Supplementary Fig. S5). The IPSL ESMs perform slightly better than BNU-ESM, but their large variances cause low skill scores (0.11 ± 0.04) as well (Fig. 3a; Supplementary Fig. S5). Higher skill scores are found for the HadGEM2 ESMs (0.85 ± 0.10) and the MIROC ESMs (0.64 ± 0.14) (Fig. 3b; Supplementary Fig. S5).

### Magnitude of forest biomass uncertainty

The biases of masked global forest total carbon mass in each biomass compartment, including the lumped total biomass from all forest compartments and the biomass in leaves, in wood, and in roots, are investigated along with the combined wood and roots (“Wood + Root”) to avoid inconsistent definitions of root biomass between ESMs14. Results demonstrate that the biases of modeled global forest total carbon mass at Ff = 0 ranges from −20.3 Pg C in HadGEM2-CC to 134.7 Pg C in IPSL-CM5A-MR (Fig. 4a). In terms of individual forest compartments, the combined Wood + Root biomass has biases ranging from −19.0 Pg C in HadGEM2-CC to 130.4 Pg C in IPSL-CM5A-MR (Fig. 4b); the biases of biomass in leaves ranges from −1.3 Pg C in HadGEM2-CC to 0.4 Pg C in IPSL-CM5A-MR (Fig. 4c); the biases of biomass in wood ranges from −5.7 Pg C in HadGEM2-CC to 120.0 Pg C in BNU-ESM (Fig. 4d); the biases of biomass in roots ranges from −13.3 Pg C in HadGEM2-CC to 25.8 Pg C in IPSL-CM5A-MR (Fig. 4e). A further examination on utilizing detailed biomass data at grid cell level rather than the grid area-weighted level from the HadGEM2-ES model results illustrates relatively smaller biases, though the bias of forest total biomass changes from negative (−13.4 Pg C) by utilizing grid area-averaged biomass data to positive (10.5 Pg C) by utilizing detailed biomass data at the grid cell level (see Supplementary Table S2). Compared to the mean value of forest total biomass from all obsESMs, that from all ESMs is overpredicted by 47.8 Pg C; for individual forest compartments, the mean value from all ESMs shows overpredicted forest biomass in Wood + Root by 43.4 Pg C, in wood by 41.5 Pg C, and in roots by 1.9 Pg C, while the biomass in leaves is underpredicted by 0.2 Pg C. Note that all ESMs predict the biomass in leaves within the uncertainty range of obsESMs, while BNU-ESM and the IPSL ESMs predict too much forest total biomass and biomass in Wood + Root due to overestimated biomass in wood (Fig. 4d). The institutional mean values of forest biomass for each modeling center show enlarged magnitude of bias in total mass (7.7 Pg C) and biomass in Wood + Root (10.2 Pg C) compared to multi-model mean values. The increased bias of biomass in wood (14.8 Pg C) is the major cause of the increasing discrepancies, even though the institutional mean biomass in roots is reduced (−4.57 Pg C). Details of extratropical forest carbon mass across 30° N–80° N for each ESM and for each modeling center is shown in Supplementary Table S2.

According to the masked forest biomass in ESMs and obsESMs, the magnitude of biases varies over a wide range. To minimize the influences of various masks applied to ESMs, we evaluate the model uncertainty in terms of relative errors (ER, see Eq. (2) in Methods) so that underestimates or overestimates in model results can be quantified in a standardized way. The multi-model ER average from the ERs of all ESMs shows that ESMs simulate excess forest total biomass (ERtotal = 0.84), biomass in wood (ERwood = 1.15), and biomass in Wood + Root (ERwood+root = 0.86), whereas underestimated biomass in leaves (ERleaf = −0.06) and biomass in roots (ERroot = −0.10) are found at Ff = 0 (Fig. 5a; institutional-averaged results are shown in Supplementary Fig. 6). The overestimation of global forest total carbon mass is mainly a result of overestimated biomass in wood. All IPSL ESMs significantly overestimate forest biomass in all compartments (ERtotal = 0.97–1.95, ERleaf = 0.02–0.11, ERwood = 1.01–2.08, ERroot = 0.82–1.69, and ERwood+root = 0.96–1.99), while HadGEM2-CC simulates too little forest biomass in all compartments (ERtotal = −0.28, ERleaf = −0.34, ERwood = −0.11, ERroot = −0.84, and ERwood+root = −0.28). BNU-ESM exhibits the largest overestimates in forest total carbon amount by more than three times the observational estimate (ERtotal = 3.03) due to extremely overestimated biomass mass in wood (ERwood = 4.41). For individual forest compartments, MIROC-ESM-CHEM has the smallest deviation from observations for forest total biomass (ERtotal = 0.08) and for biomass in Wood + Root (ERwood+root = 0.02); IPSL-CM5B-LR shows the smallest deviation for biomass in leaves (ERleaf = 0.02); HadGEM2-ES shows the smallest deviation for biomass in wood (ERwood = 0.01); BNU-ESM shows the smallest deviation for biomass in roots (ERroot = 0.04). Similar to the magnitude of forest carbon mass, utilizing detailed PFT-level biomass data from HadGEM2-ES results in better model performance with smaller absolute ER magnitude (except for biomass in wood) compared to using grid area-weighted biomass products (Fig. 5a). The ER value changes from −0.19 to 0.15 for forest total biomass, from −0.18 to 0.16 for biomass in Wood + Root, from −0.26 to −0.02 for biomass in leaves, from 0.01 to 0.44 for biomass in wood, and from −0.83 to −0.77 for biomass in roots.

All ESMs exhibit positive global-averaged relative errors at grid cell level (ERgrid, see Eq. (3) in Methods) in all compartments at Ff = 0 (Fig. 5b; institutional-averaged results are shown in Supplementary Fig. 6), indicating all ESMs overestimate forest biomass. The magnitude of absolute ERgrid values can be two to three orders of magnitude larger than ER. The largest relative errors for forest total biomass (ERgrid,total) and for individual forest compartments (ERgrid,leaf, ERgrid,wood, ERgrid,root, and ERgrid,wood+root) at grid cell level are found in IPSL-CM5A-LR (ERgrid,total = 281.0, ERgrid,leaf = 55.4, ERgrid,wood = 327.3, ERgrid,root = 207.7, ERgrid,wood+root = 287.3). BNU-ESM has the smallest relative error for biomass in leaves (ERgrid,leaf = 20.6) and in roots (ERgrid,root = 10.1), whereas HadGEM2-CC has the smallest relative errors for forest total biomass (ERgrid,total = 61.68) and biomass in wood (ERgrid,wood = 99.12) and in Wood + Root (ERgrid,wood+root = 74.08) at grid cell level. These huge relative errors at grid cell level imply that ESMs exhibit substantial overestimates at certain grid cells in which observed forest biomass is very small and thus large variations of forest biomass at grid cell level are expected for all ESMs (see Supplementary Table S3).

### Carbon allocation to forest compartments

Global carbon mass allocated in forest biomass compartments for all but the IPSL ESMs is consistently over-apportioned to wood but under-apportioned to leaves and roots (Fig. 6). The carbon allocation to forest compartments for all obsESMs ranges from 93.7–97.4% in Wood + Root, 2.6–6.3% in leaves, 67.9–83.3% in wood, and 13.9–25.9% in roots, while the mean values of all obsESMs with one standard deviation are 96.5% ± 1.5%, 3.5% ± 1.5%, 87.1% ± 7.9%, and 9.4% ± 8.5%, respectively. All IPSL ESMs apportion carbon mass to wood and to roots, with an institutional average of 76.96% ± 0.59% for the former and 20.73% ± 0.19% for the latter, the closest to the observed values. The institutional averages from the two MIROC ESMs and from the two HadGEM2 ESMs are found to allocate the carbon mass in Wood + Root (95.25% ± 0.04% for HadGEM2 ESMs and 95.22% ± 0.09% for MIROC ESMs) and in leaves (4.75% ± 0.04% for HadGEM2 ESMs and 4.78% ± 0.09% for MIROC ESMs) the closest to the observations. These results suggest that the HadGEM2 ESMs and the MIROC ESMs combine the biomass of coarse roots into the woody biomass pool, so that too much carbon is apportioned to wood and too little is apportioned to roots; on the contrary, the IPSL ESMs likely define the carbon mass in roots as the sum of coarse and fine root carbon mass since the carbon allocations are closer to the observed values.

### Spin-up impacts

The ratios of ERgrid for four forest types (see Supplementary Table S6) during the PI period over that during the contemporary period are calculated as an index (SR, see Eq. (4) in Methods) for evaluating the uncertainty caused by the spin-up processes. For individual forest compartments at Ff = 0, BNU-ESM shows that SR values are close to unity in forest total biomass (SRtotal = 1.02) and biomass in leaves (SRleaf = 0.76), in wood (SRwood = 1.03), and in roots (SRroot = 0.51). Similar to BNU-ESM, the mean SR values of all forest compartments simulated by the IPSL ESMs are also close to unity (SRtotal = 0.95 ± 0.05, SRleaf = 1.08 ± 0.06, SRwood = 0.94 ± 0.06, SRroot = 0.95 ± 0.05) (Fig. 8). These results imply the uncertainty may have been introduced at the beginning of the simulations in the ESMs due to the initial conditions derived from spin-up output. Similar unity SR values are also found in individual forest types, especially for the broadleaf and for the deciduous forests for the ESMs from these two modeling centers. On the contrary, moderate to substantial differences of ERgrid values for the HadGEM2 ESMs (SRtotal = 1.92 ± 0.43, SRleaf = 2.49 ± 0.21, SRwood = 1.85 ± 0.45, SRroot = 2.17 ± 0.23) and the MIROC ESMs (SRtotal = 5.18 ± 0.82, SRleaf = 3.91 ± 0.43, SRwood = 6.42 ± 1.66, SRroot = 4.94 ± 0.42), especially MIROC-ESM-CHEM (SRtotal = 5.76, SRleaf = 4.21, SRwood = 7.60, SRroot = 5.23), suggest uncertainty carried from the spin-up outputs due to vegetation parameterizations are minimized in these ESMs.

## Discussion

The present study evaluates the uncertainty of forest biomass across northern extratropical latitudes from eight coupled ESM simulations. Instead of regridding all ESM outputs and a recent observation-based data to a common grid resolution that induces additional uncertainty, we aggregate the fine-resolution observational data to the same grid resolution of each ESM so that each ESM output can be compared fairly to the aggregated observations. We examine the sources of uncertainty of forest biomass estimates through the forest fractions, the magnitude of carbon mass differences, the carbon mass allocations in forest compartments, and the biases carried from the spin-up results. Inconsistent definitions of PFTs between ESMs and observations contribute to the uncertainty of forest spatial distributions and thus large variations of forest fractions and biomass in each grid cell are found in ESMs33. After masks are applied, such inconsistencies cause significant differences in effective forest area among ESMs and among obsESMs. We also find that the unmasked total carbon mass in each aggregated observation data set for different ESM grid resolutions is the same but becomes smaller after forest-only masks are applied to grid cells (see Supplementary Fig. S9). The magnitude of discrepancy in modeled forest fraction and total biomass can be up to 0.98 and up to 1.63 Pg C, respectively, compared to the aggregated observations at the grid cell level. The bias of masked forest total biomass estimates in the northern extratropical region ranges from −20.3 Pg C to 134.7 Pg C and is mainly attributed to the bias in the wood compartment (−5.7 Pg C to 120.0 Pg C). In addition, CMIP5 models typically do not output the varying carbon density for all available PFTs within a grid cell. Some of the forest biomass uncertainty results from aggregation of PFT-level carbon pools to the grid cell level for purposes of reporting. This causes the loss of carbon density heterogeneity at the grid cell level and tends to lead to larger biases in carbon mass estimates. Analysis of HadGEM2-ES outputs with detailed PFT-level carbon pools demonstrates a reduction in modeled forest total carbon mass bias of 2.9 Pg C compared to the grid cell-level data reported in the CMIP5 archive. Hence, we suggest adding detailed PFT-level output of carbon stock and flux variables in CMIP6 simulations so that the uncertainty in future carbon mass estimates reported to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change can be attributed to ecosystem processes for each PFT. While causes of individual variations among model estimates of biomass are difficult to ascertain, multiple sources of uncertainty, as described below, contribute, including prognostic vegetation dynamics, land use and land cover change, and carbon allocation.

For the uncertainty of forest total carbon mass, our analyses show that four of the ESM simulations, if observational uncertainty is considered, overestimate the total carbon mass mainly due to overestimated biomass in wood. Since all ESMs consider carbon cycle feedbacks and land use change, the overestimates are possibly associated with biases in PFT structures and parameters in the DGVM and the land cover change setup in the ESMs. Traditionally, climate models use a prescribed coverage of land surface types, based on present-day observed land cover, but increasingly, ESMs are simulating the vegetation coverage internally using process-based DGVMs embedded within their land-surface scheme. This means that the models may incur biases and errors in the coverage of vegetated lands (e.g. trees, grasses, etc.), but such DGVMs offer the advantage that changes in vegetation cover in response to climate and CO2 can be simulated interactively. It has been shown that future changes in land cover induced by climate change can be just as big as changes induced by anthropogenic land-use change, and so such schemes are vital to fully capture changes in the terrestrial carbon cycle34. In our results, BNU-ESM used a DGVM to provide prognostic, climate-driven land cover change, but did not consider anthropogenic land use change; the three IPSL ESMs simulated the forest biomass with prescribed land use change but without prognostic land cover change; the two MIROC ESMs and the two HadGEM2 ESMs employed DGVMs for prognostic land cover change and incorporated prescribed land use change impacts (see Supplementary Table S1). Our analyses show that BNU-ESM and the IPSL ESMs exhibit large uncertainty in forest biomass estimates (institutional mean ERtotal = 3.03 and 1.54, respectively), while relatively smaller biases are found in the HadGEM2 ESMs and in the MIROC ESMs (institutional mean ERtotal = −0.11 and 0.13, respectively). Furthermore, Northern Hemisphere extratropical forest distributions in Whittaker climate space are mostly consistent with aggregated observations, except that BNU-ESM shows broadleaf deciduous trees below −10 °C, where needleleaf deciduous trees are found in the observations (see Supplementary Fig. S8). According to the Whittaker diagrams, vegetation models from all modeling centers, except BNU-ESM, capture biome distributions reasonably well; however, too little forest biomass was simulated by ESMs in low temperature zones, i.e., needleleaf or needleleaf deciduous trees were underrepresented in the ESMs. For BNU-ESM, forest biomass was underestimated over northern Asia because too much carbon was allocated to C3 arctic grass instead of to needleleaf forests. Biases in modeled climatic conditions also induce uncertainty in forest biomass estimates. The Whittaker diagrams of observed biomass in modeled climate space show a bias towards warmer temperatures over land in regions where needleleaf deciduous trees are located in the observations. These results suggest that (1) better PFT representations in the DGVM, (2) land use change, and (3) reducing biases in climatic conditions that affect plant productivity should all be considered in simulations to reduce the uncertainty of future forest carbon pool estimates. Further simulations for each model can help improve the understanding of the impact on biomass uncertainty due individually to the DGVM, land use changes, and climatic conditions.

In addition to the DGVM and land cover changes, the allometric equations of forest carbon mass also contribute to the simulated forest biomass uncertainty in ESMs. For example, the simulations from the HadGEM2 ESMs indiscriminately allocate exactly the same carbon mass in roots and in leaves (see Supplementary Table S2), which induces uncertainty in forest biomass estimates in these two simulations. Better representations of the dynamic carbon allocation schemes apportioning the carbon mass in leaves and roots should reduce the uncertainty of forest biomass estimates in ESMs13,35. Furthermore, all simulations except those from the IPSL ESMs consistently allocate too much carbon mass to wood but too little to leaves and roots. Although the proportions of carbon mass in wood and in roots simulated by the IPSL ESMs are within the observational uncertainty (Fig. 6c,d), the HadGEM2 ESMs and the MIROC ESMs have better carbon allocations for combined wood and root biomass (Fig. 6a). This result, which has also been shown in Negrón-Juárez et al.14, suggests the HadGEM2 ESMs and the MIROC ESMs include the coarse roots in the wood pool, while the IPSL ESMs report the coarse roots biomass as a part of root biomass. Therefore, it is more appropriate to compare the combined biomass in wood and in roots when estimating the uncertainty between ESMs, for individual models and the aggregated observations. Overestimation of wood and total biomass may also be attributable to positive biases in net primary productivity, poor parameterization of fire frequency and intensity36, lack of insect and disease processes in ESMs37, simplified characterization of canopy structure, overestimation of recruitment, and underestimation of tree mortality. As terrestrial ecosystem models evolve to simulate vegetation dynamics and demography38,39,40,41, development of improved and explicit representations of stand-level processes–including recruitment, succession, growth, mortality, and various kinds of disturbance–must be informed by systematic evaluation and benchmarking42,43,44, using in situ and remotely sensed observational data for burned area45, forest inventories46, primary production47, plant traits48, productivity and carbon residence times49, disturbance and extreme events50,51, mortality rates52, and other key forest energy and carbon fluxes53 and functional responses54,55.

Examination of the spin-up effect suggests that the uncertainty introduced at the beginning of the simulation in ESMs plays an important role in influencing the forest biomass estimates. Such spin-up issues have also been reported in soil carbon storage in CMIP5 outputs56. BNU-ESM and the IPSL ESMs simulate almost a unity SR value, demonstrating that the initial vegetation conditions from the spin-up processes in these models have substantial impact on present forest biomass estimates. As discussed above, the DGVM, land cover changes, and better representations and parameterizations of productivity, respiration and carbon turnover processes in the land models should be considered to improve model performance. Recent studies have shown that, in addition to mismatches in modeled compared to observed net primary production, carbon turnover processes are a major source of uncertainty in global vegetation models4,57. Simplified and not adequately represented turnover processes are contributing significantly to unrealistic magnitudes and spatial patterns of simulated biomass and to uncertainty in the response of vegetation carbon stocks to climate change4,57.

However, in addition to the uncertainties in the models, the observation-based product exhibits uncertainties as well. Sources of these uncertainties in the observation-based biomass estimate include the sensitivity of the radar signal to properties other than vegetation structure, the influence of non-forest vegetation on the signal, uncertainties in additional datasets (e.g. allometric databases, land cover, etc.) used for conversion of satellite measurements to biomass estimates, and the necromass component of vegetation in the observed growing stock volume. The application of allometric relationships derived on a leaf type level from a biomass compartment database has introduced relatively high uncertainties in observation-based estimates of leaf and root biomass compared to a moderate uncertainty in wood biomass. The uncertainty of the observation-based total biomass, however, is considered to be a conservative estimate and independent evaluations using upscaled forest inventory data have demonstrated the validity of the applied product at regional scales24,58,59.

Since the uncertainty of extratropical forest biomass estimates in ESMs can be from more factors in addition to the causes mentioned in this study, we suggest that the community utilize software packages to evaluate the performance of coupled ESMs whenever there is an improvement or modification in models. These tools can benefit the community by informing future development of ESMs. For instance, the International Land Model Benchmarking (ILAMB) project44 provides a means to not only provide the statistics of model outputs but also quickly and easily evaluate how model performance changes with each modification in ESMs through skill score spreads. Because of uncertainties in observational data, ILAMB incorporated multiple observational data sets for the same variable to provide the user with some indication of that observational uncertainty. With an increasing or a decreasing score, researchers are able to determine whether a modification in ESMs will reduce the model uncertainty in forest biomass estimates.

## Conclusion

We evaluated forest biomass from eight CMIP5 ESMs using a biomass dataset for the northern extratropical latitudes that was synthesized from radar remote sensing of stem volume and ground-based observations of wood density and allometric relationships between biomass compartments. ESMs exhibited large biases in forest distribution, forest fraction, and carbon mass that contributed to the overall uncertainty in forest total biomass. We showed that forest total biomass is positively correlated with precipitation variations with temperature becoming equally important at higher latitudes. Finally, we showed that uncertainties in the pre-industrial forest biomass (spin-up) persisted into the contemporary period in most ESMs. To improve future analysis of vegetation biomass, we recommend saving and archiving PFT-level carbon stocks and fluxes, which can be used for analysis and benchmarking. Routine evaluation of model biomass in all compartments and of the spatial distribution of vegetation types is important for identifying sources of uncertainty in carbon cycle predictions.

## Methods

### Model outputs

Outputs of eight Earth system model (ESM) simulations from the CMIP5 archive with the historical experiment “r1i1p1” (see Supplementary Table S1) are selected based on the data availability of carbon density of individual plant functional types (PFTs) for all biomass compartments including leaf, wood, root, and the total biomass. The model outputs contain climatic variables as well as detailed information about forest biomass for various PFTs (see Supplementary Table S4). Two climatic variables, the precipitation rate (PR) and the surface temperature (TAS), are analyzed to investigate the spatial correlations of forest biomass and climate. The forest biomass in CMIP5 archive stores a single carbon density derived from the area-weighted carbon density of all vegetation types in each grid cell. Hence, forests and grasses have the same carbon density value as long as their area fractions are non-zero within a grid cell. In this study, forest carbon density (kg C m−2) is converted into carbon mass (petagrams of carbon, Pg C). Data during 1982–2005 is averaged to represent the climatological mean status during the contemporary period, while that during the pre-industrial period is based on the climatological mean status during 1861–1885.

### Observation-based data sets

The applied observation-based forest biomass product24 (available from http://www.bgc-jena.mpg.de/geodb/projects/Home.php) provides vegetation carbon density and its uncertainty covering northern boreal and temperate forests (30°N–80°N) at 0.01° resolution. First, forest biomass in stems was derived from a radar remote sensing growing stock volume (GSV) product58,59, which integrates observations between October 2009 and February 2011, and information on wood density. Subsequently, the other biomass compartments (branches, roots, foliage) have been estimated based on a database of allometric relationships to stems. Furthermore, precipitation flux and surface temperature from the Global Soil Wetness Project Phase 3 (GSWP3) forcing data products (0.5° × 0.5°)60 serve as the observational climatic data. For land cover types, we adopt the land cover definitions from 1 km × 1 km global land cover classification for the year 2000 (GLC2000) database31 (see Supplementary Table S4). We reproject GLC2000 (applying nearest-neighbor interpolation) in order to match the resolution of the observation-based biomass product. To compare to each ESM’s output, this product is further upscaled to eight data sets with coarser grid resolutions that are the same as the eight selected CMIP5 model simulations (see Supplementary Table S1) while retaining the PFT-level biomass distributions (i.e., different carbon densities instead of a single area-weighted carbon density for each PFT within a grid cell).

We verify the data availability at each grid cell for the upscaled observational data sets and the ESM output. A grid cell is masked out when either observational or modeled forest biomass is unavailable at that grid cell. Due to different PFT distributions and grid resolutions, the available grid cells vary in each ESM. The total carbon stock estimated from the upscaled observational data sets, even though they are upscaled from the same product, are hence different. Therefore, we use relative errors, in addition to the total carbon stock, to evaluate the performance of each ESM output.

### Land cover types and forest fractions

The forest fractions for observation-based data are determined by GLC2000. The GLC2000 data is upscaled to coarser resolutions, varying from 1.25° × 1.875° to 2.8125° × 2.8125° (see Supplementary Table S1), according to each model’s setup. We classify four major forest types including broadleaf evergreen trees, needleleaf evergreen trees, broadleaf deciduous trees, and needleleaf deciduous trees (see Supplementary Table S5) when evaluating modeled forest distributions in Whittaker climate space. Alternatively, we combine the first ten classes in GLC2000 as the “lumped” forest type24 to quantify the uncertainty caused by spin-up processes in ESMs (see Supplementary Table S6). For each ESM output, modeled forest fractions are compared to that from those upscaled GLC2000 data at the same grid resolution. Uncertainty due to different forest definitions between GLC2000 and ESMs can be minimized by applying a common-grid land mask with various forest fraction thresholds (Ff). The common-grid land mask keeps only the grid cells where both model outputs and observations have non-zero forest fractions as well as valid biomass values of forest types. Ff ranges from 0 to 0.9 with an increment of 0.1. Grid cells with forest fractions lower than or equal to Ff are masked out in both model outputs and observations. A higher Ff means that the carbon pool in grid cells composed of a larger proportion of forest-only biomass, i.e., the uncertainty caused by assigning a fixed carbon density for all PFTs in the ESMs is reduced.

### Spatial correlations and skill scores

Spatial differences between model outputs and observations are compared in terms of forest fractions and forest total carbon mass. We apply a 11 × 11 moving window to compute the spatial correlation between forest total carbon mass and precipitation rate as well as that between forest total carbon mass and surface temperature at the default forest fraction threshold (Ff = 0) over extratropical regions in the Northern Hemisphere. The spread of biomass in the precipitation–temperature space is lumped into two (broadleaf and needleleaf trees) and four categories (broadleaf evergreen, needleleaf evergreen, broadleaf deciduous, and needleleaf deciduous trees) depending on the PFT definitions in each model. A Taylor diagram61 is also produced to assess the spatial uncertainty due to Ff through the correlation, the root-mean-square difference and the ratio of the variances of forest total carbon mass from observations and from model outputs. Due to different spatial resolutions in each model, we use standardized deviations from each model’s variance normalized by the variance from the observations upscaled to its resolution. Model performance is evaluated through the skill scores (S)61:

$${S}_{i}=\frac{{\rm{4}}\times (1+{R}_{i})}{(\frac{{\sigma }_{i}}{{\sigma }_{r}}+\frac{{\sigma }_{r}}{{\sigma }_{i}})({\rm{1}}+{R}_{0})}$$
(1)

where Ri is the spatial pattern correlation of forest total carbon mass between model i and the upscaled observation at its resolution, σi the variance of modeled forest total carbon mass, σr the variance of observed forest total carbon mass. We compute ten different Ri values for each model i by varying Ff from 0 to 0.9 with an increment of 0.1. The maximum value of Ri among the ten Ff cases is assigned to R0.

### Relative error

To quantify the modeled forest biomass uncertainty, we measure the relative error (ER) with the common-grid land mask applied to evaluate the discrepancies of the magnitude of forest carbon amount between models and that from upscaled observational data:

$$ER=\frac{\sum _{i=1}^{n}{y}_{i}-\sum _{i=1}^{n}{x}_{i}}{\sum _{i=1}^{n}{x}_{i}}$$
(2)

where n is the total available grid cells, yi the global forest total biomass from model outputs and xi the global forest total biomass upscaled from the observational MPI–BGIv3 data set at grid cell i. The magnitude of observed global forest total biomass depends on data availability in each model output with the common-grid land mask applied. The multi-model average of forest total biomass is calculated based on the average of modeled outputs and upscaled observations for all ESMs using Eq. (2). In addition to analyzing the grid area-weighted results, we further examine the forest biomass with detailed carbon mass in individual PFTs at each grid cell from the HadGEM2-ES outputs. Grid cell-level uncertainty is computed by averaging ER over n available grid cells at each grid cell i:

$$E{R}_{grid}=\frac{1}{n}\,\sum _{i=1}^{n}({y}_{i}-{x}_{i})$$
(3)

In the present study, we evaluate ER for forest total biomass (ERtotal) as well as biomass in leaves (ERleaf), wood (ERwood), and roots (ERroot). Because of the inconsistent definition of roots in ESMs, we also compute ER for the combined wood and roots biomass (ERwood+root). A model has less uncertainty in forest carbon mass estimates when ER approaches zero.

### Spin-up effects

The impact of the initial-to-equilibrium state from spin-up inputs on simulated forest biomass can be evaluated by the ratios of ERgrid values during the PI period and the modern time, i.e.,

$$SR=\frac{E{R}_{grid,PI}}{E{R}_{grid,contemporary}}$$
(4)

Note that we utilize the observations in contemporary years as the reference values for both ERgrid,PI and ERgrid,contemporary due to the absence of forest biomass information during the PI period. Nevertheless, it is suggested that higher possibilities of uncertainty in future carbon pool projections may be influenced by the inputs from spin-up results when SR values approach unity.

## References

1. 1.

Sitch, S. et al. Evaluation of the terrestrial carbon cycle, future plant geography and climate-carbon cycle feedbacks using five Dynamic Global Vegetation Models (DGVMs). Glob. Chang. Biol. 14, 2015–2039 (2008).

2. 2.

Friedlingstein, P., Joel, G., Field, C. B. & Fung, I. Y. Toward an allocation scheme for global terrestrial carbon models. Glob. Chang. Biol. 5, 755–770 (1999).

3. 3.

Pan, Y. et al. A large and persistent carbon sink in the world’s forests. Science 333, 988–993 (2011).

4. 4.

Friend, A. D. et al. Carbon residence time dominates uncertainty in terrestrial vegetation responses to future climate and atmospheric CO2. Proc. Natl. Acad. Sci. USA 111, 3280–3285 (2014).

5. 5.

Le Quéré, C. et al. Global carbon budget 2016. Earth Syst. Sci. Data 8, 605–649 (2016).

6. 6.

Jobbágy, E. G. & Jackson, R. B. The vertical distribution of soil organic carbon and its relation to climate and vegetation. Ecol. Appl. 10, 423–436 (2000).

7. 7.

Malhi, Y., Meir, P. & Brown, S. Forests, carbon and global climate. Philos. Trans. A Math. Phys. Eng. Sci. 360, 1567–1591 (2002).

8. 8.

Sedjo, R. & Sohngen, B. Carbon Sequestration in Forests and Soils. Annu. Rev. Resour. Econ. 4, 127–144 (2012).

9. 9.

Watson, R. T. et al. Land Use, Land-Use Change, and Forestry: A Special Report of the Intergovernmental Panel on Climate Change. (Cambridge University Press, 2000).

10. 10.

U.S. EPA. Inventory of US greenhouse gas emissions and sinks: 1990–2014. (U.S. EPA, 2016).

11. 11.

Litton, C. M., Raich, J. W. & Ryan, M. G. Carbon allocation in forest ecosystems. Glob. Chang. Biol. 13, 2089–2109 (2007).

12. 12.

Franklin, O. et al. Modeling carbon allocation in trees: a search for principles. Tree Physiol. 32, 648–666 (2012).

13. 13.

De Kauwe, M. G. et al. Where does the carbon go? A model–data intercomparison of vegetation carbon allocation and turnover processes at two temperate forest free-air CO2 enrichment sites. New Phytol. 203, 883–899 (2014).

14. 14.

Negrón-Juárez, R. I., Koven, C. D., Riley, W. J., Knox, R. G. & Chambers, J. Q. Observed allocations of productivity and biomass, and turnover times in tropical forests are not accurately represented in CMIP5 Earth system models. Environ. Res. Lett. 10, 064017 (2015).

15. 15.

Doughty, C. E. et al. Allocation trade-offs dominate the response of tropical forest growth to seasonal and interannual drought. Ecology 95, 2192–2201 (2014).

16. 16.

Doughty, C. E. et al. Drought impact on forest carbon dynamics and fluxes in Amazonia. Nature 519, 78–82 (2015).

17. 17.

Saatchi, S. S. & Houghton, R. A. Dos Santos Alvalá, R. C., Soares, J. V. & Yu, Y. Distribution of aboveground live biomass in the Amazon basin. Glob. Chang. Biol. 13, 816–837 (2007).

18. 18.

Saatchi, S. S. et al. Benchmark map of forest carbon stocks in tropical regions across three continents. Proc. Natl. Acad. Sci. USA 108, 9899–9904 (2011).

19. 19.

Schimel, D. et al. Observing terrestrial ecosystems and the carbon cycle from space. Glob. Chang. Biol. 21, 1762–1776 (2015).

20. 20.

Jackson, R. B., Banner, J. L., Jobbágy, E. G., Pockman, W. T. & Wall, D. H. Ecosystem carbon loss with woody plant invasion of grasslands. Nature 418, 623–626 (2002).

21. 21.

Bradley, B. A., Houghton, R. A. & Mustard, J. F. Invasive grass reduces aboveground carbon stocks in shrublands of the Western US. Glob. Chang. Biol (2006).

22. 22.

Hurtt, G. C. et al. Linking models and data on vegetation structure. J. Geophys. Res. 115, G00E10 (2010).

23. 23.

Baccini, A. et al. Estimated carbon dioxide emissions from tropical deforestation improved by carbon-density maps. Nat. Clim. Chang. 2, 182 (2012).

24. 24.

Thurner, M. et al. Carbon stock and density of northern boreal and temperate forests. Glob. Ecol. Biogeogr. 23, 297–310 (2014).

25. 25.

Pacala, S. W. et al. Consistent land- and atmosphere-based U.S. carbon sink estimates. Science 292, 2316–2320 (2001).

26. 26.

Shvidenko, A., Schepaschenko, D., McCallum, I. & Nilsson, S. Can the uncertainty of full carbon accounting of forest ecosystems be made acceptable to policymakers? Clim. Change 103, 137–157 (2010).

27. 27.

Bradshaw, C. J. A. & Warkentin, I. G. Global estimates of boreal forest carbon stocks and flux. Glob. Planet. Change 128, 24–30 (2015).

28. 28.

Houghton, R. A., Lawrence, K. T., Hackler, J. L. & Brown, S. The spatial distribution of forest biomass in the Brazilian Amazon: a comparison of estimates. Glob. Chang. Biol. 7, 731–746 (2001).

29. 29.

Boudreau, J. et al. Regional aboveground forest biomass using airborne and spaceborne LiDAR in Québec. Remote Sens. Environ. 112, 3876–3890 (2008).

30. 30.

Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An Overview of CMIP5 and the Experiment Design. Bull. Am. Meteorol. Soc. 93, 485–498 (2012).

31. 31.

Joint Research Centre, E. C. Global Land Cover 2000 database. http://forobs.jrc.ec.europa.eu/products/glc2000/glc2000.php (2003).

32. 32.

Whittaker, R. H. Communities and Ecosystems. (Macmillan, 1975).

33. 33.

Poulter, B., Ciais, P. & Hodson, E. Plant functional type mapping for earth system models. Geosci. Model Dev. 4, 993–1010 (2011).

34. 34.

Davies-Barnard, T., Valdes, P. J., Singarayer, J. S., Wiltshire, A. J. & Jones, C. D. Quantifying the relative importance of land cover change from climate and land use in the representative concentration pathways. Global Biogeochem. Cycles 29, 842–853 (2015).

35. 35.

Mao, J. et al. Evaluating the Community Land Model in a pine stand with shading manipulations and 13CO2 labeling. Biogeosciences 13, 641–657 (2016).

36. 36.

Hantson, S. et al. The status and challenge of global fire modelling. Biogeosciences 13, 3359–3375 (2016).

37. 37.

Hicke, J. A. et al. Effects of biotic disturbances on forest carbon cycling in the United States and Canada. Glob. Chang. Biol. 18, 7–34 (2012).

38. 38.

Moorcroft, P. R., Hurtt, G. C. & Pacala, S. W. A method for scaling vegetation dynamics: the ecosystem demography model (ED). Ecol. Monogr (2001).

39. 39.

Medvigy, D., Wofsy, S. C., Munger, J. W., Hollinger, D. Y. & Moorcroft, P. R. Mechanistic scaling of ecosystem function and dynamics in space and time: Ecosystem Demography model version 2. J. Geophys. Res. 114, G01002 (2009).

40. 40.

Fisher, R. A. et al. Taking off the training wheels: the properties of a dynamic vegetation model without climate envelopes, CLM4.5(ED). Geosci. Model Dev. 8, 3593–3619 (2015).

41. 41.

Fisher, R. A. et al. Vegetation demographics in Earth System Models: A review of progress and priorities. Glob. Chang. Biol. 24, 35–54 (2018).

42. 42.

Randerson, J. T. et al. Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models. Glob. Chang. Biol. 15, 2462–2484 (2009).

43. 43.

Luo, Y. Q. et al. A framework for benchmarking land models. Biogeosciences 9, 3857–3874 (2012).

44. 44.

Hoffman, F. M. et al. International Land Model Benchmarking (ILAMB) 2016 Workshop Report. U.S. Department of Energy, Office of Science; https://doi.org/10.2172/1330803 (2017).

45. 45.

Giglio, L., Randerson, J. T. & Werf, G. R. Analysis of daily, monthly, and annual burned area using the fourth-generation global fire emissions database (GFED4). J. Geophys. Res. Biogeo. 118, 317–328 (2013).

46. 46.

O’Connell, B. M. et al. The Forest Inventory and Analysis Database: Database description and user guide version 6.0. 1 for Phase 2. US Department of Agriculture, Forest Service 748 (2014).

47. 47.

Running, S. W. et al. A Continuous Satellite-Derived Measure of Global Terrestrial Primary Production. Bioscience 54, 547–560 (2004).

48. 48.

Kattge, J. et al. TRY–a global database of plant traits. Glob. Chang. Biol. 17, 2905–2935 (2011).

49. 49.

Thurner, M. et al. Large-scale variation in boreal and temperate forest carbon turnover rate related to climate. Geophys. Res. Lett. 43, GL068794 (2016).

50. 50.

Hansen, M. C. et al. High-resolution global maps of 21st-century forest cover change. Science 342, 850–853 (2013).

51. 51.

Frank, D. et al. Effects of climate extremes on the terrestrial carbon cycle: concepts, processes and potential future impacts. Glob. Chang. Biol. 21, 2861–2880 (2015).

52. 52.

van Mantgem, P. J. et al. Widespread increase of tree mortality rates in the western United States. Science 323, 521–524 (2009).

53. 53.

Williams, M. et al. Improving land surface models with FLUXNET data. Biogeosciences 6, 1341–1359 (2009).

54. 54.

Dukes, J. S. et al. Responses of grassland production to single and multiple global environmental changes. PLoS Biol. 3, e319 (2005).

55. 55.

Reichstein, M., Bahn, M., Mahecha, M. D., Kattge, J. & Baldocchi, D. D. Linking plant and ecosystem functional biogeography. Proc. Natl. Acad. Sci. USA 111, 13697–13702 (2014).

56. 56.

Exbrayat, J.-F., Pitman, A. J. & Abramowitz, G. Response of microbial decomposition to spin-up explains CMIP5 soil carbon range until 2100. Geosci. Model Dev. 7, 2683–2692 (2014).

57. 57.

Thurner, M. et al. Evaluation of climate-related carbon turnover processes in global vegetation models for boreal and temperate forests. Glob. Chang. Biol. 23, 3076–3091 (2017).

58. 58.

Santoro, M. et al. Retrieval of growing stock volume in boreal forest using hyper-temporal series of Envisat ASAR ScanSAR backscatter measurements. Remote Sens. Environ. 115, 490–507 (2011).

59. 59.

Santoro, M. et al. Forest growing stock volume of the northern hemisphere: Spatially explicit estimates for 2010 derived from Envisat ASAR. Remote Sens. Environ. 168, 316–334 (2015).

60. 60.

Kim, H. GSWP3 - Global Soil Wetness Project Phase 3, 10.20783/DIAS.501 (2017).

61. 61.

Taylor, K. E. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res. 106, 7183–7192 (2001).

## Acknowledgements

This work was supported through the Reducing Uncertainties in Biogeochemical Interactions through Synthesis and Computation Scientific Focus Area (RUBISCO SFA), which is sponsored by the Regional and Global Climate Modeling (RGCM) Program in the Climate and Environmental Sciences Division (CESD) of the Office of Biological and Environmental Research (BER) in the U.S. Department of Energy Office of Science. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory (ORNL), which is managed by UT-Battelle, LLC for the U.S. Department of Energy under Contract No. DE-AC05–00OR22725 and was performed at ORNL. C.D.J was supported by the Joint UK BEIS/Defra Met Office Hadley Centre Climate Programme (GA01101) and M.T. was supported by the Vetenskapsrådet grant 621-2014-4266 of the Swedish Research Council. We acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups (listed in Supplementary Table S1) for producing and making available their model output. For CMIP the U.S. Department of Energy’s Program for Climate Model Diagnosis and Intercomparison provides coordinating support and led development of software infrastructure in partnership with the Global Organization for Earth System Science Portals. We acknowledge the kind assistance of Prof. Shilong Piao from the Department of Ecology at Peking University for sharing the plotting scripts for global spatial correlation maps.

## Author information

Authors

### Contributions

J.M. conceived the study. J.M. and F.M.H. recommended analytical and graphical methods. C.-E.Y. performed the analysis, generated graphical results, and prepared the manuscript in consultation with J.M., D.M.R., F.M.H., and J.S.F. M.T. prepared the observational data. C.D.J. prepared the detailed PFT-level biomass data from the HadGEM2-ES model. All authors reviewed and edited the manuscript.

### Corresponding authors

Correspondence to Cheng-En Yang, Jiafu Mao or Forrest M. Hoffman.

## Ethics declarations

### Competing Interests

The authors declare no competing interests.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Yang, CE., Mao, J., Hoffman, F.M. et al. Uncertainty Quantification of Extratropical Forest Biomass in CMIP5 Models over the Northern Hemisphere. Sci Rep 8, 10962 (2018). https://doi.org/10.1038/s41598-018-29227-7

• Accepted:

• Published: