Field experiments underestimate aboveground biomass response to drought

Researchers use both experiments and observations to study the impacts of climate change on ecosystems, but results from these contrasting approaches have not been systematically compared for droughts. Using a meta-analysis and accounting for potential confounding factors, we demonstrate that aboveground biomass responded only about half as much to experimentally imposed drought events as to natural droughts. Our findings indicate that experimental results may underestimate climate change impacts and highlight the need to integrate results across approaches.

The exact sample size (n) for each experimental group/condition, given as a discrete number and unit of measurement A statement on whether measurements were taken from distinct samples or whether the same sample was measured repeatedly The statistical test(s) used AND whether they are one-or two-sided Only common tests should be described solely by name; describe more complex techniques in the Methods section.
A description of all covariates tested A description of any assumptions or corrections, such as tests of normality and adjustment for multiple comparisons A full description of the statistical parameters including central tendency (e.g. means) or other basic estimates (e.g. regression coefficient) AND variation (e.g. standard deviation) or associated estimates of uncertainty (e.g. confidence intervals) For null hypothesis testing, the test statistic (e.g. F, t, r) with confidence intervals, effect sizes, degrees of freedom and P value noted Give P values as exact values whenever suitable.
For Bayesian analysis, information on the choice of priors and Markov chain Monte Carlo settings For hierarchical and complex designs, identification of the appropriate level for tests and full reporting of outcomes Estimates of effect sizes (e.g. Cohen's d, Pearson's r), indicating how they were calculated Our web collection on statistics for biologists contains articles on many of the points above.

Software and code
Policy information about availability of computer code Data collection We extracted data manually from published papers. When the data were presented in a figure, we used Web Plot Digitizer (version 4.2) to read the data. Aridity index data were extracted as described in the Data availability statement in the "Data" box below.

Data analysis
Data analyses were done in the R programming environment (version 4.1.0). We used the metafor package (version 3.0-2) for the metaanalytic mixed-effects models, and to test for publication bias. In each meta-analysis, the MuMIn package (version 1.43.17) was used for making an information-theoretic model selection based on AICc values to identify the minimum adequate model. We used DHARMa package (version 0.1.5) functions for testing overdispersion and homogeneity of residual variances. For each meta-analytic model, we fitted an equivalent linear mixed-effects model using the nlme package (version 3.1-149) to extract ANOVA tables, and computed R-squared values using the r2glmm package (version 0.1.2). We tested whether experimental and observational studies differed in site aridity, drought length, drought severity, and aboveground biomass. For site aridity and drought length we used the glmmTMB package (version 1.1.2.3) and the lme4 package (version 1.1-27.1), respectively, while the differences in drought severity and biomass were tested using the nlme package. The computer code (R scripts) of the analyses is available in Figshare with the identifier https://doi.org/10.6084/m9.figshare.17881073.
For manuscripts utilizing custom algorithms or software that are central to the research but not yet described in published literature, software must be made available to editors and reviewers. We strongly encourage code deposition in a community repository (e.g. GitHub). See the Nature Portfolio guidelines for submitting code & software for further information.
nature portfolio | reporting summary

March 2021
Data Policy information about availability of data All manuscripts must include a data availability statement. This statement should provide the following information, where applicable: -Accession codes, unique identifiers, or web links for publicly available datasets -A description of any restrictions on data availability -For clinical datasets or third party data, please ensure that the statement adheres to our policy All studies must disclose on these points even when the disclosure is negative.

Study description
We compared the responses of aboveground biomass to experimentally applied versus observed drought events in a systematic review using hierarchical meta-analyses. We tested for the effects of potential confounding factors such as drought severity (% reduction in yearly precipitation), drought length (years), and site aridity (mean annual precipitation divided by mean annual potential evapotranspiration). We used log response ratio (lnRR) as an effect size metric. We weighted data by the number of replications in our focal meta-analysis, but we also conducted additional meta-analyses with different weightings, and for data not used in the focal analysis, to test the robustness of our results.

Research sample
In total, 239 data points were extracted from 114 published papers, and 158 data points of them (from 80 studies) were included in our focal meta-analysis. A data point was a natural or experimental drought event reported in a particular study. Data of different sites, or land use, etc., from the same study were collected as distinct data points, but data points from the same study received a common study ID, and study ID was treated as a random effect in statistical tests. For each study site, we extracted aridity index from Global Aridity Index and Potential Evapotranspiration (ET0) Climate Database v2 (available at https://doi.org/10.6084/ m9.figshare.7504448.v3).

Sampling strategy
We conducted a systematic literature search in the ISI Web of Science (WoS; since 1975) for published results on drought effects on aboveground plant production from studies conducted in grasslands or shrublands. For the exact search terms we used please see the Methods section. This yielded 2187 papers, which were screened using the following criteria (established before the start of the screening): The research was conducted in (semi-)natural grasslands or shrublands. The paper reported precipitation reduction relative to the control (non-drought year(s) in observational studies and no treatment in experimental studies), and plant production expressed as aboveground net primary production (ANPP), aboveground plant biomass (in grassland studies only), or percentage plant cover for control and drought. We also included 27 studies meeting these criteria from the references of WoS records and previous reviews. In total, this resulted in 114 studies. Thus, sample size was determined by the number of studies available in the literature worldwide and by our inclusion criteria. Literature search and paper screening were done by G. Kröel-Dulay.

Data collection
From the studies, we collected the study site, latitude, longitude, mean annual temperature (MAT) and precipitation (MAP), study type (experimental or observational), drought length (years), vegetation type (grassland or shrubland), and yearly precipitation for both the control and drought. From precipitation data, we calculated drought severity as % reduction in yearly precipitation in response to drought relative to the control. For production, we compiled the mean, replication, and if the study reported, a variance estimate (standard deviation, standard error of the mean, or 95% confidence interval) for control and drought. Data were extracted from the text, tables or figures of the published papers, and typed into an Excel sheet. When the data were presented in a figure, we used Web Plot Digitizer to read the data. The 114 published papers provided 239 data points. Data collection from the papers was done by A. Mojzes. For each study site, we extracted aridity index from Global Aridity Index and Potential Evapotranspiration (ET0) Climate Database v2.
Timing and spatial scale We covered the period from 1975 to 13 January 2020 in the WoS search. Additional studies from cited references go back to 1937.
Regarding the spatial coverage, we searched for papers from all parts of the world, without any geographic restriction. Since the data were collected from published papers (except for aridity index), the spatial and temporal scales, as well as the frequency and periodicity of sampling were determined by the particular study (these were study specific). Aridity index data covered the period of 1970-2000 (aggregated on annual basis).

Data exclusions
During screening of the papers, we excluded the studies that did not meet our inclusion criteria summarised above in the "Sampling strategy" box. For more details on data exclusion, please see the PRISMA flow chart (Supplementary Fig. 1) and the Methods section. From our focal meta-analysis, we excluded the studies from wet sites, shrublands, or that estimated plant cover, because these were rare and very unequally distributed between experiments and observations (but the excluded data points were analysed separately).