Synthesis of primary ecological data is often assumed to achieve a notion of ‘generality’, through the quantification of overall effect sizes and consistency among studies, and has become a dominant research approach in ecology. Unfortunately, ecologists rarely define either the generality of their findings, their estimand (the target of estimation) or the population of interest. Given that generality is fundamental to science, and the urgent need for scientific understanding to curb global scale ecological breakdown, loose usage of the term ‘generality’ is problematic. In other disciplines, generality is defined as comprising both generalizability—extending an inference about an estimand from the sample to the population—and transferability—the validity of estimand predictions in a different sampling unit or population. We review current practice in ecological synthesis and demonstrate that, when researchers fail to define the assumptions underpinning generalizations and transfers of effect sizes, generality often misses its target. We provide guidance for communicating nuanced inferences and maximizing the impact of syntheses both within and beyond academia. We propose pathways to generality applicable to ecological syntheses, including the development of quantitative and qualitative criteria with which to license the transfer of estimands from both primary and synthetic studies.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Houlahan, J. E., McKinney, S. T., Anderson, T. M. & McGill, B. J. The priority of prediction in ecological understanding. Oikos 126, 1–7 (2017).
Lawton, J. H. Are there general laws in ecology? Oikos 84, 177–192 (1999).
Elliott-Graves, A. Generality and causal interdependence in ecology. Philos. Sci. 85, 1102–1114 (2018).
Fox, J. W. The many roads to generality in ecology. Philos. Top. 9, 83–104 (2019).
McGill, B. J. et al. Species abundance distributions: moving beyond single prediction theories to integration within an ecological framework. Ecol. Lett. 10, 995–1015 (2007).
MacArthur, R. H. & Wilson, E. O. An equilibrium theory of insular zoogeography. Evolution 17, 373–387 (1963).
Gurevitch, J., Fox, G. A., Wardle, G. M., Inderjit & Taub, D. Emergent insights from the synthesis of conceptual frameworks for biological invasions. Ecol. Lett. 14, 407–418 (2011).
Borer, E. T. et al. Finding generality in ecology: a model for globally distributed experiments. Methods Ecol. Evol. 5, 65–73 (2014).
Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. Meta-analysis and the science of research synthesis. Nature 555, 175–182 (2018).
Anderson, S. C. et al. Trends in ecology and conservation over eight decades. Front. Ecol. Environ. 19, 274–282 (2021).
Kneale, D., Thomas, J., O’Mara-Eves, A. & Wiggins, R. How can additional secondary data analysis of observational data enhance the generalisability of meta-analytic evidence for local public health decision making? Res. Synth. Methods 10, 44–56 (2019).
Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R. & Dalton, C. M. Debunking myths and urban legends about meta-analysis. Organ. Res. Methods 14, 306–331 (2011).
Polit, D. F. & Beck, C. T. Generalization in quantitative and qualitative research: myths and strategies. Int. J. Nurs. Stud. 47, 1451–1458 (2010).
Cardinale, B. J., Gonzalez, A., Allington, G. R. H. & Loreau, M. Is local biodiversity declining or not? A summary of the debate over analysis of species richness time trends. Biol. Conserv. 219, 175–183 (2018).
Lundberg, I., Johnson, R. & Stewart, B. M. What is your estimand? Defining the target quantity connects statistical evidence to theory. Am. Sociol. Rev. 86, 532–565 (2021).
Lawrance, R. et al. What is an estimand & how does it relate to quantifying the effect of treatment on patient-reported quality of life outcomes in clinical trials? J. Patient-Rep. Outcomes 4, 68 (2020).
Findley, M. G., Kikuta, K. & Denly, M. External validity. Annu. Rev. Polit. Sci. 24, 365–393 (2021).
Pearl, J. & Bareinboim, E. External validity: from do-calculus to transportability across populations. Stat. Sci. 29, 579–595 (2014).
Westreich, D., Edwards, J. K., Lesko, C. R., Cole, S. R. & Stuart, E. A. Target validity and the hierarchy of study designs. Am. J. Epidemiol. 188, 438–443 (2019).
Carpenter, C. J. Meta-analyzing apples and oranges: how to make applesauce instead of fruit salad. Hum. Commun. Res. 46, 322–333 (2020).
Rohrer, J. M. & Arslan, R. C. Precise answers to vague questions: issues with interactions. Adv. Methods Pract. Psychol. Sci. 4, 1–19 (2021).
Breslow, N. E. & Clayton, D. G. Approximate inference in generalized linear mixed models. J. Am. Stat. Assoc. 88, 9–25 (1993).
Koricheva, J. & Gurevitch, J. Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102, 828–844 (2014).
Gonzalez, A. et al. Estimating local biodiversity change: a critique of papers claiming no net loss of local diversity. Ecology 97, 1949–1960 (2016).
Konno, K. et al. Ignoring non-English-language studies may bias ecological meta-analyses. Ecol. Evol. 10, 6373–6384 (2020).
Nakagawa, S. et al. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol. Evol. 13, 4–21 (2022).
Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638–641 (1979).
Leung, B. et al. Clustered versus catastrophic global vertebrate declines. Nature 588, 267–271 (2020).
Rothman, K. J., Gallacher, J. E. J. & Hatch, E. E. Why representativeness should be avoided. Int. J. Epidemiol. 42, 1012–1014 (2013).
Spake, R. et al. Implications of scale dependence for cross-study syntheses of biodiversity differences. Ecol. Lett. 24, 374–390 (2021).
Spake, R. & Doncaster, C. P. Use of meta-analysis in forest biodiversity research: key challenges and considerations. For. Ecol. Manag. 400, 429–437 (2017).
Christie, A. P. et al. Simple study designs in ecology produce inaccurate estimates of biodiversity responses. J. Appl. Ecol. 56, 2742–2754 (2019).
Nakagawa, S., Noble, D. W. A., Senior, A. M. & Lagisz, M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15, 18 (2017).
Higgins, J. P. T. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002).
Schielzeth, H. & Nakagawa, S. Conditional repeatability and the variance explained by reaction norm variation in random slope models. Methods Ecol. Evol. 13, 1214–1223 (2022).
Nakagawa, S. et al. The orchard plot: cultivating a forest plot for use in ecology, evolution, and beyond. Res. Synth. Methods 12, 4–12 (2021).
Lorah, J. Effect size measures for multilevel models: definition, interpretation, and TIMSS example. Large-Scale Assess. Educ. 6, 8 (2018).
O’Connor, M. I. et al. A general biodiversity–function relationship is mediated by trophic level. Oikos 126, 18–31 (2017).
Ojha, M., Naidu, D. G. T. & Bagchi, S. Meta-analysis of induced anti-herbivore defence traits in plants from 647 manipulative experiments with natural and simulated herbivory. J. Ecol. 110, 799–816 (2022).
Dodds, K. C. et al. Material type influences the abundance but not richness of colonising organisms on marine structures. J. Environ. Manag. 307, 114549 (2022).
Dornelas, M. et al. Assemblage time series reveal biodiversity change but not systematic loss. Science 344, 296–299 (2014).
Senior, A. M. et al. Heterogeneity in ecological and evolutionary meta- analyses: its magnitude and implications. Ecology 97, 3293–3299 (2016).
Blowes, S. A. et al. The geography of biodiversity change in marine and terrestrial assemblages. Science 366, 339–345 (2019).
Nakagawa, S. & Cuthill, I. C. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. 82, 591–605 (2007).
Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3–8 (1976).
Glass, G. V. Meta‐analysis at 25: a personal history. Education in Two Worlds https://ed2worlds.blogspot.com/2022/07/meta-analysis-at-25-personal-history.html (2000).
Cooper, H. M. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowl. Soc. 1, 104–126 (1988).
Soranno, P. A. et al. Cross-scale interactions: quantifying multi-scaled cause-effect relationships in macrosystems. Front. Ecol. Environ. 12, 65–73 (2014).
Gerstner, K. et al. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol. Evol. 8, 777–784 (2017).
Hortal, J. et al. Seven shortfalls that beset large-scale knowledge of biodiversity. Annu. Rev. Ecol. Evol. Syst. 46, 523–549 (2015).
Simons, D. J., Shoda, Y. & Lindsay, D. S. Constraints on Generality (CoG): a proposed addition to all empirical papers. Perspect. Psychol. Sci. 12, 1123–1128 (2017).
Yarkoni, T. The generalizability crisis. Behav. Brain Sci. https://doi.org/10.1017/S0140525X20001685 (2020).
Lopez, P. M., Subramanian, S. V. & Schooling, C. M. Effect measure modification conceptualized using selection diagrams as mediation by mechanisms of varying population-level relevance. J. Clin. Epidemiol. 113, 123–128 (2019).
Campbell, D. T. in Advances in QuasiExperimental Design and Analysis (ed. Trochim, W.) 67–77 (Jossey-Bass, 1986).
Spake, R. et al. Meta‐analysis of management effects on biodiversity in plantation and secondary forests of Japan. Conserv. Sci. Pract. 1, e14 (2019).
Forest Ecosystem Diversity Basic Survey (in Japanese) (Forestry Agency of Japan, 2019); https://www.rinya.maff.go.jp/j/keikaku/tayouseichousa/index.html
Ito, S., Ishigamia, S., Mizoue, N. & Buckley, G. P. Maintaining plant species composition and diversity of understory vegetation under strip-clearcutting forestry in conifer plantations in Kyushu, southern Japan. For. Ecol. Manag. 231, 234–241 (2006).
Utsugi, E. et al. Hardwood recruitment into conifer plantations in Japan: effects of thinning and distance from neighboring hardwood forests. For. Ecol. Manag. 237, 15–28 (2006).
Kominami, Y. et al. Classification of bird-dispersed plants by fruiting phenology, fruit size, and growth form in a primary lucidophyllous forest: an analysis, with implications for the conservation of fruit–bird interactions. Ornthological Sci. 2, 3–23 (2003).
Tsujino, R. & Matsui, K. Forest regeneration inhibition in a mixed broadleaf-conifer forest under sika deer pressure. J. For. Res. 27, 230–235 (2021).
Spake, R., Soga, M., Catford, J. A. & Eigenbrod, F. Applying the stress-gradient hypothesis to curb the spread of invasive bamboo. J. Appl. Ecol. 58, 1993–2003 (2021).
Mize, T. D. Best practices for estimating, interpreting, and presenting nonlinear interaction effects. Sociol. Sci. 6, 81–117 (2019).
Karaca-Mandic, P., Norton, E. C. & Dowd, B. Interaction terms in nonlinear models. Health Serv. Res. 47, 255–274 (2012).
Spake, R. et al. Forest damage by deer depends on cross-scale interactions between climate, deer density and landscape structure. J. Appl. Ecol. 57, 1376–1390 (2020).
McCabe, C. J., Kim, D. S. & King, K. M. Improving present practices in the visual display of interactions. Adv. Methods Pract. Psychol. Sci. 1, 147–165 (2018).
Shackelford, G. E. et al. Dynamic meta-analysis: a method of using global evidence for local decision making. BMC Biol. 19, 33 (2021).
Christie, A. P. et al. Innovation and forward‐thinking are needed to improve traditional synthesis methods: a response to Pescott and Stewart. J. Appl. Ecol. 59, 1191–1197 (2022).
Haddaway, N. R. et al. EviAtlas: a tool for visualising evidence synthesis databases. Environ. Evid. 8, 22 (2019).
Delory, B. M., Li, M., Topp, C. N. & Lobet, G. archiDART v3.0: a new data analysis pipeline allowing the topological analysis of plant root systems. F1000Research 7, 22 (2018).
Perkel, J. M. The future of scientific figures. Nature 554, 133–134 (2018).
Weaver, S. & Gleeson, M. P. The importance of the domain of applicability in QSAR modeling. J. Mol. Graph. Model. 26, 1315–1326 (2008).
Sutton, C. et al. Identifying domains of applicability of machine learning models for materials science. Nat. Commun. 11, 4428 (2020).
Meyer, H. & Pebesma, E. Predicting into unknown space? Estimating the area of applicability of spatial prediction models. Methods Ecol. Evol. 12, 1620–1633 (2021).
Pearl, J. & Bareinboim, E. Transportability of causal and statistical relations: a formal approach. In 2011 IEEE 11th International Conference on Data Mining Workshops https://doi.org/10.1109/ICDMW.2011.169 (IEEE, 2011).
Munthe-Kaas, H., Nøkleby, H. & Nguyen, L. Systematic mapping of checklists for assessing transferability. Syst. Rev. 8, 22 (2019).
Dekkers, O. M., von Elm, E., Algra, A., Romijn, J. A. & Vandenbroucke, J. P. How to assess the external validity of therapeutic trials: a conceptual approach. Int. J. Epidemiol. 39, 89–94 (2010).
Schloemer, T. & Schröder-Bäck, P. Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis. Implement. Sci. 13, 88 (2018).
Fernandez-Hermida, J. R., Calafat, A., Becoña, E., Tsertsvadze, A. & Foxcroft, D. R. Assessment of generalizability, applicability and predictability (GAP) for evaluating external validity in studies of universal family-based prevention of alcohol misuse in young people: systematic methodological review of randomized controlled trials. Addiction 107, 1570–1579 (2012).
Avellar, S. A. et al. External validity: the next step for systematic reviews? Eval. Rev. 41, 283–325 (2017).
Bareinboim, E. & Pearl, J. A general algorithm for deciding transportability of experimental results. J. Causal Inference 1, 107–134 (2013).
Degtiar, I. & Rose, S. A review of generalizability and transportability. Preprint at https://doi.org/10.48550/arXiv.2102.11904 (2021).
Bareinboim, E. & Pearl, J. Meta-transportability of causal effects: a formal approach. J. Mach. Learn. Res. 31, 135–143 (2013).
Jamieson, D. Scientific uncertainty: how do we know when to communicate research findings to the public? Sci. Total Environ. 184, 103–107 (1996).
Burchett, H. E. D., Mayhew, S. H., Lavis, J. N. & Dobrow, M. J. When can research from one setting be useful in another? Understanding perceptions of the applicability and transferability of research. Health Promot. Int. 28, 418–430 (2013).
Forscher, P. et al. Build up big-team science. Nature 601, 505–507 (2022).
Whalen, M. A. et al. Climate drives the geography of marine consumption by changing predator communities. Proc. Natl Acad. Sci. USA 117, 28160–28166 (2020).
Moshontz, H. et al. The Psychological Science Accelerator: advancing psychology through a distributed collaborative network. Adv. Methods Pract. Psychol. Sci. 1, 501–515 (2018).
Marschner, I. C. A general framework for the analysis of adaptive experiments. Stat. Sci. 36, 465–492 (2021).
Clark, M. Shrinkage in Mixed Effects Models https://m-clark.github.io/posts/2019-05-14-shrinkage-in-mixed-models/ (2019).
Gurevitch, J. & Hedges, L. V. Statistical issues in ecological meta-analyses. Ecology 80, 1142–1149 (1999).
Mengersen, K., Gurevitch, J. & Schmid, C. H. in Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, U. et al.) 300–312 (Princeton Univ. Press, 2013).
Hudson, L. N. et al. The database of the PREDICTS (Projecting Responses of Ecological Diversity In Changing Terrestrial Systems) project. Ecol. Evol. 7, 145–188 (2017).
Dornelas, M. et al. BioTIME: a database of biodiversity time series for the Anthropocene. Glob. Ecol. Biogeogr. 27, 760–786 (2018).
Salguero-Gómez, R. et al. The COMPADRE Plant Matrix Database: an open online repository for plant demography. J. Ecol. 103, 202–218 (2015).
Salguero-Gómez, R. et al. COMADRE: a global data base of animal demography. J. Anim. Ecol. 85, 371–384 (2016).
Pastor, D. A. & Lazowski, R. A. On the multilevel nature of meta-analysis: a tutorial, comparison of software programs, and discussion of analytic choices. Multivar. Behav. Res. 53, 74–89 (2018).
We thank J. Chase for informative discussion of concepts. R.S. is grateful for funding from the German Centre for Integrative Biodiversity Research – iDiv - Halle-Jena-Leipzig. J.M.B. was funded under UKCEH National Capability project 06895. C.T.C. was supported by a Marie Skłodowska-Curie Individual Fellowship (no. 891052).
The authors declare no competing interests.
Peer review information
Nature Ecology & Evolution thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix S1 is a table of the aims and scope of the 50 most cited journals in ecology. Appendix S2 provides details of an analysis (a meta-regression of sapling abundance on thinning intensity in coniferous forests of Japan).
About this article
Cite this article
Spake, R., O’Dea, R.E., Nakagawa, S. et al. Improving quantitative synthesis to achieve generality in ecology. Nat Ecol Evol 6, 1818–1828 (2022). https://doi.org/10.1038/s41559-022-01891-z
This article is cited by
Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences
Environmental Evidence (2023)
CABI Agriculture and Bioscience (2023)