Abstract
Peer review is central to the scientific process and scientists’ career advancement, but bias at various stages of the review process disadvantages some authors. Here we use peer review data from 312,740 biological sciences manuscripts across 31 studies to (1) examine evidence for differential peer review outcomes based on author demographics, (2) evaluate the efficacy of solutions to reduce bias and (3) describe the current landscape of peer review policies for 541 ecology and evolution journals. We found notably worse review outcomes (for example, lower overall acceptance rates) for authors whose institutional affiliations were in Asia, for authors whose country’s primary language is not English and in countries with relatively low Human Development Indices. We found few data evaluating efficacy of interventions outside of reducing gender bias through double-blind review or diversifying reviewer/editorial boards. Despite evidence for review outcome gaps based on author demographics, few journals currently implement policies intended to mitigate bias (for example, 15.9% of journals practised double-blind review and 2.03% had reviewer guidelines that mentioned social justice issues). The lack of demographic equity signals an urgent need to better understand and implement evidence-based bias mitigation strategies.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The datasets generated and analysed during the current study are available on GitHub at https://github.com/CourtneyLDavis/Peer-Review-Perpetuates-Barriers. Data are also archived at Figshare (https://doi.org/10.6084/m9.figshare.21865830)49. Additional datasets that support the findings of this study are available from the United Nations Development Programme (the HDI; https://hdr.undp.org/en/content/download-data), The CIA World Factbook (continent, language; https://www.cia.gov/the-world-factbook/), Ethnologue (language; https://www.ethnologue.com/), The European Commission’s Eurobarometer Survey (language; https://data.europa.eu/data/datasets/s1049_77_1_ebs386?locale=en), Cambridge Encyclopedia of Language38 and Clarivate Journal Citation Reports (journal attributes; https://jcr.clarivate.com/jcr/home).
Code availability
R scripts used to analyse data and generate figures during the current study are available on GitHub at https://github.com/CourtneyLDavis/Peer-Review-Perpetuates-Barriers.
References
Silbiger, N. J. & Stubler, A. D. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ 7, e8247 (2019).
Peters, D. P. & Ceci, S. J. Peer-review practices of psychological journals: the fate of published articles, submitted again. Behav. Brain Sci. 5, 187–195 (1982).
Tregenza, T. Gender bias in the refereeing process? Trends Ecol. Evol. 17, 349–350 (2002).
Burns, C. S. & Fox, C. W. Language and socioeconomics predict geographic variation in peer review outcomes at an ecology journal. Scientometrics 113, 1113–1127 (2017).
Fox, C. W. & Paine, C. E. T. Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution. Ecol. Evol. 9, 3599–3619 (2019).
Llorens, A. et al. Gender bias in academia: a lifetime problem that needs solutions. Neuron 109, 2047–2074 (2021).
Ceci, S. J. & Williams, W. M. Understanding current causes of women’s underrepresentation in science. Proc. Natl Acad. Sci. USA 108, 3157–3162 (2011).
Brodie, S. et al. Equity in science: advocating for a triple-blind review system. Trends Ecol. Evol. 36, 957–959 (2021).
Helmer, M., Schottdorf, M., Neef, A. & Battaglia, D. Gender bias in scholarly peer review. eLife 6, e21718 (2017).
Squazzoni, F. et al. Peer review and gender bias: a study on 145 scholarly journals. Sci. Adv. 7, eabd0299 (2021).
Godlee, F. & Dickerson, K. in Peer Review in Health Sciences (eds Jefferson, T., & Godlee, F.) 91–117 (Wiley, 2003).
Haffar, S., Bazerbachi, F. & Murad, M. H. Peer review bias: a critical review. Mayo Clin. Proc. 94, 670–676 (2019).
Shoham, N. & Pitman, A. Open versus blind peer review: is anonymity better than transparency? BJPsych Adv. 27, 247–254 (2021).
Budden, A. E. et al. Double-blind review favours increased representation of female authors. Trends Ecol. Evol. 23, 4–6 (2008).
Whittaker, R. J. Journal review and gender equality: a critical comment on Budden et al. Trends Ecol. Evol. 23, 478–479 (2008).
Budden, A. E. et al. Response to Whittaker: challenges in testing for gender bias. Trends Ecol. Evol. 23, 480–481 (2008).
Webb, T. J., O’Hara, B. & Freckleton, R. P. Does double-blind review benefit female authors? Trends Ecol. Evol. 23, 351–353 (2008).
O’Hara, B. Double-blind review: let diversity reign. Nature 452, 28 (2008).
Naqvi, K. R. Double-blind review: the paw print is a giveaway. Nature 452, 28 (2008).
Maas, B. et al. Women and Global South strikingly underrepresented among top-publishing ecologists. Conserv. Lett. 14, e12797 (2021).
McGillivray, B. & De Ranieri, E. Uptake and outcome of manuscripts in Nature journals by review model and author characteristics. Res. Integr. Peer Rev. 3, 1–12 (2018).
Alam, M. et al. Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi-rater study. Br. J. Dermatol. 165, 563–567 (2011).
Jagsi, R. et al. Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. Int. J. Radiat. Oncol. Biol. Phys. 89, 940–946 (2014).
Fox, C. W. et al. Double-blind peer review—an experiment. Funct. Ecol. 33, 4–6 (2019).
Lloyd, M. E. Gender factors in reviewer recommendations for manuscript preparation. J. Appl. Behav. Anal. 23, 539–543 (1990).
Ramírez-Castañeda, V. Disadvantages in preparing and publishing scientific papers caused by the dominance of the English language in science: the case of Colombian researchers in biological sciences. PLoS ONE 15, 1–15 (2020).
Ehara, S. & Takahashi, K. Reasons for rejection of manuscripts submitted to AJR by international authors. Am. J. Roentgenol. 188, 113–116 (2007).
Harris, M., Macinko, J., Jimenez, G., Mahfoud, M. & Anderson, C. Does a research article’s country of origin affect perception of its quality and relevance? A national trial of US public health researchers. BMJ Open 5, 1–10 (2015).
Harris, M. et al. Explicit bias toward high-income-country research: a randomized, blinded, crossover experiment of English clinicians. Health Aff. 36, 1997–2004 (2017).
Steinpreis, R. E., Anders, K. A. & Ritzke, D. The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: a national empirical study. Sex. Roles 41, 509–528 (1999).
Mehta, D. et al. Ways to increase equity, diversity and inclusion. eLife 9, 1–4 (2020).
Else, H. & Perkel, J. M. The giant plan to track diversity in research journals. Nature 602, 566–570 (2022).
Stefanoudis, P. V. et al. Turning the tide of parachute science. Curr. Biol. 31, R184–R185 (2021).
Stossel, T. P. & Stossel, S. C. Declining American representation in leading clinical-research journals. N. Engl. J. Med. 322, 739–742 (1990).
The World Factbook. Central Intelligence Agency https://www.cia.gov/ (2021).
Amano, T. & Sutherland, W. J. Four barriers to the global understanding of biodiversity conservation: wealth, language, geographical location and security. Proc. R. Soc. B 280, 20122649 (2013).
Eberhard, D. M., Simons, G. F. & Fennig, C. D. Ethnologue: Languages of the World (SIL International, 2022).
Crystal, D. The Cambridge Encyclopedia of Language (Cambridge Univ. Press, 2010).
Special Eurobarometer 386: Europeans and their languages. European Commission https://data.europa.eu/data/datasets/s1049_77_1_ebs386?locale=en (2014).
Zhang, X. Effect of reviewer’s origin on peer review: China vs. non-China. Learn. Publ. 25, 265–270 (2012).
Walker, R., Barros, B., Conejo, R., Neumann, K. & Telefont, M. Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study. F1000Research 4, 21 (2015).
Brooks, M. E. et al. glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. R. J. 9, 378–400 (2017).
R: a language and environment for statistical computing (R Foundation for Statistical Computing, 2020).
Hartig, F. DHARMa: residual diagnostics for hierarchical (multi-level/mixed) regression models. R package version 0.4.4 http://florianhartig.github.io/DHARMa/ (2021).
Ludecke, D., Ben-Shachar, M. S., Patil, I., Waggoner, P. & Makowski, D. performance: an R package for assessment, comparison and testing of statistical models. J. Open Source Softw. 6, 3139 (2021).
Bolker, B., & R Core Team. bbmle: tools for general maximum likelihood estimation. R package version 1.0.24 https://CRAN.R-project.org/package=bbmle (2021).
Hothorn, T., Bretz, F. & Westfall, P. Simultaneous inference in general parametric models. Biom. J. 50, 346–363 (2008).
Lenth, R. V. emmeans: Estimated marginal means, aka least-squares means. R package version 1.7.1-1 https://CRAN.R-project.org/package=emmeans (2021).
Smith, O. M. et al. Peer review perpetuates barriers for historically excluded groups. figshare https://doi.org/10.6084/m9.figshare.21865830 (2023).
Tennekes, M. tmap: thematic maps in R. J. Stat. Softw. 84, 1–39 (2018).
Acknowledgements
We thank authors who sent us additional data we requested (fully documented in Supplementary Data 4), especially S. Burns, L. Fabrigar, M. Flemming, C. Fox, B. McGillivray, T. Paine, R. Petty and T. Tregenza. V. Pan assisted with data collection. A. Hayes and Z. Katz provided input on data collection. Y. Ge, L. Rasmussen and J. Reganold provided feedback on earlier versions of this article. This work was funded by the Michigan State University Presidential Postdoctoral Fellowship in Ecology, Evolutionary and Behavior awarded to O.M.S.
Author information
Authors and Affiliations
Contributions
All authors conceptualized the study. O.M.S. led the project. All authors collected data with greatest contributions from O.M.S. and C.L.D. O.M.S. analysed the data under guidance from K.L.D., W.L. and C.L.D. O.M.S., R.B.P., R.W., J.C.J., N.N. and C.L.D. made figures. O.M.S., R.B.P., B.F., L.N.J. and C.L.D. drafted paper sections with greatest contribution from O.M.S. O.M.S., K.L.D., R.B.P., R.W., K.C.D., B.F., J.C.J., L.N.J., W.L., N.N., E.E.C., A.H., C.M., A.N.S., O.J.U., M.L.Y. and C.L.D edited the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Ecology & Evolution thanks Sean Burns, Martin Nunez and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Overview of the peer review process covered by studies in our meta-analysis.
a-b Individual journals may use different labels for aEditor-in-Chief and bAssociate Editors. Clock symbol indicates time data were available for that peer review stage.
Extended Data Fig. 2 Corresponding authors whose institutional affiliations are in countries with lower Human Development Indices (HDI) have lower percentages of manuscripts sent to review, but review scores do not differ by HDI.
Estimated means and 95% confidence intervals for a, initial decisions for corresponding authors, and b, review scores for first, corresponding, and last authors. The study underlying the regression in b required that all authors were from the same country, so the values are identical across authorship positions. *** P < 0.001 where indicated (two-sided tests). a, See Supplementary Table 32 for sample sizes, statistical tests used, and exact P-values. b, See Supplementary Tables 30, 32, and 34 for sample sizes, statistical tests used, and exact P-values.
Extended Data Fig. 3 Double-blind review can close gaps in overall acceptance by authors’ affiliations’ countries’ Human Development Indices and primary language.
a, corresponding author overall acceptance increases with increasing HDI at a faster rate under single- compared to double-blind peer review. b, single-blind review favors authors whose institutional affiliations are in countries where English is a primary language. Estimated means and 95% confidence intervals. HDI = Human Development Index, EPL = English primary language, ENPL = English not a primary language. * P < 0.050 for the interaction between peer review model and HDI/language (two-sided tests). See Supplementary Table 53 for sample sizes, statistical tests used, and exact P-values.
Extended Data Fig. 4 Double-blind review can lead to differential outcomes by author demographics.
Estimated means and 95% confidence intervals predicting review outcomes (initial decisions, review scores [lower are better], post-initial review decisions, final decisions, and overall decisions) by author demographics interacting with review model (double- vs single-blind review) for a, first, b, corresponding, and c, last authors. Review scores with data from one study per demographic category show the underlying mean. Single-blind = circles, double-blind = squares. · P < 0.10, * P < 0.05, ** P < 0.01, *** P < 0.001 when reported for the interaction between peer review model and demographic (two-sided tests). See Supplementary Tables 41–52 for sample sizes, statistical tests used, and exact P-values.
Extended Data Fig. 5 Ecology and evolution journals are taking few actions to reduce bias in peer review.
a, Many journals require or suggest the use of paid language editing services by authors for whom English is not a primary language. b, Few journals prompt authors to suggest diverse reviewers on their websites. c, But of journals that do ask authors to suggest diverse reviewers on their websites, geography and institution were the most often mentioned axes of diversity (panel shows percent of all journals that mentioned that axis of diversity, even if they did not prompt authors to suggest diverse reviewers). d, Single-blind review is the most common model. e, Only a small percentage of journals publish reviews alongside accepted manuscripts. f, Many journals do not have their own reviewer guidelines. g, Most reviewer guidelines do not mention social justice issues related to demographic equity, including the evaluation of English. Concentric circles show major publisher (outer-most ring, n = 190 journals), society journal published by a major publisher (n = 120 journals), minor publisher (n = 153 journals), and society journal published by minor publishers (inner-most ring, n = 78 journals). Major publishers included Springer, Wiley, Elsevier, Taylor & Francis, and Oxford University Press. Pub = publisher.
Extended Data Fig. 6 Journals and Editors-in-Chief (EICs) are concentrated in a few countries/regions regardless of publisher size.
a, b, Major publishers not affiliated with societies. c, d, Major publishers affiliated with societies. e, f, Minor publishers not affiliated with societies. g, h, Minor publishers affiliated with societies. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Major publishers included Springer, Wiley, Elsevier, Taylor & Francis, and Oxford University Press. Base maps were provided by the ‘tmap’ package in R50. Soc = society.
Extended Data Fig. 7 Journals and Editors-in-Chief (EICs) are concentrated in Western and Central Europe.
a, b, Major publishers not affiliated with societies. c, d, Major publishers affiliated with societies. e, f, Minor publishers not affiliated with societies. g, h, Minor publishers affiliated with societies. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50. Soc = society.
Extended Data Fig. 8 Ecology and evolution journals (n = 541) are taking few actions to reduce bias in peer review.
a, A few major publishers comprise a large proportion of journals. b, Many journals require or suggest the use of paid language editing services by authors for whom English is not a primary language. c, Few journals prompt authors to suggest diverse reviewers on their websites. d, But of journals that do ask authors to suggest diverse reviewers on their websites, geography and institution were the most often mentioned axes of diversity (panel shows percent of all journals that mentioned that axis of diversity, even if they did not prompt authors to suggest diverse reviewers). e, Single-blind review is the most common model. f, Only a small percentage of journals publish reviews alongside accepted manuscripts. g, Many journals do not have their own reviewer guidelines. h, Most reviewer guidelines do not mention social justice issues related to demographic equity, including the evaluation of English.
Extended Data Fig. 9 Journals and Editors-in-Chief (EICs) are concentrated in a few countries/regions.
When standardizing by population size, a, Australia and Europe have the greatest number of journals per one million people; and b, North America, Australia, and Europe have the greatest number of EICs per one million people; and among Europe, c, Switzerland has the greatest number of journals per one million people. d, Western/Central Europe and Scandinavia tend to have the greatest numbers of EICs per one million people. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50.
Extended Data Fig. 10 Journals and Editors-in-Chief (EICs) are heavily concentrated in Western Europe, particularly for higher impact journals.
a, c, e, g, i, journal locations and b, d, f, h, j, EICs’ institutional affiliations’ locations for a, b, all journals, and journal impact factors c, d, >10, e, f, 3–10, g, h, <3, and i, j, not available (not Journal Citation Reports [JCR] indexed or has not been in JCR long enough). Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50. JIF = journal impact factor.
Supplementary information
Supplementary Information
Supplementary Tables 1–72, Figs. 1–10 and references.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Smith, O.M., Davis, K.L., Pizza, R.B. et al. Peer review perpetuates barriers for historically excluded groups. Nat Ecol Evol 7, 512–523 (2023). https://doi.org/10.1038/s41559-023-01999-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41559-023-01999-w
This article is cited by
-
Scientific publishing has a language problem
Nature Human Behaviour (2023)
