Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Peer review perpetuates barriers for historically excluded groups

Abstract

Peer review is central to the scientific process and scientists’ career advancement, but bias at various stages of the review process disadvantages some authors. Here we use peer review data from 312,740 biological sciences manuscripts across 31 studies to (1) examine evidence for differential peer review outcomes based on author demographics, (2) evaluate the efficacy of solutions to reduce bias and (3) describe the current landscape of peer review policies for 541 ecology and evolution journals. We found notably worse review outcomes (for example, lower overall acceptance rates) for authors whose institutional affiliations were in Asia, for authors whose country’s primary language is not English and in countries with relatively low Human Development Indices. We found few data evaluating efficacy of interventions outside of reducing gender bias through double-blind review or diversifying reviewer/editorial boards. Despite evidence for review outcome gaps based on author demographics, few journals currently implement policies intended to mitigate bias (for example, 15.9% of journals practised double-blind review and 2.03% had reviewer guidelines that mentioned social justice issues). The lack of demographic equity signals an urgent need to better understand and implement evidence-based bias mitigation strategies.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Bias can filter scientists from different demographics across the review process, leading to fewer published manuscripts by historically excluded groups.
Fig. 2: Author demographics predict review outcomes across the peer review process.
Fig. 3: Authors in countries with lower HDI and where English is not a primary language have lower overall acceptance rates and submit their manuscripts to more journals before acceptance.
Fig. 4: Double-blind review and editor/reviewer homophily can lead to differential outcomes by author assumed gender.
Fig. 5: Ecology and evolution journals are taking a few actions to reduce bias in peer review.
Fig. 6: Journals and EICs are concentrated in few countries/regions.

Similar content being viewed by others

Data availability

The datasets generated and analysed during the current study are available on GitHub at https://github.com/CourtneyLDavis/Peer-Review-Perpetuates-Barriers. Data are also archived at Figshare (https://doi.org/10.6084/m9.figshare.21865830)49. Additional datasets that support the findings of this study are available from the United Nations Development Programme (the HDI; https://hdr.undp.org/en/content/download-data), The CIA World Factbook (continent, language; https://www.cia.gov/the-world-factbook/), Ethnologue (language; https://www.ethnologue.com/), The European Commission’s Eurobarometer Survey (language; https://data.europa.eu/data/datasets/s1049_77_1_ebs386?locale=en), Cambridge Encyclopedia of Language38 and Clarivate Journal Citation Reports (journal attributes; https://jcr.clarivate.com/jcr/home).

Code availability

R scripts used to analyse data and generate figures during the current study are available on GitHub at https://github.com/CourtneyLDavis/Peer-Review-Perpetuates-Barriers.

References

  1. Silbiger, N. J. & Stubler, A. D. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ 7, e8247 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Peters, D. P. & Ceci, S. J. Peer-review practices of psychological journals: the fate of published articles, submitted again. Behav. Brain Sci. 5, 187–195 (1982).

    Article  Google Scholar 

  3. Tregenza, T. Gender bias in the refereeing process? Trends Ecol. Evol. 17, 349–350 (2002).

    Article  Google Scholar 

  4. Burns, C. S. & Fox, C. W. Language and socioeconomics predict geographic variation in peer review outcomes at an ecology journal. Scientometrics 113, 1113–1127 (2017).

    Article  Google Scholar 

  5. Fox, C. W. & Paine, C. E. T. Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution. Ecol. Evol. 9, 3599–3619 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Llorens, A. et al. Gender bias in academia: a lifetime problem that needs solutions. Neuron 109, 2047–2074 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Ceci, S. J. & Williams, W. M. Understanding current causes of women’s underrepresentation in science. Proc. Natl Acad. Sci. USA 108, 3157–3162 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Brodie, S. et al. Equity in science: advocating for a triple-blind review system. Trends Ecol. Evol. 36, 957–959 (2021).

    Article  PubMed  Google Scholar 

  9. Helmer, M., Schottdorf, M., Neef, A. & Battaglia, D. Gender bias in scholarly peer review. eLife 6, e21718 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Squazzoni, F. et al. Peer review and gender bias: a study on 145 scholarly journals. Sci. Adv. 7, eabd0299 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Godlee, F. & Dickerson, K. in Peer Review in Health Sciences (eds Jefferson, T., & Godlee, F.) 91–117 (Wiley, 2003).

  12. Haffar, S., Bazerbachi, F. & Murad, M. H. Peer review bias: a critical review. Mayo Clin. Proc. 94, 670–676 (2019).

    Article  PubMed  Google Scholar 

  13. Shoham, N. & Pitman, A. Open versus blind peer review: is anonymity better than transparency? BJPsych Adv. 27, 247–254 (2021).

    Article  Google Scholar 

  14. Budden, A. E. et al. Double-blind review favours increased representation of female authors. Trends Ecol. Evol. 23, 4–6 (2008).

    Article  PubMed  Google Scholar 

  15. Whittaker, R. J. Journal review and gender equality: a critical comment on Budden et al. Trends Ecol. Evol. 23, 478–479 (2008).

    Article  PubMed  Google Scholar 

  16. Budden, A. E. et al. Response to Whittaker: challenges in testing for gender bias. Trends Ecol. Evol. 23, 480–481 (2008).

    Article  Google Scholar 

  17. Webb, T. J., O’Hara, B. & Freckleton, R. P. Does double-blind review benefit female authors? Trends Ecol. Evol. 23, 351–353 (2008).

    Article  PubMed  Google Scholar 

  18. O’Hara, B. Double-blind review: let diversity reign. Nature 452, 28 (2008).

    Article  PubMed  Google Scholar 

  19. Naqvi, K. R. Double-blind review: the paw print is a giveaway. Nature 452, 28 (2008).

    Article  CAS  PubMed  Google Scholar 

  20. Maas, B. et al. Women and Global South strikingly underrepresented among top-publishing ecologists. Conserv. Lett. 14, e12797 (2021).

    Article  Google Scholar 

  21. McGillivray, B. & De Ranieri, E. Uptake and outcome of manuscripts in Nature journals by review model and author characteristics. Res. Integr. Peer Rev. 3, 1–12 (2018).

    Article  Google Scholar 

  22. Alam, M. et al. Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi-rater study. Br. J. Dermatol. 165, 563–567 (2011).

    Article  CAS  PubMed  Google Scholar 

  23. Jagsi, R. et al. Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. Int. J. Radiat. Oncol. Biol. Phys. 89, 940–946 (2014).

    Article  PubMed  Google Scholar 

  24. Fox, C. W. et al. Double-blind peer review—an experiment. Funct. Ecol. 33, 4–6 (2019).

    Article  Google Scholar 

  25. Lloyd, M. E. Gender factors in reviewer recommendations for manuscript preparation. J. Appl. Behav. Anal. 23, 539–543 (1990).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Ramírez-Castañeda, V. Disadvantages in preparing and publishing scientific papers caused by the dominance of the English language in science: the case of Colombian researchers in biological sciences. PLoS ONE 15, 1–15 (2020).

    Article  Google Scholar 

  27. Ehara, S. & Takahashi, K. Reasons for rejection of manuscripts submitted to AJR by international authors. Am. J. Roentgenol. 188, 113–116 (2007).

    Article  Google Scholar 

  28. Harris, M., Macinko, J., Jimenez, G., Mahfoud, M. & Anderson, C. Does a research article’s country of origin affect perception of its quality and relevance? A national trial of US public health researchers. BMJ Open 5, 1–10 (2015).

    Article  Google Scholar 

  29. Harris, M. et al. Explicit bias toward high-income-country research: a randomized, blinded, crossover experiment of English clinicians. Health Aff. 36, 1997–2004 (2017).

    Article  Google Scholar 

  30. Steinpreis, R. E., Anders, K. A. & Ritzke, D. The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: a national empirical study. Sex. Roles 41, 509–528 (1999).

    Article  Google Scholar 

  31. Mehta, D. et al. Ways to increase equity, diversity and inclusion. eLife 9, 1–4 (2020).

    Article  CAS  Google Scholar 

  32. Else, H. & Perkel, J. M. The giant plan to track diversity in research journals. Nature 602, 566–570 (2022).

    Article  CAS  PubMed  Google Scholar 

  33. Stefanoudis, P. V. et al. Turning the tide of parachute science. Curr. Biol. 31, R184–R185 (2021).

    Article  CAS  PubMed  Google Scholar 

  34. Stossel, T. P. & Stossel, S. C. Declining American representation in leading clinical-research journals. N. Engl. J. Med. 322, 739–742 (1990).

    Article  CAS  PubMed  Google Scholar 

  35. The World Factbook. Central Intelligence Agency https://www.cia.gov/ (2021).

  36. Amano, T. & Sutherland, W. J. Four barriers to the global understanding of biodiversity conservation: wealth, language, geographical location and security. Proc. R. Soc. B 280, 20122649 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  37. Eberhard, D. M., Simons, G. F. & Fennig, C. D. Ethnologue: Languages of the World (SIL International, 2022).

  38. Crystal, D. The Cambridge Encyclopedia of Language (Cambridge Univ. Press, 2010).

  39. Special Eurobarometer 386: Europeans and their languages. European Commission https://data.europa.eu/data/datasets/s1049_77_1_ebs386?locale=en (2014).

  40. Zhang, X. Effect of reviewer’s origin on peer review: China vs. non-China. Learn. Publ. 25, 265–270 (2012).

    Article  CAS  Google Scholar 

  41. Walker, R., Barros, B., Conejo, R., Neumann, K. & Telefont, M. Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study. F1000Research 4, 21 (2015).

    PubMed  PubMed Central  Google Scholar 

  42. Brooks, M. E. et al. glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. R. J. 9, 378–400 (2017).

    Article  Google Scholar 

  43. R: a language and environment for statistical computing (R Foundation for Statistical Computing, 2020).

  44. Hartig, F. DHARMa: residual diagnostics for hierarchical (multi-level/mixed) regression models. R package version 0.4.4 http://florianhartig.github.io/DHARMa/ (2021).

  45. Ludecke, D., Ben-Shachar, M. S., Patil, I., Waggoner, P. & Makowski, D. performance: an R package for assessment, comparison and testing of statistical models. J. Open Source Softw. 6, 3139 (2021).

    Article  Google Scholar 

  46. Bolker, B., & R Core Team. bbmle: tools for general maximum likelihood estimation. R package version 1.0.24 https://CRAN.R-project.org/package=bbmle (2021).

  47. Hothorn, T., Bretz, F. & Westfall, P. Simultaneous inference in general parametric models. Biom. J. 50, 346–363 (2008).

    Article  PubMed  Google Scholar 

  48. Lenth, R. V. emmeans: Estimated marginal means, aka least-squares means. R package version 1.7.1-1 https://CRAN.R-project.org/package=emmeans (2021).

  49. Smith, O. M. et al. Peer review perpetuates barriers for historically excluded groups. figshare https://doi.org/10.6084/m9.figshare.21865830 (2023).

  50. Tennekes, M. tmap: thematic maps in R. J. Stat. Softw. 84, 1–39 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

We thank authors who sent us additional data we requested (fully documented in Supplementary Data 4), especially S. Burns, L. Fabrigar, M. Flemming, C. Fox, B. McGillivray, T. Paine, R. Petty and T. Tregenza. V. Pan assisted with data collection. A. Hayes and Z. Katz provided input on data collection. Y. Ge, L. Rasmussen and J. Reganold provided feedback on earlier versions of this article. This work was funded by the Michigan State University Presidential Postdoctoral Fellowship in Ecology, Evolutionary and Behavior awarded to O.M.S.

Author information

Authors and Affiliations

Authors

Contributions

All authors conceptualized the study. O.M.S. led the project. All authors collected data with greatest contributions from O.M.S. and C.L.D. O.M.S. analysed the data under guidance from K.L.D., W.L. and C.L.D. O.M.S., R.B.P., R.W., J.C.J., N.N. and C.L.D. made figures. O.M.S., R.B.P., B.F., L.N.J. and C.L.D. drafted paper sections with greatest contribution from O.M.S. O.M.S., K.L.D., R.B.P., R.W., K.C.D., B.F., J.C.J., L.N.J., W.L., N.N., E.E.C., A.H., C.M., A.N.S., O.J.U., M.L.Y. and C.L.D edited the paper.

Corresponding author

Correspondence to Olivia M. Smith.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Ecology & Evolution thanks Sean Burns, Martin Nunez and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Overview of the peer review process covered by studies in our meta-analysis.

a-b Individual journals may use different labels for aEditor-in-Chief and bAssociate Editors. Clock symbol indicates time data were available for that peer review stage.

Extended Data Fig. 2 Corresponding authors whose institutional affiliations are in countries with lower Human Development Indices (HDI) have lower percentages of manuscripts sent to review, but review scores do not differ by HDI.

Estimated means and 95% confidence intervals for a, initial decisions for corresponding authors, and b, review scores for first, corresponding, and last authors. The study underlying the regression in b required that all authors were from the same country, so the values are identical across authorship positions. *** P < 0.001 where indicated (two-sided tests). a, See Supplementary Table 32 for sample sizes, statistical tests used, and exact P-values. b, See Supplementary Tables 30, 32, and 34 for sample sizes, statistical tests used, and exact P-values.

Extended Data Fig. 3 Double-blind review can close gaps in overall acceptance by authors’ affiliations’ countries’ Human Development Indices and primary language.

a, corresponding author overall acceptance increases with increasing HDI at a faster rate under single- compared to double-blind peer review. b, single-blind review favors authors whose institutional affiliations are in countries where English is a primary language. Estimated means and 95% confidence intervals. HDI = Human Development Index, EPL = English primary language, ENPL = English not a primary language. * P < 0.050 for the interaction between peer review model and HDI/language (two-sided tests). See Supplementary Table 53 for sample sizes, statistical tests used, and exact P-values.

Extended Data Fig. 4 Double-blind review can lead to differential outcomes by author demographics.

Estimated means and 95% confidence intervals predicting review outcomes (initial decisions, review scores [lower are better], post-initial review decisions, final decisions, and overall decisions) by author demographics interacting with review model (double- vs single-blind review) for a, first, b, corresponding, and c, last authors. Review scores with data from one study per demographic category show the underlying mean. Single-blind = circles, double-blind = squares. · P < 0.10, * P < 0.05, ** P < 0.01, *** P < 0.001 when reported for the interaction between peer review model and demographic (two-sided tests). See Supplementary Tables 4152 for sample sizes, statistical tests used, and exact P-values.

Extended Data Fig. 5 Ecology and evolution journals are taking few actions to reduce bias in peer review.

a, Many journals require or suggest the use of paid language editing services by authors for whom English is not a primary language. b, Few journals prompt authors to suggest diverse reviewers on their websites. c, But of journals that do ask authors to suggest diverse reviewers on their websites, geography and institution were the most often mentioned axes of diversity (panel shows percent of all journals that mentioned that axis of diversity, even if they did not prompt authors to suggest diverse reviewers). d, Single-blind review is the most common model. e, Only a small percentage of journals publish reviews alongside accepted manuscripts. f, Many journals do not have their own reviewer guidelines. g, Most reviewer guidelines do not mention social justice issues related to demographic equity, including the evaluation of English. Concentric circles show major publisher (outer-most ring, n = 190 journals), society journal published by a major publisher (n = 120 journals), minor publisher (n = 153 journals), and society journal published by minor publishers (inner-most ring, n = 78 journals). Major publishers included Springer, Wiley, Elsevier, Taylor & Francis, and Oxford University Press. Pub = publisher.

Extended Data Fig. 6 Journals and Editors-in-Chief (EICs) are concentrated in a few countries/regions regardless of publisher size.

a, b, Major publishers not affiliated with societies. c, d, Major publishers affiliated with societies. e, f, Minor publishers not affiliated with societies. g, h, Minor publishers affiliated with societies. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Major publishers included Springer, Wiley, Elsevier, Taylor & Francis, and Oxford University Press. Base maps were provided by the ‘tmap’ package in R50. Soc = society.

Extended Data Fig. 7 Journals and Editors-in-Chief (EICs) are concentrated in Western and Central Europe.

a, b, Major publishers not affiliated with societies. c, d, Major publishers affiliated with societies. e, f, Minor publishers not affiliated with societies. g, h, Minor publishers affiliated with societies. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50. Soc = society.

Extended Data Fig. 8 Ecology and evolution journals (n = 541) are taking few actions to reduce bias in peer review.

a, A few major publishers comprise a large proportion of journals. b, Many journals require or suggest the use of paid language editing services by authors for whom English is not a primary language. c, Few journals prompt authors to suggest diverse reviewers on their websites. d, But of journals that do ask authors to suggest diverse reviewers on their websites, geography and institution were the most often mentioned axes of diversity (panel shows percent of all journals that mentioned that axis of diversity, even if they did not prompt authors to suggest diverse reviewers). e, Single-blind review is the most common model. f, Only a small percentage of journals publish reviews alongside accepted manuscripts. g, Many journals do not have their own reviewer guidelines. h, Most reviewer guidelines do not mention social justice issues related to demographic equity, including the evaluation of English.

Extended Data Fig. 9 Journals and Editors-in-Chief (EICs) are concentrated in a few countries/regions.

When standardizing by population size, a, Australia and Europe have the greatest number of journals per one million people; and b, North America, Australia, and Europe have the greatest number of EICs per one million people; and among Europe, c, Switzerland has the greatest number of journals per one million people. d, Western/Central Europe and Scandinavia tend to have the greatest numbers of EICs per one million people. Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50.

Extended Data Fig. 10 Journals and Editors-in-Chief (EICs) are heavily concentrated in Western Europe, particularly for higher impact journals.

a, c, e, g, i, journal locations and b, d, f, h, j, EICs’ institutional affiliations’ locations for a, b, all journals, and journal impact factors c, d, >10, e, f, 3–10, g, h, <3, and i, j, not available (not Journal Citation Reports [JCR] indexed or has not been in JCR long enough). Warmer colors indicate more journals or EICs. Note that the total number of EICs exceeds the total number of journals because some journals have more than one EIC. Base maps were provided by the ‘tmap’ package in R50. JIF = journal impact factor.

Supplementary information

Supplementary Information

Supplementary Tables 1–72, Figs. 1–10 and references.

Reporting Summary

Peer Review File

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Smith, O.M., Davis, K.L., Pizza, R.B. et al. Peer review perpetuates barriers for historically excluded groups. Nat Ecol Evol 7, 512–523 (2023). https://doi.org/10.1038/s41559-023-01999-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41559-023-01999-w

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing