Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Little race or gender bias in an experiment of initial review of NIH R01 grant proposals

Abstract

Many granting agencies allow reviewers to know the identity of a proposal’s principal investigator (PI), which opens the possibility that reviewers discriminate on the basis of PI race and gender. We investigated this experimentally with 48 NIH R01 grant proposals, representing a broad range of NIH-funded science. We modified PI names to create separate white male, white female, black male and black female versions of each proposal, and 412 scientists each submitted initial reviews for 3 proposals. We find little to no race or gender bias in initial R01 evaluations, and additionally find that any bias that might have been present must be negligible in size. This conclusion was robust to a wide array of statistical model specifications. Pragmatically, important bias may be present in other aspects of the granting process, but our evidence suggests that it is not present in the initial round of R01 reviews.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Estimated differences between the Overall Impact scores given to proposals with white male PIs and each of white female, black male and black female PIs.
Fig. 2: Sensitivity of our results to alternative analytic models.
Fig. 3: The relative rate of word use in the written critiques given to white male PIs and each of white female, black male and black female PIs.

Code availability

All code used in this paper can be accessed at our project page at https://osf.io/c5csm/

Data availability

Our data and materials have been deposited at https://osf.io/uy7vq/, which also includes our preregistered protocol. The modified grant proposals have not been deposited for confidentiality reasons. Modified grant proposals can be obtained by contacting the corresponding authors, who will seek permission to share these materials from the research teams that prepared the proposals.

References

  1. 1.

    Ginther, D. K. et al. Race, ethnicity, and NIH research awards. Science 333, 1015–1019 (2011).

    CAS  Article  Google Scholar 

  2. 2.

    Ceci, S. J. & Williams, W. M. Understanding current causes of women’s underrepresentation in science. Proc. Natl Acad. Sci. USA 108, 3157–3162 (2011).

    CAS  Article  Google Scholar 

  3. 3.

    Pohlhaus, J. R., Jiang, H., Wagner, R. M., Schaffer, W. T. & Pinn, V. W. Sex differences in application, success, and funding rates for NIH extramural programs. Acad. Med. 86, 759–767 (2011).

    Article  Google Scholar 

  4. 4.

    Lakens, D. Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Soc. Psychol. Personal. Sci. 8, 355–362 (2017).

    Article  Google Scholar 

  5. 5.

    Kaatz, A., Magua, W., Zimmerman, D. R. & Carnes, M. A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution. Acad. Med. 90, 69–75 (2015).

    Article  Google Scholar 

  6. 6.

    Crowne, D. P. & Marlowe, D. A new scale of social desirability independent of psychopathology. J. Consult. Psychol. 24, 349–354 (1960).

    CAS  Article  Google Scholar 

  7. 7.

    Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Science faculty’s subtle gender biases favor male students. Proc. Natl Acad. Sci. USA 109, 16474–16479 (2012).

    CAS  Article  Google Scholar 

  8. 8.

    Steinpreis, R. E., Anders, K. A. & Ritzke, D. The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: a national empirical study. Sex Roles 41, 509–528 (1999).

    Article  Google Scholar 

  9. 9.

    Lerner, J. S. & Tetlock, P. E. Accounting for the effects of accountability. Psychol. Bull. 125, 255–275 (1999).

    CAS  Article  Google Scholar 

  10. 10.

    Tomkins, A., Zhang, M. & Heavlin, W. D. Reviewer bias in single- versus double-blind peer review. Proc. Natl Acad. Sci. USA 114, 12708–12713 (2017).

    CAS  Article  Google Scholar 

  11. 11.

    Peterson, D. A. M. Author gender and editorial outcomes at Political Behavior. PS Polit. Sci. Polit. 51, 866–869 (2018).

    Article  Google Scholar 

  12. 12.

    Samuels, D. Gender and editorial outcomes at Comparative Political Studies. PS Polit. Sci. Polit. 51, 854–858 (2018).

    Article  Google Scholar 

  13. 13.

    König, T. & Ropers, G. Gender and editorial outcomes at the American Political Science Review. PS Polit. Sci. Polit. 51, 849–853 (2018).

    Article  Google Scholar 

  14. 14.

    Tudor, C. L. & Yashar, D. J. Gender and the editorial process: World Politics, 2007–2017. PS Polit. Sci. Polit. 51, 870–880 (2018).

    Article  Google Scholar 

  15. 15.

    Nedal, D. K. & Nexon, D. H. Gender in the International Studies Quarterly Review process. PS Polit. Sci. Polit. 51, 859–865 (2018).

    Article  Google Scholar 

  16. 16.

    Williams, W. M. & Ceci, S. J. National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track. Proc. Natl Acad. Sci. USA 112, 5360–5365 (2015).

    CAS  Article  Google Scholar 

  17. 17.

    Link, A. M. US and non-US submissions: an analysis of reviewer bias. JAMA 280, 246–247 (1998).

    CAS  Article  Google Scholar 

  18. 18.

    Bertrand, M. & Mullainathan, S. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. Am. Econ. Rev. 94, 991–1013 (2004).

    Article  Google Scholar 

  19. 19.

    Bates, D., Mächler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1–48 (2015).

    Article  Google Scholar 

  20. 20.

    Brauer, M. & Curtin, J. J. Linear mixed-effects models and the analysis of nonindependent data: a unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychol. Med. 23, 389–411 (2018).

    Google Scholar 

  21. 21.

    Halekoh, U. & Højsgaard, S. A Kenward–Roger approximation and parametric bootstrap methods for tests in linear mixed models—the R package pbkrtest. J. Stat. Softw. 59, 1–32 (2014).

    Article  Google Scholar 

  22. 22.

    Fox, J. & Weisberg, S. An R Companion to Applied Regression (SAGE, Los Angeles, CA, USA, 2011).

  23. 23.

    Neale, M. C. et al. OpenMx 2.0: extended structural equation and statistical modeling. Psychometrika 81, 535–549 (2016).

    Article  Google Scholar 

  24. 24.

    Simonsohn, U., Simmons, J. P. & Nelson, L. D. Specification curve: descriptive and inferential statistics on all reasonable specifications. SSRN Electron. J. https://doi.org/10.2139/ssrn.2694998 (2015).

  25. 25.

    Pordes, R. et al. The Open Science Grid. J. Phys. Conf. Ser. 78, 12057 (2007).

    Article  Google Scholar 

  26. 26.

    Sfiligoi, I. et al. The Pilot Way to Grid Resources Using glideinWMS. In 2009 WRI World Congress on Computer Science and Information Engineering (eds Burgin, M. et al.) 428–432 (IEEE, 2009).

  27. 27.

    Tausczik, Y. R. & Pennebaker, J. W. The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29, 24–54 (2009).

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge E. Brandt, K. Lange, D. (Dianne) Lee, J. Marsh, V. Martinez, C. Mitamura, N. Mohan, Y. Lee, C. Henriques, R. Grzenia, K. Scott, S. Staples, D. Statz and P. Rienke for their help in conducting this research. We also acknowledge M. Carnes, C. Ford, A. Kaatz and J. Raclaw for their help in the design of the research. Finally, we acknowledge J. Westfall for his helpful comments on our analyses and J. Fox for his advice on the car package. This research was supported by NIH grant 5R01GM111002-02 to P.G.D. Part of this research was conducted using technical resources provided by the Open Science Grid25,26, which is supported by the National Science Foundation award 1148698 and the US Department of Energy’s Office of Science. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

P.G.D. conceived the research. All authors designed the research. P.S.F. and W.T.L.C. supervised the preparation of materials and data collection. P.S.F. prepared the preregistration. All authors revised the preregistration. P.S.F. and M.B. analysed the data. P.S.F. wrote the first draft of the manuscript. All authors revised the paper.

Corresponding authors

Correspondence to Patrick S. Forscher or Patricia G. Devine.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary Tables 1–4 and Supplementary Figures 1–6

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Forscher, P.S., Cox, W.T.L., Brauer, M. et al. Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nat Hum Behav 3, 257–264 (2019). https://doi.org/10.1038/s41562-018-0517-y

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing