Bias against women in science is a well-studied and well-documented phenomenon. But some cases may not be as clear cut as they first seem. A study1 published this week claimed that female researchers in the Netherlands are more likely than men to lose out when applying for grants. The paper gained widespread support on social media, but some commenters quickly raised doubts. In a blog post, Casper Albers, a statistician at the University of Groningen in the Netherlands, argued that the authors had fallen victim to a common statistical error, which negates the main finding. But the paper’s lead author Romy van der Lee, a psychologist at Leiden University in the Netherlands, says that she stands behind their conclusion that gender affects success.

The study, published in the Proceedings of the National Academy of Sciences (PNAS), followed 2,823 early-career researchers who applied for a prestigious grant awarded by the Netherlands Organisation for Scientific Research (NWO) between 2010 and 2012. Overall, 14.9% of all female applicants were successful, compared to 17.7% of men — a difference that the authors calculated to be only just statistically significant. The study also found that, in evaluations of applications, women received lower scores than did men for “quality of researcher” but not for “quality of proposal”. The researchers also noted instances of male-centred language (such as the use of male pronouns) in written communications. The authors conclude that the findings add up to “compelling evidence of gender bias”.

In light of the published results, the NWO, which commissioned the study, announced soon after that it would review its practices and assessment criteria. The study was also quickly picked up on social media. Henry Otgaar, a forensic psychologist at Maastrich University in the Netherlands, tweeted:

Asifa Majid, a linguist at Radboud University in Nijmegen, the Netherlands, tweeted:

But other researchers expressed doubt. In his blog post, Albers disputed the key finding that women are at a disadvantage when it comes to getting a grant. When he looked at the success rates broken down by discipline, he noted that relatively large numbers of women applied for grants in fields with a low rate of success, such as medical sciences. Conversely, fields where grant-success rates were relatively high — such as chemical sciences — tended to have a bigger share of male applicants. There was no sign of statistically significant bias against women in any single field, Albers writes. In fact, women enjoyed higher success rates in four out of the nine disciplines — but, again, the differences weren’t significant.

Albers argues that the PNAS paper exemplifies Simpson’s paradox, which describes cases in which seemingly significant differences can disappear when all variables are considered. One of the most famous examples is from the University of California at Berkeley. In 1973, the university came under fire because it seemingly admitted a higher proportion of male applicants (44%) than female ones (35%). But further research showed that women were more likely to apply to especially competitive departments for which the acceptance rate was low. “It really struck me how similar [the PNAS paper] was to the Berkeley case,” Albers said in an interview with Nature.

Van der Lee acknowledges that heightened competition partly explains why women overall were less likely than men to win grants, but she says that was just one part of the analysis. The totality of the paper, she says, still makes it clear that gender matters. The fact that there is less money available in disciplines with a large proportion of female applicants is one sign that women are at a disadvantage, she says.

Albers says that he has no complaints about the other aspects of the PNAS paper, including the analysis of gender-based language. If the authors had focused on just the qualitative findings, “that would have been more compelling,” he says.

For more, see www.nature.com/socialselection.