Working Scientist podcast: Inside the NIH grant-review process

Julie Gould:

Happy 2019! I hope you’ve all managed to take some time to celebrate. As it’s a new year and new years often come with a makeover in one form or another, the Nature Careers team decided to give the podcast a makeover. As well as a new name, we’ve also got a new format. So, instead of our monthly episodes, we’re going to be producing more episodes in 2019 and grouping them together into different series, featuring six weekly episodes followed by a short break. So, here’s series one – funding – and as an added extra, each episode in this series will end with a ten-minute sponsored slot from the European Research Council. So, without any further ado, let’s go....

(Theme music)

Hello, I’m Julie Gould and this is Working Scientist, a Nature Careers podcast.

Grant funding plays such an overwhelming role in the career of an academic scientist, and the funders are all too aware of it. Now, I know that all researchers spend many sleepless nights and cups of coffee writing grant proposals, so when I first started doing the research for this series, I wanted to find the best experts to give you the best tips on how to write the best grant proposals to make things a little bit easier for you.

But then I came across a research paper that made me stop and reflect. In March 2018, Elizabeth Pier – who was then a PhD student at the University of Wisconsin, Madison, in the Educational Psychology department – published a paper as part of her thesis in PNAS.

The paper was entitled "Low agreement among reviewers evaluating the same NIH grant applications." When I first read this title, I thought, "Wait a minute, I thought the idea of the whole funding process was that the top proposals were being funded, the ones where everyone in the peer review system agreed that these were the best ideas supported by the best researchers to do the work."

But clearly, this title shows that there’s something else going on in the background, so I wanted to find out more. The research was funded by the National Institutes of Health, which commissioned an independent study to examine the potential for bias to enter into the peer review process.

The overarching goal of the whole project was to look for evidence of gender or racial bias, based on the characteristics of the PI or the application, and where in the process these biases might enter.

Now, this particular piece of research from Pier is just one of the studies. It recreated a peer review panel to see how these meetings unfold and how they affect the decision-making process. Using previously accepted NIH project proposals, Pier explored this as part of her research. But before we go any further, it’s worth me outlining some of the basic steps of how the NIH proposal review system works.

Now, just so you’re clear, these steps are the bare bones and they miss out a lot of the details, but they should give a flavour of what happens once you hit submit.

So, the NIH uses a two-stage review process.

In the first stage, between two and five reviewers individually evaluate each grant application, and they rate them using the NIH’s nine-point scale, with one for exceptional and nine for poor. They also record what they feel are the application’s strengths and weaknesses.

The reviewers will then meet for what’s called a study section meeting to discuss their preliminary ratings. The discussion only looks at the top half of all the applications they have evaluated. The study section members then collectively assign a final rating, and this is averaged into a final priority score.

So, that’s stage one. Then in the second stage, members of the NIH advisory councils use this priority score and the written critiques from the reviewers to make funding recommendations to the Director of the NIH institute or centre that awards the funding.

So, given all that, I spoke to Elizabeth Pier who now works as a research manager at Education Analytics to find out more about her research.

Elizabeth Pier:

In this particular study, I was really interested in looking at the degree of agreement between different reviewers and what is even happening before the reviewers come together and how are reviewers going about scoring these applications based on their assessments. So, another way of putting that is: Are the reviewers agreeing not only on the score that they assign, but are they also identifying similar strengths and weaknesses in the critiques that they write prior to the meeting, and also what’s the relationship between that numeric score and the written evaluation?

Julie Gould:

So, there were some sobering results...

Elizabeth Pier:

We found that numerically speaking, there really was no agreement between the different individual reviewers in the score that they assigned to the proposals. We also found that when we were looking at the relationship between the strengths and weaknesses, written proposal, and the score that was assigned, we did see a relationship between the number of weaknesses that a reviewer would identify in their critique and the score that the reviewer assigned, but that relationship between the weaknesses and the score doesn’t hold up between different reviewers.

Julie Gould:

Another way of saying this is that the individual reviewers were really consistent – the more weaknesses they identified in the proposal, the lower the score awarded. But unfortunately, it appeared that each reviewer had a different idea of what a weakness is and what score that meant the proposal would ultimately be given. So, what this means is…

Elizabeth Pier:

...we can’t really compare the evaluations of different reviewers and the degree of disagreement that we see in the scores seems to be a reflection of a different sense of calibration in what constitutes a bad score versus a good score.

Julie Gould:

The reviewers do come together for a meeting to discuss the papers based on the initial reviews, and in the meetings that Elizabeth Pier recreated…

Elizabeth Pier:

As you would predict and as people told us based on their intuition participating in these kinds of meetings, the range of scores does get smaller after discussions, so there’s a degree of consensus building within individual peer review panels, but the agreement between different panels actually got wider after discussion, and we had a unique opportunity here because we had four different panels that were evaluating the same applications.

So in practice each application is only evaluated by one study section but for the purposes of this study we exploited that we had these four different groups looking at the same proposals.

And so, in the process of building consensus within a given panel, different panels actually went further apart.

Julie Gould:

So really the outcome that you’re coming to is that it’s potentially better that these reviewers don’t meet?

Elizabeth Pier:

Our studies haven’t indicated any value or benefit in the sense of improving the consistency or reliability of the process.

Julie Gould:

But what about the variability in the quality of the proposal being discussed, doesn’t that make a difference?

Elizabeth Pier:

But we had to ask people to donate their applications and the summary statement that they received to us and the donations that we received just happened to be funded. And so, we tried to say that above a certain quality threshold, our results suggest that it’s essentially a random process and the meeting doesn’t seem to remove that randomness.

However, I will say a caveat to that caveat is that the applications that get discussed in the meeting have already gone through triage, so only the top 50% of applications based on their preliminary score even get discussed in the meeting. So, what we are talking about is given that top 50% of proposals, after you’ve already excluded the ones that really have no chance of being funded initially, there really is a lot of randomness, but even more so, there’s already randomness such that the applications that have been weeded out so to speak and don’t get the opportunity to be discussed in the meeting might actually have a lot of merits. #

Had it been assigned to a different panel with different reviewer it very well could have gone on to be discussed.

Julie Gould:

So, what you’re saying really is that luck plays a very large role in whether or not your research gets funded.

Elizabeth Pier:

Yes, that is what our results suggest. Above a certain degree, if you have a relatively competitive application, there aren’t any major issues that would immediately disqualify it to any kind of representative effort in the field, there’s a great deal of randomness and luck that we find in determining who does and does not get funding.

Julie Gould:

So, what does that mean for all those people who are spending all these hours and hours and hours and hours and hours on getting their funding applications organised and sorted and written up? I mean my heart goes out to them...

Elizabeth Pier:

Yes, my heart does as well. I mean it’s one of the reasons I studied this for my dissertation because it’s incredibly important for individual careers and also incredibly important for the progress of science, right?

We want to make sure that the most deserving ideas are getting rewarded and funded and that it’s not just picking out of a hat.

So, I mean there are a couple of pieces of silver lining. I think that we see evidence, especially as grants get resubmitted, that being responsive to reviewers’ critiques can play a strong role in conveying to reviewers improvement over time, and so there is something to be said for if you get rejected or you don’t get funded, having some tenacity and resubmitting that application and doing everything you can to address the reviewers’ critiques and feedback.

It can make a potential difference. I also think that it’s important for folks not to take it personally. As academics, we definitely are used to rejection and used to plenty of times when we think we have really great ideas and reviewers of manuscripts or of grant applications don’t seem to agree with us, so I would encourage people to take a little bit of solace in that it’s not necessarily a reflection of the quality of the ideas but it’s more kind of a feature of the process.

Julie Gould:

How would you suggest then that the process is improved?

Elizabeth Pier:

There should be some assessments of whether what some scholars have called a "modified lottery system" could do.

So, the idea being that there’s some initial screening process that experts do conduct to make sure, like I said, they’re kind of weeding out any really problematic proposals, things that are just wildly out of left field in terms of being feasible to complete given the budget or things like that.

And then after that kind of initial screen then it really is just a random selection. And the reason I think that would be an improvement is because if the process is already random above a certain quality threshold, which our study suggests it is, we might as well save the money and the time involved to convene thousands of people and spend millions of dollars to have these meetings if the outcome is essentially the same as a random process.

Julie Gould:

Now, we’ll touch on the idea of a lottery-style funding system later on in the series, but what we can say now is that change is going to be slow – it always is in academia.

Is there anything that can be done in the meantime, before this lottery style system or something completely different is created?

Elizabeth Pier:

Starting to accept the fact that it’s not a completely objective process, that humans are fallible, they are subjective, and when you’re asking experts to make very complex judgements about the potential likelihood of success of a project, that’s a really difficult decision that’s going to bring in a lot of heuristics and biases that go into their decision making.

Julie Gould:

My final question to Elizabeth was what advice have you got for anyone who’s currently writing a grant proposal to the NIH.

Elizabeth Pier:

One piece of advice, which is probably pretty obvious but I will say is backed by our findings, is that weaknesses are much more predictive of the score that reviewers will assign, rather than strengths. So, what that means is minimise as many weaknesses as you can.

Julie Gould:

So, after all of that, I’m intrigued. What do you think? How would you feel about a more lottery-style funding system? Please send in your thoughts to the Nature Careers team which you can do via Twitter, Facebook and LinkedIn. And over the next two episodes, I’ll be speaking to different experts on how to minimise the number of weaknesses within your funding application, in the hopes – fingers crossed – that you’ll have a bit more success and a bit more luck.

Now, that’s all for this section of our Working Scientist podcast. We now have a slot sponsored by and featuring the work of the European Research Council. Thanks for listening. I’m Julie Gould.

(Theme music)

Jean-Pierre Bourguignon:

So, my name is Jean-Pierre Bourguignon and my title is President of the European Research Council, which of course is supported by the European Union as through the European Commission.

I’m a French mathematician, I should say. I spent most of my career in the CNRS (Centre National de la Recherche Scientifique).

My field was differential geometry, but did a lot of work actually at the boundary of theoretical physics, general relativity and Dirac operators and these kind of topics, but still always as a mathematician.

The European Research Council is actually an interesting story. It was created in 2007, so it’s now 11 years of age, and it was a long process. Myself, the first time I heard about the possibility of having an ERC was 1995, and it was a long effort by the scientific community, and step after step we had to convince people in the Commission, people in the European Council – namely the countries – that they should support such a project, but still it has a lot of very specific characteristics, particularly the power which has been given to its scientific council is considerable.

It really was an innovation and the council has the responsibility of deciding on how to spend the money and how to do the evaluation. This is unique in the setting of the European Commission, that a group of 22 scientists are given such a responsibility and of course as President of the European Research Council, I have some very specific ones, which is to confirm the list of people who are granted and really guarantee the quality of the work done.

The mission of the ERC was really to make Europe more attractive, to be a place where science can develop really in the most ambitious way and to push the ambition, particularly of young people, upward, that is to make them independent early enough and to take their vision on board.

You know, we are at the stage of giving 1100 research grants this year, which is of course a very significant amount of money. The budget is now really over €2 billion per year, and we are covering all fields of science – that is physical sciences, engineering, including maths, computer science, and so on, life sciences, social sciences and humanities.

If you want to know what you are doing, you need to talk and meet and discuss with the people you are funding and so I do travel a lot, particularly in Europe, to meet the people we call our grantees – the people who get the grants from the ERC – and this part of my job is really extremely worthwhile and extremely rewarding because the selection process is a very tough one – the typical success rate at this moment is 13%.

And it means that people have all proposed very ambitious projects that are conditioned to be successful at the ERC, high risk again, we need to encourage the panels who are selecting people to really accept to take risks. And that’s one thing I hear regularly from grantees, telling me, ‘I submitted a very similar project to my national agency but then I was not funded, it was considered too risky, then I submitted to the ERC, and then the ERC funds me,’ so it makes a big difference.

Another component which is very important in our strategy is the fact that the clause, which has been put in place by the European Research Council.

Really, we have three categories at the moment for the individual grants, which is the starting grants, consolidator grants and advanced grants, and it means timed to PhD.

So, starting grants – to 2 to 7 years, consolidator grants – 7 to 12 years, and advanced grants – there is no condition, it just means people who are already confirmed, and while doing that it means that in the end we are dedicating typically two thirds of our budget to the younger people, people who are typically below 40 years of age.

Very often people get the belief that really if you are not from one of the leading research institutions in Europe, then you have no chance. This is not the case. I mean the institution which is your host institution is not part of the evaluation, which is really the key for the evaluation is the project.

You have to show that you have thought of what kind of resources will be needed, and you describe them, but this is not the institution as such which is very important.

So, it means that in particular in terms of the support we give, part of it could be also buying expensive equipment if you need them and if it’s not available in your institution.

So, we consider the project, not just as helping the people, but helping the people also to set up the environment which will make it possible to get the project through.

So, this is sometimes one misconception that people have, that they get the feeling that if they are coming from a smaller place, they have no chance.

The number of institutions the ERC has been signing with is close to 800 now, so of course it’s quite a significant number of institutions based in Europe and of course some of the leading ones got more grants than others, but definitely even small institutions have been very successful at the ERC.

One of the key things that the ERC is doing is empowering researchers. This is something very, very important for us, and a very good example for this is one of the specific characteristics of the ERC programme which is called portability, but the host institution is not part of the selection criteria – it’s just here to make sure that there is a legal body that is able to receive the contract and it gives a lot of power to the researchers.

And one of the typical powers, what I mentioned, portability, which means that the researcher can change the host institution if he or she feels that it’s not given the proper treatment, or maybe they could give other personal reasons to do that.

This is the whole philosophy behind it – we really want the researchers in the driver’s seat at the level of the council but also at the level of how they run their contract, and of course there is an institution behind it because you want to be sure that there is a legal basis for this, but we really want the researchers to be able to do their research in the best possible conditions.

The map I have in front of me, which is the map of the world, has on top of it one of our mottos which is ‘Open to the world.’

One of the conditions to be funded is that you have to spend at least 50% of your time in Europe, but you can be from any country, and we want to be sure that Europe is the leader to tackle some of the most challenging scientific problems.

At the very beginning, we could notice that the percentage of women who were applying to the ERC was less than the percentage of women in the scientific community, and we felt this was definitely not adequate.

Also, we had for the ones who applied, the success rate was definitely lower than the success rate for men. Through very sustained efforts, and identifying the issue from the very beginning, I think we made very significant progress.

So, first of all, the percentage of women applying to the ERC has been steadily growing.

We are now basically at the level where the percentage of women applying to the ERC is very similar to the level of the percentage of women in the age group of the different goals we have.

So, from that point of view, I think we really achieved something which means that there’s no some kind of resistance or reluctance of women to apply, so this is one step.

And then, of course very important but I think the two are linked – the fact that in recent years, the ERC women have been on average more successful than men.

It’s a very slight difference but since we started the situation shows the opposite.

We are very pleased that all the efforts we made, particularly to tackle complicit bias or various other things have been more or less successful.

I’ve been visiting many, many countries in Europe, in particular countries in the "EU13," because I feel you need to understand the real situation people are exposed to in the various countries, and they are the ones who joined Europe the most recently, and most of them located in the eastern part of Europe and it’s very important to realise that actually that situation can be quite different from one country to the next.

It has to do with teaching load, it has to do with the power structures in the institutes, it has to do of course with the support which is available to people, so for me it was very, very important to meet the researchers because that’s for me the key point.

Also, to meet the authorities in these countries and to understand in which environment they are operating because I think that’s the very best way.

We, at the level of the European Research Council, have also introduced some help, in particular by encouraging the various countries, could be also regions, to support possibilities of researchers in typically underrepresented regions (it could be EU13 countries) – to really give them some possibility of spending some time with the support of their countries or their region, in ERC teams, so that they understand what it takes to submit a proposal, but also to understand better, to really also test their ideas with other people so that then they have a much better idea what it takes to submit a proposal and therefore they are better prepared personally, not just intellectually, to really submit a proposal in good conditions because they have seen what a difference it makes and also what also kind of effort you have to put in if you want to be successful.