Working Scientist podcast: The grant funding lottery and how to fix it

Julie Gould:

Hello, I’m Julie Gould and this is Working Scientist, a Nature Careers podcast. This is the final episode of our series on funding, but just a quick note, don’t forget that there’s also a final ten-minute sponsored slot at the end of this Working Scientist podcast from the European Research Council.

Now, throughout this series, we’ve heard a lot about funding – what’s the best way to prepare for writing a grant, how to write that grant, how to make sure it gets read, how to prepare for an interview should you have one, and then we looked a little bit broader at the funding environment.

Now, one of the things that I found really interesting, if we look back at the very first episode, is something that Elizabeth Pier said about what her research suggested....

Elizabeth Pier:

Given that top 50% of proposals, after you’ve already excluded the ones that really have no chance of being funded initially, there really is a lot of randomness, but even more so, there’s already randomness, such that the applications that have been weeded out, so to speak, and don’t get the opportunity to be discussed in the meeting, might actually have a lot of merits.

Had it been assigned to a different panel with different reviewers, it very well could have gone on to be discussed.

Julie Gould:

So, what you’re saying really is that luck plays a very large role in whether or not your research gets funded.

Elizabeth Pier:

Yes, that is what our results suggest.

Julie Gould:

And then add to that what Michael Teitelbaum mentioned in our fifth episode, that the NIH has experienced a period of flat funding for the last couple of decades, which has added stress to the system.

Michael Teitelbaum:

In the 1990s, was a decision by the US Congress and the presidential leadership of both parties to double funding over a five-year period for the National Institutes of Health, a massive increase for five years, averaging about 14% per year, that then was followed by flat funding for subsequent years.

Julie Gould:

As Michael mentioned, it’s difficult to tell whether or not you’re going to be in a boom/bust cycle when you’re actually in it, but this prolonged period of flat funding might not be part of a cycle at all. It might be a new norm.

Ferric Fang:

And I think for a long time, people thought this is going to be cyclical, and things are good and then they’re bad and then they’re good again, and we just have to wait. But I think it’s gradually dawned on people that it’s not cyclical in any kind of an orderly way, and that it may be the new normal for scientific funding, where there’s a shortage of funding for the size of the workforce and there’s a problem with job opportunities for new trainees, and this is something that I think is belatedly being addressed.

Julie Gould:

So, that was Ferric Fang and he’s a professor at the University of Washington in Seattle, and he, like many others, is concerned that the current funding system in the United States isn’t working. So, in a time when there’s inadequate funding for the size of the scientific workforce and the researchers are spending increasing amounts of time applying for this funding, what is the best way of allocating not-enough money to more researchers than the system can support? So, Ferric and a colleague of his, Arturo Casadevall, suggest that a modified lottery system, like the one Libby Pier suggested in the first episode of this series, could be the answer.

Ferric Fang:

And we came up with the idea of a two-tiered lottery system where initially there could be a review to divide grants into these two hypothetical stacks of high-quality grants and then the others, and the other grants could be sent back to be revised and hopefully improved and many of them could come back and eventually enter the lottery. And then you would have the other grants which are all judged to be of high enough quality to be supported, and then you would see how much funding was available, and you would randomly then prioritise the grants and you would fund accordingly. And you could introduce lots of nuances into the system, in terms of the number of grants that any given investigator could have in the lottery.

Julie Gould:

Now, as well as the benefits of reducing the amount of money and time spent on peer review, Ferric and Arturo argue that it could have wider implications for the entire funding environment.

Ferric Fang:

And a school that had a large number of researchers could be reasonably certain, based on laws of probability, that they would get a fairly predictable amount of funding based on the meritorious work that their researcher were doing, even though there would be little fluctuations. I think because of the large numbers it would even out. Another thing you could do is go to policymakers and say this is the amount of meritorious proposals that our scientific enterprise is producing and yet we’re only funding a small percentage of them, and this could be the basis for making more rational assessments of how much research funding should really be allocated in a budget.

Julie Gould:

I asked Ferric what he thought people might think of this modified lottery-style funding.

Ferric Fang:

I think a lot of people’s initial reaction to it would be that it would be leaving the future of the scientific enterprise to chance. But it’s no more irrational than trying to hedge your bets when you’re trying to invest economic resources for your future and trying to figure out how to make a diversified portfolio. We really want to make sure that our blind spots, in terms of our biases, aren’t preventing us from funding ideas that could really be transformative for society in the future.

Julie Gould:

Now actually, this modified lottery system does already exist. So, to fund any innovative and transformative research, the Health Research Council in New Zealand set up their explorer grant, which operates as a modified lottery system. Vernon Choy, who’s a director of Research Investments and Contracts at the Health Research Council, told me a little bit about their system and how it’s working for them.

Vernon Choy:

So, the way that it works is we do use a panel, but the panel does not discuss the ranking of the applications that come through to us. What they do is they provide us with an opinion on whether the application is transformative and we do have a particular definition of transformative. So, they must decide whether the application is transformative and they must also decide whether the application or the research proposed is viable. So, having reached the point where the panel agree that an application is fundable and meets the requirements of the explorer grant guidelines, then those applications go into a pool or thunderball and then we use a random number generator to allocate the funds to those applications using the random numbers that are then ranked, and then we fund according to that random rank to fit within the budget available for that particular round.

Julie Gould:

And how has this particular mode of allocating funding been received by the scientists and the health researchers in New Zealand?

Vernon Choy:

Well it’s surprising. At the time, we felt that this was going to be highly controversial, and in some respects, it was and still is, and obviously there has been a continued interest internationally in the explorer grant, but from our point of view, our researchers have accepted both the way that we allocate the funds and also the way that we determine eligibility or fundability. We did a survey back in 2017 of people that had applied for the fund to gauge their thoughts on both the format, the allocation method, the processes overall, and basically, we had quite good from everybody. One of the things – and I haven’t talked about this – but one of the things that we do is the applications are anonymised so that in determining whether an application is eligible, there’s no bias against any particular institution or against any particular team of researchers. When the process was first announced and we had a huge number of applications, and one of the reasons we were told was well this was a fund that allowed young, inexperienced researchers to compete against senior researchers and because there was no bias towards the experienced researchers. The other thing that we’ve investigated is the gender balance in the applications because of the anonymisation, and I would like to say that there was no gender bias in these applications, but from our initial look at numbers, there is still a slight bias towards men – not a huge one, only 3-4% – but it’s still slightly different between men and women. So, that’s difficult to say why that might be. Potentially it could be the style of text and the way that people write, but apart from that we’re quite happy with the explorer grant so far, and I’m expecting that the funds that we have available to allocate this way will increase.

Julie Gould:

So, time will only tell whether or not this is really a great system, and maybe expanding it further will give people a better idea of how it will work across a larger research system. But there are others who are taking different approaches, and one of these was by Johan Bollen who’s a professor at Indiana University. He and his colleague Marten Scheffer, out of sheer frustration with the time-consuming and expensive funding system that’s currently in place, thought well what if we just give everybody a pot of money at the beginning of the year, and then implement a rule where everybody has to redistribute a certain percentage of their money to another scientist. So, they’ve called it ‘self-organised funding allocation’ or SOFA for short. And here’s Johan describing how it works.

Johan Bollen:

Essentially, you’re a young researcher, you’ve just been hired as an assistant professor and at the end of the year you receive a fixed and unconditional amount of base funding in your research funding account at the university, and you know that the re-donation fraction is 50%, which means that you can keep 50% of that and then the other 50% you’d have to donate to other researchers of your choosing. You log into a website that could be run by the National Science Foundation and you enter the names. There could even be a pull-down list. There could even be, I wouldn’t call it a recommendations system, but an order completion system where you enter the names of the scientists that you would like to donate a fraction of that 50% to, and when the system has determined that you have completed the list of names and the relative fractions and it adds up the 50% of the money you have received, you hit submit and you’re done. The next year you receive the same base amount and perhaps funding from other scientists that saw you speak at a conference or that read your paper and really liked the work that you do and would like to support it. You add it all up, again you take 50% for your own research needs and the other 50% again, you log into the website and you enter the names of the individuals and how much money, or percentage of the money that you’re supposed to donate, that you wish to donate to them and then hit submit, and you’re done for another round.

Julie Gould:

But how would you then decide who to give your money to? I mean so you want to get rid of the time-consuming grant proposal writing – yes, I know it can be a painful process – but then how does a person decide who to give their money to if they don’t have all these grant proposals to read.

Johan Bollen:

This question is asked lots – how do you know who to give your money to – and the thing is that as scientists you’re supposed to know who does the most exciting work in your area. I mean that’s how we write our papers. If you look at the bibliographies in our papers, our references etc., they’re essentially a testament to the obligation that we have to stay abreast of the developments in our area. You’re not very good as a scientist if you don’t know about the work that’s happening in your research area. And so that same assumption is true if people would have to make decisions about who to pass their money on to, and so you can actually show mathematically that under the right conditions, this process of the money being passed from one person to the next could lead to convergence of funding across the entire community that reflects all of the knowledge in the system, not just of one particular individual, but of all individuals that participate in the system.

Julie Gould:

What would stop people from just funding their colleagues, their collaborators or even their friends?

Johan Bollen:

First of all, I don’t know whether that’s such a bad thing to begin with. People do collaborate and they don’t just collaborate within institutions, they collaborate externally, but if you’re really concerned about it, you could very easily enforce the exact same kind of conflict of interests rules that we have right now with respect to the submitting and review of proposals. For example, you could introduce a rule that you couldn’t donate to people within your same institution and, for example, that you couldn’t donate to the same people more than two years in a row. You could even mandate that a given fraction of your money goes to underrepresented groups. So, there’s a lot of social distortions that you could fix very easily by limiting on the basis of very reasonable arguments who to donate the money to.

Julie Gould:

And what about the early career researchers, those researchers that are just starting off in their career in science. How do they promote themselves in order to get some of the funding from other people?

Johan Bollen:

Well, first of all, everybody receives the same amount of funding regardless of your merit or how well-known you are, everybody receives the same base amount, so all of those young researchers have the base amount to begin with. Then of course there’s a challenge in getting your name out and convincing the community at large that you’re doing good work. That involves going to conferences, giving presentations, getting in touch with your colleagues. These are the kind of things that young researchers do anyway, but now of course it would be crucial to getting their name out. So, I think it would benefit the overwhelming majority of early career researchers.

Julie Gould:

Nobody really knows how this scientific funding system is going to organise itself over the coming years, but I would be really keen to hear your thoughts. What do you think of this concept of a self-organised funding system or even the modified lottery system which is already in place in New Zealand? Or have you got any experiments or paradigm-shifting ideas of how the funding system could be changed? If you have, get in touch – we would really like to hear from you. Something else we’d like to hear from you about is what series would you like to have on the Working Scientist podcast? So, we’ve now finished our series on funding, but what else do you want to know about? Each series will have five or six different episodes with a variety of experts on that particular topic, but we’d like to get your input into how to shape our future series. So, if you have any thoughts or burning desires about what you’d like to know more about then get in touch with the Nature Careers team. I want to give one final thank you to everybody who has contributed, so that would be Johan Bollen, Vernon Choy and Ferric Fang from this episode, as well as Michael Teitelbaum, James Wilsdon, Peter Gorsuch, Anne-Marie Coriat, Jernej Zupanc and Elizabeth Pier. Thank you again for contributing your thoughts and ideas to this series.

And that is the end of this series on the Working Scientist podcast, but before you go, just a reminder that there is another last sponsored slot by and featuring the work of the European Research Council, and in this slot we hear from the President of the ERC, Jean-Pierre Bourguignon, and then also from Professor Helen Tremlett from the University of British Columbia in Canada. Thanks for listening. I’m Julie Gould.

Jean-Pierre Bourguigno:

My name is Jean-Pierre Bourguignon, and my function is to be the President of the European Research Council. I’ve been in this position for five years now and I still have one year to go in my mandate. The search for my successor has started.

So, in the sense that when you have reached such a level of success, the first priority is of course making sure that you still are in a good position to continue with this success, and the main priority of 2019 will be to revisit basically every way we do the evaluation because we know we have some challenges. For example, for some of the panels we have reached a size which means that we have to think of organising slightly differently because to do a good job as evaluators, you cannot have too many applications because then you cannot dedicate enough attention to them. So, we are really going to go through a very, very thorough check of all our evaluation systems, of course, taking advantage of all the knowledge accumulated with the scientific people who are members of our panels for evaluation, but also really trying to get advantage over not being too frozen, too rigid or too persistent on the way we structure these… I’ve said we cover old domains of science, but science is changing all the time, so you want to be sure that you adapt to the new emerging fields quickly enough that you bring on board all the right competent people. So, this is really for the immediate future because that’s a priority for 2019, and we want also to announce the new way we want to do the evaluation early enough so that the scientific community will be ready for when it will be put in place in 2021, and the scientific community has absorbed these changes, understood them, and can really adopt them and in particular that we will be able to continue to convince the very best scientists in the world to participate in the evaluation.

Well, first of all, I mean ‘open to the world’ is one of our mottos. It means, of course, already that we have on board scientists from I think about 80 nationalities, so it means it’s not just Europeans who are a part of it. But of course, another part is for the ERC to interact with agencies in other countries in the world. We have already 11 countries with which we have signed agreements. For the moment, these agreements are of the type that researchers from these countries funded by these agencies can visit and spend time in some of the ERC teams.

Helen Tremlett:

So, my name is Helen Tremlett. I’m a professor in the faculty of medicine and neurology at the University of British Columbia in Canada, and I’m Canada Research Chair in Neuroepidemiology and Multiple Sclerosis, and I’m a British citizen and a Canadian citizen. I’ve been here since 2001. I was part of a programme between the Canadian government and the ERC, which enabled Canada Research Chair holders to spend time in the lab of someone who holds ERC funding. So, it was a great opportunity to bring together individuals who have complimentary skills and can learn from each other and develop collaboration over the long term. And it was a wonderful opportunity. I was based at the Max Planck Institute on the edge of Munich and they were focused on the gut microbiome in multiple sclerosis.

It was so exciting. So, 2011, I can even remember that day. Nature published a paper and they were looking at the animal model of multiple sclerosis and how the gut microbiome may be influential in terms of triggering the animal model of multiple sclerosis. I had no idea that people were even thinking about this, and this led me down a whole new research path and now I’m actually coordinating principal investigator on a study where we’re collecting stool samples from children with multiple sclerosis and controls across Canada and across the US, and so it was thrilling for me to spend time in the lab whose work had really pushed me onto this path, so it was a lot of fun.

So, there’s no additional funds attached to it, but it just meant that it was a formal opportunity and your salary was continued as such without a break. You didn’t have to take it as a sabbatical leave or anything like that. And I was just there for two months, but it was a really great two months.

Jean-Pierre Bourguignon:

During my time we have signed agreements with China, with India, with Brazil, with Australia, with Japan, so of course, these are countries which worldwide play a very critical role. I should also mention South Africa with which we have also developed a very interesting collaboration. Still, we want for the future to actually have more tools. For example, for the moment, the tools we have are only the ones I describe – namely visits by scientists from these places to visit ERC teams. We hope that in the next framework programmes, some more agility will be given to the scientific council, and having the possibility to also accompany researchers from our teams who want to visit abroad. One of the very simple principles of international collaboration is typically reciprocity – that is what you make possible in one direction should be possible in the other direction. For the moment, as you heard, the only possibility is people from these countries to come and visit Europe. We would like also to help and accompany researchers from Europe who have got the ERC contracts to also be helped when they want to visit researchers from other countries outside Europe.

Something we reintroduced very recently are the so-called Synergy calls, so it’s a different call from the other ones. They are really for individual principal investigators, as we name them. In the case of Synergy, it’s really to encourage more ambitious, more global projects with two, three or four PIs (principal investigators). Of course, the idea is not to create a consortium. It’s really the idea that people come up with a truly challenging scientific problem they want to address, and we call it Synergy because we want them to really convince us that they are really the right group of people to tackle this. So in particular, we see this as a very specific place where interdisciplinary work can be developed. So, in a sense we wanted to create such a space where really people who need resources and skills and knowledge, expertise from different fields, can come together to tackle a very well-identified problem and to do that together. And so, this has been we have run only one such call so far for the year 2018. We just published the results. So, 27 projects have been supported. I didn’t mention globally the number of projects we have supported – we are typically at 9,000 projects overall supported – but the Synergy project is for very interesting new challenges. So, this is another dimension that ERC and the scientific council wants to tackle – that is to acknowledge the great importance for the future development of research of interdisciplinary work, that people need to learn how to work together but the way we do it is again under the very strict bottom-up philosophy. We just want people to come up to us, come forward with ambitious projects and very challenging problems they want to tackle and to try and convince the evaluators that they are the right people to do that and that they have assembled really the people who can do that in the best possible way. So, this dimension of Synergy is also that we want to be sure that Europe is the leader to tackle some of the most challenging scientific problems.