About five years ago, when I was director of the Danish National Research Foundation in Copenhagen, I held focus groups to ask postdocs and early-career researchers how funders might further their work. Members of the board and I spoke with more than 400 young scientists and kept hearing the same depressing refrain: many were writing grants not for work they really wanted to do, but for projects they thought could get funded. Often, they were not even bringing their best ideas to the table.
And why would they? Grant review tends to be biased against innovation; researchers’ best shot at funding is proposing the same sort of work that they have already proved they can do. Although there is some evidence to suggest that peer review can distinguish solid research from poor research, it is not clear that it can identify the very best — especially as falling funding rates demand that reviewers make finer and finer distinctions when selecting which projects to support.
One way to improve the situation is for funders to try different schemes and share their experiences. The Villum Fonden is the largest philanthropic foundation in Denmark for the support of technical and natural-science research. Such foundations have more leeway than organizations funded by taxpayers to experiment with different ways of selecting which research to finance.
Two years ago, when I was director of science at the foundation, we set up a project that we hoped would support innovative ideas by evaluating applications in an unusual way. Assessment of research proposals would be blinded and based on a three-page description. Evaluators would have no information on the applicant’s background or publishing record. By coincidence, I learned that the Volkswagen Foundation in Hanover, Germany, was running a similar scheme; we both hoped to gather evidence on how grant review worked. Each foundation had, independently, dubbed its new scheme ‘Experiment’.
In January 2017, the Villum Experiment called for “science so risky that applicants would not normally consider putting forward the project for funding”. We committed about 15% of our annual funds to this sort of research. We recruited evaluators whom we thought (by reputation) would be particularly able to judge risky ideas — for example, people we knew to have discussed new ways of funding research. They ranked each application they read. Each reviewer was also given one ‘golden ticket’ — a right to fund an application, no matter what their fellow reviewers said.
Funding rates at both foundations were just over 10% of the applications submitted for this call. Recipients included both postdocs and department heads, and about one-third of successful applicants were under the age of 40. So far, the Villum Foundation has awarded 39 grants of up to two years each, and the Volkswagen scheme 96 of up to 18 months; overall, each grant is worth from about €120,000 (US$148,000) to €250,000. In the Villum model, 31% were funded on the basis of golden tickets. Although all golden-ticket grants scored better than most others in this call, about half would not have been funded if based on cumulative scores from all reviewers. In the Volkswagen scheme, 11% were golden tickets, none of which would have been funded otherwise.
In a survey, about half of the recipients said that had the call for unorthodox ideas not been anonymous, they would not have proposed their winning idea — they didn’t think they had a shot if judged on their publishing track records. Reviewers said that they liked evaluating ideas without knowing the applicant’s past performance.
There are wrinkles to iron out. Some reviewers are concerned that if junior researchers’ risky ideas don’t work out, promising scholars will have missed a chance to pursue more-conservative projects. Others warn that recipients might not be qualified to carry out their plans. It is too soon to know, and we want to learn more. The second round of applications closes on 21 March.
Meanwhile, we want to work out how to bring in more ideas. We asked applicants what might have kept colleagues from applying. Answers included discomfort with risky projects; concerns that funding decisions would be haphazard; short-duration and limited funds; the inability to simply reuse another application; and a perception that ideas were either not good or risky enough. The numbers are too small to be certain, but there are signs that men are more likely to get funded. Both foundations plan to tweak how applicants and reviewers are recruited — for example, using ungendered text in the call for proposals — and will continue to monitor diversity.
What interests me most about the experiment is the prospect of better understanding peer review to improve the process. The Bill & Melinda Gates Foundation uses blind review for awards in its Grand Challenges Explorations programme, and New Zealand’s Health Research Council uses a random-number generator to prioritize ‘Explorer’ grant proposals that have fulfilled certain criteria. The global RAND Corporation and an international panel convened by the Canadian Institutes of Health Research have compiled an overview of review approaches and the — limited — empirical evidence for them.
To paraphrase Winston Churchill, grant peer review might be the worst system, except all the others. Given the massive resources dedicated to it, we need a better evidence base to guide its evolution.
Nature 555, 143 (2018)