Access
This article is part of Nature's premium content.
Published online 22 February 2008 | Nature 451, 1035- (2008) | doi:10.1038/4511035a
News
Revamp for NIH grants
US funding body receives recommendations for improving its peer-review process.
Scientists applying to the US National Institutes of Health (NIH) for grants could be accelerated through the painful peer-review process under recommendations proposed on 21 February aimed at overhauling the decades-old system.
Currently, a grant proposal can take 18 months to pass through the system, waiting in line behind older applications, most of which must go back and forth to the applicants for rewrites and amendments before approval.
To read this story in full you will need to login or make a payment (see right).
Comments
Reader comments are usually moderated after posting. If you find something offensive or inappropriate, you can speed this process by clicking 'Report this comment' (or, if that doesn't work for you, email redesign@nature.com). For more controversial topics, we reserve the right to moderate before comments are published.
The proposed change in NIH grant review methodology is certain to make things worse, and to reward superficiality at the expense of solid, critical work. Reviewers are as faillible as applicants, and the response section of resubmissions is often the most meainingful part of the grant. It gives the applicant a chance to explain why a criticism is wrong. And what is the point of having lots of funding for 30-years-olds and leaving them without a grant or a job when they are 45?
I agree with most of Claud Wasterlain's comments, particularly about eliminating the "Introduction" and an over concern for new investigators. One could argue that artificially supporting new investigators simply worsens the situation in which we have too many investigators in the first place. However, the biggest problem is that indirect costs are too high and eat up far too much of the budget. Indirects make research a money-making enterprise for institutions, and that is bad for science because it encourages unsustainable growth and a destructive system in which past performance counts for little and the only thing that matters is current funding. While the costs are real in one sense, it is not clear to me that they should all be borne by the taxpayers. I do applaud shorter proposals. Our current system punishes the innovative and immaginative and because of the over reliance on mechanistic, hypothesis-driven research forces knowledge ever deeper along known paths rather than forging new ones. Thus we continue to learn more and more about less and less, as some wag characterized our system. Meanwhile, we still have not cured cancer, perhaps because we continue to pursue the usual suspects.
NIH already has several grant structures designed to support new or junior investigators and there are other startup funds available on a competitive basis. In my view, the RO1 playing field should be level. For reviewers to not see the previous critique is a change that could be argued either way. I doubt that it is a significant problem in fairness.The qualifications, effort, and performance of reviewers are far more important and they should be evaluated on a regular basis. I agree with Dr Hurst and would add that the stature of investigators would be greatly elevated if every institution had to document some accountability to its investigators for the use of indirect funds. If this was required, I am sure systems for accountability could be created that would be constructive for all parties.
I agree with both, Drs. Hurst and Wasterlain. There are several avenues for young investigators to obtain good amount of funding, including the NIH, NSF various state sponsored funds etc. In addition, there are many charities which specifically support young investigators, young meaning, people without R01s, to prepare or reapply for NIH grants. Other than an institutional and occasionally state sponsored BRIDGE funds, there is hardly any support to mid-career scientists who have a cadre of graduate, post-doctoral and experienced technicians to support their research, provided they are funded, although they might have a good track record and proven project. Considering the grant review is mostly based on one at most two reviewers, the rest are often persuaded to join the primary and secondary reviewers, the evaluation is often subjective or hasty. The researchers who think shorter grant will relieve of their review burden are mistaken, as the SRO's are likely to load them with more proposals to review! What we all need is inflation adjusted appropriation and a cap on how many grants, at most 3, a PI can be funded on simultaneously. I disagree with the committee which suggested a large fraction of the proposals should be dismissed as "not appropriate for further consideration", considering all ideas evolve and improve upon criticism and re-evaluation, the educated person who put in huge effort with personal sacrifice to prepare a proposal should be given reconsideration. If the committee's suggestion stand as it is, I am sure the investigators will come up with creative way to resubmit their application. BL
I suspect that the unexpected outcome of this proposal will be the opposite of the intention. Without preliminary data most panel members and reviewers will not believe that an innovative idea has any credibility. The outcome will be that research that follows well-worn paths will get funded because everyone will agree this is reasonable. The unexpected and innovative ideas will not get funded because they truly are unexpected
If the goal of NIH is to fund innovative grants and investigators the new system should consider a design that can be improved by iteration. One approach would be to select reviewers based on their success in recognizing innovative grants. NIH could pay these “experts†for extended service by providing them with a small grant sufficient to support a lab manager and cull reviewers with poor grades. Another area that needs to be addressed is the format used to recognize the most innovative researchers. Most grants are reviewed based on written proposals, but when NIH’s explicit goal was to identify the most innovative researchers using Pioneer Awards their review panels used oral interviews. Finally, the new system needs to meet the standard of scientific practice and have a “control group†to determine if it is working. I suggest a small group of “placebo grants†be funded based on lottery. If the new system beats the lottery, then the cost benefit of beating the lottery will be known. If it doesn’t, NIH will need to take a hard look at how it evaluates and assesses innovation.
In his book "Science Observed" Jevons noted that asking a researcher about the funding system is like asking a bird about aerodynamics (1). This is well illustrated by some of the "reforms" of the peer review system proposed above. Nowhere is there recognition that peer review as it currently operates is highly error-prone. Nowhere is there recognition of a need for system redesign taking error-proneness into account. Despite lip-service to the contrary, granting agencies assess projects, not people. In the final analysis, the agencies consider it better that a less able researcher carry out an approved project than a more able researcher carry out an unapproved project. Indeed, the agencies hope with the funding carrot to coerce more able researchers to carry out approved projects. For the less able researchers this is not a problem. They just have to write an honest application stating what they want to do and why they want to do it. On the other hand, the more able researchers, who can see beyond the conventional wisdom, have serious difficulties. Grant writing is a marketing exercise that, more often than not, requires that their "best" ideas be discarded because, almost by definition, these ideas are difficult to understand and communicate (if not, the less able researchers would have already thought of them). Thus, the more able researchers are tested, not on their abilities to come up with innovative ideas, but on their abilities to tune into the conventional wisdom and write proposals with an appropriate degree of marketing spin. Many able researchers, and especially the most able, find this not only distasteful, but impossible. People find this difficult to understand. Why can't the most able researchers just write a simple application and then, when they have the money, use it to do the work they really want to do? Unfortunately (or fortunately, depending on your perspective), the most able researchers have one common attribute, integrity. They can no more discard this than a tortoise can discard its shell. In summary, the peer review system as it currently operates discriminates against the most able researchers, thereby achieving the very opposite of what is desired. If true, this means that over past decades peer review has progressively "dumbed down" the Professoriat, thus impairing the process of scientific discovery and, more seriously, decreasing the quality of "expert" advice available to governments (e.g. on matters such as bioterrorism). A possible solution has been on the table for some decades, but would appear to play no role in current discourse (2). (1) Jevons, F. R. (1973) Science Observed. Allen & Unwin, London. (2) Forsdyke, D. R. (2000) Tomorrow's Cures Today? Harwood Academic, Amsterdam.
I find myself in agreement with much of what is stated above. The introduction is an essential part of a resubmission. Moreover, without having access to previous review comments funding becomes a moving target. If we assume reviewers are competent and tendering constructive comments addressing these points should be sufficient to improve the proposal. Second, I think the preliminary data section is the most important section of an RO1. This is where we as writers make our case for the ideas contained within the Aims and the reviews can directly judge their merits. Finally an RO1 should be an RO1 defining different classes of RO1s based on the PI is establishing a multi-class system which peer-review was not meant to be. Regardless of your career stage or track record an RO1 is an RO1 and they should be scored collectively using the same standards. What I find funny is that the peer-review "system" is never a problem when funding levels are adequate to support 20% funding of all submitted grants but when the funding drops near the 10% mark somehow this is the fault of peer-review. The only problem I see is money and I fail to see how any the proposed changes in peer review are designed to fix the budget issues we currently face.
I'm surprised to see so much griping about the idea of having new investigators compete against each other. The average age at which a PI receives the first NIH grant is now 40. Although there are some mechanisms to help new investigators, I think many grants are awarded based on past performance rather than the content of the current grant, which clearly puts new investigators at a disadvantange. (Of course, I say this as an unfunded new investigator) Considering the fact that new investigators need funding to receive tenure, I think it makes sense to have a pool in which new investigators compete.
Along the lines of Dr. Shannon's comment, as well as Dr. Lokeshwar, I think there are two identifiable problems: 1) Institutes eat too much in indirect costs, meaning grantees must ask for more money. This expands the budget; 35-40% of every grant just disapears. 2) Tenure is almost invariably dependent on an RO1. There maybe multiple sources of funding for innovative ideas, but you have to play the game to stay in the game. Perhaps the NIH et al. need to address some concerns at the level of the institutes they're funding, rather than the PIs themselves.
I think Anthony Firulli makes a very good point- the system just doesn't work very well when the payline is below about 20% because its simply not possible for reviewers to discriminate between grants that are all excellent (I think we can probably identify the top quarter but to really discriminate between that group is not feasible). So, while I think some of the proposals that NIH is considering may be useful, I doubt that any change in the mechanism of assessment will help with that problem. I would add that I think that Donald Forsdyke's comment misses an important point, I see no reason why the best investigators with the most innovative ideas should not be able to explain those ideas in a way that is persuasive to other scientists (and, in fact, everyone). My experience is that people who do a poor job of explaining things in grants usually do so not because they are proposing work that is so brilliant that the rest of us don't understand what they are on about but rather because they are actually not very good ideas in the first place or the person presenting them couldn't be bothered to really make the case. Since the NIH (and other funding agencies) use public money to support our work, it is our responsibility to make it clear what we are doing and why it is important and if a scientist can't do that, I don't think they should be supported.
The NIH might do well to look at the system in the UK, which has one of the most cost-effective scientific funding systems for biosciences in the world (measured by, for example, citations/cost), that grant project applications are 6-8 pages long (versus R01 25 pages), they more typically fund 1-2 postdocs and run for three years. Also, no resubmission is allowed (although a proposal can be shopped around to three different funding agencies in the biosciences). Having spent years writing grants in both the US and UK, I find that smaller grants, while requiring more frequent applications, are less stressful to write or have rejected. From a policy point of view, they can be spread around further to include more junior investigators.