Published online 13 July 2011 | Nature 475, 141 (2011) | doi:10.1038/475141a

Column: World View

The dubious benefits of broader impact

Assessments of the wider value of research are unpopular. Proposed changes will only produce more hype and hypocrisy, says Daniel Sarewitz.

Since 1997, it has not been sufficient for US researchers seeking grants from the National Science Foundation (NSF) to merely explain the intellectual merit of their proposal. They must also justify their work in terms of a variety of 'broader impacts'.

Politicians worldwide no longer accept that public investments in science automatically bring social benefits. They increasingly expect research expenditure to be justified on its potential contribution to social and economic goals. In the United States, this expectation has resulted in the NSF's notorious Criterion 2.

Criterion 2 is used by peer reviewers to check that projects will promote education and training, broaden participation, improve infrastructure for research and education, disseminate knowledge or deliver more general social benefits. Yet, according to a review by the National Science Board (the NSF's advisory and oversight body), the criterion "can be very confusing to the research community, which continues to express frustration in interpreting and thus responding effectively".

Last month, the board published a revised criterion, and scientists had until this week to provide comments to the NSF before the final version is issued. But Criterion 2.1, as it might be called, is just as confusing and counterproductive as its predecessor.

At the heart of the new approach is "a broad set of important national goals". Some address education, training and diversity; others highlight institutional factors ("partnerships between academia and industry"); yet others focus on the particular goals of "economic competitiveness" and "national security". The new Criterion 2 would require that all proposals provide "a compelling description of how the project or the [principal investigator] will advance" one or more of the goals.

The nine goals seem at best arbitrary, and at worst an exercise in political triangulation. How else to explain the absence of such important aims as better energy technology, more effective environmental management, reinvigorated manufacturing, reduced vulnerability to natural and technological hazards, reversal of urban-infrastructure decay or improved performance of the research system? These are the sorts of goal that continue to justify public investments in fundamental research.

Yet, more troubling than the goals themselves is the problem of democratic legitimacy. In applying Criterion 2, peer-review panels will often need to choose between projects of equal intellectual merit that serve different national goals. Who gave such panels the authority to decide, for example, whether a claim to advance participation of minorities is more or less important than one to advance national security?

This problem is exacerbated by issues of expertise. To convincingly assess how a particular research project might contribute to national goals could be more difficult than the proposed project itself. Neither project leaders nor peer-review panels are likely to have sufficient expertise to really understand a single project's capacity to connect to a persistent challenge such as increasing the nation's science literacy or economic competitiveness.

Individual projects are the wrong lever to bring NSF research into line with national goals. It is not surprising, however, that the NSF and the science board made this mistake — the agency's public image is dominated by the idea of the individual scientist, advancing the frontiers of knowledge. As its website explains, the "NSF's task of identifying and funding work at the frontiers of science and engineering is not a 'top-down' process. NSF operates from the 'bottom up,' keeping close track of research around the United States and the world, maintaining constant contact with the research community."

“Individual projects are the wrong lever to bring NSF research into line with national goals.”

Yet the NSF has engaged in ongoing organizational experiments over the past 40 years, aiming to overcome the limits of single-investigator, peer-reviewed science. From massive Engineering Research Centers and Science and Technology Centers that address complex, interdisciplinary problems, to small Rapid Response Research grants to get funds quickly to researchers working on urgent questions, and programmes that push university academics to engage seriously in education, the NSF is committed to top-down behavioural modification of the scientific community, often driven by the vision of agency leaders and linked to national challenges such as climate change or emerging opportunities such as nanotechnology.

Motivating researchers to reflect on their role in society and their claim to public support is a worthy goal. But to do so in the brutal competition for grant money will yield not serious analysis, but hype, cynicism and hypocrisy. The NSF's capacity to meet broad national goals is best pursued through strategic design and implementation of its programmes, and best assessed at the programme-performance level. Individual projects and scientists should be held accountable to specific programmatic goals, not vague national ones. For example, if an NSF initiative aims to provide information for decision-makers, proposals should have to provide evidence that there is actually a customer for the results of the proposed work. Criterion 2 needs to be flexible and tailored to the goals of particular NSF programmes. Otherwise, it will remain confusing and frustrating for scientists and politicians alike. 

Daniel Sarewitz is co-director of the Consortium for Science, Policy and Outcomes at Arizona State University, and is based in Washington DC.

Commenting is now closed.