washington

A committee of the US National Academy of Sciences said last week that an effective assessment of research performance must take into account three factors: the quality of the research, its contribution to world leadership in various fields, and its relevance to the goals of funding agencies.

The Committee on Science, Engineering and Public Policy (COSEPUP) said that, despite serious doubts, research performance can be assessed — applied research easily, basic research with more difficulty.

The report is a response to attempts by research agencies to implement the Government Performance and Results Act (GPRA), which requires federal agencies to measure how their programmes benefit taxpayers.

The committee, chaired by Phillip A. Griffiths, director of the Institute for Advanced Study in Princeton, cites the pros and cons of common measures of research quality such as bibliometric analysis, economic rate of return, peer review, case studies, retrospective analysis, and benchmarking.

All feature in evaluations. But a magic element is needed, it says: “expert review”, an assessment of the “quality-relevance-leadership troika”. Making such an assessment requires unique and seldom-used combinations of expertise, it adds.

The 80-page report stresses the development of research talent in fields critical to agency missions. Most performance plans have lacked such goals, says COSEPUP. But it emphasizes that education and training are essential in any research performance plan.

COSEPUP warns that focusing on short-term results can undermine the whole purpose of evaluation. This echoes the National Science Foundation's struggles to comply with GPRA. The foundation was the first agency to release a performance plan geared to its fiscal year 2000 budget proposals.

The committee also suggests extending GPRA by developing assessments of programmes involving several agencies — such as global change research, or the information technology initiative announced earlier this month in the Clinton administration's 2000 research and development budget.

COSEPUP says the administration should designate lead agencies to ensure that interagency programmes are well coordinated, a task that usually falls to the Office of Science and Technology Policy (OSTP) as the operating arm of the National Science and Technology Council.

OSTP has submitted its own GPRA strategic plan, but has yet to release any performance plan. It may need to if GPRA assessors in Congress adopt COSEPUP's coordination recommendation. In a final recommendation, COSEPUP says that researchers must take a leading role for the whole process to work.

In its original form, agencies had to submit strategic plans to GPRA by September 1997, performance plans by spring 1998, and performance reports by March 2000. All performance plans are now in, although some remain in outline form, and many are being revised.

Committee member Morris Tanenbaum, former chairman of the AT&T telecommunicationss corporation, says GPRA is critical to give researchers a better understanding of government and Congress a better understanding of research.