Italy's universities have undergone their first comprehensive evaluation. Pictured are students at the University of Bologna, founded in 1088 ad and considered Europe's oldest. Credit: Silvia Morara/Corbis

Exam season is over for Italian universities, and the results are in. An ambitious evaluation exercise has ranked the country's universities and research institutes according to the quality of their scientific output. The data will be used to award extra funds to the best-ranking centres. But some critics say that the report's methodology is not reliable and may lead to distortions in budget allocation.

The National Agency for the Evaluation of the University System and Research (ANVUR) presented its first report in Rome on 16 July. The University of Padua stands out as the best large university overall, making the top spot in 7 out of 14 research areas. The University of Trento tops the ranking for medium ones, and the Scuola Superiore Sant'Anna in Pisa, an elite applied-science school, took the first place among small universities.

“There were no big surprises,” says ANVUR president Stefano Fantoni. “Italian universities have a good average scientific output, particularly in chemistry, physics and industrial engineering, but with great differences. Small universities work better than large ones, and the best ones are mostly in the north”.

ANVUR was established by a 2006 law with the goal of increasing meritocracy in Italian research, and was modelled after organizations and systems such as the French evaluation agency for research and higher education (Aeres) and the UK Research Excellence Framework (REF).

In November 2011, after five years of preparation, ANVUR embarked on an assessment of 95 universities, 21 research agencies or institutes, and 17 inter-university consortia. All faculty members and researchers were asked to submit a sample of their 'research products' from the period 2004–10: university employees had to provide three examples, and researchers with no teaching commitments had to give six.

Piece by piece

In addition to academic papers, accepted products included book chapters, patent applications, databases, conference proceedings and, for the humanities, works of art or exhibition projects. The agency received a total of almost 185,000 such items, which were then evaluated by 450 experts divided into 14 groups corresponding to different research areas.

Each group chose its own methodology, using both bibliometrics (a quantitative analysis of the impact of journal papers) and peer review — with the help of about 14,000 external reviewers, of whom more than 4,000 were from outside Italy. “Bibliometrics was prevalent in hard sciences, life sciences and engineering, whereas for social sciences and humanities only peer review was used,” says Sergio Benedetto, a telecommunications specialist formerly at the Polytechnic University of Turin who led the evaluation process for ANVUR.

Data were then distilled into scores assigned to individual institutes. Rankings for large, medium and small universities were made in each research area, with separate rankings for research agencies and consortia.

There were some surprises among the rankings of the largest research agencies. Italy's main scientific body, the National Research Council (CNR), ranked well below the National Institute of Geophysics and Volcanology (INGV), the National Institute of Nuclear Physics (INFN) and the National Institute for Astrophysics (INAF) in terms of overall quality. Benedetto explained at the presentation that this was because the CNR had submitted significantly fewer products than requested. But CNR president Luigi Nicolais said that his agency is not involved solely in research: it also oversees other concerns, such as technology transfer, which ANVUR failed to recognize.

Statistical error?

ANVUR's methodology was also criticized by a group of researchers and professors who for months have been voicing their concerns on a website called ROARS (Return on Academic Research). Giuseppe De Nicolao, a data analyst at the University of Pavia and a ROARS contributor, says, “Bibliometrics may be useful for large-scale statistics at the level of whole countries or large research institutes, but not for automatic evaluation of papers.”

Benedetto disagrees. “We were not so naive. We combined different bibliometrics systems, using both Journal Impact Factor and citation indexes, and resorting to peer review whenever their results were not consistent.” He notes that peer review is also being criticized widely for being unreliable, and that using it alone would be too costly and time consuming.

ANVUR data on universities will be used by Italy's education and research ministry to award about €540 million (US$711 million) in 'prize funds', part of the government's 2013 university budget. But the exact criteria for translating ANVUR rankings into shares of those funds are still being worked out, a spokesperson for the ministry told Nature.

It is unclear what effect the evaluation will have for research agencies and other institutes. Research minister Maria Chiara Carrozza said that such organizations may receive a limited amount of prize funds on the basis of ANVUR data, but that a further re-analysis is needed to make sure the data “take into account the great differences among these institutes”. The ANVUR rankings may play a part in a continuing debate about the possibility of merging some of those institutes to save costs.