A group of researchers at work in the laboratory at the Cibio Department of the University of Trento on February 18, 2020. As in previous evaluations, this and other universities in the north of the country on the whole fared better than those in the south. Credit: Alessio Coser/University Of Trento via Getty Images.
Italian universities are improving when it comes to attracting talented young researchers and engaging with society, according to the National Agency for the Evaluation of Universities and Research Institutes (ANVUR), which has announced the results from its latest periodic assessment.
But some scientists are critical of the exercise, known as the Research Quality Evaluation (VQR), arguing that the agency's data are based on crude and easily manipulable criteria. "These quantitative criteria have little to do with science and culture," says Patrizio Dimitri, a geneticist at La Sapienza University in Rome.
ANVUR was set up in 2006 to try and improve the meritocracy of Italian research by providing scores for individual institutions that the research ministry (now MUR) could use to allocate part of its budget. The latest VQR is the third installment of that process, and the largest. Involving around 600 experts and more than 11,000 external reviewers, it analysed more than 180,000 papers, monographs and book chapters produced between 2015 and 2019 by about 65,000 researchers in 134 institutions, including universities and research institutes.
Among the headline results, the University of Milan was judged on average to produce the highest quality research of any institution, while La Sapienza in Rome recorded the best score when it came to total research output. As in previous iterations of the VQR, universities in the north of the country on the whole fared significantly better than those in the south.
ANVUR president, Antonio Uricchio, insists that the VQR is not designed to generate league tables, but simply to provide the research ministry with data – which, he says, it will use to distribute about 30% of the roughly €8bn it allocates to universities every year.
Rather than focus on rankings, Uricchio is keen to stress the progress that he says the Italian research system has made as a whole in recent years. He points out that the average quality score is higher for recently appointed researchers than it is for more established ones. In contrast with the common perception that talented researchers are leaving Italy in droves, he says that this result shows institutions are managing to recruit “able young people”.
Uricchio also takes pride in the data concerning what is known as universities' “third mission”. The idea is to quantify the impact of universities on society and the economy more broadly by assessing the extent of patent production, creation of spin-off companies and public engagement, among other things. Missing from the previous two VQR rounds as a mechanism for distributing funding, he says that analyses of case studies presented by institutions revealed a "growing attention" to such activities. Teaching is assessed in a separate exercise.
Mauro Perretti, an expert in immunopharmacology at Queen Mary University of London, agrees it is important that ANVUR assesses whether research "has relevance and value to society". He is also encouraged by what he says is a greater emphasis in the current VQR on the use of peer review to assess scientific research as opposed to bibliometric criteria such as citation counts – a change he himself recommended as chair of an independent panel set up in 2018 to review the work of ANVUR. "I think all said it is a better evaluation than previous ones," he says.
However, editors of a website known as Return on Academic Research and School (ROARS) argued in 2020 that while a ministerial decree setting out the guidelines for the current VQR sought to limit the use of bibliometric criteria, ANVUR brought the criteria back in by allowing reviewers considerable scope to choose between peer review and bibliometrics.
One of the ROARS editors, Giuseppe De Nicolao of the University of Pavia, says that the funds to be given out directly on the basis of the VQR scores will not have a significant impact, because the funding is largely proportional to the size of those institutions – as it would be without any ranking. What stands to make more of a difference, he maintains, is another funding stream dedicated specifically to “departments of excellence”. Uricchio says that ANVUR will send MUR a list of the top 350 departments in the country in May, and that it will then be down to the ministry to distribute some €0.3bn per year among them. De Nicolao describes the process as “Darwinian”, because lower-ranked departments in this case will get nothing. He says that he and his fellow editors requested details on how these funds were carved up for the previous VQR but were turned down on privacy grounds. “It is a completely opaque procedure,” he says.
