Relationship between bibliometric indicators and university ranking positions

A growing interest for demonstrating prestige and status of higher education institutions has spurred the establishment of several international ranking systems. A major percentage of these rankings include parameters related to scientific productivity. Here, we examined the differences between diverse rankings as well as correlation with bibliometric parameters and disciplines for the top universities. We investigated the top 300 universities from four international rankings, the Times Higher Education World University Ranking (THE), the QS World University Rankings (QS) the ShanghaiRanking-Academic Ranking of World Universities (ARWU) and the U.S.News Best Global Universities Ranking (USNews). The assessed parameters include ranking positions, size related and bibliometrics-related indicators of each selected ranking. The weight of scientometric parameters ranges between 20% (QS) and 75% (USNews). The most important parameters defining ranking positions include citations, international reputation, and the number of researchers, but the correlation strength varies among ranking systems. The absolute number of publications and citations are particularly important in ARWU and USNews rankings, and scientific category normalized (field weighted) citation impact is central in THE and USNews rankings. Our results confirm that universities having outstanding results in rankings using size-independent indicators (QS and THE) compared to others have significantly lower number of students. High impact research can improve position in ARWU and USNews ranking lists. Regarding to different disciplines, the main results show that outstanding universities in THE ranking have higher publication activity in social sciences and universities which perform better in USNews and QS ranking have more publications in science, technology, and medicine fields and lower score in social sciences. In brief, here we present a comprehensive analysis of the correlation between scientometric parameters and university ranking positions, as well as the performance of outstanding universities and their correlation with different disciplines, to help decision makers select parameters for strengthening and to attract the interest of prospective students and their parents via a better understanding of the functions of different ranks.


Results
Setting up a matched ranking for top universities.The ranking presented by the Times Higher Education (THE) magazine performs its own data collection for the included universities.It had common roots with QS ranking between 2004-2009, but in 2010 it switched to a different methodology.THE is based on 13 performance indicators grouped into five areas: teaching (30%), research (30%), citations (30%), knowledge transfer (2.5%), and international outlook (7.5%).All together 13 indicators are used, and out of these the total weight of bibliometric indicators is 38.5%.All indicators are normalized to university size or scientific area.Elsevier's SciVal (based on Scopus data) is used as publication and citations source.
The QS-World University Ranking is published by the Quacquarelli Symonds (QS) company.Universities are ranked based on six key metrics in the ranking: academic reputation (40%), employer reputation (10%), faculty/ student ratio (20%), citations per faculty (20%), international faculty ratio (5%), and international student ratio (5%).There is only one bibliometric indicator (citations per faculty-20%) which is normalized to the scientific area, and self-citations are excluded.Elsevier's SciVal (based on Scopus data) is used as publication and citations source.
The Academic Ranking of World Universities (ARWU, also called ShanghaiRanking) was first published in 2003.Since 2009 the ARWU has been published by ShanghaiRanking Consultancy, which is a fully independent organization.It ranks the first 1000 universities, using six indicators, including quality of education (10%), quality of faculty (40%), research output (40%), and per capita performance (10%).The two normalized bibliometric indicators have a total weight of 40%, and one bibliometric related (number of highly cited researchers) indicator weights 20%.Clarivate's InCites database (based on Web of Science Core Collection data) is used as publication and citations source.
The U.S.News Best Global Universities Ranking was launched by U.S.News & World Report in 2014.It ranks universities based on thirteen indicators, including global research reputation (12.5%), regional research reputation (12.5%), and eleven bibliometric indicators (75%).It contains size-dependent indicators, but some indicators are normalized.Clarivate's InCites database (based on Web of Science Core Collection data) is used as publication and citations source.
Differences in ranking positions vs. determined parameters.Next, we investigated the correlation between the ranking position differences and the determined scientometric parameters.In this analysis, high correlation values mean high importance for this parameter and low correlations mean small importance in determining ranking differences between the four investigated rankings.All together 126 correlations were evaluated, and in 99 cases we found significant correlations (p < 0.05).
The most significant correlations were observed in case of the THE citations and the USNews normalized citation impact indicators.Although both of these indicators represent the citation impact of publications relative to their scientific discipline, but they use two different data sources, Web of Science and Scopus.The two indicators show higher correlation in THE-QS and in THE-ARWU pairs proving that universities with higher discipline-specific citation reach better ranking in THE compared to QS or ARWU.Albeit with lower correlation values, these universities also have better ranks in USNews.When comparing THE and USNews, THE has a positive value also linked to the higher weight of citation in THE (30%).
In case of THE-QS and of the ARWU-USNews pairs, the differences do not show correlation with university size while in case of the QS/ARWU vs THE/USNews pairs the differences show high correlation values.These results support the notion that QS and THE rankings are better for small universities with high impact while larger universities can reach better scores in ARWU and USNews rankings.
The detailed analysis results are shown in Fig. 2.
Comparing all determined parameters to each other.We also correlated all the investigated parameters to each other across all included universities.In this, 441 associations were checked, 399 of which had a significant correlation (p < 0.05).
Particularly high correlations can be observed for USNews scientometric parameters including the number of publications in the top 10% and top 1%, and citations in the top 10% and top 1%.These results provide evidence that the overall impact of these parameters is amplified by their influence on the other parameters.Of note, the percentage of top 10% and top 1% are also positively correlated.
Similarly, significant correlations can be observed between the size-dependent parameters of ARWU and USNews.The complete results for all parameters are depicted in Fig. 3.

Outstanding universities.
A particularly interesting analysis involves the "outstanding universities", which have excellent position in one ranking but mediocre position in another.All together twelve outstanding university groups were compared with the control groups (which include all non-outstanding universities) for the 22 determined parameters.The analysis includes universities outstanding in THE vs. the three other rankings (Fig. 4A), universities outstanding in QS vs. the three other rankings (Fig. 4B), universities outstanding in ARWU compared to the other universities (Fig. 4C), and universities outstanding in USNews compared to other rankings (Fig. 4D).Note that because outstanding universities have better positions (lower number), the ranking difference is negative for almost each parameter.
We also studied the typical disciplines of the universities selected with this method.We examined the differences of OpenAlex root level concepts scores (19 concepts and three concept groups) between outstanding university groups and control groups.In this analysis, we found significant differences (Mann-Whitney p < 0.01, two-tailed) in case of 98 outstanding groups (see result in Fig. 5A-D).The main results show that outstanding universities in THE ranking have higher score in social sciences and universities which perform better in USNews and QS ranking have higher score in science, technology, and medicine fields and lower score in social sciences.
The university size parameter was also examined when comparing outstanding and control university groups.Results show that universities outstanding in QS and THE compared to ARWU and USNews have significantly differed with lower number of students (Mann-Whitney p < 0.01 two-tailed) and conversely, universities outstanding in ARWU and USNews compared to THE and QS had higher median of number of students (Fig. 6).

Discussion
Here, we have determined the effects of scientometric parameters in four international ranking systems.Our results confirm previous observations that there are reasonable similarities between the rankings 5 .Of course, this does not mean that all universities perform equally in each ranking-we can also confirm the size dependence, which lead to better result for larger universities 14   emphasize size and/or quality parameters, while smaller universities can reach better positions in staff normalized rankings.
We have determined the effect of scientometric indicators on ranking positions.Previously, it has been shown that ranking scores correlate with the publication output and citations of a university 15 .It was also established and extended that broader field coverage is also an advantage in rankings 16 .Our results confirmed that scientometric indicators play a major role in rankings.We have found significant correlations between almost all bibliometricrelated indicators and positions in each examined ranking.Our results also show that the absolute number of publications and citations are particularly important in ARWU and USNews rankings, and scientific category normalized (field weighted) citation impact is important in THE and USNews rankings.
Each ranking uses different indicators to measure the performance of universities, which vary from one ranking to another.Remarkably, many of these different parameters used for the rankings are not truly independent and we found strong correlations between examined parameters in some cases.These similarities can be divided to two groups: first, the correlation is actually high because of the natural phenomenon that the indicators can be similar in different rankings, e.g., ARWU publications and USNews publications or THE citation and USNews normalized citation impact show similar data from different sources which have significant  Low correlation values mean small importance in determining ranking differences between the four investigated rankings.RankDiffs are the ranking position differences of a university between two rankings (ranked place of first ranking minus ranked place in the second ranking).RankDiff value is low (negative) if the university is ranked higher in the first ranking than in the second, and high (positive) if it is ranked higher in the second ranking.Significant differences (p < 0.05) are marked with bold correlation coefficients.(The colors show the distance from the zero.Green to Yellow < 0; Yellow to Red 0 <).
Vol:.( 1234567890) We have to mention several criticisms of university rankings formulated by different research groups.Vernon et al. summarized several doubts about the rankings, suggesting that the significance of the reputation questionnaire should be kept below 10% 12 .QS has the highest weight of reputation indicator (50%) of all the rankings we examined, followed by THE (33%) and USNews (25%).Daraio et al. abridged the main criticisms of the rankings into following four issues: monodimensionality, i.e. the rankings focus mainly on research among the education, research, and third missions of universities; statistical robustness, i.e. statistical problems of the individual indicators; dependence on university size and subject mix; and lack of consideration of the input-output structure 13 .As our results show, that ARWU and USNews rankings have high correlation with university size, while THE and QS rankings use normalized parameters.
We have to note a limitation of our study: the university ranking websites usually do not provide detailed information on the indicators, so approximations had to be used in some cases.In the THE ranking, the "Research" and "International Outlook" pillars, whose values are publicly available, were calculated by combining bibliometric and other indicators, so the effect of bibliometric parameters can be distorted in the correlation calculations.In the USNews ranking, the order of each indicator was available instead of the score values, and this contains less information.The exact ranking position in the ARWU list can only be calculated as an approximation, which also causes some bias.A second limitation is the use of different university names.In some ranking sites the English name only while in other sites the local language versions are used.We tried to identify each university in each list precisely, but we cannot completely exclude the possibility of a mismatch.Notably, as we found strong correlation between number-related parameters, it seems that at least for the THE, USNews, and QS rankings this potential negligible.Finally, although we aimed to make all data openly available, the copyright of the original data sources prohibited this goal.
Unfortunately, in our study it was not possible to give a perfect guide, as one of the limitations of our project is that we only looked at bibliometric indicators (and the university's size).However, each ranking attempts to find the best universities by weighting different indicators.In our study, we found that, even when using the most objective bibliometric indicators, significant differences in the performance of several universities in each ranking can be detected when using certain types of indicators, and that publication differences in the disciplines    also affect the performance in each ranking.Based on these considerations, it is likely that the use of disciplinespecific rankings would be more effective than overall rankings for the study of education, research and services.University ranking systems are on the rise and influence the perceived prestige of a university.Here, we have determined the effects of scientometric parameters on university ranking positions.Notably, there are also other reasons why one group of universities have significantly different results in rankings, e.g., diversified territorial and educational contexts can cause structural biases 17 .In this study we have identified factors significantly related to the outstanding status.Overall, strong publication activity is an important factor in each ranking, but significant differences in ranking places depend on both the selected indicators used by the ranking and the publication and citation characteristics of the universities.

Methods
Setting up a matched ranking for top universities.We selected four internationally recognized rankings from three continents, Europe, North-America and Asia.A common characteristic of these rankings was the availability of detailed ranking data.We used the most recent version of the four chosen rankings including the Times Higher Education World University Ranking 2022 edition (THE), the QS World University Rankings 2022 edition (QS) the ShanghaiRanking-Academic Ranking of World Universities 2021 edition (ARWU), and the U.S.News Best Global Universities Ranking 2022 edition (USNews).

Recomputing ranking scores for scientometric indicators.
Because most ranking websites do not publish exact ranking results, only bins, we had to calculate the ranking positions of the universities according to the ranking's published methodology.For example, in ARWU, the total scores were available only for the first 100 universities, for the other universities the total scores could not be derived from the public data, so we used estimation of the total score based on the values of each indicator using their methodology.
We selected bibliometrics-related indicator values from the four rankings, which were available on the webpage of the rankings.This includes altogether twenty indicators (one indicator from QS, three indicators from THE, four indicators from ARWU, and 12 indicators from USNews) which are described in detail in Table 1.Notably, the total weight of scientometric parameters is 60% in THE, 60% in ARWU, 20% in QS, and 75% in USNews.
We also collected available data on university sizes, which was the "Number of Students" parameter from THE and "Enrollment" value from USNews.These two parameters were the same for most universities, but they originate from different data collection processes.-

Figure 1 .
Figure 1.Spearman rank correlation of ranking positions and selected indicator values/ranks.High correlations show that the given indicator/rank affects the rank position in different rankings.Continuous indicators are scores, where higher values mean better positions, while USNews indicators are ranks, where the lower value means a better position.Best rank was computed by using the best positions across all four ranking for each university and reflects the power of each feature in predicting the best position of a university.Significant differences (p < 0.05) are marked with bold correlation coefficients.(The colors show the distance from the zero.Green to Yellow < 0; Yellow to Red 0 <).

Figure 2 .
Figure2.Spearman rank correlation between ranking position differences and the investigated scientometric parameters.High correlation values mean high importance of this parameter in determining the rankingspecific positions.Low correlation values mean small importance in determining ranking differences between the four investigated rankings.RankDiffs are the ranking position differences of a university between two rankings (ranked place of first ranking minus ranked place in the second ranking).RankDiff value is low (negative) if the university is ranked higher in the first ranking than in the second, and high (positive) if it is ranked higher in the second ranking.Significant differences (p < 0.05) are marked with bold correlation coefficients.(The colors show the distance from the zero.Green to Yellow < 0; Yellow to Red 0 <).

Figure 3 .
Figure 3. Spearman rank correlations between all the investigated parameters across all included universities.High correlations show that the given indicator/rank affects the other indicator/rank.Significant differences (p < 0.05) are marked with bold correlation coefficients."n" means the number of universities where both indicator/rank data were available.(The colors show the distance from the zero.Green to Yellow < 0; Yellow to Red 0 <).

Figure 4 .Figure 5 .
Figure 4. Differences of medians of indicator ranks between outstanding university groups compared to control groups comprising all non-outstanding universities in the given pair.Outstanding universities were those which had a ranking advantage of at least 100 positions compared to position in the other ranking.Outstanding universities and control groups were compared using the Mann-Whitney two-sample rank-sum test.Only those median differences are shown where the two group significantly differed (p < 0.01). )

THE calculated rank QS calculated rank ARWU calculated rank USNews rank Best Rank ConƟnuous indicators (scores) THE FTE of Students
. Our results clearly show that smaller universities have better results in size independent rankings including the THE and QS, while bigger universities can perform better in USNews and ARWU rankings which both have size-dependent indicators.Overall, large universities can
Figure6.Differences of medians of university size parameters between outstanding university groups compared to control groups comprising all non-outstanding universities in the given pair.Outstanding universities were those which had a ranking advantage of at least 100 positions compared to the other ranking.Outstanding universities and control groups were compared using the Mann-Whitney two-sample rank-sum test.Only those median differences are shown where the two group significantly differed (p < 0.01).