Introduction

The concept of the impact factor was proposed by Garfield1 about 60 years ago. Since then, the impact factor has become the most commonly used assessment tool for ranking and evaluating scientific journals. However, the impact factor suffers from a number of shortcomings and limitations2,3,4,5. These shortcomings include: (i) Because the JCR impact factor is only an average citation received, it does not possess the ability to discriminate the shapes of citation distribution curves, leading to cases in which the same impact factor may correspond to different journals with different citation distributions. (ii) For the same reason, the JCR impact factor as an index of citations per publication usually rewards low productivity and penalizes high productivity6. (iii) The JCR impact factor is relatively easily manipulated by increasing self-citations. On the other hand, Hirsch launched a new study direction of scientometrics by proposing a novel index, now known as the h-index6. The h-index was quickly adopted in various research areas. For example, the h-index or h-type indices were used for evaluating physicists7, evaluating the 100 most prolific economists8 and for evaluating chemical research groups correlated with peer judgment9. Recent studies of the h-index were reviewed10,11,12,13.

Meanwhile, a number of researchers explored the possibility of using the h-index to complement or correct the traditional impact factor. Indeed, the h-index or h-type indices were used to complement journal impact factors or to rank scientific journals. For example, Braun, Glänzel, & Schubert suggested that the h-type index may be a useful complement to journal impact factors14. Schubert and Glänzel performed a systematic analysis of h-type indices for journals15. A theoretical model of the dependence of h-type indices on the number of publications and the average citation rate was tested successfully on empirical samples of journal h-indices15. Vanclay ranked forestry journals using the h-index16. Relationship between the h-index and impact factor in the power-law model was studied by Egghe, Liang, & Rousseau17.

Despite these beneficial exploring researches, the h-index suffers from a number of inherent shortcomings. Probably, the most notable shortcoming of the h-index is that only h-squared citations can be inferred from the h-index, which completely ignores excess citations18 and h-tail citations19,20. The h-index by itself does not carry information for excess and h-tail citations, which can play an even more dominant role than the h-index in determining the shape of citation distribution curve. Ignoring the contributions from the excess and h-tail citations usually either under-estimates or over-estimates the academic performance of the scientist or the journal under study. To solve this problem, recently a new h-type index was proposed, called the h′-index21. One of the main merits of the h′-index is that it takes h-squared, excess and tail citations all into account. The h′-index possesses the ability to distinguish between different shapes of the citation distribution curves. The proposal of the h′-index inspired us to explore the possibility of using the h′-index to complement and correct the traditional impact factor and this is the aim of the present paper.

Results

Derivation of an alternative impact factor, IF′

To complement the h-index6, the e-index18 and t-index20 were proposed. The e-index is the square root of the excess citations over h2 in the h-core. The t-index is the square root of the h-tail citations19. Therefore, the area under the citation distribution curve is divided by the h-index into three parts, representing the h2, the excess citations (e2) and h-tail citations (t2), respectively. To capture the main shape of the citation distribution curve, the head-tail ratio, denoted by r, was defined21

The three cases of r > 1, r = 1 and r < 1 correspond to three types of the citation distribution functions. The shapes of citation distribution functions for r > 1 are peaked and for r < 1 the shapes of the citation functions are flat with a long tail, whereas for r = 1 the citation functions are roughly symmetrical with respect to the diagonal line of the coordinate system. A number of authors attempted to apply the h-index to complement or correct the JCR impact factor14,15,16. It should be pointed out that we cannot simply use the h-index to complement or correct the JCR impact factor. This is due to the fact that the h-index by itself does not carry information for the excess and h-tail citations, which can play an even more dominant role than the h-index in determining the shape of citation distribution curve. As pointed out previously21, when r > 1, especially r ≫ 1, the h-index under-estimates the academic performance of a scientific journal,whereas when r < 1, especially r ≪ 1, the h-index over-estimates the academic performance of a scientific journal. When r = 1, the h-index properly reflects the academic performance of a scientific journal.

To provide an alternative evaluation of the academic performance of a scientific journal, we propose a novel impact factor, denoted by IF′, to complement the JCR impact factor, defined by

where h′ is the h′-index21.

The citations received by all papers in the h-core, denoted by Ch-core, are

where citj are the citations received by the jth paper. Letting e2 denote the excess citations within the h-core, we find18

where R is the R-index22. So,

Meanwhile, the t-index was defined by20

where Ctotal is the number of total citations received by all papers published. Finally, we have

and

Based on the WoS database, all the published items for the 227 journals of biochemistry and molecular biology in 2009 and 2010 were retrieved. The retrieval resulted in 111,002 records, which were downloaded and saved as Excel files. Finally, we calculated the h-index, e-index and t-index of the 227 journals according to the citation data derived from the 111,002 records. Of note, all the calculations of the h-index, e-index and t-index were within a time window from the publication year of items to 2011. The parameter Ctotal is the total citations received by all items published in 2009 and 2010 during the time window from the publication year of items to 2011. The parameter Ch-core is the citations received by all items in the h-core of the citation distribution curve. Then IF′ can be calculated using eqs. (2) or (8). The JCR impact factor for the year 2011 is the ratio of the total citations received in 2011 by all items published in 2009 and 2010 over the total numbers of citable items published in 2009 and 201023.

Correlation between IF′ and the JCR impact factor

Based on the data derived from the JCR and WoS, the related parameters including the total citations Ctotal, the h-index, e-index, t-index, the head-tail ration r and IF′ are calculated for each of the 227 journals in the category of biochemistry and molecular biology. To save printing space, only the front ten journals in the alphabetic order are listed in Table 1. The related data for all the 227 journals are listed in the Appendix-1. As we can see from Table 1 that the head-tail ratio r for nine of the ten journals listed is all less than 1. Indeed, for almost all of the 227 journals listed in the Appendix-1, the values of r are less than 1. To study the relation between the two impact factors, the correlation between IF′ and the JCR impact factor is shown in Fig. 1. It is seen that IF′ is highly linearly and positively correlated with the JCR impact factor, as reflected by the fact that the correlation coefficient is as high as 0.89. Of note, IF′ and the JCR impact factor have roughly the same order of magnitude. The above two features make IF′ relatively easy to be accepted by the academic community.

Table 1 The front ten journals in the category of biochemistry and molecular biology in the alphabetic ordera
Full size table
Figure 1
figure 1

The correlation between IF′ and the JCR impact factor (IF).

The related data of IFs and IF were derived from the JCR and the WoS for the 227 journals in the category of biochemistry and molecular biology. Note that IF′ and IF is highly linear and positively correlated, as reflected by the fact that the correlation coefficient is as high as 0.89. Of note, IF′ and IF have roughly the same order of magnitude.

Full size image

Applications of IF′

As an application, we show here the rankings of the 227 journals in the category of biochemistry and molecular biology based on IF′. For comparison, rankings are also provided based on traditional JCR impact factor. To save printing space, only the top ten journals are listed in 2Table 3. The rankings for all the 227 journals are listed in the Appendix-2. As we can see from Table 3 that the two rankings overlapped in 8 of the top 10 journals (80%), except that orders of some journals were different. This fact indicates that IF′ is basically consistent with the JCR impact factor, confirming the observation in Fig. 1. The famous journal Cell is clearly ranked No. 1 in both rankings. Of note, the journal Nucleic Acids Research is ranked No. 15 in the rankings based on the JCR impact factor, whereas it is ranked No. 3 in that based on IF′. On the contrary, the EMBO Journal is ranked No.10 in the JCR ranking, whereas it is ranked No. 24 in that based on IF′. Furthermore, the journal Current Biology is ranked No. 8 in the JCR-rankings, whereas it is ranked No. 40 in the rankings based on IF′. The variations in both ranks are determined mainly by their shapes of citation distribution curves. Of note, according to the data listed in the Appendeix-1, the head-tail ratios for the journals Nucleic Acids Research, EMBO Journal and Current Biology are 0.597, 0.245 and 0.231, respectively. It is obvious that the shape of the citation distribution curve for the journal Nucleic Acids Research is relatively sharp, whereas those for the journals EMBO Journal and Current Biology are relatively flat with a longer tail.

Table 2 Comparison between two citation distributions for four pairs of journals with appropriately equal JCR IF, h-index and publication numbera
Full size table
Table 3 The top ten journals in the rankings based on IF′ and JCR impact factora
Full size table

In the analysis above, we show that with these journals, IF′ has a one-to-one correspondence, whereas the JCR impact factor has a one-to-multiple correspondence. This is due to the fact that the JCR impact factor does not possess the ability to discriminate the shapes of the citation distribution functions, whereas the shapes of the citation distribution curves are properly taken into account by IF′. As a consequence, the JCR impact factor as an index of citations per publication usually rewards low productivity and penalizes high productivity6.

Discussion

The JCR impact factor is the averaged number of citations received per publication. As pointed out by Hirsch6, measuring the average citations per publication, as does the JCR impact factor, usually rewards low productivity and penalizes high productivity. This is due to the fact that such indices do not possess the ability to distinguish between different shapes of citation distribution curves. To illustrate this point, let us consider two journals among the 227 journals. The first one is the Journal of Physiology and Biochemistry (JPB) and the second one is the Journal of Liposome Research (JLR). The JCR impact factor and the number of publications of both journals are roughly equal. The related data of the two journals are listed in Table 2. Their citation distribution curves are shown in Fig. 2. Note that the shapes of the two curves are different, as clearly shown in Fig. 2. The shape of the citation distribution curve for JPB is relatively sharp, whereas that for JLR is relatively flat. Unfortunately, the JCR impact factor cannot distinguish between the two cases, resulting in almost identical impact factor values (1.711 vs. 1.707). On the contrary, IF′ is able to distinguish between the academic performance of the two journals,resulting in the values of IF′ (3.658 vs.1.867).This example shows that IF′ results in distinct values for different journal citation distributions, whereas different citation distribution curves may correspond to the same JCR impact factor value. In other words, IF′ and journals have a relation of one-to-one correspondence, whereas the JCR impact factor and journals have a relation of one-to-multiple correspondence. Table 2 shows that in addition to the numbers of publication, the total citations, the average citations per publication (i.e., the JCR impact factor) and the h-indices are all roughly equal to each other for the two journals. Of note, the four indices mentioned above are all the single-number evaluation indices currently available and widely used nowadays. Unfortunately, none of them is capable of discriminating the academic performance of the two journals. On the contrary, IF′ (or the h′-index) is a single-number evaluation index that is able to distinguish between the academic performance of journals with different citation distribution. In addition to the two journals mentioned above, other three similar pairs of journals are also listed in Table 2.

Figure 2
figure 2

Comparison between two citation distribution curves for two journals with approximately equal JCR impact factor and number of publications.

Note that the citation distribution curve for the journal JPB is relatively peaked, indicating that a part of its publications are highly cited, whereas that for the journal JLR is relatively flat, indicating that most of its publications are relatively lowly cited. Although the total citations are approximately equal, the academic performance of the two journals is different. The JCR impact factor cannot distinguish between the two journals, with almost identical JCR impact factor IF ≈ 1.7. On the contrary, IF′ possesses the ability to distinguish between the two journals with IF′ ≈ 3.6 and IF′ ≈ 1.8 for JPB and JLR, respectively.

Full size image

Of note, the example above is a typical case, rather than a rare event, because we can find a number of similar cases, as shown in Table 2. Let us further consider two or more journals with roughly equal JCR impact factors, but the numbers of publication may be different. Such cases occur more frequently than the example shown above. Although their JCR impact factors are roughly equal, their citation distribution curves may be different, even quite different. The traditional JCR impact factor lacks the ability to discriminate different shapes of the citation distribution curves, whereas IF′ proposed in this paper does not have this drawback. In this regard, IF′ is an alternative to the JCR impact factor with unique features that reward journals with highly-cited papers, but disregard lowly-cited papers. We need to point out that whether journals with such peaked citation distribution are preferable to those having uniform distribution is arguable. However, current studies on the citation distributions show that the citation distributions obey the so-called power law24,25,26, stretched exponential24,27,28, lognormal24,29,30,31 and modified Bessel function24,32. According to the parameters involved in these functions, the citation distributions are usually skewed, rather than uniform.

In summary, IF′ is proposed as an alternative to the JCR impact factor with some unique features. (i) IF′ is designed to possess the ability to distinguish between different shapes of the citation distribution curves. It gives larger values of IF′ to those with relatively sharp citation distribution curves, whereas lower values to those with flat ones. This is in agreement with the common point of view of academic evaluation33,34. (ii) Citation information of three years is considered by IF′, rather than 2 years for JCR impact factor and therefore IF′ carries more citation information. (iii) IF′ is basically consistent with the JCR impact factor and the two impact factors have roughly the same order of magnitude. These features make IF′ relatively easy to be accepted by the academic community. IF′ may properly reflect the academic performance of a scientific journal in a way that is different from the JCR impact factor with some unique features that reward journals with highly cited papers and therefore it could be used to complement the JCR impact factor.

Methods

As an example, the journals in the category of biochemistry and molecular biology were studied. According to the Thomson Reuters Journal Citation Reports (JCR) Science Edition 2011, there are 290 journals listed in the subject category BIOCHEMISTRY & MOLECULAR BIOLOGY. Among these 290 journals, a few of them changed their titles during the period of 2006 to 2010. Some journals published fewer than 100 articles, whereas some were not continuously indexed by the Web of Science database (WoS) during the period of 2006–2010. All the journals mentioned above were excluded from the current study. Consequently, 227 journals were remained for the present study. We downloaded the Journal Summary List of the subject category BIOCHEMISTRY & MOLECULAR BIOLOGY in the JCR Science Edition 2011 and extracted the Impact Factor values of the 227 journals. The detailed titles and related data for the 227 journals are listed in the Appendix-1.