Article | Open Access | Published:

A proposal for a novel impact factor as an alternative to the JCR impact factor

Scientific Reports volume 3, Article number: 3410 (2013) | Download Citation

Abstract

One disadvantage of the JCR impact factor, the most commonly used assessment tool for ranking and evaluating scientific journals, is its inability in distinguishing among different shapes of citation distribution curves, leading to unfair evaluation of journals in some cases. This paper aims to put forward an alternative impact factor (IF′) that can properly reflect citation distributions. The two impact factors are linearly and positively correlated, and have roughly the same order of magnitude. Because of the ability of IF′ in distinguishing among different shapes of citation distribution curves, IF′ may properly reflect the academic performance of a scientific journal in a way that is different from the JCR impact factor with some unique features that reward journals with highly cited papers. Therefore, it is suggested that IF′ could be used to complement the JCR impact factor.

Introduction

The concept of the impact factor was proposed by Garfield1 about 60 years ago. Since then, the impact factor has become the most commonly used assessment tool for ranking and evaluating scientific journals. However, the impact factor suffers from a number of shortcomings and limitations2,3,4,5. These shortcomings include: (i) Because the JCR impact factor is only an average citation received, it does not possess the ability to discriminate the shapes of citation distribution curves, leading to cases in which the same impact factor may correspond to different journals with different citation distributions. (ii) For the same reason, the JCR impact factor as an index of citations per publication usually rewards low productivity, and penalizes high productivity6. (iii) The JCR impact factor is relatively easily manipulated by increasing self-citations. On the other hand, Hirsch launched a new study direction of scientometrics by proposing a novel index, now known as the h-index6. The h-index was quickly adopted in various research areas. For example, the h-index or h-type indices were used for evaluating physicists7, evaluating the 100 most prolific economists8, and for evaluating chemical research groups correlated with peer judgment9. Recent studies of the h-index were reviewed10,11,12,13.

Meanwhile, a number of researchers explored the possibility of using the h-index to complement or correct the traditional impact factor. Indeed, the h-index or h-type indices were used to complement journal impact factors or to rank scientific journals. For example, Braun, Glänzel, & Schubert suggested that the h-type index may be a useful complement to journal impact factors14. Schubert and Glänzel performed a systematic analysis of h-type indices for journals15. A theoretical model of the dependence of h-type indices on the number of publications and the average citation rate was tested successfully on empirical samples of journal h-indices15. Vanclay ranked forestry journals using the h-index16. Relationship between the h-index and impact factor in the power-law model was studied by Egghe, Liang, & Rousseau17.

Despite these beneficial exploring researches, the h-index suffers from a number of inherent shortcomings. Probably, the most notable shortcoming of the h-index is that only h-squared citations can be inferred from the h-index, which completely ignores excess citations18 and h-tail citations19,20. The h-index by itself does not carry information for excess and h-tail citations, which can play an even more dominant role than the h-index in determining the shape of citation distribution curve. Ignoring the contributions from the excess and h-tail citations usually either under-estimates or over-estimates the academic performance of the scientist or the journal under study. To solve this problem, recently a new h-type index was proposed, called the h′-index21. One of the main merits of the h′-index is that it takes h-squared, excess and tail citations all into account. The h′-index possesses the ability to distinguish between different shapes of the citation distribution curves. The proposal of the h′-index inspired us to explore the possibility of using the h′-index to complement and correct the traditional impact factor, and this is the aim of the present paper.

Results

Derivation of an alternative impact factor, IF′

To complement the h-index6, the e-index18 and t-index20 were proposed. The e-index is the square root of the excess citations over h2 in the h-core. The t-index is the square root of the h-tail citations19. Therefore, the area under the citation distribution curve is divided by the h-index into three parts, representing the h2, the excess citations (e2) and h-tail citations (t2), respectively. To capture the main shape of the citation distribution curve, the head-tail ratio, denoted by r, was defined21 The three cases of r > 1, r = 1 and r < 1 correspond to three types of the citation distribution functions. The shapes of citation distribution functions for r > 1 are peaked, and for r < 1 the shapes of the citation functions are flat with a long tail, whereas for r = 1 the citation functions are roughly symmetrical with respect to the diagonal line of the coordinate system. A number of authors attempted to apply the h-index to complement or correct the JCR impact factor14,15,16. It should be pointed out that we cannot simply use the h-index to complement or correct the JCR impact factor. This is due to the fact that the h-index by itself does not carry information for the excess and h-tail citations, which can play an even more dominant role than the h-index in determining the shape of citation distribution curve. As pointed out previously21, when r > 1, especially r 1, the h-index under-estimates the academic performance of a scientific journal,whereas when r < 1, especially r 1, the h-index over-estimates the academic performance of a scientific journal. When r = 1, the h-index properly reflects the academic performance of a scientific journal.

To provide an alternative evaluation of the academic performance of a scientific journal, we propose a novel impact factor, denoted by IF′, to complement the JCR impact factor, defined by where h′ is the h′-index21.

The citations received by all papers in the h-core, denoted by Ch-core, are where citj are the citations received by the jth paper. Letting e2 denote the excess citations within the h-core, we find18 where R is the R-index22. So, Meanwhile, the t-index was defined by20 where Ctotal is the number of total citations received by all papers published. Finally, we have and Based on the WoS database, all the published items for the 227 journals of biochemistry and molecular biology in 2009 and 2010 were retrieved. The retrieval resulted in 111,002 records, which were downloaded and saved as Excel files. Finally, we calculated the h-index, e-index and t-index of the 227 journals according to the citation data derived from the 111,002 records. Of note, all the calculations of the h-index, e-index and t-index were within a time window from the publication year of items to 2011. The parameter Ctotal is the total citations received by all items published in 2009 and 2010 during the time window from the publication year of items to 2011. The parameter Ch-core is the citations received by all items in the h-core of the citation distribution curve. Then IF′ can be calculated using eqs. (2) or (8). The JCR impact factor for the year 2011 is the ratio of the total citations received in 2011 by all items published in 2009 and 2010 over the total numbers of citable items published in 2009 and 201023.

Correlation between IF′ and the JCR impact factor

Based on the data derived from the JCR and WoS, the related parameters including the total citations Ctotal, the h-index, e-index, t-index, the head-tail ration r and IF′ are calculated for each of the 227 journals in the category of biochemistry and molecular biology. To save printing space, only the front ten journals in the alphabetic order are listed in Table 1. The related data for all the 227 journals are listed in the Appendix-1. As we can see from Table 1 that the head-tail ratio r for nine of the ten journals listed is all less than 1. Indeed, for almost all of the 227 journals listed in the Appendix-1, the values of r are less than 1. To study the relation between the two impact factors, the correlation between IF′ and the JCR impact factor is shown in Fig. 1. It is seen that IF′ is highly linearly and positively correlated with the JCR impact factor, as reflected by the fact that the correlation coefficient is as high as 0.89. Of note, IF′ and the JCR impact factor have roughly the same order of magnitude. The above two features make IF′ relatively easy to be accepted by the academic community.

Table 1: The front ten journals in the category of biochemistry and molecular biology in the alphabetic ordera
Figure 1: The correlation between IF′ and the JCR impact factor (IF).
Figure 1

The related data of IFs and IF were derived from the JCR and the WoS for the 227 journals in the category of biochemistry and molecular biology. Note that IF′ and IF is highly linear and positively correlated, as reflected by the fact that the correlation coefficient is as high as 0.89. Of note, IF′ and IF have roughly the same order of magnitude.

Applications of IF′

As an application, we show here the rankings of the 227 journals in the category of biochemistry and molecular biology based on IF′. For comparison, rankings are also provided based on traditional JCR impact factor. To save printing space, only the top ten journals are listed in Table 3. The rankings for all the 227 journals are listed in the Appendix-2. As we can see from Table 3 that the two rankings overlapped in 8 of the top 10 journals (80%), except that orders of some journals were different. This fact indicates that IF′ is basically consistent with the JCR impact factor, confirming the observation in Fig. 1. The famous journal Cell is clearly ranked No. 1 in both rankings. Of note, the journal Nucleic Acids Research is ranked No. 15 in the rankings based on the JCR impact factor, whereas it is ranked No. 3 in that based on IF′. On the contrary, the EMBO Journal is ranked No.10 in the JCR ranking, whereas it is ranked No. 24 in that based on IF′. Furthermore, the journal Current Biology is ranked No. 8 in the JCR-rankings, whereas it is ranked No. 40 in the rankings based on IF′. The variations in both ranks are determined mainly by their shapes of citation distribution curves. Of note, according to the data listed in the Appendeix-1, the head-tail ratios for the journals Nucleic Acids Research, EMBO Journal and Current Biology are 0.597, 0.245 and 0.231, respectively. It is obvious that the shape of the citation distribution curve for the journal Nucleic Acids Research is relatively sharp, whereas those for the journals EMBO Journal and Current Biology are relatively flat with a longer tail.

Table 2: Comparison between two citation distributions for four pairs of journals with appropriately equal JCR IF, h-index and publication numbera
Table 3: The top ten journals in the rankings based on IF′ and JCR impact factora

In the analysis above, we show that with these journals, IF′ has a one-to-one correspondence, whereas the JCR impact factor has a one-to-multiple correspondence. This is due to the fact that the JCR impact factor does not possess the ability to discriminate the shapes of the citation distribution functions, whereas the shapes of the citation distribution curves are properly taken into account by IF′. As a consequence, the JCR impact factor as an index of citations per publication usually rewards low productivity, and penalizes high productivity6.

Discussion

The JCR impact factor is the averaged number of citations received per publication. As pointed out by Hirsch6, measuring the average citations per publication, as does the JCR impact factor, usually rewards low productivity, and penalizes high productivity. This is due to the fact that such indices do not possess the ability to distinguish between different shapes of citation distribution curves. To illustrate this point, let us consider two journals among the 227 journals. The first one is the Journal of Physiology and Biochemistry (JPB) and the second one is the Journal of Liposome Research (JLR). The JCR impact factor and the number of publications of both journals are roughly equal. The related data of the two journals are listed in Table 2. Their citation distribution curves are shown in Fig. 2. Note that the shapes of the two curves are different, as clearly shown in Fig. 2. The shape of the citation distribution curve for JPB is relatively sharp, whereas that for JLR is relatively flat. Unfortunately, the JCR impact factor cannot distinguish between the two cases, resulting in almost identical impact factor values (1.711 vs. 1.707). On the contrary, IF′ is able to distinguish between the academic performance of the two journals,resulting in the values of IF′ (3.658 vs.1.867).This example shows that IF′ results in distinct values for different journal citation distributions, whereas different citation distribution curves may correspond to the same JCR impact factor value. In other words, IF′ and journals have a relation of one-to-one correspondence, whereas the JCR impact factor and journals have a relation of one-to-multiple correspondence. Table 2 shows that in addition to the numbers of publication, the total citations, the average citations per publication (i.e., the JCR impact factor) and the h-indices are all roughly equal to each other for the two journals. Of note, the four indices mentioned above are all the single-number evaluation indices currently available and widely used nowadays. Unfortunately, none of them is capable of discriminating the academic performance of the two journals. On the contrary, IF′ (or the h′-index) is a single-number evaluation index that is able to distinguish between the academic performance of journals with different citation distribution. In addition to the two journals mentioned above, other three similar pairs of journals are also listed in Table 2.

Figure 2: Comparison between two citation distribution curves for two journals with approximately equal JCR impact factor and number of publications.
Figure 2

Note that the citation distribution curve for the journal JPB is relatively peaked, indicating that a part of its publications are highly cited, whereas that for the journal JLR is relatively flat, indicating that most of its publications are relatively lowly cited. Although the total citations are approximately equal, the academic performance of the two journals is different. The JCR impact factor cannot distinguish between the two journals, with almost identical JCR impact factor IF ≈ 1.7. On the contrary, IF′ possesses the ability to distinguish between the two journals with IF′ ≈ 3.6 and IF′ ≈ 1.8 for JPB and JLR, respectively.

Of note, the example above is a typical case, rather than a rare event, because we can find a number of similar cases, as shown in Table 2. Let us further consider two or more journals with roughly equal JCR impact factors, but the numbers of publication may be different. Such cases occur more frequently than the example shown above. Although their JCR impact factors are roughly equal, their citation distribution curves may be different, even quite different. The traditional JCR impact factor lacks the ability to discriminate different shapes of the citation distribution curves, whereas IF′ proposed in this paper does not have this drawback. In this regard, IF′ is an alternative to the JCR impact factor with unique features that reward journals with highly-cited papers, but disregard lowly-cited papers. We need to point out that whether journals with such peaked citation distribution are preferable to those having uniform distribution is arguable. However, current studies on the citation distributions show that the citation distributions obey the so-called power law24,25,26, stretched exponential24,27,28, lognormal24,29,30,31 and modified Bessel function24,32. According to the parameters involved in these functions, the citation distributions are usually skewed, rather than uniform.

In summary, IF′ is proposed as an alternative to the JCR impact factor with some unique features. (i) IF′ is designed to possess the ability to distinguish between different shapes of the citation distribution curves. It gives larger values of IF′ to those with relatively sharp citation distribution curves, whereas lower values to those with flat ones. This is in agreement with the common point of view of academic evaluation33,34. (ii) Citation information of three years is considered by IF′, rather than 2 years for JCR impact factor, and therefore IF′ carries more citation information. (iii) IF′ is basically consistent with the JCR impact factor, and the two impact factors have roughly the same order of magnitude. These features make IF′ relatively easy to be accepted by the academic community. IF′ may properly reflect the academic performance of a scientific journal in a way that is different from the JCR impact factor with some unique features that reward journals with highly cited papers, and therefore it could be used to complement the JCR impact factor.

Methods

As an example, the journals in the category of biochemistry and molecular biology were studied. According to the Thomson Reuters Journal Citation Reports (JCR) Science Edition 2011, there are 290 journals listed in the subject category BIOCHEMISTRY & MOLECULAR BIOLOGY. Among these 290 journals, a few of them changed their titles during the period of 2006 to 2010. Some journals published fewer than 100 articles, whereas some were not continuously indexed by the Web of Science database (WoS) during the period of 2006–2010. All the journals mentioned above were excluded from the current study. Consequently, 227 journals were remained for the present study. We downloaded the Journal Summary List of the subject category BIOCHEMISTRY & MOLECULAR BIOLOGY in the JCR Science Edition 2011 and extracted the Impact Factor values of the 227 journals. The detailed titles and related data for the 227 journals are listed in the Appendix-1.

References

  1. 1.

    Citation indexes to science: a new dimension in documentation through association of ideas. Science. 122, 108–111 (1955).

  2. 2.

    , & The journal impact factor: A misnamed, misleading, misused measure. Cancer Genet. Cytogenet. 104, 77–81 (1998).

  3. 3.

    , & The “impact factor” revisited. Biomed Digit Libr. 2, 7. 10.1186/1742–5581-2-7. (2005).

  4. 4.

    & What the impact factor means for surgery journals. World J. Surg. 30, 1368–1370 (2006).

  5. 5.

    , , , & Refining dermatology journal impact factors using pagerank. J. Am. Acad. Dermatol. 57, 116–119 (2007).

  6. 6.

    An index to quantify an individual's scientific research output. Proc. Natl. Acad. Sci. U. S. A. 102, 16569–16572 (2005).

  7. 7.

    A case study of the Hirsch index for 26 non-prominent physicists. Ann. Phys.-Berlin. 16, 640–652 (2007).

  8. 8.

    The h-index and its alternatives: An application to the 100 most prolific economists. Scientometrics. 80, 317–324 (2009).

  9. 9.

    Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics. 67, 491–502 (2006).

  10. 10.

    & The state of h index research: Is the h index the ideal way to measure research performance? EMBO Rep. 10, 2–6 (2009).

  11. 11.

    The Hirsch index and related impact measures. Annu. Rev. Inform. Sci. Technol. 44, 65–114 (2010).

  12. 12.

    , & The diffusion of H-related literature. J. Informetr. 5, 583–593 (2011).

  13. 13.

    , & The h-bubble. J. Informetr. 7, 294–300 (2013).

  14. 14.

    , & A Hirsch-type index for journals. Scientometrics. 69, 169–173 (2006).

  15. 15.

    & A systematic analysis of Hirsch-type indices for journals. J. Informetr. 1, 179–184 (2007).

  16. 16.

    Ranking forestry journals using the h-index. J. Informetr. 2, 326–334 (2008).

  17. 17.

    , & A relation between h-index and impact factor in the power-law model. J. Am. Soc. Inf. Sci. Technol. 60, 2362–2365 (2009).

  18. 18.

    The e-index, complementing the h-index for excess citations. PLoS One. 4, e5429 (2009).

  19. 19.

    & Probing the h-core: an investigation of the tail-core ratio for rank distributions. Scientometrics. 84, 431–439 (2010).

  20. 20.

    A novel triangle mapping technique to study the h-index based citation distribution. Sci Rep. 3, 1023 (2013).

  21. 21.

    The h′-index, effectively improving the h-index based on the citation distribution. PLoS One. 8, e59912 (2013).

  22. 22.

    , , & The R- and AR-indices: Complementing the h-index. Chin. Sci. Bull. 5, 855–863 (2007).

  23. 23.

    Thomson Reuters. Journal citation reports help. (2012-05-22).

  24. 24.

    & A Reverse Engineering Approach to the Suppression of Citation Biases Reveals Universal Properties of Citation Distributions. PLoS One. 7, e33833. 10.1371/journal.pone.0033833 (2012).

  25. 25.

    How popular is your paper? An empirical study of the citation distribution. Eur. Phys. J. B. 4, 131–134 (1998).

  26. 26.

    The Skewness of Science. J. Am. Soc. Inf. Sci. 43, 628–638 (1992).

  27. 27.

    & Stretched exponential distributions in nature and economy: “fat tails” with characteristic scales. Eur. Phys. J. B. 2, 525–539 (1998).

  28. 28.

    , & Modeling a century of citation distributions. J. Informetr. 3, 296–303 (2009).

  29. 29.

    , & Effectiveness of journal ranking schemes as a tool for locating information. PLoS One. 3, e1683. 10.1371/journal.pone.0001683 (2008).

  30. 30.

    , & Universality of citation distributions: Toward an objective measure of scientific impact. Proc. Natl. Acad. Sci. U. S. A. 105, 17268–17272 (2008).

  31. 31.

    , & Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. J. Am. Soc. Inf. Sci. Technol. 61, 1377–1385 (2010).

  32. 32.

    Two-step competition process leads to quasi power-law income distributions. Application to scientific publication and citation distributions. Physica A. 298, 530–536 (2001).

  33. 33.

    & Scientific Output and Recognition - Study in Operation of Reward System in Science. Am. Sociol. Rev. 32, 377–390 (1967).

  34. 34.

    , & The h index research output measurement: Two approaches to enhance its accuracy. J. Informetr. 4, 407–414 (2010).

Download references

Acknowledgements

We thank Dr. F Gao for the help in preparing Fig. 1 and Fig. 2. The present study was supported in part by the Research Foundation of the ISTIC-Thomson Reuters Joint Lab for Scientometrics Research (No. IT2010014) and National Social Science Foundation of China (No. 11BTQ022), both to ZGY.

Author information

Affiliations

  1. Library, Tianjin University, Tianjin 300072, China

    • Zu-Guo Yang
  2. Department of Physics, Tianjin University, Tianjin 300072, China

    • Chun-Ting Zhang

Authors

  1. Search for Zu-Guo Yang in:

  2. Search for Chun-Ting Zhang in:

Contributions

Conceived and designed the experiments: C.T.Z. Performed the experiments: Z.G.Y. Analyzed the data: C.T.Z. and Z.G.Y. Wrote the paper: C.T.Z.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Chun-Ting Zhang.

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    Journal title, JCR impact factor and IF'

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/srep03410

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.