Introduction

It has always been difficult to judge the importance of scientific papers or journals, and the impact of these on users and customers is a very fickle concept. Impact factor (IF), a term familiar to most people in the dental, medical, and scientific academic communities, appeared about 30 or more years ago (http://www.isinet.com/essays/journalcitationreports/7html/),1,2,3 and has come to be regarded by some as of increasing importance, yet few fully understand its meaning or implications. Perhaps the most important use of the IF is as a useful tool for the evaluation of journals but, since it is also employed in some situations in the process of academic evaluation, considerable controversy surrounds it. The 'impact' of a publication on the community can bear little relationship to the 'impact factor'.

Definitions

The journal impact factor (IF) is published annually by the ISI (a private, profit-oriented commercial Philadelphia-based organisation which was formerly termed the Institute for Scientific Information; http://www.isinet.com). Access to the ISI databank is not free of charge.

The IF is a measure applied to a journal and is a measure of the frequency with which the 'average article' has been cited in a particular year or period. The key idea behind the IF is citations, and the number of them. Citations are the papers and book chapters listed in the references at the end of a scientific paper. Journal citation patterns (ie which author was being cited and where) have been studied since the 1920s and, since the 1960s, the Science Citation Index (SCI) and the Journal Citation Reports (JCR) have been produced based upon computer-compiled statistical reports.

The citation index makes it feasible to produce computer-compiled statistical reports not only on the output of journals but also in terms of their citation frequency. The JCR provides quantitative tools for ranking, evaluating, categorising, and comparing journals. The JCR uses the IF, along with other criteria, to compare, evaluate and rank journals; it can also indicate the largest journals, the 'hottest' journals, what publications a journal cites and which publications cite the journal itself. The IF is calculated by dividing the number of current citations to articles published in a specific journal in the previous two-year period by the total number of articles published in the same journal in the corresponding two-year period (Table 1). The annual JCR IF is a ratio between citations and recent citable items published.

Table 1 Calculation of a journal impact factor
Full size table

The impact factors are published every September. So, in September 2003, the IF for a journal for 2002 would have been calculated from the number of citations in the year 2002 to articles published in the journal in 2000 and 2001, divided by the number of articles published in the journal in 2000 and 2001. The IF is thus calculated from the number of times the articles are cited divided by the number of articles in that journal that could be cited.

It should be noted that, when the IF is calculated, the numerator is the total number of citations to virtually any item in the journal, all types of articles, such as editorials, letters and abstracts from meetings are included in the numerator — the cited items. However, the denominator is the number of articles only and includes only normal articles, notes and reviews — the citable items. The IF can be affected, for example, by a large correspondence section or by publishing controversial editorials. If a journal publishes more additional items, they become included in the numerator and thus the impact factor may well increase significantly. In contrast, an increase in citable items can have the opposite effect. For example when, in 1997, the Lancet divided its 'Letters' section into 'Correspondence' and 'Research Letters' — the latter being peer-reviewed and hence 'citable' for the denominator, the increase in the denominator led to a fall in IF from about 17 to about 12.

Only journals covered by the SCI database are scanned for their reference lists, only a very small proportion (about 5,000 science and technology titles) of the world total of over 100,000 English language journals and chiefly those from the USA are favoured, and tend to enjoy higher IF than non-English language journals. Books and book chapters are not scanned for their bibliographies, or included in any IF calculation.

Where can impact factors be found?

Journals often advertise their IF on their websites and publicity fliers (eg http://www.elsevier.com/locate/oraloncology). Otherwise, a university library or a library of one of the larger institutions can access them either on CD or on the Internet (http://www.isinet.com/isi or http://wos.mimas.ac.uk).

Impact factors of dentally-related journals

The impact factors for 2002 for the first 20 of the 49 dental journals evaluated in JCR are shown in Table 2. The IFs ranged from 1.047 to 2.956, and no other journal in this field had an IF ≤1 Interestingly, two of the three journals with the highest IFs are review journals ie they contain essentially no original research.

Table 2 Impact factors for Dentistry, Oral Surgery and Medicine, 2002
Full size table

Uses of impact factor

The IF can be helpful to evaluate a journal's relative 'importance', especially when compared to others in the same field but, as noted above, this can be comparable to comparing apples with pears. The IF is useful in clarifying the significance of absolute (or total) citation frequencies. It eliminates some of the bias of such counts which favour larger journals over small ones, of older journals over newer ones or of frequently issued journals over those less frequently issued. However, larger journals have a larger citable body of literature than smaller or younger journals and, generally speaking, the larger the number of previously published articles, the more often a journal will be cited and the higher the IF will be.

Publishers often use IF for marketing (many fliers give the journal IF) or in identifying opportunities for new journal launches or taking decisions on whether to expand, merge or discontinue existing titles. Authors sometimes use IF to decide where to publish and to discover other journals in their specialty. Other uses of IF are discussed below.

Care in the use of impact factors

Informed and careful use of IF data is essential. Ill-formed conclusions based on IF statistics may arise unless several caveats are considered.

Criticisms levelled against IF have been well documented4 and the main ones are summarised here.

The scientific field to which the journal belongs influences IF. The scientific field to which the journal belongs influences the IF greatly; ISI recognises this and warns against making comparisons between fields. For example, the highest impact factor in the ISI subject category 'Dentistry' is 2.956, whereas that in 'Oncology' is CA Cancer J Clin, with a massive IF of 32.886. But does that really mean that Oncology is 15 times as good or important as Dentistry? Different disciplines have widely differing citation practices, and general journals are at a particular advantage over more specialist journals.

Scientific journals generally rank higher than clinical journals. Scientific journals generally rank higher than clinical journals in the IF league, partly due to the fact that scientific papers tend to cite only scientific and not clinical articles, whereas clinical papers tend to cite both scientific and clinical articles.

Self-citation is also possible, increasing the IF. Self-citation is also possible, so authors happily cite previous papers in the same journal and editors may cite editorials. Some journals have been known to try to manipulate the IF by writing to authors asking them to add references to articles published in that journal.

Errors, misprints and inconsistencies in citations can distort the IF. Errors, misprints and inconsistencies in citations can distort the IF, and cause damage. Seglen4 suggests that misprints in lists of references may affect up to one-quarter of references. Errors in the published IF have caused considerable damage to at least two journals in the dental field.

IFs are biased toward journals that mainly publish review articles. IFs are biased toward journals that are review journals or mainly publish review articles, since those tend to be cited more frequently — often in authors' introductions.5 Amongst the dental journals with the highest IFs (Table 2) are the review journals Critical Reviews in Oral Biology and Medicine, and Periodontology 2000.

Multi-author and consortia articles sometimes pose a problem. Multi-author and consortia articles sometimes pose a problem regarding who — or what — should be cited, and how. For example, the International Human Genome Sequencing Consortium described the sequencing of the human genome in a paper in Nature in 2001 — probably the most important scientific advance ever and of phenomenal impact, but the IF was low.6

Greater availability tends to raise the IF. Free electronic access, or the inclusion of a journal as part of the membership to a society and therefore greater availability, tends to raise the IF of a journal. Accessibility may mean that the source is cited but it does not necessarily follow that the most appropriate or 'best' reference has been chosen to be cited. For example, in the SCI subject category 'Dentistry', the journal with the highest impact factor (Table 2) is the Journal of Dental Research. Apart from the high quality, the fact that J Dent Res is taken by most dental researchers, has surely helped it become the most highly cited journal.

Controversial or poor papers may increase the IF.7 It is worth remembering that a paper may be cited as an example of poor research or may be highly cited if it covers a controversial topic eg suggesting that HIV is not the cause of AIDS.

Citation counts in JCR do not distinguish between letters, reviews, or original research. So, if a journal publishes a large number of letters, there will usually be a temporary increase in citations of those letters. Letters to the Lancet may indeed be cited more often that letters to JAMA or vice versa, but the overall citation count and IF recorded would not take into account this artefact.

A change in journal title may adversely affect the IF. In the first year after a change, the IF is not available for the new title unless the data for old and new can be unified, and in the second year, the IF is split. The new title may rank lower than expected and the old title may rank higher than expected because only one year of source data are included in the calculation. For example, the journal Oral Oncology, once a daughter journal of the European Journal of Cancer, was undercited for several subsequent years, since many authors and journals were mistakenly still quoting it as Eur J Cancer rather than as Oral Oncology. Thus the IF for Eur J Cancer was artificially inflated and that for Oral Oncology was falsely low.

Contentious uses of impact factors

Citation analysis, in the hands of non-experts, can be an extremely blunt instrument.8 Probably the most serious criticism of IFs relates to their erroneous and potentially dangerous use to determine 'author impact'. In some countries and institutions, academic administrators significantly (and controversially) use IF as a convenient tool in the process of deciding on promotion and tenure, ie the IFs of journals in which candidates have published are used to help determine the impact or importance of their research. IFs can represent an all too convenient shortcut to bypass proper appraisal, and the work involved in obtaining the more meaningful information of citation counts for individual articles and authors. By extension, and perhaps erroneously, IFs may be taken as an indication of a person's scientific worth. By further extension, there is a possibility of IFs being used to compare institutions.

However, it cannot be over-emphasised that the IF was created with the intent of comparing journals, not authors or individual articles. The IF can be used to provide a gross approximation of the prestige of journals in which individuals have published, but this is best done in conjunction with other considerations such as peer review, productivity, and subject specialty citation rates.

Nevertheless, despite these problems, many bodies responsible for hiring, promotion, tenure, or the assessment of research groups and grant proposals for funding and resource allocation continue to evaluate the quality of an individual's research and publications by looking at the IF of the journals in which they have published. The widespread belief that the IF is representative of an individual author (or article) has been countered by the finding that citations of individual articles in a journal show a very skewed distribution.4

Authors consequently feel significant pressure to submit papers to a journal with a high IF, whether or not that journal is the most appropriate platform for their work. Many believe that the higher the journal IF that published their paper, the more their paper will be cited, but this is a myth.4

Conclusions

The impact factor is a useful tool for evaluation of journals, but it must be used very carefully. Considerations include the amount of review articles, letters or other types of material published in the journal, variations between disciplines, and item-by-item impact.

The IF is of questionable value in certain circumstances and the extent to which the IF of a journal is appropriate for the evaluation of the quality of a specific article or journal and particularly of individual and collective research achievements is undoubtedly highly questionable.9,10,11,12,13,14 Certainly, the actual impact on the community of an article is not necessarily related to the IF.