How to evaluate journal impact factors

Sir

Journal impact factors, published annually by the Institute of Scientific Information (ISI) in two Journal Citation Reports (JCR) editions (science and social science), are a well known — though not uncontested¹,² — means of evaluating and comparing the scientific impact of journals. A major disadvantage of the JCR is the limited number of evaluated journals. For example, the 1996 JCR on CD-ROM (science edition) lists 4,779 journals, roughly half of them with biomedical orientation.

In contrast, about 4,000 journals are continuously indexed in Medline, nearly 3,500 in Embase and at least 4,600 in Biosis, making a total (without duplicate titles) of about 7,000 unique journals in these three biomedical databases (data retrieved from Serline).

There is, however, an easy way to construct impact factors also for journals not included in the JCR. Online database searches can be carried out to determine the number of published articles in a given journal and the number of citations to that journal, according to ISI's definition of a journal's impact factor (number of citations in year X to papers published in the journal in question in years X−1 and X−2, divided by the number of counted articles).

The table shows constructed impact factors (CIF) for several biomedical journals not included in the JCR 1996 science edition (No. 1-11). In addition, to confirm the reliability of the method, the constructed as well as the JCR-derived impact factors (JCR-IF) are shown for a couple of journals included by JCR (No. 12-17). All online searching was carried out at the German host DIMDI (Deutsches Institut für Medizinische Dokumentation und Information). Numbers of articles published in 1994 and 1995 were retrieved by searching the databases Medline, Cancerlit, Healthstar, Embase, Biosis and Scisearch simultaneously (DIMDI's ‘superbase’ modus), followed by elimination of duplicate articles using the ‘check duplicates’ command of DIMDI's retrieval language. Editorials, letters, news and meeting reports were excluded from searches because they are not subject to ISI's citation analyses³. Numbers of citations were retrieved from ISI's database Scisearch which — in addition to bibliographic data — contains all the references cited in the indexed articles, whether the cited journal is being indexed in Scisearch or not.

The slight differences between constructed and JCR-derived impact factors may be due to the following: (i) for retrieval of published articles the document types ‘article’, ‘journal article’ and ‘review’ were used, which may have a wider definition in non-ISI databases, thus leading to higher article counts (and lower CIFs) than the more restrictive use of the term ‘research article’ seen in JCR 4; and (ii) Scisearch comprises more journals than JCR (personal communication from ISI) which may lead to higher citations.

Impact factors can only give some hints with respect to the importance of a journal in its field. Nevertheless, because there is permanent pressure on scientists to prove the impact of their work and to publish in journals with measurable impact factors, the method described might also be helpful in evaluating those journals not listed in JCR.

Table 1 Constructed impact factors for journals not in JCR