Correspondence

Nature 415, 731-732 (14 February 2002) | doi:10.1038/415731a

The impact-factors debate: the ISI's uses and limits

Henk F. Moed1

  1. Centre for Science and Technology Studies (CWTS), Leiden University, PO Box 9555, 2300 RB Leiden, The Netherlands.

Top

Towards a critical, informative, accurate and policy-relevant bibliometrics.

Sir

Your Opinion article "Errors in Citation Statistics" (Nature 415, 101; 2002) identified how journal impact factors compiled by the Institute for Scientific Information (ISI) are sometimes included as variables in mathematical formulae and directly influence funding decisions by individual research departments. Such use is inappropriate and counterproductive. In addition, the understandably negative reactions of the scientific community towards this type of use mask the great potential of bibliometric methods.

You also uncovered a particular type of inaccuracy in the ISI's citation rate of a paper by a consortium. This is the tip of an iceberg: we have detected errors skewing results among papers, authors, journals and countries, as well as other sources of error, in a large study we have carried out (details available from H. F. M. at Email: moed@cwts.leidenuniv.nl). Scientists subjected to bibliometric assessment and policy officials using it should be aware of these limitations and problems, so that they can properly evaluate and use it (see Box 1).

Bibliometric indicators reflect scientific impact, not quality, and provide useful supplementary tools in the evaluation of academic research, provided that they have a sufficiently high level of sophistication; that their pitfalls are taken into account; and that they are combined with more qualitative types of information. Their application can stimulate useful discussion among scientists and research managers about publication strategies and research directions; help peer-reviewers to make quality judgements; and enable policy officials and science administrators to raise critical questions about many aspects of scientific activity and to provide insight for policy (funding) decisions.

Individual scientists may wish to assess their publication strategies, or examine the impact of their work on related fields. Managers of research departments may wish to compare the performance of their department with those of competitors and assess their collaboration strategies. A review committee may have the difficult task of assessing a country's performance in a particular field compared with that of others. The dean of a medical faculty may wish to examine whether it can qualify as a 'research' faculty. A science minister may wish to assess procedures for submitting proposals for research projects to a national research council. Journal publishers or editors may wish to assess the position of their journals, or find suitable reviewers for submitted manuscripts.

For all these needs, context-specific bibliometric indicators were developed as supplementary tools. Yet an indicator that is useful in one context may be inappropriate in another. For instance, in a field in which international journals are dominant channels of written communication, journal impact factors are useful measures if calculated accurately. But such measures have no value in assessing individual scientists or research departments. There can be no direct relationship between statistics such as journal impact factors and policy decisions. Such statistics provide indications and category classifications rather than precise measurements. They need to be adjusted and fine-tuned in close interaction with users and with the scientists who are being evaluated, which may require a long development process. Other types of information should also be taken into account.

In our institute's huge analysis of more than 20 million cited references matched to 8 million target articles extracted from the Science Citation Index (SCI) and related ISI citation indexes, we found that when data are derived from 'simple' or 'standard' citation-matching procedures, citation statistics at the level of individuals, research groups, journals and countries are strongly affected by sloppy referencing, editorial characteristics of scientific journals, referencing conventions in scholarly subfields, language problems, author-identification problems, unfamiliarity with foreign author names and ISI data-capturing conventions. The overall number of discrepant cited references is about 7% of the number of citations obtained in a simple matching procedure similar to that applied by the ISI in establishing citation links in the Web of Science and calculating statistics for its newsletter Science Watch. Typical examples of strongly affected entities are 'consortium' papers; journals with dual volume-numbering systems or combined volumes; journals published in different versions applying different article-numbering systems; and authors from non-English-speaking countries.

This 7% of lost citations skews the distribution of discrepant citations, making some statistics highly inaccurate. For instance, a group of scientists collaborating in a consortium may lose all their joint impact; authors from China or Spain may lose 13% and 8% of their overall citations, respectively; journals publishing combined volumes, such as Applied Surface Science and Physica B, lose 15–20%. When Spanish or Chinese authors publish the main part of their output in these two journals, the percentage of 'lost' citations can easily rise to 25–30%.

Although the ISI is a monopoly supplier of the SCI and Web of Science, it is not a monopoly supplier of bibliometric statistics derived from these bibliographic information products.

Bibliographic and bibliometric use are two distinct types of use of scientific information, each with its own set of operational and quality criteria. The ISI's information products are primarily developed for bibliographic use. When conducted properly, bibliometrics can unravel relationships that were previously unknown, and put new issues on the political agenda. It can be informative in providing condensed overviews of publication and citation frequencies, and accurate if proper data-collection procedures are applied.

Anyone confronted with bibliometric statistics derived from the SCI, intended to be applied at the level of individuals or research groups or departments, should know the answers to the questions summarised in the Box.

These minimum criteria are crucial for assessment of the accuracy, validity and usefulness of bibliometric statistics.