Scientific journals are ranked according to the average number of citations to the content that they publish — that is, the widely known journal impact factor (IF). Yet the citation distribution of articles in a journal is skewed by the relatively few ‘hits’, which makes the average number of citations a weak measure of the central tendency of the citation distribution. Still, despite the large number of recent stories about the many shortcomings of the IF, the metric remains in vogue as a measure of journal prestige. Scientists can be obsessed with imperfect rankings as much as anyone else.

The IF's unfathomed popularity when ranking journals and in helping readers to select what to read and authors to decide where to send their work should not be reason for complacency. The misuse of the IF ought to be challenged. Foremost, using a journal's IF to assess the quality of a particular paper or scientist published in the journal is akin to illiteracy in statistics. Second, achieving scientific merit is not a popularity contest, yet the careless use of citation numbers can make it seem so. Moreover, we would also like to draw attention to the fact that, especially for IFs below 10, the metric is ineffectual as a measure of journal selectivity. In other words, for over 95% of journals the IF is not informative of the fraction of published papers that are in the top 1% most-cited papers in the journal's research area — a new metric that we name ‘impact quotient’ or IQ (Fig. 1). For example, eLife, Cancer Research, PLoS Biology and Plant Cell all have 2016 IFs within half a point of each other, yet their IQs differ vastly (up to fourfold). Similarly, for Cell and the European Heart Journal, about 20% of the papers that each published in 2014–2015 were in the top 1% most-cited in, respectively, molecular biology and genetics, and clinical medicine; yet their 2016 IFs differ by a factor of about two.

Figure 1: IQ versus IF.
figure 1

For relatively large values of the IF, the metric is indicative of journal selectivity — as measured by the IQ, the fraction of published papers (articles, reviews and conference proceedings) that are in the top 1% most-cited papers in the journal's research area. For IF ≲ 10, IFs and IQs do not correlate. The data include all journals that received a 2016 IF and published 48 or more papers in 2014–2015 (except for Nature Reviews Drug Discovery, Nature Reviews Cancer and CA: A Cancer Journal for Clinicians, which fall outside the plotted range). Data from Clarivate Analytics' Journal Citation Reports and Web of Science Core Collection, as of 25 June 2017. In Web of Science, journals are assigned to 1 of 22 research areas (four of which are named in the legend), except for multidisciplinary journals, whose papers are assigned individually to the research area most represented in the paper's reference list.

Full size image

Arguably, the two most serious shortcomings of the IF — it being significantly affected by the skewed shape of the citation distribution and by the variability in citation rates across research fields — can be alleviated by journal metrics based on the citation distribution of a broad research area and on percentiles rather than averages. The IQ is one such metric, and it has a number of advantages: it ranges from 0% to 100% (the IF doesn't have an upper boundary and typically increases over time), it only counts research articles and reviews (it is thus unaffected by citations to news and opinion articles, which inflate the IF), and for most journals with an IF above 10 the IQ and the IF have similar values (Fig. 1). And as with the IF, the IQ is simple to understand and calculate.

Beyond citation-based metrics, journals should be best assessed through a range of quantitative and qualitative parameters. This is the spirit of the Declaration on Research Assessment (known as DORA), which Nature journals have recently signed (go.nature.com/2qIA81E). Nature Research also provides a range of journal metrics for the broad family of Nature journals (go.nature.com/2arq7OM).

Papers published in highly selective journals typically receive broader exposure, which in turn attracts more citations. The journal IF inflates such an advantage. When using citation-based statistics, let's instead count highly cited papers, regardless of their actual citation numbers. It's a matter of IQ.