The Nature Index gauges the performance of countries and research institutions by tracking the number of papers they publish in reputable journals (www.natureindex.com). We suggest that using absolute numbers and journal reputation can yield misleading results (see R. Haunschild and L. Bornmann Scientometrics (in the press) and at http://doi.org/xrg; 2014).

For example, the index ranks the Chinese Academy of Sciences above Harvard University in Cambridge, Massachusetts. Using the complete publication output from each of these in 2013 (31,428 and 17,836 articles, respectively, according to Thomson Reuters InCites), we calculate that only 8% (2,661) of papers from the Chinese Academy of Sciences contributed to the Nature Index, whereas 14% (2,555 papers) of Harvard papers contributed. This relative perspective is important, given that an institution with a high publication output would be expected to publish more papers in reputable journals.

In our view, it would also be better to measure the performance of countries and institutions on the basis of individual papers, rather than on the journals in which they are published (see http://am.ascb.org/dora). This is because the quality of a journal (as measured by peers or citations) is not a reliable proxy for the quality of each paper it publishes.

To illustrate this point, we counted citations over 5 years for papers published in 2008 in Applied Physics Letters — the journal contributing most articles to the Nature Index — using an in-house bibliometrics database maintained by the Max Planck Digital Library in Munich, Germany. We found that some 40% of the papers in Applied Physics Letters accounted for about 80% of the citations to this journal, suggesting an uneven distribution of quality.

Footnote 1