How do we recognize a good scientist? There is an entire industry — bibliometrics — that would have us believe that it is easy: count journal articles, sort them according to the impact factors of the journals, and count all the citations.
Science managers and politicians seem especially fond of such ways to assess ‘scientific quality’. But many scientists also accept them, and use them in hiring and funding decisions. They are drawn to the alleged objectivity of bibliometrics. Indeed, one sometimes hears that scientists should be especially ready to apply scientific methods to their own output. However, scientists will also be aware that no good science can be built on bad data, and we are in a unique position to judge the quality of the raw data of bibliometrics, because we generate them through our citation behaviour.
The underlying assumption of bibliometrics is that, by citing, scientists are engaging in an ongoing poll to elect the best-quality academic papers. But we know the real reasons that we cite. Chiefly, it is to refer to results from other people, our own earlier work or a method; to give credit to partial results towards the same goal; to back up some terminology; to provide background reading for less familiar ideas; and sometimes to criticize.
There are less honourable reasons, too: to boost a friend’s citation statistics; to satisfy a potential big-shot referee; and to give the impression that there is a community interested in the topic by stuffing the introduction with irrelevant citations to everybody, often recycled from earlier papers.
None of these citations — for good reasons or bad — express the opinion that the paper in question is a remarkable scientific achievement.
Consequently, highly cited papers often contain popular (but otherwise unimpressive) concepts or methods. If you have a favourite well-cited paper, it is a sobering experience to check 20 random citations. They typically contain little appreciation for the quality of the work.
To be sure, selection for an academic job guided mainly by citation statistics or papers in high-impact journals will get better results than flipping a coin. But it is blind to the difference between someone who creatively develops a research agenda — and is likely to be doing that in ten years — and someone who grinds out papers in a narrow, fashionable subfield.
“When we believe that we will be judged by silly criteria we will adapt and behave in silly ways.”
Many negative effects of bibliometrics come not from using it, but from the anticipation that it will be used. When we believe that we will be judged by silly criteria, we will adapt and behave in silly ways.
A good example is the distortion in the journal landscape — and with it the changes in the style of papers — that arose when the journal impact factor began to be taken seriously as a proxy for reputation.
For example, when Physical Review Letters (PRL) split from Physical Review, it was intended to allow speedier publication of short announcements, which had previously been sent as unrefereed letters to the editor of Physical Review.
It is easier to reach high impact with this format, so the ‘reputation’ shifted from the standard journal to the letters section. Although there is no reason that shorter papers should be scientifically better than long ones, many authors now happily mutilate their work to stay under PRL’s page limit, rendering papers less readable and less useful.
Another example is the way Nature became the top journal for experimental physicists. Life scientists are more numerous and use more citations than physicists, so the impact factors of Science and Nature, which cover all sciences, easily beat that of any non-review physics journal. Despite the higher impact factor, there is no reason why a paper written for a broad audience should be scientifically more valuable than one with an in-depth technical discussion. In fact, in pitching for such an audience, authors often leave out the tricky parts, keep technical terms out of their titles, and overstate their conclusions in broad terms.
What can we do? Simply, individual scientists must resist the trend of making bibliometrics a central plank in their decision-making processes. And we must make this public, perhaps by stating in job adverts that papers will be judged by scientific merit and not by journal impact factor.
Once a hiring decision is made, we should resist the temptation to justify it by quoting the candidate’s bibliometrics to administrators. This reinforces the damaging idea that hiring decisions could be made by administrators in the first place, and makes it harder to justify decisions that do not follow the metrics the next time round.
As the tyranny of bibliometrics tightens its grip, it is having a disastrous effect on the model of science presented to young researchers. For example, a master’s student of mine moved to a renowned research institute for his PhD. Like many institutes, this one boasts of its performance in terms of publications in high-impact journals. So my student was told: “If you cannot write up your research in a form suitable for Nature or Science or Physical Review Letters, don’t bother to even do it.” Such advice, driven by the appeal of metrics to funders, is common but horribly misguided.
If we raise scientists to be driven by such extrinsic motivation alone, then why should they not follow the logic to its natural conclusion, and run off to become well-paid bankers instead?
- Journal name:
- Date published: