to nature home page debates5 November 1998
 
home
search



The invisible hand of peer review

STEVAN HARNAD

The refereed journal literature needs to be freed from both paper and its associated production costs, but not from the process of peer review, whose "invisible hand" is what maintains its quality. Is there a way to continue providing this quality at no cost to the reader?

Human nature alas cannot be relied upon to police itself. Standards of human performance need to be actively maintained lest they regress on the mean, or even meaner. In learned inquiry, the traditional mechanism of quality control has been "peer review,"1, in which specialists submit their work to qualified adjudicators ('editors'), who in turn select specialists ('referees') to advise them as to whether the material has potential and if so, what further work is needed to make it publishable. A paper is only accepted when the requisite revisions are made to the satisfaction of both editor and referees.

Pitfalls of peer policing

This quality control system is far from infallible. Editors can botch or bias the referee selection (most editors would admit that a deliberate bad choice of referees can ensure the acceptance or rejection of any paper) or they can misinterpret or misapply referees' advice. Referees can fail to be sufficiently expert, informed, conscientious or fair.

Nor are authors always conscientious in applying the advice of peer review. Indeed, virtually every paper eventually manages to get published, somewhere2,3. This produces a hierarchy among journals, based on the rigour of their peer review, with an unrefereed vanity press at the bottom. Persistent authors can start at the top and work their way down until their paper finds its own level, consuming considerable time and resources, some donated by referees and some paid for by the journals' publishers. Referees may even find themselves asked to review the same paper, sometimes unaltered, for several different journals.

Yet no one has demonstrated a viable alternative to having experts judge the work of their peers, let alone one that is at least as effective in maintaining the quality of the literature4. Thus, despite its flaws, this system has prevailed to the present day.

Self policing?

Alternatives have, of course, been proposed from time to time, based invariably on weakening the constraints of classical peer review in some way, the most radical being to do away with it altogether: Let authors police themselves, let every paper be published, and let the reader decide what is to be taken seriously. This would amount to discarding the current hierarchical filter which has served both to direct authors' revisions and to guide readers' navigation of an ever-swelling literature5.

Here is a thought-experiment to test such a laissez-faire system: Imagine someone near and dear to you is ill with a serious but potentially treatable disease. Would you prefer to have them treated on the basis of the refereed medical literature or on the basis of an unfiltered free-for-all where the distinction between reliable expertise and ignorance, incompetence or charlatanism is left entirely to the reader, on a paper by paper basis? (If we are not prepared to generalise the verdict of this thought-experiment to all of scholarly research, then we must go on to ask ourselves how seriously we take the acquisition of knowledge.)

A variant on this scenario is about to be undertaken by the British Medical Journal [http://www.bmj.com/cgi/shtml/misc/peer/index.shtml]. Taking a cue from developments on both the Internet and TV chat-shows: submitted papers will be publicly posted, unrefereed, and all readers will be invited to submit commentaries. These will then be used in lieu of referees' reports to decide on formal publication.

Expert opinion or opinion poll?

Is this peer review? It is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). Experts in any given speciality are a scarce resource, already over-harvested by classical peer review. One wonders who would have the time or inclination to act as journeyman referees in such a system, particularly if the entire raw literature were appearing online. Do we really want to entrust such a crucial function to those who have nothing more pressing to do with their time than to rummage through unfiltered free-flow? And considering all that hangs on being published in refereed journals, it is easy to imagine ways in which authors could manipulate such a system to their own advantage.

Even if we can settle the "peer" question, is peer commentary the same as peer review? Many researchers would feel inhibited in publicly pointing out flaws in the manuscript of someone who might be refereeing their next grant application. (How many people would vote in elections if voting were not anonymous?) In addition, hard-pressed expert scientists are unlikely to expend the same time and effort over a public commentary – which would be one of many, to be used who knows how by who knows whom – that they would when personally invited by an editor to referee a particular paper because of their specific expertise.

Peer commentary vs. peer review

I have had over two decades' worth of direct experience sampling the differences between peer review and peer commentary1,4. My experience has been based on umpiring a printed journal (Behavioral and Brain Sciences, published by Cambridge University Press, [http://www.princeton.edu/~harnad/bbs.html]) as well as an online journal (Psycoloquy, sponsored by the American Psychological Association, [http://www.princeton.edu/~harnad/psyc.html], both providing "open peer commentary"6 over and above peer review. The two journals are rigorously refereed, but only those papers (less than 25% of those submitted) that successfully pass that first filter of revisions and re-refereeing go on to run the gauntlet of open peer commentary, a powerful and important supplement to peer review, but certainly no substitute for it. Referee reports, some written for the editor's eyes only, all written for at most the author and fellow referees, are nothing like public commentaries, which are intended for the eyes of the entire learned community.

Successful test-site in Los Alamos

In fields such as Physics, where rejection rates are lower (perhaps because authors are more disciplined and realistic in their initial choice of journal!), there is less difference between the unrefereed preprint literature and the refereed reprint literature. The remarkable Los Alamos Physics Archive [http://xxx.lanl.gov] (supported by the US National Science Foundation and Department of Energy) is a free, public repository for a growing proportion of the current physics literature, mirrored in 15 countries and growing steadily. It has over 35,000 daily users and 25,000 new papers archived annually. Most papers are initially deposited as unrefereed preprints, and for some (no one knows how many), their authors never even bother replacing them with the final revised draft that is accepted for print publication. Yet Los Alamos is actively used and cited by the world's physics community7-9.

Some may be tempted to infer from this that refereeing could be safely jettisoned after all, but that would be to forget that even in the unrefereed sectors of Los Alamos the 'invisible hand' of peer review is exerting its standard-maintaining influence. Every paper deposited there is destined for a peer reviewed journal. The authors accordingly all know in advance that they will be answerable to editors and referees, which constrains how their papers are written in the first place. Remove that constraint, let the authors be answerable to no one but the general users of the Archive, and human nature would soon prevail, standards devolving inexorably toward those of the free-for-all chat-groups of Usenet [http://tile.net/news/listed.html], until someone re-invented peer review and quality control.

A subversive proposal

Yet conservatism about quality control10,11, can be reconciled with radicalism about a toll-free literature like the Los Alamos Archive, without sacrificing any of the benefits of peer review12-14. Refereed paper journals are paid for by subscription, and their electronic versions by site license or pay-per-view. Both the medium (print) and these methods of cost recovery restrict readers' access to the literature, whereas authors, who contribute their papers for free, would without exception prefer if everyone everywhere had full, free access to them, if that were possible.

With the end of the paper era this has become possible: An online-only refereed journal literature would be much less expensive to publish (less than 1/3 of the current price per page) once it was paper-free15 but it would still not be entirely cost-free, however, as peer review and editing would still need to be financed16.

If those residual costs were simply paid at the author's end (out of institutional publication funds redirected from cancelled library journal subscriptions) instead of the reader's end, then all the papers could be made freely accessible to everyone. They could be distributed through discipline-specific archives (for example CogPrints [http://cogprints.soton.ac.uk]) and eventually through a single international, interdisciplinary archive with many mirror sites worldwide (why not Los Alamos?). That archive would have both an unrefereed preprint sector and a refereed, published, reprint sector, tagged by journal name. Journal publishers would continue to provide the quality control, while the public archive would handle both submission and distribution.

Streamlining peer review for the airwaves

Peer review is medium-independent, but online-only distribution will mean it can be provided more cheaply and efficiently. It will also be more equitable and effective. Papers can be submitted electronically and archived on the Web, either publicly or in restricted access sites for referees only. This would eliminate the need to mail hard copies around the world17.

Journal editors will be able to approach a larger population of potential referees, who can be invited to respond if they have the time and inclination. Referee reports can be returned by email or deposited through password-controlled Web interfaces. These reports can be revised, published and linked to the accepted article if the referee wishes, along with further commentaries (both refereed and unrefereed), and authors' responses. Furthermore, authors can update and revise their original articles as often as necessary.

Galactic hitch-hiking, PostGutenberg

Learned inquiry is a continuum18. Reports of its findings, both informal and formal, unrefereed and refereed, are milestones, not gravestones. As such, they need only be reliably sign-posted. The discerning hitch-hiker in the PostGutenberg Galaxy19 can take care of the rest.


References

1. Harnad, S. Rational disagreement in peer review. Sci. Tech. Human Val. 10, 55-62 (1985).
2. Lock, S. A difficult balance: editorial peer review in medicine, (Nuffield Provincial Hospitals Trust. London, 1985).
3. Harnad, S. Policing the paper chase. Nature 322, 24-25 (1986).
4. Harnad, S. (ed.) Peer commentary on peer review: A case study in scientific quality control (Cambridge University Press, NY, 1982).
5. Hitchcock, S., Quek, F., Carr, L., Hall, W., Witbrock, A. & Tarr, I. Linking everything to everything: journal publishing myth or reality? The open journal project (c. 5 Nov. 1998) <http://journals.ecs.soton.ac.uk/IFIP-ICCC97.html> (1997).
6. Harnad, S. Creative disagreement. The Sciences 19, 18-20 (1979).
7. Youngen, G. Citation patterns of the physics preprint literature with special emphasis on the preprints available electronically. UIUC Physics and Astronomy library [online] (c. 5 Nov. 1998) <http://www.physics.uiuc.edu/Physics/library/preprint.html> (1997)
8. Ginsparg, P. First steps towards electronic research communication. Computers in Physics 8, 390-396 (1994).
[http://xxx.lanl.gov/blurb/]
9. Ginsparg, P. Winners and losers in the global research village. xxx (c. 5 Nov. 1998) < http://xxx.lanl.gov/blurb/pg96unesco.html> (1996).
10. Harnad, S. Implementing peer review on the net: Scientific quality control in scholarly electronic journals. in Scholarly publishing: The electronic frontier. (Peek, R. & Newby, G. eds) 103-118 (MIT Press, Cambridge MA, 1996).
[ http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/
harnad96.peer.review.html
]
11. Harnad, S. Learned inquiry and the net: The role of peer review, peer commentary and copyright. Antiquity 71, 1042-1048 (1997).
[http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html]
12. Okerson, A. & O'Donnell, J. (eds) Scholarly journals at the crossroads; a subversive proposal for electronic publishing. (Association of Research Libraries, Washington, DC, 1995).
[http://www.arl.org/scomm/subversive/index.html & ftp://ftp.cogsci.soton.ac.uk/pub/psycoloquy/Subversive.Proposal]
13. Harnad, S. For whom the gate tolls? Free the online-only refereed literature. Am. Sci. Forum (c. 5 Nov. 1998) <http://amsci-forum.amsci.org/scripts/wa.exe?A2=ind98&L=
september-forum&F=lf&S=&P=47
> (1998).
14. Harnad, S. On-line journals and financial fire-walls. Nature 395, 127-128 (1998).
[/server-java/Propub/nature/395127A0.frameset]
15. Walker, T.J. Free Internet Access to Traditional Journals. American Scientist 86, 463 (1998).
[http://www.amsci.org/amsci/articles/98articles/walker.html and see http://amsci-forum.amsci.org/archives/september-forum.html]
16. Odlyzko, A.M. The economics of electronic journals. in Technology and Scholarly Communication (eds Ekman R. & Quandt, R. University of California Press, Los Angeles, 1998).
[http://www.research.att.com/~amo/doc/economics.journals.txt]
17. Pope, S. & Miller, L. Using the web for peer review and publication of scientific journals. Conservation ecology [online], (c. 5 Nov. 1998) <http://www.consecol.org/Journal/consortium.html> (1998).
18. Harnad, S. Scholarly skywriting and the prepublication continuum of scientific inquiry. Psycholog. Sci. 1, 342-343 (1990).
[http://www.cogsci.soton.ac.uk/~harnad/Papers/
Harnad/harnad90.skywriting.html
]
19. Harnad, S. Post-Gutenberg galaxy: the fourth revolution in the means of production of knowledge. Public-Access Comp. Syst. Rev. 2, 39-53 (1991).
[http://www.cogsci.soton.ac.uk/~harnad/Papers/
Harnad/harnad91.postgutenberg.html
]

Stevan Harnad
Multimedia Research Group Electronics and Computer Science Department,
Southampton University,
Highfield,
Southampton SO17 1BJ
United Kingdom

http://www.princeton.edu/~harnad/intpub.html
http://cogsci.soton.ac.uk/~harnad/intpub.html

 

Macmillan MagazinesNature © Macmillan Publishers Ltd 1998 Registered No. 785998 England.