Technical solutions: Wisdom of the crowds

Nature (2006) | doi:10.1038/nature04992

Scientific publishers should let their online readers become reviewers.

Who are the peers in peer review? In journals such as Nature, they usually have a PhD and work in a field relevant to the paper under consideration. If they are academics, they may be tenured professors, usually people on a relatively short list of experts who have agreed to review papers. This is a little élitist, but credentials such as PhDs and tenure are given in part to reward those things – experience, insight, brains and the respect of other researchers – that also make for wise advice. The process is not perfect, for reasons ranging from cronyism to capriciousness, yet long experience has shown it to be better than the alternatives.

But now a new kind of peer review is emerging online, outside the scientific community, and it's worth asking if there are lessons for science. In the Internet age, 'peer' is coming to mean everyman more than professional of equal rank. Consider the rise of 'peer-to-peer' networks and the user-created content of 'peer production', such as the millions of blogs that now complement mainstream media.

Perhaps the most dramatic example is Wikipedia, an online encyclopaedia written and edited by more than 100,000 volunteers. It is, says its founder Jimmy Wales, not so much anti-élitist as it is 'anti-credentialist'. Wikipedia's editors don't have to have PhDs or any sort of professional affiliation; their contributions are considered on their merit, regardless of who they are or how they have become knowledgeable. If what they write stands up to inspection, it remains; otherwise it goes. Everyone – readers and contributors alike – sees the same 'Edit this page' button, inviting correction and amplification. Aside from very obscure entries and very controversial ones, this usually results in a continuous improvement in quality, as clear prose drives out muddy phrasing and many eyes fact-check assertions.

Ruling this world of open publishing is the egalitarian hierarchy of Google, which establishes relevance and merit based on incoming links from other sites. The authority of a citation is determined by how many others cited it. It is citation analysis 'crowdsourced' to the collective wisdom of anyone with a computer. It sounds like chaos, but in reality it's the closest thing to an oracle the world has ever seen. It uses a form of peer review, but not one that a scientific journal would recognize.

Two types of 'peer'?

Are these two different uses of 'peer' – in science, a professor; online, anyone – just a case of sloppy semantics, or could the wisdom of online crowds be a model for scientific review? The answer depends on what you want. As it stands, peer review in journals primarily decides if a paper is novel and interesting enough to satisfy both the space constraints of a printed publication and the journal's mission – less 'Is it right?' and more 'Is it right for the readers of journal X?' Of course, peer review can also catch errors and suggest improvements, but its main role is to give the journal's editors a thumbs-up or -down.

Online, plenty of websites let their readers decide what makes the front page. Slashdot and Digg, for instance, are remarkably vibrant and useful news services where users submit stories and others vote them up or down. Readers can also comment on the stories, providing a real-time counterpoint that is often as illuminating as the original article.

But although this online method feels a bit like peer review, it's not yet a substitute for the current process of publishing original scientific research. For starters, stories submitted to these collective news services have already been published elsewhere – the aggregators serve mostly as reading clubs that filter the news for interesting nuggets, rather than determining what should be published.

Yet there are elements of this model that could work for science. Scientific peer review is typically a process of 'pre-filtering' – deciding which of the many papers submitted should be published. By contrast, Digg is a 'post-filter', deciding which of the many papers published are most interesting to a group of readers. Today, scientific publishing does the first kind of filtering pretty well, and the second hardly at all. Word of mouth aside, citation analysis, tracked in databases such as ISI and Scopus, is the only good way to determine what the collective wisdom has decided are the most important papers, and this takes years to emerge. Is there a faster and better way?

Publishing experiments

Several online publications are trying to find out. Naboj aims to be a post-filter on physics papers published on the ArXiv online repository of physics preprints. It currently consists only of readers rating papers on a five-point scale, but the rankings are better than nothing. Philica, an online journal started earlier this year, goes further: it publishes any paper submitted, but ranks them based on open peer review by any reader.

The free online Public Library of Science (PLoS) journals are planning to extend this model by adopting conventions from the blogosphere: an open comment area for each paper, 'trackbacks' that show which sites are linking to it, and perhaps a reader ratings scheme. Michael Eisen, a genomics researcher at the Lawrence Berkeley National Laboratory and one of PloS's founders, says the hope is to capture some of the post-publication wisdom already found in academia, but rarely accessible to others.

PLoS still uses expert researchers to review papers before publication, but the editors realize that these scientists often have little time to really dig into a paper. By contrast, readers of a paper after publication may also have an opinion, and many (especially graduate students) have the time to evaluate the paper in depth. The online environment means there's no reason not to record it.

Such a record would have the effect not only of continuing peer review after publication, but also of making it easier to find important work in a blizzard of papers – they're the ones that are being buzzed about. It is also easier to ignore poor work that slipped through peer review – these are the papers with the withering comments and poor ratings.

Best of all, such an open peer-review process taps into something that already exists: journal clubs. Every day, thousands of researchers and students are discussing the latest papers, but their insights and opinions are not recorded and shared widely. This information needs only to be collected, organized and distributed to become far more useful. It's now possible to tap such collective intelligence online by doing to scientific publishing what the web has already done to mainstream media: democratizing it.

So the rise of the online 'peer' has shown that there is another way of tapping collective wisdom. But it's not going to eliminate traditional peer review anytime soon. The reason why can be explained in the economic terms of scarcity and abundance. Closed peer review works best in scarce environments, where many papers fight for a few coveted journal slots. Open peer review works best in an abundant environment of online journals with unlimited space or the post-publication marketplace of opinion across all work.

In the scarce world of limited pages in top journals, prestige is earned through those journals' high standard and exclusivity. That comes, in part, from the process, which involves impressing the very discriminating combination of an editor and a few respected researchers. Defining 'peer' relatively narrowly is part of the game. It's not always fair or efficient, but in a world ruled by reputation, having successfully run that gauntlet is proof of at least some kind of fitness.

But in the abundance market of online journals or that of post-publication filtering, where each paper is competing with all the other papers in its field, it's more sensible to define 'peer' as broadly as possible, to maximize the power of collective intelligence. In that market, prestige is just one factor in many determining relevance for a reader, and the more filtering aids that can be brought to bear, the better. From that perspective, these are exciting times. The experiments of Nature, PLoS journals and others will reveal where and how these techniques work best. But Wikipedia and Digg have already demonstrated that they do work.

RELATED LINKS:

ArXiv

Digg

ISI

Naboj

Philica

PLoS

Scopus

Slashdot

Wikipedia

Chris Anderson is the editor-in-chief of Wired magazine and author of The Long Tail: Why the Future of Business is Selling Less of More, to be published in July 2006. He can be contacted through www.wired.com.

Visit our peer-to-peer blog to read and post comments about this article.

Extra navigation

.
ADVERTISEMENT