*This post may or may not have been peer reviewed*

Bohannon's recent article in Science has renewed the classic debate about the value of peer review. Peer review is a crucial step in the process of publishing a work of any sort. Although an author or a group of authors is credited with creating the study itself, the study supposedly "gains legitimacy" if it is reviewed by other esteemed colleagues in the same field. Flaws in the peer review process have been described for quite some time. Peters and Ceci's 1982 article criticizes peer review for focusing too much on reputation. They used previously published articles (from prestigious institutions) and simply changed the names of the authors and institutions (to fictitious people and institutions), finding that the vast majority of papers were rejected on the grounds of "serious methodological flaws" and only a few were flagged as resubmissions.

Could the reputation of the author have something to do with the results? Of course, there was also the controversy surrounding the famous 2010 study finding the existence of arsenic-based life. Individuals were falsely driven to believe that the study was legitimate, having intuitively trusted the findings because they were "peer reviewed." The Sokal hoax is an example from outside the sciences, in which Alan Sokal successfully published a paper that was computer-generated in a cultural studies journal. Differing from the anecdotal examples of peer review's failure, Goodman et al found in 1994 that peer reviewing increased the quality of reporting the results of medical research, although they did not examine the effect on the accuracy of the research itself. Peer reviewing not only lets the author publish his or her work at an earlier time, but also mold a study to what will be most successful in all areas. Most would agree that peer review is not a perfect practice, so the real question is how the process can be improved.

One problem with suggesting improvements to the peer review process is that it is comparatively difficult to measure benefits of peer review, whereas it is easier to give examples of peer review failing (mostly due to not being able to isolate peer review as a variable for success). In spite of that difficulty, a close examination of the flaws of peer review suggests that there are a variety of potential improvements. Peer review is often conducted with anonymous reviewers, and it has been criticized as a result for both a lack of transparency and a lack of accountability. Thus, one potential improvement is an opening up of the review process; however, Rooyen et al. found in 1999 that opening up peer review did not have a significant effect on review quality, although it did increase the likelihood that reviewers would decline to participate. Considering that the resources of reviewers can oftentimes be stretched thin, the potential for driving away reviewers is important to consider.

However, Walsh et al. in 2000 concluded that opening up the review process was feasible even given fewer reviewers and more time spent on each review. Thus, making peer review a public process can still be considered a viable option. Others have proposed more fundamental changes. Eisen proposes that reviewers should comment on technical merits and significance, and that the paper be classified by research area and level of interest instead of by prestige of journal. Of course, there are also ethical arguments for opening up the review process. Not only would reviewers be rewarded and accountable for their work, but authors would also be able to more easily engage in dialogue with reviewers about their papers. Although one can maximize the benefit of peer review by opening up the process to the public eye, it would come at the sacrifice of a fair evaluation. In fact, ethical considerations have caused the BMJ (British Medical Journal) to be committed to opening its review process.

Change usually takes time, especially to something as fundamental as peer review. However, there seems to be a significant push for a change in the process to occur, and that can be enough to spark a substantial, permanent, if gradual, alteration of peer review.

References:

References:

Van Rooyen, Susan. "Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial." BMJ (2010).

Eisen, Michael. "Peer review is f***ed up - let's fix it." It is NOT Junk (2011).

Walsh, Elizabeth. "Open peer review: a randomised controlled trial." British Journal of Psychology (2000).

McCook, Alison. "Is Peer Review Broken?" The Scientist (2006).

Goodman, Steven N. "Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine." Annals of Internal Medicine (1994).

Humphrey, Wilhelm N.O. "Textual Discourses: Subcultural Constructive Theory in the Works of Madonna." Elsewhere (2000).

Dawkins, Richard. "Postmodernism Disrobed." The Richard Dawkins Foundation for Reason and Science (2007).

Vergano, Dan. "Glowing reviews on 'arseniclife' spurred NASA's embrace." USA Today (2013).

Peters, Douglas P. "Peer-review practices of psychological journals: The fate of published articles, submitted again." Cambridge University Press (2010).

Bohannon, John. "Who's Afraid of Peer Review?" Science Magazine (2013).

Image Credit: Center for Scientific Review (Wikipedia), The Yorck Project (Wikipedia)