Systems: Trusting data's quality

Nature (2006) | doi:10.1038/nature04993

Database publication presents unique challenges for the peer reviewer

The reader of a scientific paper in a high-quality journal knows that the information has been vetted by a formal process of peer review, moderated by editors. But the face of publishing is changing, and peer review of databases is becoming an increasingly important facet of scientific data curation. The Signaling Gateway's molecule pages represent an important, innovative experiment in applying models of peer review developed in journals to the much newer world of scientific databases.

Databases are a tricky business: too little information and they have no use; too much and the user is at a loss as to what is relevant and what may be background noise. Peer review offers a way to find the middle ground, ensuring that only the most relevant data are included and that a complete picture is given. Users can then trust not only the quality of the information but also be confident that they have the complete picture.

Signaling Gateway

The Signaling Gateway molecule database is an open-access relational database aiming to summarize all relevant published information pertaining to almost 4,000 cell-signalling molecules. Up to now, 241 have been published. As far as I am aware, it is the only open-access database of its kind to be authored by specialists and subjected to anonymous peer review in a process similar to that used by traditional journals. The procedure presents unique advantages and challenges.

Individual molecule pages contain both unreviewed, automated data and peer-reviewed author-entered data. The automated data integrate information about the molecule from external database records, such as DNA and protein sequence information, structural information, and basic biophysical and biochemical properties. The author-entered component consists both of free text and interlinked formalized descriptions of all biochemical and cellular states of a particular molecule.

The author-entered component is equivalent to a review article, with 'States', 'Transitions' and 'Functions' sections representing the information contained in a free-text summary section in a highly structured and interconnected way. This makes it usable by data modellers as well as by the scientific community in general.

Peer review follows a similar pattern to that of a commissioned review article. Most vetting is at the author application and invitation stage, when we apply stringent criteria to find suitable authors. More than half of unsolicited applications to author molecule pages are declined by this process.

Once a molecule page is submitted, we recruit anonymous peer reviewers who ensure that the information presented is both correct and unbiased. Where controversy exists in a field, peer review, and if necessary arbitration by the editorial board, ensures that the final version represents the different perspectives, so readers are able to judge for themselves.

Molecule pages can be rejected after peer review and revision, if the account is incomprehensible, unbalanced or incorrect. Because articles are commissioned reviews, rejection rates are significantly lower than those of journals publishing primary research, as for any "conventional" review journal. About 5% of molecule pages are rejected after peer review.

Because of the nature of the material, the peer-review process is conducted entirely online, with reviewers entering their comments directly on to the molecule page alongside the relevant content. We give reviewers anonymous user accounts, and they either accept or comment on each individual section containing author-entered data.

Learning curve

The display of the data in the molecule pages is new to many reviewers, particularly in the States and Transitions sections, where covalent modifications and interactions with binding partners are broken down and represented systematically. The learning curve for the reviewer can be greater than that of traditional peer review, as at times reviewers struggle to grasp exactly what it is they are being asked to review.

Most peer reviewers take their role seriously and often contribute immensely to the finished piece. Authors are usually grateful for the constructive feedback. Yet because the reviewers' contributions are anonymous, their contributions go largely unrecognized. Is it really necessary for peer review to be anonymous for material that is summarizing published information? How could peer reviewers who contribute significantly to the finished product be acknowledged?

A fundamental aspect of scientific databases is that information can be captured and published quickly, presenting a topical view of continually changing data. A major drawback in adding peer review to the process of database curation is the slower rate at which data can be added. It can take several months for articles to be peer reviewed, revised and edited. In fields that move quickly, data may already be outdated by the time they are published.

We are in the process of introducing wiki social-software tools to the molecule pages. Wikis allow any registered user to edit information, so the tool would sit alongside the peer-reviewed data and could offer a viable solution to the issue of data curation speed. It must be made clear to the user, however, which information has been peer reviewed and which has not. Wikis in the molecule pages will provide a platform for the signalling community to become more involved, but their success will depend largely on whether the community uses them.

RELATED LINKS:

AfCS-Nature signaling gateway

Molecule pages so far published

Beta-Catenin molecule page: an example of a published, peer-reviewed molecule page

Brenda Riley is associate editor of the AfCS Nature Signaling Gateway. She can be contacted at signaling@nature.com.

Visit our peer-to-peer blog to read and post comments about this article.

Extra navigation

.
ADVERTISEMENT