Scientists must be attentive to ensure the integrity of the published record and find new ways to facilitate scientific debate in the electronic age.
In our era of up-to-the-minute reporting and social networking, newly published scientific discoveries—especially those appearing in high-profile journals—are reaching larger scientific audiences and the general public with greater efficiency. However, these online media are also being used more and more to broadcast criticisms of a study's technical merit or conclusions. This changing landscape provides two lessons: first, scientists need to do all they can to prevent erroneous science from being published in the first place; second, they need to make better use of the online environment to maximize transparency and scientific communication.
As case studies, two recent high-profile papers have received widespread criticism for apparent lapses in scientific rigor and have attracted the interest of many chemical biologists because of their provocative chemical content and the public debate they have initiated. A paper describing a 'reactome array' promised a new method to examine metabolic networks and identify enzymatic substrates (Science 326, 252–257, 2009). After its publication, several blogs and news stories highlighted major concerns regarding the chemical reactions described in the paper (Science 327, 22–23, 2010); upon further review by the authors, the paper was subsequently retracted (Science 330, 912, 2010). A more recent paper reported a bacterium that could grow on arsenic and claimed that arsenates may substitute for phosphates within the DNA of this organism (Science doi:10.1126/science.1197258, 2010). Early press coverage of the story—fueled by a NASA press release—heralded a major breakthrough. Yet, since publication, the paper has been the focus of substantial criticism, voiced in the blogosphere and in the news. In both cases, many critics pointed to deficiencies in how the chemical aspects of these studies were both conducted and assessed; other examples indicate that deficiencies in validation of biological materials or datasets are equally likely to lead to faulty conclusions (Nature 464, 480–481, 2010).
Authors are ultimately responsible for what they publish, and therefore they need to exercise care in the design, execution and communication of their studies. Quality control starts with principal investigators, who must educate their students and postdoctoral associates on their field's standards and insist on these standards at every step of a project. This can be a challenge for authors working on interdisciplinary studies that often extend beyond a single laboratory's expertise. Yet it is incumbent upon investigators to recognize potential deficiencies and actively seek out collaborators or consult expert colleagues to ensure that all technical aspects of their work are fundamentally sound and that the interpretation of their data is valid.
These cases particularly highlight the need for careful molecular characterization of the chemical entities—from small molecules to biopolymers—used in scientific studies. Such vigilance is crucial for scientists working at interfaces, such as chemical biologists, materials scientists and nanotechnologists, where the experimental samples are increasingly complex but where numerous technologies are available for rigorous chemical characterization. Although most journals have molecular characterization guidelines that must be satisfied prior to publication (see, for example, http://www.nature.com/nchembio/authors/submit/index.html#ch), authors of all studies with chemical content should recognize that molecular characterization is absolutely essential, independent of journal requirements. Rigorous characterization at each stage of a project is not simply necessary for validating key results; it is useful too. It can prevent failed experiments and costly research delays by providing unambiguous insight into the molecules (or impurities) in experimental samples and simplify the writing of manuscripts.
The above-mentioned recent examples should also remind editors and referees of their roles in ensuring that manuscripts are technically sound before publication. Editors facilitate this process by enforcing journal and community standards and by selecting qualified referees with diverse expertise to ensure that all technical aspects of a manuscript are properly assessed (Nat. Chem. Biol. 6, 245, 2010). Referees must closely examine the paper's data—including the methods and supplementary information sections—and assess whether the authors' interpretations are supported by their data. A renewed commitment to high standards during peer review simply acknowledges that scientific projects can be complex and that a collaborative effort involving critical but balanced perspectives only enhances published work and prevents later concerns (Nat. Chem. Biol. 7, 1, 2011).
Though the coverage of recent controversial studies has centered primarily on author responsibilities and peer review, it also has challenged the traditional process of post-publication criticism of science. Debate through scientific correspondence is typically mediated by journal editors and involves authors responding to the written comments of a critic, followed by referees providing their views of whether the critical comment and response should be published. Many feel this process is the most scientific way of managing debates about technical or interpretive aspects of published work. However, the pace of formal correspondence is falling out of step with our increasingly digital age.
Scientists are already finding diverse outlets for expanding scientific conversations surrounding published papers, through news stories, blog posts and the commenting sections associated with these new media. In some communities, scientific criticism has found a voice in these places out of necessity, as some journals do not publish a scientific correspondence section at all. In parallel, online products such as Faculty of 1000 (http://www.f1000.com/), which bring together leading scientists to evaluate and comment on published studies, are widely viewed as useful technologies that lower the barriers to international discussion of important or controversial papers.
Recent cases have awakened scientists to the realities of contemporary media culture and simultaneously have encouraged the use of online technologies for scientific communication and discussion (Nature 468, 867, 2010). Nature and several other journals have introduced a functionality that allows registered users to post comments on the online versions of published research papers (Nature 464, 466, 2010). This commenting functionality, which links debates directly to the original study, may offer a more immediate outlet for scientific criticism that is complementary to traditional modes of correspondence. Wider adoption of these technologies offers an important first step in facilitating scientific criticism. Scientists should embrace these new avenues while maintaining the scholarly and thoughtful level of debate familiar to traditional scientific correspondence.
About this article
Nature Chemical Biology (2012)