In 1977, two jumbo jets collided at an airport on the island of Tenerife. The highly regarded pilot of a KLM plane was under pressure to depart, and ignored several signs and warnings that a Pan Am jumbo had not yet cleared the runway. He took off in mist without proper clearance and 583 people were killed. The disaster happened because of a failure of the checks and balances put in place to deal with low visibility, arrogance, stress and the tendency to neglect conflicting information. These measures have since been improved, and commercial aviation is safer today not because it pointed the accusing finger at the erring pilot, but because it drew hard lessons.
Psychology, my field of science, recently uncovered a disastrous case of prolonged misconduct by Diederik Stapel, a highly regarded scientist at Tilburg University in the Netherlands (see Nature 479, 15; 2011). In response, we too must critically consider the circumstances under which the misconduct took place, with the aim of improving checks and balances to avoid a repeat.
The committee that investigated Stapel's misconduct has yet to identify the research papers tainted by his fraud, but it has already noted that the closed culture that characterizes much psychology research greatly aided Stapel's deceptions. One may argue that his misconduct is exceptional, regardless of research culture. However, the minor transgressions that all scientists are tempted to commit, as pointed out by Jennifer Crocker in a World View last month (see Nature 479, 151; 2011), are more likely when there is less scrutiny.
The interim report of the investigating committee revealed that Stapel often refused to share his research data with colleagues, even co-authors on papers. To scientists in other fields, this may seem extraordinary; to psychologists it is sadly common practice.
In a 2006 study published in American Psychologist, I helped to show that almost three-quarters of researchers who had published a paper in a high-impact psychology journal had not shared their data (J. M. Wicherts et al. Am. Psychol. 61, 726–728; 2006). Several data sets, authors said, had been misplaced, whereas others were kept secret because they were part of ongoing work, or because of ethical rules meant to protect participants' privacy. Such confidentiality has long been the most common excuse that psychologists offer for not sharing data, but in practice, most simply fail to document their data in a way that allows others to quickly and easily check their work. It is not unusual for data that are shared to list variables only as VAR00001 through VAR00019, with no further explanation.
It is not just misconduct that flourishes in such secrecy. So too do the common and more insidious failings of error and bias in data analysis — for example, the use of incorrect tests, reporting errors due to similarly named variables, favouring results that confirm a hypothesis and overly positive reporting of statistical outcomes.
“To scientists in other fields, not sharing data may seem extraordinary; to psychologists it is sadly common practice.”
It is striking that psychology researchers go to great lengths to blind their data collection to possible confounding effects, including the expectations of participants, observers and experimenters, but seem oblivious to problems in the subsequent analysis and reporting. In aviation, a co-pilot checks the pilot's every step, and actions are recorded in a 'black box' to reconstruct any errors. In psychology, co-authors rarely verify a study's analysis, which is effectively conducted inside a box. Readers of published papers are shown dense summaries of results — without proper data archiving, they can only hope for the best.
Psychology's culture of secrecy produces substandard science. Reanalyses of statistics in published psychology papers show frequent errors, and the more reluctant authors are to share their data, the more likely it is that their papers will contain mistakes. Or to put it another way — the results that most need checking cannot be checked.
How to lift the veil of secrecy? Obligatory archiving of raw data in online appendices to journal articles or in repositories should be a precondition for publication. This would not only help to uncover misconduct — curious patterns in Stapel's data led to his downfall — but would also help to prevent and later correct honest mistakes and unduly positive reporting.
With online publishing, data can often be published alongside the researchers' chosen statistical analyses and their summary of the results. As part of the growing concern over scientific openness, grant-giving organizations, academic publishers and professional organizations including the American Psychological Organization are already considering such options. Yes, there are practical problems, including the need to keep data or participants confidential, but these can be solved by embargoes on releasing data for longitudinal studies, guidelines for preprocessing raw data, proper anonymity and exemptions where necessary.
As a first step towards this new way of publishing results, my close colleagues and I have implemented a 'co-pilot' model for our statistical analyses, in which we share data between us for double-checking and preventing embarrassing errors. And perhaps the wider mood is changing. Last week, a colleague, who had previously been reluctant to share, sent me the data from his latest work without me asking. And the analyses proved excellent.
- Journal name:
- Date published: