Jon Roemer
Daniel Kahneman wants psychologists to spend more time replicating each others' work.
Nobel prize-winner Daniel Kahneman has issued a strongly worded call to one group of psychologists to restore the credibility of their field by creating a replication ring to check each others’ results.
Kahneman, a psychologist at Princeton University in New Jersey, addressed his open e-mail to researchers who work on social priming, the study of how subtle cues can unconsciously influence our thoughts or behaviour. For example, volunteers might walk more slowly down a corridor after seeing words related to old age1, or fare better in general-knowledge tests after writing down the attributes of a typical professor2.
Such tests are widely used in psychology, and Kahneman counts himself as a “general believer” in priming effects. But in his e-mail, seen by Nature, he writes that there is a “train wreck looming” for the field, due to a “storm of doubt” about the robustness of priming results.
Under fire
This scepticism has been fed by failed attempts to replicate classic priming studies, increasing concerns about replicability in psychology more broadly (see 'Bad Copy'), and the exposure of fraudulent social psychologists such as Diederik Stapel, Dirk Smeesters and Lawrence Sanna, who used priming techniques in their work.
“For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research,” Kahneman writes. “I believe that you should collectively do something about this mess.”
Kahneman’s chief concern is that graduate students who have conducted priming research may find it difficult to get jobs after being associated with a field that is being visibly questioned.
“Kahneman is a hard man to ignore. I suspect that everybody who got a message from him read it immediately,” says Brian Nosek, a social psychologist at the University of Virginia in Charlottesville.
David Funder, at the University of California, Riverside, and president-elect of the Society for Personality and Social Psychology, worries that the debate about priming has descended into angry defensiveness rather than a scientific discussion about data. “I think the e-mail hits exactly the right tone,” he says. “If this doesn’t work, I don’t know what will.”
Hal Pashler, a cognitive psychologist at the University of California, San Diego, says that several groups, including his own, have already tried to replicate well-known social-priming findings, but have not been able to reproduce any of the effects. “These are quite simple experiments and the replication attempts are well powered, so it is all very puzzling. The field needs to get to the bottom of this, and the quicker the better.”
Chain of replication
To address this problem, Kahneman recommends that established social psychologists set up a “daisy chain” of replications. Each lab would try to repeat a priming effect demonstrated by its neighbour, supervised by someone from the replicated lab. Both parties would record every detail of the methods, commit beforehand to publish the results, and make all data openly available.
Kahneman thinks that such collaborations are necessary because priming effects are subtle, and could be undermined by small experimental changes.
Norbert Schwarz, a social psychologist at the University of Michigan in Ann Arbor who received the e-mail, says that priming studies attract sceptical attention because their results are often surprising, not necessarily because they are scientifically flawed.. “There is no empirical evidence that work in this area is more or less replicable than work in other areas,” he says, although the “iconic status” of individual findings has distracted from a larger body of supportive evidence.
“You can think of this as psychology’s version of the climate-change debate,” says Schwarz. “The consensus of the vast majority of psychologists closely familiar with work in this area gets drowned out by claims of a few persistent priming sceptics.”
Still, Schwarz broadly supports Kahneman’s suggestion. “I will participate in such a daisy-chain if the field decides that it is something that should be implemented,” says Schwarz, but not if it is “merely directed at one single area of research”.
“I hope that this becomes part of a broader movement in psychology to be more self-critical, and to see if there are gaps in the way we do everyday science,” says Nosek. “I suspect those who are really committed to doing the best science possible will say that this or some alternative is a good idea.”
- Journal name:
- Nature
- DOI:
- doi:10.1038/nature.2012.11535
The field of psychologists is rather hard to validate, isn't it? Scientific researches and tests have to be validated by other universities and institutes. What Daniel Kahneman suggests seems to make very much sense indeed. Open letters usually have a good effect and I certainly hope this one will have a big impact on the psychologists community. If only a few institutes/universities adopt and stick to the daisy chain, it makes less sense, but if the majority adopts it, it should be a great step forward! I plan to publish the open letter for the German community to spread the word in hope that other will find this suggestion useful to adopt.
It's gene expression in gonadotropin releasing hormone neurosecretory cells of brain tissue that links sensory stimuli directly to changes in nutrient chemical-dependent species-specific social priming and behaviors. Ben LSW
Readers may be unaware that we have published numerous articles over the last decade that would suggest that to large degree equipment error might be responsible for many unstable findings across the field (Plant et al 2002 onwards).
This is not a new problem in computer-based studies and counter intuitively one which is actually getting worse year-on-year. We hope shortly to be summarising this decade worth of work in a CABN special issue paper which should be due out next month. In the meantime readers are strongly advised to visit our website for an overview of the issues: http://www.blackboxtoolkit.com.
We will be presenting at SCiP and Psychonomics in Minneapolis in the next few days if readers would like to talk to us about the issues in person.
I remember an article in Science or Nature saying that the Chinese government started to clamp down on the hundreds of bogus "stem cell clinics" that offer whatever therapies they can convince people to pay good cash for. Not too different from placenta supplements or Mamushi Powder.
Posted on behalf of Daniel Kahneman:
I write to complain about the irresponsible and damaging title that was affixed to Ed Yong’s piece on October 3. The headline asserts that Nobel laureates challenges … to clean up their act. There is no challenge in my letter, and certainly not a challenge for anyone to â€ৌlean up their act.” Instead, I offered friendly advice to colleagues whose work I respect, about an image problem they face and how they might deal with it. The misleading title outraged many of my friends, and probably caused real damage by making it harder for priming researchers to address my suggestion. I would not have expected misleading headlines from “Nature,” and hope you will be kind enough to publish this correction.
Regards,
Daniel Kahneman
From behavioral epigenetics it is now clearer that an environmental drive evolved from that of nutrient chemical ingestion in unicellular organisms to that of socialization in insects. The honeybee model organism exemplifies that fact. What the queen eats determines her pheromone production and everything else about the interactions in the colony. It is also clear that, in mammals, food odors and pheromones cause changes in hormones that have developmental affects on behavior in nutrient-dependent reproductively fit individuals.
Social scientists may want to include knowledge of the basic principles of biology and levels of biological organization in their studies of social priming. In all vertebrates, for example, the effects of social odors on hormones cause unconscious affects on the development of behavior. A sensory stimulus-> effect-> hormone-> affect approach could incorporate what is known about the requirement for gene, cell, tissue, organ, organ system reciprocity in adaptive evolution, and help to ensure that no missing steps in this critical path lead to results that cannot be replicated because the requirements for adaptive evolution of behavior via ecological, social, neurogenic, and socio-cognitive niche construction were not considered.
For example, social priming doesn't begin with a visual stimulus or a hormone like oxytocin in mammals. It begins with gene activation by a sensory stimulus that epigenetically effects intracellular signaling and stochastic gene expression. It's gene expression in gonadotropin releasing hormone neurosecretory cells of brain tissue that links sensory stimuli directly to changes in nutrient chemical-dependent species-specific social priming and behaviors — via downstream effects on other hormones in all vertebrates. Starting with something that affects behavior in individuals or groups from one species and expecting to find reproducible results at different developmental stages in another species virtually ensures missing the involvement of genetic predispositions and experience-dependent epigenetic effects.
Replication attempts are doomed to fail if only for the fact that each individual of each group tested has slightly different genetic predispositions and experiences that epigenetically effect hormones that affect behavior during development. No matter what measure of behavior is used, if a model for behavioral development is not used there's likely to be a problem with replication.