When your research won't reproduce, publish!
Cognitive psychologist William S. Horton studies language at Northwestern University in Evanston, Illinois. But last October, he did something unusual — he co-authored a paper that had failed to replicate some of his earlier results. He explains that it was a tough decision, but has had a positive outcome.
What are your research interests?
As a graduate student, I worked on language use in conversations with a researcher who was investigating whether effective communication requires consideration of shared information. He found evidence that people are egocentric and initially give more weight to their own knowledge in conversations with others, and make language adjustments only later on the basis of feedback. As a postdoc, I had worked with someone who believed that we always keep track of 'common ground' or shared knowledge and are not egocentric initially. I developed a model that bridges both perspectives.
How did your research evolve?
I started to look at the role of memory in how people establish common ground in conversations. I showed that common ground need not be a conversation goal because other people can function as cues to retrieve relevant experiences from memory (1114–1139; 2007). Lang. Cogn. Process 22,
Was it controversial?
There wasn't a strong reaction one way or another. Sarah Brown-Schmidt, a psychologist at the University of Illinois at Urbana-Champaign, wanted to build on my memory theory in her research. She recreated the experiment, but it did not replicate my results. I gave her my materials so that she could try again. That experiment failed, too, and she asked me to be a co-author in a failure-to-replicate paper.
Were you worried about doing so?
The main con was putting my name on a publication that called my earlier work into question. On top of that, I was concerned about how I would talk about this result and what it would mean for my career. The decision would have been much harder had I not yet had tenure. The pros were that it was the right thing to do and that I would be able to help to put the finding into context. Sarah and I both had a sincere interest in making clear that although this study didn't replicate my results, the idea still has worth.
What did the failure-to-replicate study find?
We found no evidence that memories established in the context of other individuals helps in the recognition of shared information during subsequent interactions.
You have had a positive response to the publication. Was that surprising?
Yes. The journal I originally published in chose not to review it, so we went to PLoS ONE (e109035; 2014) which encourages the publication of negative results. The study got picked up on Twitter, Reddit and CBC radio. I was surprised that others found it so noteworthy. and PLoS ONE 9,
Do you think that more researchers should publish the findings of replication attempts?
There is an increasing effort, at least in psychology, to document replications in open-source databases, such as the Reproducibility Project. Some top-tier psychology journals have adopted pre-registration reporting, in which the methods and data-analysis plans are reviewed before replication is attempted, to smooth the review process. To what extent the original author is part of that process is pretty open.
Will you try again to validate your theory?
I still very much believe in it and have other results that support it, but I may look for new ways to address the same questions.
Where do you go from here?
I'm interested in seeing how this paper gets cited. I believe in the accumulation of findings. Not every result is going to hold up. That's just how science works.