To the Editor — We appreciate the Social Sciences Replication Project (SSRP) team’s work on these replications1. Naturally, we were disappointed to learn that our study 1 (ref. 2) did not replicate. Nevertheless, this is part of science and how it moves forward.
Our paper was motivated by a question about why individuals in conditions of scarcity engage in certain financial behaviours, such as excessive borrowing. Previous explanations suggested these behaviours stemmed from the personality traits of the poor or structural barriers they face. We tested a different explanation — that resource scarcity itself can lead to these behaviours. We suggested that various forms of resource scarcity would have similar effects, and that a scarcity mindset would lead to attentional shifts that might drive behaviours such as over-borrowing.
When the SSRP team contacted us, we welcomed the opportunity to have an independent replication of our study. We invited them to replicate all studies in the paper, but this was beyond the scope of their efforts. Their decision to replicate study 1 from each paper, they explained, was that “the first experiment typically provide[s] the first evidence of an hypothesized effect and the robustness of this effect is then typically demonstrated in the additional experiments.” In our paper, study 2 filled that role. While study 1 tested a peripheral hypothesis about cognitive fatigue, study 2 was the first to test the central hypothesis that scarcity leads to over-borrowing and all subsequent studies replicated that effect.
The replication efforts made us want to revisit all of our results. We therefore conducted pre-registered replications of all five studies3. We invite other research teams to independently replicate these studies.
We replicated three key results in this replication3. (1) Scarcity itself leads to over-borrowing (the motivating hypothesis for the paper). (2) This is true for multiple kinds of resources. (3) Scarcity leads to greater focus. But we found weaker evidence for the hypothesis that scarcity-induced focus leads to neglect.
Finally, we found no evidence that scarcity-induced focus leads to cognitive fatigue on subsequent tasks (study 1). Based on the SSRP’s findings1 and our own3, we believe that the original result2 was a false positive.
Replication efforts that focus on a cross-section of studies provide useful overviews of the literature. However, they do not permit a deeper dive into individual research projects. Furthermore, the criteria used to select studies can overlook the most central hypotheses in a paper, as was the case with ours. Ultimately, to build a more reproducible social science, we need to understand which hypotheses and theories are robust.
The SSRP highlights the need to replicate studies before publication. We replicated the most central findings, but were less vigilant about the introductory study. Today, we would have attempted to replicate that study as well — especially as it tests a hypothesis that was not central to our paper. Greater awareness and care of the kind raised by the SSRP should help more carefully balance central and peripheral claims in ways that increase publications’ reliability.
Camerer, C. F. et al. Nat. Hum. Behav. https://doi.org/10.1038/s41562-018-0399-z (2018).
Shah, A. K., Mullainathan, S. & Shafir, E. Science 338, 682–685 (2012).
Shah, A. K., Mullainathan, S. & Shafir, E. An exercise in self-replication: Replicating, Shah, Mullainathan, and Shafir (2012). Working paper. https://osf.io/7z38r/ (2018).
The authors declare no competing interests.
About this article
Cite this article
Shah, A.K., Mullainathan, S. & Shafir, E. An opportunity for self-replication. Nat Hum Behav 2, 603 (2018). https://doi.org/10.1038/s41562-018-0405-5
This article is cited by
Nature Human Behaviour (2022)