To the Editor — In an experiment run on Amazon Mechanical Turk (MTurk), Camerer et al.1 failed to replicate the finding in study 8 of Rand et al.2 that promoting intuition via a recall induction increased cooperation. Before Camerer et al.1 began collecting data to replicate our study2, I cited prior evidence2,3 that experienced participants would not exhibit the original effect, and that many MTurkers are now experienced. At my request, Camerer et al.1 collected data on prior experience. As described in more detail below, examining these data shows that a large majority of their participants were experienced — and that analysing only inexperienced participants yields an effect size comparable to the original effect.

MTurk participants have become much more experienced with economic games since 2010, when our study 8 was run (this was before MTurk became widely used by academics). My colleagues and I have argued that experienced participants are typically more aware of — and attentive to — the strategic details of the game they are playing2,3. This attentiveness may enhance treatment effects based on variation in experimental details (like some MTurk experiments4,5 successfully replicated by Camerer et al.1) — but has been theorized to reduce the application of daily-life intuitions to atypical (one-shot anonymous) lab experiments2,3.

Accordingly, we showed that such experience undermines the intuition–cooperation effect in study 9 of our original paper2, which used a similar procedure to study 8 but was run nearly two years later. Study 9 found a non-significant overall effect of promoting intuition, but a significant interaction with experience: intuition increased cooperation only among participants who lacked prior experience with economic games. Subsequent work3 provided further evidence, showing that the time-pressure effect on cooperation decreased over two years as MTurk participants became increasingly experienced. It is therefore not surprising — and actually consistent with past results — that Camerer et al.1 failed to replicate study 8’s results using the original methods.

Before their data collection, I informed Camerer et al.1 that “[I]t’s been well documented that MTurkers are now highly experienced with economic game paradigms”, and “Study 9 of our paper shows that non-naïvety specifically undermines the treatment effect you are replicating.” I therefore asked that they include our standard question assessing prior experience with economic games, enabling them to compare “people that answered 1 [naïve] to everyone else [non-naïve]”. Camerer et al.1 collected these data, but did not analyse them.

These data show that 82.8% of the participants in Camerer et al.1 had prior experience with economic games. When analysing only the 367 naïve participants, the effect size was similar to our original study: Tobit regression coefficient on a ‘promote intuition’ dummy of b = 9.37, compared with b = 10.95 in the original study. (The naïve-only effect does not reach statistical significance, z = 1.61, P = 0.054 one-tailed given clear directional hypothesis; however, when including only naïve participants, the analysis is under-powered to detect an effect of the anticipated (and observed) size.)

Importantly, I am not arguing that it is now impossible to study intuition and cooperation on MTurk. It just requires more powerful manipulations. For example, Levine et al.6 conceptually replicate the present results by directly instructing participants to decide using emotion versus reason (rather than our more indirect recall induction); and Everett et al.7 and Isler et al.8 reproduce our original time-constraint results using improved methods that avoid non-compliance. In sum, the weight of the evidence confirms our initial conclusion that intuition promotes cooperation in social dilemmas, but reliably reproducing these effects requires updated methods.