Studies that use a homogeneous pool of subjects, particularly undergraduates, must be interpreted with caution.
University students are an easy source of research subjects; they are abundant on most campuses, are less likely to have day jobs (giving them more free time and making them more willing to work for low pay) and many are even required to participate in experiments for course credit. It is easy to see why they have come to be the predominant study population for many psychology and neuroscience researchers. However, a recent survey1 of the psychology and behavioral economics literature suggests that American college students are outliers, quite atypical of the world population. This finding raises serious questions about how broadly we can interpret conclusions based on such a narrow and nonrepresentative sample of the population. Critically, it suggests that many experimental results that purport to support conclusions about all of humanity may be no more (or less) generally relevant than those based on data from any other model organism and that researchers need to do more to try and diversify their subject pool.
In their meta-analysis of the literature, Henrich and colleagues1 point out several astonishing examples of cross-cultural inconsistency. For example, the Mueller-Lyer illusion (in which the perceived length of a line segment is influenced by the shape of the terminators at each end) is a classic example of perceptual distortion that is used by many scientists to study fundamental properties of the visual system. American children and adults report a robust distortion. However, the size of the effect varies substantially among populations, with smaller effects in South African Europeans and nonexistent effects in the San (a group of foragers in the Kalahari desert).
Perhaps less shocking, but equally relevant to neuroscientists, is the diversity in responses in behavioral economics tasks. The ultimatum game is a widely used procedure in which one partner decides how much of an endowment to share and the other partner decides whether to accept the split or reject it (in which case no one gets any money). Many variations on the basic task have been tested to try and elucidate the factors that influence choices about offers and whether to accept them. Cross-cultural studies suggest that offer sizes and rejection rates vary widely between industrialized and small-scale societies. Moreover, even in American students, these patterns don't converge to adult levels until participants reach their mid-twenties (well after many students have graduated college). In a related procedure, in which subjects could punish collaborators based on the amount they contributed to a common good, participants from Western cultures were slightly more likely to exact punishment on those who didn't contribute, leading to greater cooperation. Participants from non-Western nations (including Korea, Turkey and Saudia Arabia) had an entirely different strategy; they punished those who contributed more than they did, leading to less cooperation. Taken together, these studies suggest that decision patterns in behavioral economics procedures are unlikely to generalize widely across groups of subjects.
Cognitive psychology is also vulnerable to the criticism that 'common effects' are really unique to the typical study population. For example, American college students (but not non–college educated American adults) are known to rationalize their choices after they have been made. Furthermore, although IQ is thought to be highly heritable and less susceptible to environmental influence, this appears to be true only for children of high socioeconomic status (and relatively low environmental variability). One might argue that the neural mechanisms underlying even a wide range of behaviors will be the same, regardless of subject variability. Studies that emphasize correlations between individual differences in performance and brain network activity would seem to suggest that generalization is possible across some range of behavioral variation. However, without measurements from a wider range of subjects, it is impossible to know whether or not this is the case.
The way to address these questions is to start including cross-cultural studies in cognitive experiments. However, there are substantial hurdles to overcome. Physical and financial constraints make it unlikely that scientists will be able to lug their machines to the Kalahari Desert any time soon. Even for purely behavioral work, assembling a research team with the appropriate skills and finding study populations can be daunting. Identifying appropriate tools to assess behavior may also be tricky. For example, Noam Sobel, an investigator at the Weizmann Institute of Science, found that native Israeli participants intuitively grasped the use of a visual-analog scale for rating odor pleasantness, whereas participants recently emigrated from rural Ethiopia found it completely non-intuitive.
Fundamentally, the only way to encourage true cross-cultural studies will be for funding bodies to recognize the importance of this work and to provide the resources to make it economically viable. However, there are other ways scientists can help to circumvent this problem. Researchers need to put more effort into diversifying their subject pools.
Most universities are in areas that contain individuals of diverse nationality, age and socioeconomic status who can and should be tapped for experiments. Authors should also be aware of the limitations imposed by the selection of subjects as they prepare their manuscripts, while editors and reviewers should encourage explicit mention of the characteristics of the study participants in the methods and limit speculation about what broader meaning can be inferred from the population sample.
Research with human populations has an important place in neuroscience. However, just as genetics studies typically require replication in multiple populations to avoid false positives from population stratification, behavioral neuroscientists should also consider the implications of using a small and homogeneous subject pool. Diversifying the study population might even be a tremendous opportunity to understand the truly universal principles of neuroscience.
Henrich, J., Heine, S.J. & Norenzayan, A. The weirdest people in the world? Behav. Brain Sci. in press. (http://journals.cambridge.org/images/fileUpload/documents/Henrich-BBS-D-09-00017R2_preprint.pdf)
About this article
Magyar Pszichológiai Szemle (2018)
Experimental Brain Research (2015)
Nicotine Effects on Brain Function during a Visual Oddball Task: A Comparison between Conventional and EEG-informed fMRI Analysis
Journal of Cognitive Neuroscience (2012)