Social hierarchy is an organizing principle of social life: when resources are limited, living in large social groups inevitably invites competition. To reduce competition’s costs, dominance hierarchies emerge, where individuals of lower ranks submit to individuals of higher ranks to avoid injury, be it physical, financial, or social. Navigating these dominance hierarchies is key to successful social interaction.

The neurotransmitter serotonin (5-HT) is important in establishing dominance hierarchies in various species. For example, direct modulation of 5-HT signaling in monkeys changes their dominance behavior [1]. It is unclear, however, whether serotonin directly relates to dominance in humans. In this issue of Neuropsychopharmacology, Janet et al. [2] sought to answer this question by combining computational modeling of a dominance learning task with the simultaneous measurement of 5-HT transporter (SERT) availability, which indexes serotonin activity (with positron emission tomography), and blood oxygenation, which indexes neuronal activity (with functional magnetic resonance imaging; fMRI).

Participants played a competitive decision-making game with three opponents. On each trial, the participants chose one opponent to compete against. The participants were told that the opponents were real people, and that playing ability would determine wins and losses. In reality, the opponents were fictional with fixed probabilities of winning. One opponent was likely to win, another was likely to lose, and the third was in the middle, at 50% chance. There weren’t any indicators of those win probabilities; dominance was latent and could only be inferred from the outcomes. A reinforcement learning algorithm computed the opponents’ dominance ranks as the expected value of competing against them, learned from the win/loss history. High probability of losing, for example, meant there was a low expected value of competition, resulting in a high dominance rank for that opponent. Participants quickly learned to compete against the easier-to-defeat opponents.

Janet et al. analyzed SERT levels in the dorsal raphe nucleus (DRN), which provides most of the serotonin to the rest of the brain. They also measured fMRI activity in the ventral striatum and the anterior medial prefrontal cortex (mPFC), two regions that receive direct serotonergic innervation from the DRN. The analysis revealed that interactions between the DRN, striatum, and mPFC tracked the opponents’ social dominance ranks. Activity in the striatum and mPFC correlated with two parameters from the computational model of dominance learning: the expected value of competing against the opponents, and the prediction errors from surprising outcomes (e.g., a win when a loss was more probable).

The results further showed that greater serotonergic activity (indicated by lower SERT availability in the DRN) corresponded to greater dominance tracking in the striatum. Serotonin may drive the learning rate in this reinforcement learning problem. When the learning rate is small, new outcomes weigh less when updating expected values; a large learning rate means expectations can change quickly with new experience. The participants’ competitiveness correlated with greater serotonergic activity: being competitive may mean having a short memory.

Janet et al.’s findings have several important implications. By modeling the learning mechanisms underlying the formation of social dominance hierarchies, they provide a firm basis for further theorizing. They estimate social dominance ranks with a model-free reinforcement learning algorithm, where the value of competition is learned directly from wins and losses against the opponents. This is computationally simple, but inflexible. If the ranks change, as they often do in real-life, then the reward values must be fully (and slowly) re-learned. Future research could test for more adaptive representations, such as model-based learning or the successor representation, approaches with more flexibility than model-free algorithms (albeit at the cost of more computation). Think of map-like representations that track predictive relationships between the different locations. One possibility is that the hippocampus encodes such maps of social hierarchy. It is densely innervated with serotonin, it is important to social discrimination and hierarchy learning, and it may interact with striatum-encoded reward signals to guide social bonding [3].

Janet et al. highlight dominance as a main social dimension. Future research could consider other dimensions in a multidimensional social space. Being dominant is valuable, but so is having a coalition [4]. Affiliative behaviors help manage conflict and interact with dominance to drive social outcomes. Dominance and affiliation are both under serotonin’s influence and may be represented together in the hippocampus as a two-dimensional map [3].

Finally, Janel et al. show that social value can be estimated through explicit feedback, but social outcomes are often ambiguous. In such cases, the individual’s prior expectations may dominate beliefs. For example, people with social anxiety disorders avoid social interactions out of fear of negative outcomes, especially in ambiguous settings [5]. Abnormalities in the DRN, striatum, mPFC, and hippocampus all seem to be involved. Also, serotonin-increasing drugs, and even psychedelics, reduce social anxiety, perhaps in part by changing social representations (e.g., flattening hierarchy) and improving social connectedness. The competitive outcomes in Janet et al.’s task could be made more ambiguous and combined with serotonergic drugs to test such questions.

Of note, Janet et al. examined only young adult males who currently attend university, which limits the study of individual differences. In many species there are different male and female hierarchies. For example, while male chimpanzees aggressively compete to climb the dominance hierarchy, female chimpanzees organize their hierarchy by seniority. Other variables, including age, socioeconomic status, and culture, may also be important.

Social context appears to be unique in Janet et al.’s study, as they did not find similar effects in a closely matched non-social task. This prompts a familiar question: what makes “social” special? One possibility, suggested by the authors, is that social hierarchies are less certain than non-social hierarchies. This unpredictability could be encoded by serotonin and may affect the individuals’ perceived control over their environment.

People are complex: we’re goal-directed, multi-dimensional, nonlinear, and dynamic. We are also connected to and influenced by each other. Predicting others’ behaviors to better decide our own is a central challenge in social living—and a main reason to learn social hierarchies in the first place.