All animals possess a repertoire of innate (or instinctive1,2) behaviours, which can be performed without training. Whether such behaviours are mediated by anatomically distinct and/or genetically specified neural pathways remains unknown3,4,5. Here we report that neural representations within the mouse hypothalamus, that underlie innate social behaviours, are shaped by social experience. Oestrogen receptor 1-expressing (Esr1+) neurons in the ventrolateral subdivision of the ventromedial hypothalamus (VMHvl) control mating and fighting in rodents6,7,8. We used microendoscopy9 to image Esr1+ neuronal activity in the VMHvl of male mice engaged in these social behaviours. In sexually and socially experienced adult males, divergent and characteristic neural ensembles represented male versus female conspecifics. However, in inexperienced adult males, male and female intruders activated overlapping neuronal populations. Sex-specific neuronal ensembles gradually separated as the mice acquired social and sexual experience. In mice permitted to investigate but not to mount or attack conspecifics, ensemble divergence did not occur. However, 30 minutes of sexual experience with a female was sufficient to promote the separation of male and female ensembles and to induce an attack response 24 h later. These observations uncover an unexpected social experience-dependent component to the formation of hypothalamic neural assemblies controlling innate social behaviours. More generally, they reveal plasticity and dynamic coding in an evolutionarily ancient deep subcortical structure that is traditionally viewed as a ‘hard-wired’ system.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Tinbergen, N. The Study of Instinct (Clarendon Press, 1951)
Lorenz, K. On Aggression (Harcourt, Brace & World, 1966)
Root, C. M., Denny, C. A., Hen, R. & Axel, R. The participation of cortical amygdala in innate, odour-driven behaviour. Nature 515, 269–273 (2014)
Newman, S. W. The medial extended amygdala in male reproductive behavior. A node in the mammalian social behavior network. Ann. NY Acad. Sci. 877, 242–257 (1999)
Veening, J. G. et al. Do similar neural systems subserve aggressive and sexual behaviour in male rats? Insights from c-Fos and pharmacological studies. Eur. J. Pharmacol. 526, 226–239 (2005)
Yang, C. F. et al. Sexually dimorphic neurons in the ventromedial hypothalamus govern mating in both sexes and aggression in males. Cell 153, 896–909 (2013)
Lee, H. et al. Scalable control of mounting and attack by Esr1+ neurons in the ventromedial hypothalamus. Nature 509, 627–632 (2014)
Sano, K., Tsuda, M. C., Musatov, S., Sakamoto, T. & Ogawa, S. Differential effects of site-specific knockdown of estrogen receptor α in the medial amygdala, medial pre-optic area, and ventromedial nucleus of the hypothalamus on sexual and aggressive behavior of male mice. Eur. J. Neurosci. 37, 1308–1319 (2013)
Ghosh, K. K. et al. Miniaturized integration of a fluorescence microscope. Nat. Methods 8, 871–878 (2011)
Ziv, Y. et al. Long-term dynamics of CA1 hippocampal place codes. Nat. Neurosci. 16, 264–266 (2013)
Chen, T. W. et al. Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499, 295–300 (2013)
Thurmond, J. B. Technique for producing and measuring territorial aggression using laboratory mice. Physiol. Behav. 14, 879–881 (1975)
Geladi, P. & Kowalski, B. R. Partial least-squares regression: a tutorial. Anal. Chim. Acta 185, 1–17 (1986)
Lin, D. et al. Functional identification of an aggression locus in the mouse hypothalamus. Nature 470, 221–226 (2011)
Falkner, A. L., Dollar, P., Perona, P., Anderson, D. J. & Lin, D. Decoding ventromedial hypothalamic neural activity during male mouse aggression. J. Neurosci. 34, 5971–5984 (2014)
Falkner, A. L., Grosenick, L., Davidson, T. J., Deisseroth, K. & Lin, D. Hypothalamic control of male aggression-seeking behavior. Nat. Neurosci. 19, 596–604 (2016)
Shadlen, M. N., Britten, K. H., Newsome, W. T. & Movshon, J. A. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16, 1486–1510 (1996)
Jennings, J. H. et al. Visualizing hypothalamic network dynamics for appetitive and consummatory behaviors. Cell 160, 516–527 (2015)
Grewe, B. F. et al. Neural ensemble dynamics underlying a long-term associative memory. Nature 543, 670–675 (2017)
Efron, B., Hastie, T., Johnstone, I. & Tibshirani, R. Least angle regression. Ann. Statist. 32, 407–499 (2004)
Ogawa, S. et al. Survival of reproductive behaviors in estrogen receptor β gene-deficient (βERKO) male and female mice. Proc. Natl Acad. Sci. USA 96, 12887–12892 (1999)
Gobrogge, K. L., Liu, Y., Jia, X. & Wang, Z. Anterior hypothalamic neural activation and neurochemical associations with aggression in pair-bonded male prairie voles. J. Comp. Neurol. 502, 1109–1122 (2007)
Redondo, R. L. et al. Bidirectional switch of the valence associated with a hippocampal contextual memory engram. Nature 513, 426–430 (2014)
Xu, P. S., Lee, D. & Holy, T. E. Experience-dependent plasticity drives individual differences in pheromone-sensing neurons. Neuron 91, 878–892 (2016)
Stowers, L. & Liberles, S. D. State-dependent responses to sex pheromones in mouse. Curr. Opin. Neurobiol. 38, 74–79 (2016)
Gur, R., Tendler, A. & Wagner, S. Long-term social recognition memory is mediated by oxytocin-dependent synaptic plasticity in the medial amygdala. Biol. Psychiatry 76, 377–386 (2014)
Cooke, B. M. Steroid-dependent plasticity in the medial amygdala. Neuroscience 138, 997–1005 (2006)
Knierim, J. J. & Zhang, K. Attractor dynamics of spatially correlated neural activity in the limbic system. Annu. Rev. Neurosci. 35, 267–285 (2012)
Burak, Y. Spatial coding and attractor dynamics of grid cells in the entorhinal cortex. Curr. Opin. Neurobiol. 25, 169–175 (2014)
Moser, E. I., Kropff, E. & Moser, M.-B. Place cells, grid cells, and the brain’s spatial representation system. Annu. Rev. Neurosci. 31, 69–89 (2008)
Ogawa, S. et al. Abolition of male sexual behaviors in mice lacking estrogen receptors α and β (αβERKO). Proc. Natl Acad. Sci. USA 97, 14737–14741 (2000)
Dana, H. et al. Thy1–GCaMP6 transgenic mice for neuronal population imaging in vivo. PLoS ONE 9, e108697 (2014)
Resendez, S. L. et al. Visualization of cortical, subcortical and deep brain neural circuit dynamics during naturalistic mammalian behavior with head-mounted microscopes and chronically implanted lenses. Nat. Protocols 11, 566–597 (2016)
Dollár, P., Welinder, P . & Perona, P. Cascaded pose regression. In IEEE Conference on Computer Vision and Pattern Recognition 1078–1085 (CVPR, 2010)
Mukamel, E. A., Nimmerjahn, A. & Schnitzer, M. J. Automated analysis of cellular signals from large-scale calcium imaging data. Neuron 63, 747–760 (2009)
Machens, C. K., Romo, R. & Brody, C. D. Functional, but not anatomical, separation of “what” and “when” in prefrontal cortex. J. Neurosci. 30, 350–360 (2010)
Jolliffe, I. Principal Component Analysis (Wiley Online Library, 2002)
De Jong, S. SIMPLS: an alternative approach to partial least squares regression. Chemom. Intell. Lab. Syst. 18, 251–263 (1993)
Cristianini, N & Shawe-Taylor, J. An Introduction to Support Vector Machines and Other Kernel-based Learning Methods (Cambridge Univ. Press, 2000)
We thank X. Wang, J. S. Chang and R. Robertson for technical help, H. Lee and P. Kunwar for experimental advice, D. Senyuz for testing behaviour in wild-type mice, D.-W. Kim for pilot experiments, M. McCardle and C. Chiu for genotyping, J.Costanza for mouse colony management, G. Stuber for advice on GCaMP6s expression, Inscopix Inc. for technical support, P. Perona for mouse tracking software, L. Abbott for comments on the manuscript, R. Axel, D. Y. Tsao and M. Meister for critical feedback, X. Da and C. Chiu for laboratory management and G. Mancuso for Administrative Assistance. D.J.A. and M.J.S. are Investigators of the Howard Hughes Medical Institute and Paul G. Allen Distinguished Investigators. This work was supported in part by NIH grant no. R01MH070053, and grants from the Gordon Moore Foundation, Ellison Medical Research Foundation, Simons Foundation and Guggenheim Foundation to D.J.A. A.K. is a fellow of the Helen Hay Whitney Foundation, M.Z. is a recipient of fellowships from the NSF and L’Oréal USA Women in Science.
M.J.S. is a scientific co-founder of Inscopix Inc., which produced the miniature fluorescence microscope used in this study. R.R., A.K., M.Z., B.F.G. and D.J.A. declare no competing financial interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
Extended Data Figure 1 Properties of Esr1+ neuron responses in VMHvl during free social interactions.
Data presented are from the third day of imaging. a, Histogram of time-averaged ΔF(t)/F0 values observed in the 30 s following stimulus introduction (or the first 30 s of imaging, on baseline trials). Dashed lines indicate thresholds for identifying excited (≥2 s.d. (2σ) of baseline above the mean pre-intruder baseline) and inhibited (≤−2σ below pre-intruder baseline) cells. (n = 5 imaged mice, 5 male trials, 5 female trials, 2 toy trials, 2 baseline trials per mouse.) b, c, Average percentage of time cells were excited (b) or inhibited (c) in first 30 s of imaging. d, Example traces from single Esr1+ cells, showing example response profiles during a single trial interaction with a female conspecific. While some cells showed transient calcium responses (N1, N2), others showed slower dynamics (N3–N5), including persistent elevation or suppression of activity for the duration of the encounter with the female. We believe the latter responses reflect the ongoing activity of some VMHvl cells, as filtered by the slow dynamics of GCaMP6s. e, Autocorrelation functions computed across all day 3 trials, for each example cell in d; red line indicates half-width. f, Autocorrelation half-width computed across all day 3 trials, for all cells from all mice (n = 1,435 cells in five mice), sorted from smallest to largest, revealing a continuous range of values. Autocorrelation half-width of the five cells in d are indicated. g, Example spatial maps of excited (left; mean ΔF(t)/F0 ≥ 2σ during periods of social interaction) and inhibited (right; mean ΔF(t)/F0 ≤ −2σ during interaction across all male and female trials on the third day of imaging) cells in the presence of male and female conspecifics. h, The percentage of cells across all mice (n = 5) that were excited and/or inhibited by both males and females, compared to the percentage expected by chance assuming statistical independence. (both inhibited, P = 2.17 × 10−4; excited by female + inhibited by male, P = 0.0299).
a, Day 3 ensemble representations of intruder sex for the five mice that mounted and fought with conspecifics, projected onto the first two axes of a PLS regression against intruder sex. Traces are coloured by intruder sex identity. Percentage of variance explained by the first two PLS components is noted for each mouse. b, Projection of a single trial onto the first two principal axes of one example mouse (mouse 2), colour-coded to highlight temporal trajectory of the population activity vector during the trial. Principal axes were identified using PCA (same axes as in Fig. 1k). c, All trials from mouse 2, coloured by trial number.
a, Example video frame showing estimates of resident and intruder poses and centroids (red bounding boxes and points, respectively) produced by an automated tracker34. Green, grey, and purple bars mark the inter-mouse distances (relative to the resident) categorized as close, intermediate and far. b, Percentage of time animals spent either interacting with conspecifics or present within the three zones. Time points in which animals have recently interacted (that is, interaction has occured within ±1 s) are shown as their own category as a control for the effect of slow GCaMP dynamics. c, PCC between trials of the same intruder sex, or between male and female trials, where representations of intruders are computed by averaging each cell’s ΔF(t)/F0 across the indicated subsets of imaging time points. d, Accuracy of a linear SVM decoder for intruder sex, using representations computed as in c. Decoder accuracy was not significantly affected by distance. e, Top, the distance modulation index (m), a measurement of the extent to which cell activity is modulated by inter-animal distance, is computed from rclose (the average response of a cell when animals are close but have not recently interacted (within ±1 s)), and rfar (the average response of that cell when animals are far apart and have not recently interacted). e, Bottom, example ΔF/F traces (black) from two cells that have a high (e1, m = 0.89) and low (e2, m = 0.08) distance modulation index, and corresponding inter-animal distances (brown). f, Histograms of values of m observed in all cells that are significantly excited (above) or inhibited (below) during interaction with males or females. Note that inhibited cells are less sensitive to inter-animal distance than are excited cells, and that the distribution of m is similar for cells that are responsive to male and female intruders.
a, Steps for computing CP, a measurement of the discriminability of two conditions, given the ΔF(t)/F0 of a single cell. Given a pair of behaviours (here, male-directed attack and sniffing), we computed the distribution of ΔF(t)/F0 values for this cell for each behaviour, and then integrated each distribution to produce a pair of cumulative distribution functions (cdfs). The area under the ROC curve formed from these two distributions is defined as the cell’s CP; CP of this example cell was 0.69. Right-most panel shows CP for all cells (arrow indicates CP of the example cell); outlined bars denote cells for which CP was significantly higher than chance. b, ∆F/F traces and corresponding behaviour rasters from example cells that showed significant CPs for the specified behaviour, illustrating the relative changes in ∆F/F and behaviour. Three representative trials are shown (same examples as in Fig. 2 d–g). c, Proportion of cells significantly tuned for each examined behaviour. Overlap between blocks reflects the proportion of cells that were significantly tuned for both behaviours. Note that by this metric, cells cannot be tuned for both attack and male-directed sniffing, nor for both mounting and female-directed sniffing, because the CP of a cell is defined by comparing ΔF(t)/F0 values under these paired conditions. All cells significantly tuned for attack, mount or sniff also showed a significant CP for periods of close social interactions (including sniffing and/or mount or attack) versus periods of non-social interactions, by construction. However, some cells that showed a significant CP for close social interactions versus non-social periods did not show a significant CP for a specific behaviour (see Fig. 2h). d, Separate precision (true positives / (true positives + false positives)) and recall (true positives / (true positives + false negatives)) scores for the behaviour decoders presented in Fig. 2b, c. The F1 score (presented in Fig. 2c) is defined as 2(precision × recall) / (precision + recall).
a, Percentage of time mice spent engaging in male- and female-directed behaviours on each of five trials on three days of imaging. n = 5 mice, mean ± s.e.m. b, Cumulative time spent mounting and attacking by a cohort of unoperated, socially isolated mice that underwent two days of the standard social experience paradigm depicted in Fig. 1d (n = 8 mice). The unoperated mice exhibited a gradual appearance of mounting, and showed mounting behaviours before attack, indicating that changes in behaviour were not due to the effects of surgery or the presence of the scope. c, Data from implanted mice (n = 5), reproduced from Fig. 3g and restricted to the first 20 trials for direct comparison.
Extended Data Figure 6 Comparison of the social behaviours and neural representations in example mice that showed and did not show aggression.
a–d, Behaviour rasters (a, c) and the corresponding first two PLS components (b, d) for example mice that showed (a, b) and did not (c, d) show aggression to conspecific males over the course of the experiment.
a, Left, maximum projection maps of spatial filters of all cells from each of three days of imaging for an example mouse. Right, RGB-composite image of the three left images (day 1 = red channel, day 2 = green channel, day 3 = blue channel), showing overlap of registered filters (overlap in all three channels appears in white). Outlined are 20 example cells (out of 135) that could be identified in all three days of imaging: red/green/blue lines indicate filter outlines on first/second/third day, respectively; black points mark filter centroids. b, Histograms of average distance between filter centroids from days 1–3, in cells that could be tracked across days (red) as compared to random triplets of cells (black). Day 1–3 centroids from tracked cells were separated by an average of 2.15 ± 0.06 μm (mean ± s.e.m., n = 593 cells tracked across days in six mice, during the standard resident–intruder assay).
Extended Data Figure 8 Preference changes of Esr1+ neurons during the acquisition of social experience.
a, Responses on day 1 trial 1 are compared to active cells on day 3, for all mice and all cells that were registered across three days of imaging (n = 455 cells analysed in the five mice that showed attack/mounting by day 3 of the standard resident–intruder assay). b, Top, response properties of cells on day 3, conditioned on their responses on day 1 trial 1, grouped according to the response types on trial 1. For example, among cells that responded to both males and females on day 1 trial 1, around 26% (20 out of 78) specifically responded to males on day 3, around 26% responded specifically to females; additionally, around 35% responded to neither sex and around 13% responded to both. The percentages of these categories are summarized in c. Bottom, response properties of cells on day 1 trial 1 conditioned on their response properties on day 3, grouped according to response types on day 3. The percentages in different categories are summarized in c and d. For example, of the cells that showed male-specific responses on day 3, around 54% is derived from cells that responded to neither sex on day 1 trial 1; 20% is derived from cells that responded to both sexes, 10% is derived from initially female-specific cells and 16% is derived from initially male-specific cells. Numbers used to calculate the percentages are from a. e, Analysis of the day 1 preferences of cells that responded only to males or only to females on day 3.
a, Cumulative minutes of each behaviour by the nth trial plotted against the PCC between male and female representations on that trial, for six mice (behaviours already presented in Fig. 4a–c have been omitted). All trials from a given mouse are shown in the same colour. Solid lines are square-root fits (of the form ) of the plotted points from all mice. b, A weighted sum of cumulative minutes of each recorded behaviour (attack, mounting, anogenital sniffing and other sniffing) as well as cumulative time spent interacting with conspecifics, plotted against PCC between male and female representations; weights fit by non-negative LASSO regression. c, Bar plot of weights used to generate plot in b, fit by non-negative LASSO with sparseness parameter chosen to minimize mean-squared error on held-out data (see Methods).
To exclude the possibility that the lack of representation separation shown in Fig. 4l–o was due to the presence of the experimenter’s hand during access-restricted trials, we repeated this experiment with the intruder mouse inside a wire mesh container. a, Diagram of experimental setup. b, Percentage of time the imaged mouse spent interacting with the intruder on each of the three days (n = 2 mice). Aside from day 1, the barrier to free interactions presented by the container did not reduce the time the resident spent investigating the intruder. c, PCC between male and female representations on the third day of the assay (blue bars) showed that the separation of representations did not occur in these two imaged mice. Gray bars show the average PCC between pairs of male trials or pairs of female trials, for comparison. d, Following the three days of interactions with the intruder behind the barrier, mice were given two additional days of free social interaction, before a final day (day 6) in which intruders were again presented inside the mesh container. A third, experienced animal (mouse 18) was also tested with the mesh container. e, PCC between representations of males and females presented within the mesh container on day 6; grey bars show average PCC between pairs of male trials or pairs of female trials. Two out of three tested mice showed clear separation of male and female representations; the third (mouse 17) did not, but failed to fight or mate with conspecifics during the free social interactions on the two days preceding the day 6 test. f, Performance of an SVM decoder trained to predict intruder sex from the data on day 6, showing high accuracy in the two mice that had previously fought and mounted. These data provide additional evidence that olfactory cues from conspecifics are not sufficient to produce ensemble separation, and that behaviors occurring during free social interactions are required.
About this article
Cite this article
Remedios, R., Kennedy, A., Zelikowsky, M. et al. Social behaviour shapes hypothalamic neural ensemble representations of conspecific sex. Nature 550, 388–392 (2017). https://doi.org/10.1038/nature23885
This article is cited by
Nature Methods (2023)
Nature Neuroscience (2023)
Cortical astrocytes modulate dominance behavior in male mice by regulating synaptic excitatory and inhibitory balance
Nature Neuroscience (2023)
Nature Neuroscience (2023)