Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning


Preclinical studies of psychiatric disorders use animal models to investigate the impact of environmental factors or genetic mutations on complex traits such as decision-making and social interactions. Here, we introduce a method for the real-time analysis of the behaviour of mice housed in groups of up to four over several days and in enriched environments. The method combines computer vision through a depth-sensing infrared camera, machine learning for animal and posture identification, and radio-frequency identification to monitor the quality of mouse tracking. It tracks multiple mice accurately, extracts a list of behavioural traits of both individuals and the groups of mice, and provides a phenotypic profile for each animal. We used the method to study the impact of Shank2 and Shank3 gene mutations—mutations that are associated with autism—on mouse behaviour. Characterization and integration of data from the behavioural profiles of Shank2 and Shank3 mutant female mice revealed their distinctive activity levels and involvement in complex social interactions.

Your institute does not have access to this article

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Fig. 1: Tracking method and validation.
Fig. 2: Automatically labelled behaviours and an example of their representation.
Fig. 3: Behavioural profiles of Shank2−/− mice and Shank3−/− mice.
Fig. 4: Group of three and four dynamics for Shank2−/−, Shank3−/− and their respective wild-type littermate mice.
Fig. 5: Hyperactivity and atypical exploration strategy in Shank2−/− and Shank3−/− female mice.

Data availability

The authors declare that all the data supporting the findings of this study are available within the paper and Supplementary Information. Full datasets (databases and films) generated during and/or analysed during the current study are available in the LMT website repository,

Code availability

Full source code is available at This includes Java code and also CAD hardware resource files. Python analysis scripts are available at


  1. Tecott, L. H. & Nestler, E. J. Neurobehavioral assessment in the information age. Nat. Neurosci. 7, 462–466 (2004).

    CAS  Article  Google Scholar 

  2. Silverman, J. L., Yang, M., Lord, C. & Crawley, J. N. Behavioural phenotyping assays for mouse models of autism. Nat. Rev. Neurosci. 11, 490–502 (2010).

    CAS  Article  Google Scholar 

  3. Crawley, J. N. Designing mouse behavioral tasks relevant to autistic-like behaviors. Ment. Retard. Dev. Disabil. Res. Rev. 10, 248–258 (2004).

    Article  Google Scholar 

  4. Olsson, I. A. S. et al. Understanding behaviour: the relevance of ethological approaches in laboratory animal science. Appl. Anim. Behav. Sci. 81, 245–264 (2018).

    Article  Google Scholar 

  5. Belzung, C. & Griebel, G. Measuring normal and pathological anxiety-like behaviour in mice: a review. Behav. Brain Res. 125, 141–149 (2001).

    CAS  Article  Google Scholar 

  6. Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).

    CAS  Article  Google Scholar 

  7. Schmeisser, M. J. et al. Autistic-like behaviours and hyperactivity in mice lacking ProSAP1/Shank2. Nature 486, 256–260 (2012).

    CAS  Article  Google Scholar 

  8. Ey, E. et al. The autism ProSAP1/Shank2 mouse model displays quantitative and structural abnormalities in ultrasonic vocalisations. Behav. Brain Res. 256, 677–689 (2013).

    CAS  Article  Google Scholar 

  9. Berkel, S. et al. Mutations in the SHANK2 synaptic scaffolding gene in autism spectrum disorder and mental retardation. Nat. Genet. 42, 489–491 (2010).

    CAS  Article  Google Scholar 

  10. Leblond, C. S. et al. Genetic and functional analyses of SHANK2 mutations suggest a multiple hit model of autism spectrum disorders. PLoS Genet. 8, e100252 (2012).

    Article  Google Scholar 

  11. Leblond, C. S. et al. Meta-analysis of SHANK mutations in autism spectrum disorders: a gradient of severity in cognitive impairments. PLoS Genet. 10, e1004580 (2014).

    Article  Google Scholar 

  12. Durand, C. M. et al. Mutations in the gene encoding the synaptic scaffolding protein SHANK3 are associated with autism spectrum disorders. Nat. Genet. 39, 25–27 (2007).

    CAS  Article  Google Scholar 

  13. Bernardin, K., Elbs, A. & Stiefelhagen, R. Multiple object tracking performance metrics and evaluation in a smart room environment. In Proc. of IEEE International Workshop on Visual Surveillance (2006).

  14. Matthews, S. G., Miller, A. L., Plötz, T. & Kyriazakis, I. Automated tracking to measure behavioural changes in pigs for health and welfare monitoring. Sci. Rep. 7, 17582 (2017).

    Article  Google Scholar 

  15. Peca, J. et al. Shank3 mutant mice display autistic-like behaviours and striatal dysfunction. Nature 472, 437–442 (2011).

    CAS  Article  Google Scholar 

  16. Drapeau, E., Riad, M., Kajiwara, Y. & Buxbaum, J. D. Behavioral phenotyping of an improved mouse model of Phelan-McDermid syndrome with a complete deletion of the Shank3 gene. eNeuro 5, e0046-18 (2018).

  17. Ohayon, S., Avni, O., Taylor, A. L., Perona, P. & Roian Egnor, S. E. Automated multi-day tracking of marked mice for the analysis of social behaviour. J. Neurosci. Methods 219, 10–19 (2013).

    Article  Google Scholar 

  18. Ey, E. et al. Shank2 mutant mice display hyperactivity insensitive to methylphenidate and reduced flexibility in social motivation, but normal social recognition. Front. Mol. Neurosci. 11, 365 (2018).

    CAS  Article  Google Scholar 

  19. Pappas, A. L. et al. Deficiency of Shank2 causes mania-like behavior that responds to mood stabilizers. JCI Insight 2, 92052 (2017).

    Article  Google Scholar 

  20. Holly, K. S., Orndorff, C. O. & Murray, T. A. MATSAP: an automated analysis of stretch-attend posture in rodent behavioral experiments. Sci. Rep. 6, 31286 (2016).

    CAS  Article  Google Scholar 

  21. Torquet, N., de Chaumont, F., Faure, P., Bourgeron, T. & Ey, E. mouseTube—a database to collaboratively unravel mouse ultrasonic communication. F1000 Res. 5, 2332 (2016).

    Article  Google Scholar 

  22. Esler, A. & Grissom, N. Why we need a mouse version of a diagnostic test for autism. Spectrum (21 March 2018);

  23. Maubourguet, N., Lesne, A., Changeux, J.-P., Maskos, U. & Faure, P. Behavioral sequence analysis reveals a novel role for β2* nicotinic receptors in exploration. PLoS Comput. Biol. 4, e1000229 (2008).

    Article  Google Scholar 

  24. R Core Team R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2017);

Download references


The authors thank Y. Archambeau and P. Ollivon at the workshop of the Institut Pasteur for building the first 12 setups and advising on hardware, W. Meiniel for the mathematical proof for decisions of head/tail probability, Microsoft France for their technical support, P. Spinicelli for optical engineering and reading of the paper, R. Marée for machine learning support, B. König for advice and reading of biological experiments, J. N. Crawley for reading and providing comments on the manuscript, A. Barmpoutis for providing us with the early Kinect 2 driver and support, N. Chenouard for driving the use of the machine learning solution, P. Dugast for drawing the mice in the different behavioural events, A. Engelberg for checking the English, S. Wagner and R. Accardi for RFID advice, M. Marim for website development, and X. Montagutelli and M. Bérard for animal facility support. This work was partially funded by the Institut Pasteur, the Bettencourt-Schueller Foundation, the Cognacq–Jay Foundation, the Conny–Maeva Foundation, the ERANET–NEURON SYNPATHY program, the Agence Nationale de la Recherche through grant number ANR-10-LABX-62-IBEID, France-BioImaging infrastructure through grant number ANR-10-INBS-04 and the INCEPTION program through grant number ANR-16-CONV-0005, the Centre National de la Recherche Scientifique, the University Paris Diderot, the BioPsy Labex, the Institut National du Cancer through grant number TABAC-16–022, the Foundation for Medical Research (Equipe DEQ20130326488), the Innovative Medicines Initiative Joint Undertaking through grant agreement number 115300, resources of which are composed of financial contributions from the European Union’s Seventh Framework Program (FP7/2007–2013) and EFPIA companies in kind contribution. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations



F.d.C. created the tracking and analysis methods, developed the open-hardware setup and software. E.E. performed and analysed experiments. N.T. analysed experiments. S.D. provided real-time programming advice. T. Lagache provided the probabilistic framework for machine learning decision-making. T. Legou provided electronic support. A.I. created the blueprints. A.-M.L.S. performed the genotyping of the mice. F.d.C., E.E., N.T., P.F., T.B. and J.-C.O.-M. conceived the project and wrote the manuscript.

Corresponding authors

Correspondence to Fabrice de Chaumont, Elodie Ey or Jean-Christophe Olivo-Marin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary figures, tables, methods, results, video captions and references.

Reporting Summary

Assembly instructions

Assembly instructions for the mouse live tracker.

Supplementary Video 1

The different steps of the method, and a general overview of the system.

Supplementary Video 2

3D animations of the assembly, and how to perform the calibration of the system.

Supplementary Video 3

A case study of a typical problem: a nest that is built by the animal at an unexpected location.

Supplementary Video 4

Background height map (a view of the cage without the animals).

Supplementary Video 5

Live 3D rendering demo.

Supplementary Video 6

Data used to validate the method.

Supplementary Video 7

How the system copes with different appearances, and also with a mix of animals having different coat colours.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

de Chaumont, F., Ey, E., Torquet, N. et al. Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning. Nat Biomed Eng 3, 930–942 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:

Further reading


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing