Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning

Nature Biomedical Engineering volume 3, pages 930–942 (2019)Cite this article

Subjects

Abstract

Preclinical studies of psychiatric disorders use animal models to investigate the impact of environmental factors or genetic mutations on complex traits such as decision-making and social interactions. Here, we introduce a method for the real-time analysis of the behaviour of mice housed in groups of up to four over several days and in enriched environments. The method combines computer vision through a depth-sensing infrared camera, machine learning for animal and posture identification, and radio-frequency identification to monitor the quality of mouse tracking. It tracks multiple mice accurately, extracts a list of behavioural traits of both individuals and the groups of mice, and provides a phenotypic profile for each animal. We used the method to study the impact of Shank2 and Shank3 gene mutations—mutations that are associated with autism—on mouse behaviour. Characterization and integration of data from the behavioural profiles of Shank2 and Shank3 mutant female mice revealed their distinctive activity levels and involvement in complex social interactions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Tracking method and validation.
Fig. 2: Automatically labelled behaviours and an example of their representation.
Fig. 3: Behavioural profiles of Shank2−/− mice and Shank3−/− mice.
Fig. 4: Group of three and four dynamics for Shank2−/−, Shank3−/− and their respective wild-type littermate mice.
Fig. 5: Hyperactivity and atypical exploration strategy in Shank2−/− and Shank3−/− female mice.

Similar content being viewed by others

Data availability

The authors declare that all the data supporting the findings of this study are available within the paper and Supplementary Information. Full datasets (databases and films) generated during and/or analysed during the current study are available in the LMT website repository, https://livemousetracker.org.

Code availability

Full source code is available at http://icy.bioimageanalysis.org/plugins/livemousetracker. This includes Java code and also CAD hardware resource files. Python analysis scripts are available at https://github.com/fdechaumont/lmt-analysis.

References

  1. Tecott, L. H. & Nestler, E. J. Neurobehavioral assessment in the information age. Nat. Neurosci. 7, 462–466 (2004).

    Article  CAS  Google Scholar 

  2. Silverman, J. L., Yang, M., Lord, C. & Crawley, J. N. Behavioural phenotyping assays for mouse models of autism. Nat. Rev. Neurosci. 11, 490–502 (2010).

    Article  CAS  Google Scholar 

  3. Crawley, J. N. Designing mouse behavioral tasks relevant to autistic-like behaviors. Ment. Retard. Dev. Disabil. Res. Rev. 10, 248–258 (2004).

    Article  Google Scholar 

  4. Olsson, I. A. S. et al. Understanding behaviour: the relevance of ethological approaches in laboratory animal science. Appl. Anim. Behav. Sci. 81, 245–264 (2018).

    Article  Google Scholar 

  5. Belzung, C. & Griebel, G. Measuring normal and pathological anxiety-like behaviour in mice: a review. Behav. Brain Res. 125, 141–149 (2001).

    Article  CAS  Google Scholar 

  6. Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).

    Article  CAS  Google Scholar 

  7. Schmeisser, M. J. et al. Autistic-like behaviours and hyperactivity in mice lacking ProSAP1/Shank2. Nature 486, 256–260 (2012).

    Article  CAS  Google Scholar 

  8. Ey, E. et al. The autism ProSAP1/Shank2 mouse model displays quantitative and structural abnormalities in ultrasonic vocalisations. Behav. Brain Res. 256, 677–689 (2013).

    Article  CAS  Google Scholar 

  9. Berkel, S. et al. Mutations in the SHANK2 synaptic scaffolding gene in autism spectrum disorder and mental retardation. Nat. Genet. 42, 489–491 (2010).

    Article  CAS  Google Scholar 

  10. Leblond, C. S. et al. Genetic and functional analyses of SHANK2 mutations suggest a multiple hit model of autism spectrum disorders. PLoS Genet. 8, e100252 (2012).

    Article  Google Scholar 

  11. Leblond, C. S. et al. Meta-analysis of SHANK mutations in autism spectrum disorders: a gradient of severity in cognitive impairments. PLoS Genet. 10, e1004580 (2014).

    Article  Google Scholar 

  12. Durand, C. M. et al. Mutations in the gene encoding the synaptic scaffolding protein SHANK3 are associated with autism spectrum disorders. Nat. Genet. 39, 25–27 (2007).

    Article  CAS  Google Scholar 

  13. Bernardin, K., Elbs, A. & Stiefelhagen, R. Multiple object tracking performance metrics and evaluation in a smart room environment. In Proc. of IEEE International Workshop on Visual Surveillance (2006).

  14. Matthews, S. G., Miller, A. L., Plötz, T. & Kyriazakis, I. Automated tracking to measure behavioural changes in pigs for health and welfare monitoring. Sci. Rep. 7, 17582 (2017).

    Article  Google Scholar 

  15. Peca, J. et al. Shank3 mutant mice display autistic-like behaviours and striatal dysfunction. Nature 472, 437–442 (2011).

    Article  CAS  Google Scholar 

  16. Drapeau, E., Riad, M., Kajiwara, Y. & Buxbaum, J. D. Behavioral phenotyping of an improved mouse model of Phelan-McDermid syndrome with a complete deletion of the Shank3 gene. eNeuro 5, e0046-18 (2018).

  17. Ohayon, S., Avni, O., Taylor, A. L., Perona, P. & Roian Egnor, S. E. Automated multi-day tracking of marked mice for the analysis of social behaviour. J. Neurosci. Methods 219, 10–19 (2013).

    Article  Google Scholar 

  18. Ey, E. et al. Shank2 mutant mice display hyperactivity insensitive to methylphenidate and reduced flexibility in social motivation, but normal social recognition. Front. Mol. Neurosci. 11, 365 (2018).

    Article  CAS  Google Scholar 

  19. Pappas, A. L. et al. Deficiency of Shank2 causes mania-like behavior that responds to mood stabilizers. JCI Insight 2, 92052 (2017).

    Article  Google Scholar 

  20. Holly, K. S., Orndorff, C. O. & Murray, T. A. MATSAP: an automated analysis of stretch-attend posture in rodent behavioral experiments. Sci. Rep. 6, 31286 (2016).

    Article  CAS  Google Scholar 

  21. Torquet, N., de Chaumont, F., Faure, P., Bourgeron, T. & Ey, E. mouseTube—a database to collaboratively unravel mouse ultrasonic communication. F1000 Res. 5, 2332 (2016).

    Article  Google Scholar 

  22. Esler, A. & Grissom, N. Why we need a mouse version of a diagnostic test for autism. Spectrum (21 March 2018); https://spectrumnews.org/opinion/viewpoint/need-mouse-version-diagnostic-test-autism/

  23. Maubourguet, N., Lesne, A., Changeux, J.-P., Maskos, U. & Faure, P. Behavioral sequence analysis reveals a novel role for β2* nicotinic receptors in exploration. PLoS Comput. Biol. 4, e1000229 (2008).

    Article  Google Scholar 

  24. R Core Team R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2017); http://www.R-project.org/

Download references

Acknowledgements

The authors thank Y. Archambeau and P. Ollivon at the workshop of the Institut Pasteur for building the first 12 setups and advising on hardware, W. Meiniel for the mathematical proof for decisions of head/tail probability, Microsoft France for their technical support, P. Spinicelli for optical engineering and reading of the paper, R. Marée for machine learning support, B. König for advice and reading of biological experiments, J. N. Crawley for reading and providing comments on the manuscript, A. Barmpoutis for providing us with the early Kinect 2 driver and support, N. Chenouard for driving the use of the machine learning solution, P. Dugast for drawing the mice in the different behavioural events, A. Engelberg for checking the English, S. Wagner and R. Accardi for RFID advice, M. Marim for website development, and X. Montagutelli and M. Bérard for animal facility support. This work was partially funded by the Institut Pasteur, the Bettencourt-Schueller Foundation, the Cognacq–Jay Foundation, the Conny–Maeva Foundation, the ERANET–NEURON SYNPATHY program, the Agence Nationale de la Recherche through grant number ANR-10-LABX-62-IBEID, France-BioImaging infrastructure through grant number ANR-10-INBS-04 and the INCEPTION program through grant number ANR-16-CONV-0005, the Centre National de la Recherche Scientifique, the University Paris Diderot, the BioPsy Labex, the Institut National du Cancer through grant number TABAC-16–022, the Foundation for Medical Research (Equipe DEQ20130326488), the Innovative Medicines Initiative Joint Undertaking through grant agreement number 115300, resources of which are composed of financial contributions from the European Union’s Seventh Framework Program (FP7/2007–2013) and EFPIA companies in kind contribution. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Author notes

  1. These authors contributed equally: Fabrice de Chaumont, Elodie Ey.

  2. These authors jointly supervised this work: Philippe Faure, Thomas Bourgeron, Jean-Christophe Olivo-Marin.

Authors and Affiliations

  1. Institut Pasteur, BioImage Analysis Unit, CNRS UMR 3691, Paris, France

    Fabrice de Chaumont, Thibault Lagache, Stéphane Dallongeville & Jean-Christophe Olivo-Marin

  2. Human Genetics and Cognitive Functions, Institut Pasteur, UMR 3571 CNRS, University Paris-Diderot, Paris, France

    Elodie Ey, Anne-Marie Le Sourd & Thomas Bourgeron

  3. Sorbonne Université, CNRS UMR 8246, INSERM, Neurosciences Paris Seine - Institut de Biologie Paris-Seine, Paris, France

    Nicolas Torquet & Philippe Faure

  4. Institut Pasteur, FabLab, Center for Innovation and Technological research, Paris, France

    Albane Imbert

  5. Aix-Marseille Université, CNRS, LPL, UMR 7309, Aix-en-Provence, France

    Thierry Legou

Authors
  1. Fabrice de Chaumont
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elodie Ey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Torquet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thibault Lagache
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stéphane Dallongeville
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Albane Imbert
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thierry Legou
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Anne-Marie Le Sourd
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Philippe Faure
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Thomas Bourgeron
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jean-Christophe Olivo-Marin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

F.d.C. created the tracking and analysis methods, developed the open-hardware setup and software. E.E. performed and analysed experiments. N.T. analysed experiments. S.D. provided real-time programming advice. T. Lagache provided the probabilistic framework for machine learning decision-making. T. Legou provided electronic support. A.I. created the blueprints. A.-M.L.S. performed the genotyping of the mice. F.d.C., E.E., N.T., P.F., T.B. and J.-C.O.-M. conceived the project and wrote the manuscript.

Corresponding authors

Correspondence to Fabrice de Chaumont, Elodie Ey or Jean-Christophe Olivo-Marin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary figures, tables, methods, results, video captions and references.

Reporting Summary

Assembly instructions

Assembly instructions for the mouse live tracker.

Supplementary Video 1

The different steps of the method, and a general overview of the system.

Supplementary Video 2

3D animations of the assembly, and how to perform the calibration of the system.

Supplementary Video 3

A case study of a typical problem: a nest that is built by the animal at an unexpected location.

Supplementary Video 4

Background height map (a view of the cage without the animals).

Supplementary Video 5

Live 3D rendering demo.

Supplementary Video 6

Data used to validate the method.

Supplementary Video 7

How the system copes with different appearances, and also with a mix of animals having different coat colours.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Chaumont, F., Ey, E., Torquet, N. et al. Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning. Nat Biomed Eng 3, 930–942 (2019). https://doi.org/10.1038/s41551-019-0396-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-019-0396-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics