Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

JAABA: interactive machine learning for automatic annotation of animal behavior

Nature Methods volume 10, pages 64–67 (2013)Cite this article

Subjects

Abstract

We present a machine learning–based system for automatically computing interpretable, quantitative measures of animal behavior. Through our interactive system, users encode their intuition about behavior by annotating a small set of video frames. These manual labels are converted into classifiers that can automatically annotate behaviors in screen-scale data sets. Our general-purpose system can create a variety of accurate individual and social behavior classifiers for different organisms, including mice and adult and larval Drosophila.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: JAABA overview.
Figure 2: Behavior detector accuracy across behaviors and organism types.
Figure 3: Behavior classifiers generalize over a phenotypically diverse data set.

Similar content being viewed by others

References

  1. Branson, K., Robie, A.A., Bender, J., Perona, P. & Dickinson, M.H. Nat. Methods 6, 451–457 (2009).

    Article  CAS  Google Scholar 

  2. Dankert, H., Wang, L., Hoopfer, E.D., Anderson, D.J. & Perona, P. Nat. Methods 6, 297–303 (2009).

    Article  CAS  Google Scholar 

  3. de Chaumont, F. et al. Nat. Methods 9, 410–417 (2012).

    Article  CAS  Google Scholar 

  4. Swierczek, N.A., Giles, A.C., Rankin, C.H. & Kerr, R.A. Nat. Methods 8, 592–598 (2011).

    Article  CAS  Google Scholar 

  5. Luo, L. et al. J. Neurosci. 30, 4261–4272 (2010).

    Article  CAS  Google Scholar 

  6. Friedman, J., Hastie, T. & Tibshirani, R. Ann. Stat. 28, 337–407 (2000).

    Article  Google Scholar 

  7. Settles, B. Active learning literature survey (Computer sciences technical report 1648) 〈http://research.cs.wisc.edu/techreports/2009/TR1648.pdf〉 (Univ. Wisconsin-Madison, 2009).

  8. Gomez-Marin, A., Stephens, G.J. & Louis, M. Nat. Commun. 2, 441 (2011).

    Article  Google Scholar 

  9. Jenett, A. et al. Cell Rep. 2, 991–1001 (2012).

    Article  CAS  Google Scholar 

  10. Martin, J.R. Behav. Processes 67, 207–219 (2004).

    Article  Google Scholar 

  11. Robie, A.A., Straw, A.D. & Dickinson, M.H. J. Exp. Biol. 213, 2494–2506 (2010).

    Article  Google Scholar 

  12. Aggarwal, J.K. & Ryoo, M.S. ACM Comput. Surv. 43, 16 (2011).

    Article  Google Scholar 

  13. Dollar, P., Rabaud, V., Cottrell, G. & Belongie, S. in Visual Surveillance and Performance Evaluation of Tracking Surveillance (VS-PETS) 65–72 (IEEE, 2005).

  14. Simon, J.C. & Dickinson, M.H. PLoS ONE 5, e8793 (2010).

    Article  Google Scholar 

  15. Pfeiffer, B.D. et al. Proc. Natl. Acad. Sci. USA 105, 9715–9720 (2008).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank A. Miller and R. Egnor (HHMI JFRC) for creating the mouse behavior detectors; R. Egnor and S. Ohayon (Caltech) for providing their mouse data before publication; P. Keller for help in preparing the manuscript and helpful discussions; H. Kelstrup and C. Mirth (Instituto Gulbenkian de Ciência) for providing their larva data; G. Rubin (HHMI JFRC) for providing GAL4 lines before publication; G. Card and U. Heberlein (HHMI JFRC) for sharing flies; the Janelia Fly Olympiad Team (particularly J. Hirokawa, L. Umayam, W. Korff, M. Phillips and S. Roberts) and the Janelia Fly Facility (particularly J. MacMahon and T. Laverty) for helping in the TrpA neural activation fly screen; B. Arthur and J. Liu for help in developing the JAABA software; U. Heberlein, P. Perona, M. Zlatic, T. Jovanic, E. Hoopfer, B. Afonso, M. Gershow and C. Schusterreiter for helpful discussions; S. Albin, Y. Aso, R. Azanchi, B. Baker, G. Card, C. Dan, E. Hoopfer, V. Jayaraman, W. Korff, G. Ophir, Y. Pan and A. Seeds for participating in our study of JAABA's usability; and W. Karsh for acronym help.

Author information

Author notes

  1. Mayank Kabra and Alice A Robie: These authors contributed equally to this work.

Authors and Affiliations

  1. Howard Hughes Medical Institute, Janelia Farm Research Campus (HHMI JFRC), Ashburn, Virginia, USA

    Mayank Kabra, Alice A Robie, Marta Rivera-Alba, Steven Branson & Kristin Branson

  2. Development, Evolution and the Environment Laboratory, Instituto Gulbenkian de Ciência, Oeiras, Portugal

    Marta Rivera-Alba

  3. Department of Computer Science and Engineering, University of California, San Diego, La Jolla, California, USA

    Steven Branson

Authors
  1. Mayank Kabra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alice A Robie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marta Rivera-Alba
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Steven Branson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kristin Branson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.B. conceived of the project with help from M.K., A.A.R. and S.B. M.K. and K.B. designed and wrote the software with help from A.A.R., S.B. and M.R.-A. A.A.R. created and tested the fly behavior detectors with help from K.B. A.A.R. developed and performed the fly experiments with help from K.B. and M.K. M.R.-A. developed the larva tracker and larva behavior detectors with help from K.B. and M.K. K.B., A.A.R. and M.K. wrote the paper with help from M.R.-A.

Corresponding author

Correspondence to Kristin Branson.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–6, Supplementary Tables 1–3 and Supplementary Note (PDF 2090 kb)

Supplementary Software

MATLAB source code for JAABA. The documentation and a video tutorial are in the 'docs' folder. (ZIP 58510 kb)

Demonstration of using JAABA to train a wing-grooming classifier.

We recorded a screencast of a user beginning to train a wing-grooming classifier. In this video, the user labels one bout of frames in which the fly is wing grooming and one bout of frames in which the fly is not wing grooming. The user leaves a gap between the end of one of these bouts and the beginning of the next, as it is difficult to consistently label these frames. The user then trains an initial classifier from this first set of labels, a process that takes a few seconds. The user examines the resulting classifier's predictions on frames not in the training data. This initial classifier makes several mistakes, and the user labels a third bout to correct some mistakes. The user then retrains a new classifier and again examines the predictions. After just three bouts of training labels, the classifier is making reasonable predictions on data outside the training set. (AVI 16481 kb)

Automatic detection of 15 behaviors for adult flies.

We show the results of applying the 15 automatic fly behavior classifiers described in Figure 2 to a video of pBDPGAL4U control and R71G01 flies. For all the flies, we show clips of 500 frames. The first four clips are for two males and two females from the control lines. The last two clips show two males from R71G01. We selected clips from R71G01 because the social behaviors are rare in control lines. On each frame, we plot a triangle to indicate the selected fly's tracked position and a line to indicate its trajectory. Below the frame, we plot each of the 15 behavior classifiers' current predictions in gray (above the dashed line indicates a prediction that the behavior is occurring; below, that the behavior is not occurring). In color, on top of these gray boxes, we plot the raw outputs of the classifier, where larger bars indicate that the classifier is more confident. The names of the behaviors currently detected are written behind the fly. The video plays at on-third real time (10 f.p.s.). (AVI 21121 kb)

Behavior variation between GAL4 lines.

We show the results of applying three of the automatic fly behavior classifiers (chase, jump and righting) to the selection of GAL4 lines described in Figure 3. For each video, we show one male and one female fly for 250 frames. Data are plotted as in Supplementary Video 2. (AVI 16481 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kabra, M., Robie, A., Rivera-Alba, M. et al. JAABA: interactive machine learning for automatic annotation of animal behavior. Nat Methods 10, 64–67 (2013). https://doi.org/10.1038/nmeth.2281

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.2281

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing