Using DeepLabCut for 3D markerless pose estimation across species and behaviors

Abstract

Noninvasive behavioral tracking of animals during experiments is critical to many scientific pursuits. Extracting the poses of animals without using markers is often essential to measuring behavioral effects in biomechanics, genetics, ethology, and neuroscience. However, extracting detailed poses without markers in dynamically changing backgrounds has been challenging. We recently introduced an open-source toolbox called DeepLabCut that builds on a state-of-the-art human pose-estimation algorithm to allow a user to train a deep neural network with limited training data to precisely track user-defined features that match human labeling accuracy. Here, we provide an updated toolbox, developed as a Python package, that includes new features such as graphical user interfaces (GUIs), performance improvements, and active-learning-based network refinement. We provide a step-by-step procedure for using DeepLabCut that guides the user in creating a tailored, reusable analysis pipeline with a graphical processing unit (GPU) in 1–12 h (depending on frame size). Additionally, we provide Docker environments and Jupyter Notebooks that can be run on cloud resources such as Google Colaboratory.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Pose estimation with DeepLabCut.
Fig. 2: DeepLabCut workflow.
Fig. 3: Methods for frame selection.
Fig. 4: Labeling GUI.
Fig. 5: Evaluation of results.
Fig. 6: Refinement tools.
Fig. 7: 3D pose estimation of a cheetah.

Data and code availability

The code is fully available at https://github.com/AlexEMG/DeepLabCut. Other inquiries should be made to the corresponding author (M.W.M.).

References

  1. 1.

    Ilg, E. et al. FlowNet 2.0: evolution of optical flow estimation with deep networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1647–1655 (2017).

  2. 2.

    Toshev, A. & Szegedy, C. DeepPose: human pose estimation via deep neural networks. 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVVR) 1653–1660 (2014).

  3. 3.

    Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J. & Quillen, D. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37, 421–436 (2018).

  4. 4.

    Wainberg, M., Merico, D., Delong, A. & Frey, B. J. Deep learning in biomedicine. Nat. Biotechnol. 36, 829–838 (2018).

  5. 5.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  6. 6.

    Donahue, J. et al. DeCAF: a deep convolutional activation feature for generic visual recognition. Proceedings of the 31st International Conference on Machine Learning 647–655 (2014).

  7. 7.

    Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems (NIPS) 27, 3320–3328 (2014).

  8. 8.

    Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning 1 (MIT Press, Cambridge, MA, 2016).

  9. 9.

    Kümmerer, M., Wallis, T. S. A., Gatys, L. A. & Bethge, M. Understanding low-and high-level contributions to fixation prediction. Proceedings of the IEEE International Conference on Computer Vision 4789–4798 (2017).

  10. 10.

    Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M. & Schiele, B. DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. European Conference on Computer Vision 34–50 (2016).

  11. 11.

    Insafutdinov, E. et al. ArtTrack: articulated multi-person tracking in the wild. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1293–1301 (2017).

  12. 12.

    Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).

  13. 13.

    Nath, T. et al. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Preprint at https://www.biorxiv.org/content/10.1101/476531v1 (2018).

  14. 14.

    Dell, A. I. et al. Automated image-based tracking and its application in ecology. Trends Ecol. Evol. (Amst) 29, 417–428 (2014).

  15. 15.

    Anderson, D. J. & Perona, P. Toward a science of computational ethology. Neuron 84, 18–31 (2014).

  16. 16.

    Egnor, S. R. & Branson, K. Computational analysis of behavior. Annu. Rev. Neurosci. 39, 217–236 (2016).

  17. 17.

    Dollár, P., Welinder, P. & Perona, P. Cascaded pose regression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1078–1085 (2010).

  18. 18.

    Gomez-Marin, A., Partoune, N., Stephens, G. J. & Louis, M. Automated tracking of animal posture and movement during exploration and sensory orientation behaviors. PLoS ONE 7, 1–9 (2012).

  19. 19.

    Matsumoto, J. et al. A 3D-video-based computerized analysis of social and sexual interactions in rats. PLoS ONE 8, e78460 (2013).

  20. 20.

    Uhlmann, V., Ramdya, P., Delgado-Gonzalo, R., Benton, R. & Unser, M. FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila. PLoS ONE 12, e0173433 (2017).

  21. 21.

    Ben-Shaul, Y. OptiMouse: a comprehensive open source program for reliable detection and analysis of mouse body and nose positions. BMC Biol. 15, 41 (2017).

  22. 22.

    Winter, D. A. Biomechanics and Motor Control of Human Movement (John Wiley & Sons, 2009).

  23. 23.

    Zhou, H. & Hu, H. Human motion tracking for rehabilitation—a survey. Biomed. Signal Process. Control 3, 1–18 (2008).

  24. 24.

    Kays, R., Crofoot, M. C., Jetz, W. & Wikelski, M. Terrestrial animal tracking as an eye on life and planet. Science 348, aaa2478 (2015).

  25. 25.

    Colyer, S. L., Evans, M., Cosker, D. P. & Salo, A. I. A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system. Sports Med. Open 4, 24 (2018).

  26. 26.

    Wei, K. & Kording, K. P. Behavioral tracking gets real. Nat. Neurosci. 21, 1146–1147 (2018).

  27. 27.

    Mathis, A. & Warren, R. A. On the inference speed and video-compression robustness of DeepLabCut. Preprint at https://www.biorxiv.org/content/10.1101/457242v1 (2018).

  28. 28.

    Aguillon Rodriguez, V. et al. The International Brain Laboratory: reproducing a single decision-making behavior in mice across labs. Society for Neuroscience 2018, abstr. 613.01 (2018).

  29. 29.

    Felzenszwalb, P. F. & Huttenlocher, D. P. Pictorial structures for object recognition. Int. J. Comput. Vis 61, 55–79 (2005).

  30. 30.

    Andriluka, M., Pishchulin, L., Gehler, P. & Schiele, B. 2D human pose estimation: new benchmark and state of the art analysis. 2014 IEEE Conference on Computer Vision and Pattern Recognition 3686–3693 (2014).

  31. 31.

    Wei, S.-E., Ramakrishna, V., Kanade, T. & Sheikh, Y. Convolutional pose machines. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 4724–4732 (2016).

  32. 32.

    Cao, Z., Simon, T., Wei, S.-E. & Sheikh, Y. Realtime multi-person 2D pose estimation using part affinity fields. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1302–1310 (2017).

  33. 33.

    Simon, T., Joo, H., Matthews, I. & Sheikh, Y. Hand keypoint detection in single images using multiview bootstrapping. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 4645–4653 (2017).

  34. 34.

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (2016).

  35. 35.

    Dong, H. et al. TensorLayer: a versatile library for efficient deep learning development. Proceedings of the 25th ACM International Conference on Multimedia 1201–1204 (2017).

  36. 36.

    Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2019).

  37. 37.

    OpenCV. Open Source Computer Vision Library, https://opencv.org (2015).

  38. 38.

    Lucas, B. D. & Kanade, T. An iterative image registration technique with an application to stereo vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence, Vol. 2, 674–679 (Morgan Kaufmann, 1981).

  39. 39.

    Oliphant, T. E. Python for scientific computing. Comput. Sci. Eng. 9, 10–20 (2007).

  40. 40.

    Abadi, M. et al. TensorFlow: a system for large-scale machine learning. Preprint at https://arxiv.org/abs/1605.08695 (2016).

  41. 41.

    Merkel, D. Docker: lightweight linux containers for consistent development and deployment. Linux J. 2014, 2 (2014).

  42. 42.

    Mathis, M. W., Mathis, A. & Uchida, N. Somatosensory cortex plays an essential role in forelimb motor adaptation in mice. Neuron 93, 1493–1503.e6 (2017).

  43. 43.

    McKinney, W. Data structures for statistical computing in Python. Proceedings of the 9th Python in Science Conference 51–56 (2010).

  44. 44.

    Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).

  45. 45.

    Durbin, J. & Koopman, S. J. Time Series Analysis by State Space Methods Vol. 38 (Oxford University Press, 2012).

  46. 46.

    Seabold, S. & Perktold, J. Statsmodels: econometric and statistical modeling with Python. Proceedings of the 9th Python in Science Conference 57–61 (2010).

  47. 47.

    Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S. & Branson, K. JAABA: interactive machine learning for automatic annotation of animal behavior. Nat. Methods 10, 64 (2012).

  48. 48.

    Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface 11, (2014).

  49. 49.

    Fox, E., Jordan, M. I., Sudderth, E. B. & Willsky, A. S. Sharing features among dynamical systems with beta processes. In Advances in Neural Information Processing Systems (eds Bengio, Y. et al.) 22, 549–557 (Neural Information Processing Systems Foundation, 2009).

  50. 50.

    Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).

  51. 51.

    Vogelstein, J. T. et al. Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning. Science 344, 386–392 (2014).

  52. 52.

    Priebe, C. E., Marchette, D. J. & Healy, D. M. Integrated sensing and processing for statistical pattern recognition. Mod. Signal Process. 46, 223 (2003).

  53. 53.

    Todd, J. G., Kain, J. S. & de Bivort, B. L. Systematic exploration of unsupervised methods for mapping behavior. Phys. Biol. 14, 015002 (2017).

  54. 54.

    Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).

  55. 55.

    Pouw, W., de Jonge-Hoekstra, L. & Dixon, J. A. Stabilizing speech production through gesture-speech coordination. Preprint at https://psyarxiv.com/arzne (2018).

  56. 56.

    Wickens, A. et al. Magnetoelectric materials for miniature, wireless neural stimulation at therapeutic frequencies. Preprint at https://www.biorxiv.org/content/10.1101/461855v1 (2018).

  57. 57.

    Wilson, A. M. et al. Locomotion dynamics of hunting in wild cheetahs. Nature 498, 185–189 (2013).

  58. 58.

    Jackson, B. E., Evangelista, D. J., Ray, D. D. & Hedrick, T. L. 3D for the people: multi-camera motion capture in the field with consumer-grade cameras and open source software. Biol. Open 5, 1334–1342 (2016).

  59. 59.

    Urban, S., Leitloff, J. & Hinz, S. Improved wide-angle, fisheye and omnidirectional camera calibration. ISPRS J. Photogramm. Remote Sens 108, 72–79 (2015).

  60. 60.

    Theriault, D. H. et al. A protocol and calibration method for accurate multi-camera field videography. J. Exp. Biol. 217, 1843–1848 (2014).

Download references

Acknowledgements

DeepLabCut is an open-source tool on GitHub and has benefited from suggestions and edits by many individuals, including R. Eichler, J. Rauber, R. Warren, T. Abe, H. Wu, and J. Saunders. In particular, the authors thank R. Eichler for input on the modularized version. The authors thank the members of the Bethge Lab for providing the initial version of the Docker container. We also thank M. Li, J. Li, and D. Robson for use of the zebrafish image; B. Rogers for use of the horse images; and K. Cury for the fly images. The authors are grateful to E. Insafutdinov and C. Lassner for suggestions on how to best use the TensorFlow implementation of DeeperCut. We also thank A. Hoffmann, P. Mamidanna, and G. Kane for comments throughout this project. Last, the authors thank the Ann van Dyk Cheetah Centre (Pretoria, South Africa) for kindly providing access to their cheetahs. The authors thank NVIDIA Corporation for GPU grants to both M.W.M. and A.M. A.M. acknowledges a Marie Sklodowska-Curie International Fellowship within the 7th European Community Framework Program under grant agreement no. 622943. A.P. acknowledges an Oppenheimer Memorial Trust Fellowship and the National Research Foundation of South Africa (grant 99380). M.B. acknowledges funding from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) via the Collaborative Research Center (Projektnummer 276693517–SFB 1233: Robust Vision) and by the German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ 01IS18039A). M.W.M. acknowledges a Rowland Fellowship from the Rowland Institute at Harvard.

Author information

Conceptualization: A.M., T.N., and M.W.M. A.M., T.N., and M.W.M. wrote the code. A.P. provided the cheetah data; A.C.C. labeled the cheetah data; A.C.C., A.M., and A.P. analyzed the cheetah data. M.W.M., A.M., and T.N. wrote the manuscript with input from all authors. M.W.M. and M.B. supervised the project.

Correspondence to Mackenzie Weygandt Mathis.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information: Nature Protocols thanks Gonzalo G. de Polavieja and other anonymous reviewer(s) for their contribution to the peer review of this work.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related link

Key reference using this protocol

Mathis, A. et al. Nat. Neurosci. 21, 1281–1289 (2018): https://www.nature.com/articles/s41593-018-0209-y

Supplementary information

Supplementary Video 1

A video of the sequence described in Fig. 7c. While the images created in Fig. 7c included 256 labeled images (99% training set split, from different sessions, cameras, and perspectives), the video was created after additional training with data from different videos (908 images, 95% training set split). This network was also used for Fig. 7e.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.