Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • News & Views
  • Published:

There is no mouse: using a virtual mouse to generate training data for video-based pose estimation

Lab Animal volume 50, pages 172–173 (2021)Cite this article

Subjects

Over the last decade, deep artificial neural networks have revolutionized fields such as speech recognition, object detection, and drug discovery1. Such ‘deep learning’ algorithms learn which features of the data are relevant to perform a certain task. This makes hand-built ‘feature extractors’ unnecessary and obsolete since they are a more time consuming, inefficient, and less versatile way of dealing with this problem. Many recent, impactful applications of deep learning are based on supervised classification, where the only input needed by the model is a labeled dataset. For this reason, obtaining large labeled datasets has become the new bottleneck in training deep learning algorithms. As a consequence, the use of pre-trained models - also called transfer learning - has become widespread. In this approach, a model is trained using high quality datasets and can then be re-used with fewer labeled examples for other applications in a similar domain. For instance, the Inception algorithm, trained by Google to classify images from 1000 different categories2, is now widely used to score conceptual similarity between different sets of images3.

Supervised deep learning algorithms, as well as transfer learning, have found several applications in neuroscience. Pose estimation algorithms4,5,6,7 use deep learning to track animals’ body parts from video data. This obviates the need to apply visible markers on the animals’ body, and considerably eases the burden of manually scoring different behaviors (common in e.g. ethology). Instead of labeling many thousands of images, a researcher can now label as few as ~200 video frames, and the pose estimation algorithm then tracks the movement of these same body parts in the remaining video. Such algorithms have recently been applied to video data from various species, and have become a popular approach for analyzing rich behavioral data8.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Virtual mouse workflow.

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Nature 521, 436–444 (2015).

    Article  CAS  Google Scholar 

  2. Zaccone, G. & Karim, M. R. Deep Learning with TensorFlow: Explore neural networks and build intelligent systems with Python, 2nd Edition. (Packt Publishing Ltd, 2018).

  3. Salimans, T. et al. arXiv 1606.03498 (2016).

  4. Mathis, A. et al. Nat. Neurosci. 21, 1281 (2018).

    Article  CAS  Google Scholar 

  5. Pereira, T. D. et al. Nat. Methods s16, 117–125 (2019).

    Article  Google Scholar 

  6. Graving, J. M. et al. eLife 8, e47994 (2019).

    Article  CAS  Google Scholar 

  7. Wu, A. et al. bioRxiv 259705 (2020). https://doi.org/10.1101/2020.08.20.259705

  8. Musall, S., Urai, A. E., Sussillo, D. & Churchland, A. K. Curr. Opin. Neurobiol. 58, 229–238 (2019).

    Article  CAS  Google Scholar 

  9. Bolaños, L. A. et al. Nat. Methods 18, 378–381 (2021).

    Article  Google Scholar 

  10. Tobin, J. et al. in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 23–30 (2017). https://doi.org/10.1109/IROS.2017.8202133

  11. The International Brain Laboratory et al. eLife 63711, (2021).

  12. The International Brain Laboratory. 3D View Behavioral Training Rig. Figshare (2020). https://doi.org/10.6084/m9.figshare.13042574

  13. Community, B. O. Blender - a 3D modelling and rendering package. (2018). http://www.blender.org

  14. Bińkowski, M., Sutherland, D. J., Arbel, M. & Gretton, A. arXiv, 1801.01401 (2021).

  15. Barratt, S. & Sharma, R. arXiv, 1801.01973 (2018).

  16. de Vries, S. E. J. et al. Nat. Neurosci. 23, 138–151 (2020).

    Article  Google Scholar 

  17. Siegle, J. H. et al. Nature 592, 86–92 (2021).

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank Liam Paninski and Matthew Whiteway for comments on the manuscript. AEU is supported by the German National Academy of Sciences Leopoldina and the International Brain Research Organization.

Author information

Author notes

  1. These authors contributed equally to this work: Guido T. Meijer, Jaime Arlandis.

Authors and Affiliations

  1. Champalimaud Center for the Unknown, Lisbon, Portugal

    Guido T. Meijer & Jaime Arlandis

  2. Cognitive Psychology Unit, Institute of Psychology and Leiden Institute for Brain and Cognition, Leiden University, Leiden, The Netherlands

    Anne E. Urai

Authors
  1. Guido T. Meijer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaime Arlandis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anne E. Urai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anne E. Urai.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meijer, G.T., Arlandis, J. & Urai, A.E. There is no mouse: using a virtual mouse to generate training data for video-based pose estimation. Lab Anim 50, 172–173 (2021). https://doi.org/10.1038/s41684-021-00794-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41684-021-00794-z

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing