Abstract

Standard animal behavior paradigms incompletely mimic nature and thus limit our understanding of behavior and brain function. Virtual reality (VR) can help, but it poses challenges. Typical VR systems require movement restrictions but disrupt sensorimotor experience, causing neuronal and behavioral alterations. We report the development of FreemoVR, a VR system for freely moving animals. We validate immersive VR for mice, flies, and zebrafish. FreemoVR allows instant, disruption-free environmental reconfigurations and interactions between real organisms and computer-controlled agents. Using the FreemoVR platform, we established a height-aversion assay in mice and studied visuomotor effects in Drosophila and zebrafish. Furthermore, by photorealistically mimicking zebrafish we discovered that effective social influence depends on a prospective leader balancing its internally preferred directional choice with social interaction. FreemoVR technology facilitates detailed investigations into neural function and behavior through the precise manipulation of sensorimotor feedback loops in unrestrained animals.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    et al. Impaired spatial selectivity and intact phase precession in two-dimensional virtual reality. Nat. Neurosci. 18, 121–128 (2015).

  2. 2.

    , , & Walking modulates speed sensitivity in Drosophila motion vision. Curr. Biol. 20, 1470–1475 (2010).

  3. 3.

    & Das Reafferenzprinzip - Wechselwirkungen zwischen Zentralnervensystem und Peripherie. Naturwissenschaften 37, 464–476 (1950).

  4. 4.

    , & Flight activity alters velocity tuning of fly motion-sensitive neurons. J. Neurosci. 31, 9231–9237 (2011).

  5. 5.

    , & Cellular evidence for efference copy in Drosophila visuomotor processing. Nat. Neurosci. 18, 1247–1255 (2015).

  6. 6.

    et al. Two-photon calcium imaging in mice navigating a virtual reality environment. J. Vis. Exp. 20, e50885 (2014).

  7. 7.

    Neural basis of the spontaneous optokinetic response produced by visual inversion. J. Comp. Physiol. Psychol. 43, 482–489 (1950).

  8. 8.

    et al. Multisensory control of hippocampal spatiotemporal selectivity. Science 340, 1342–1346 (2013).

  9. 9.

    , , , & Causal influence of visual cues on hippocampal directional selectivity. Cell 164, 197–207 (2016).

  10. 10.

    , , & Intracellular dynamics of hippocampal place cells during virtual navigation. Nature 461, 941–946 (2009).

  11. 11.

    & Cellular mechanisms of spatial navigation in the medial entorhinal cortex. Nat. Neurosci. 16, 325–331 (2013).

  12. 12.

    & Engagement of neural circuits underlying 2D spatial navigation in a rodent virtual reality system. Neuron 84, 442–456 (2014).

  13. 13.

    , , & Natural whisker-guided behavior by head-fixed mice in tactile virtual reality. J. Neurosci. 34, 9537–9550 (2014).

  14. 14.

    , , , & Rats are able to navigate in virtual environments. J. Exp. Biol. 208, 561–569 (2005).

  15. 15.

    , , , & Functional imaging of hippocampal place cells at cellular resolution during virtual navigation. Nat. Neurosci. 13, 1433–1440 (2010).

  16. 16.

    , & Active flight increases the gain of visual motion processing in Drosophila. Nat. Neurosci. 13, 393–399 (2010).

  17. 17.

    et al. Multisensory control of multimodal behavior: do the legs know what the tongue is doing? PLoS One 8, e80465 (2013).

  18. 18.

    , , & Multi-camera real-time three-dimensional tracking of multiple flying animals. J. R. Soc. Interface 8, 395–409 (2011).

  19. 19.

    , , & Visual control of flight speed in Drosophila melanogaster. J. Exp. Biol. 212, 1120–1130 (2009).

  20. 20.

    , & Virtual-reality techniques resolve the visual cues used by fruit flies to evaluate object distances. Curr. Biol. 12, 1591–1594 (2002).

  21. 21.

    , & Visual control of altitude in flying Drosophila. Curr. Biol. 20, 1550–1556 (2010).

  22. 22.

    et al. Reverse engineering animal vision with virtual reality and genetics. Computer 47, 38–45 (2014).

  23. 23.

    , , , & Virtual reality system for freely-moving rodents. Preprint at (2017).

  24. 24.

    , & Distance estimation in the Mongolian gerbil: the role of dynamic depth cues. Behav. Brain Res. 14, 29–39 (1984).

  25. 25.

    & A theory of the pattern induced flight orientation of the fly Musca domestica. Kybernetik 12, 185–203 (1973).

  26. 26.

    , & Binocular interactions underlying the classic optomotor responses of flying flies. Front. Behav. Neurosci. 6, 6 (2012).

  27. 27.

    & Visual motion speed determines a behavioral switch from forward flight to expansion avoidance in Drosophila. J. Exp. Biol. 216, 719–732 (2013).

  28. 28.

    & Head and body stabilization in blowflies walking on differently structured substrates. J. Exp. Biol. 215, 1523–1532 (2012).

  29. 29.

    & Blowfly flight and optic flow. I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481–1490 (1999).

  30. 30.

    , , , & nacre encodes a zebrafish microphthalmia-related protein that regulates neural-crest-derived pigment cell fate. Development 126, 3757–3767 (1999).

  31. 31.

    et al. Brain-wide neuronal dynamics during motor adaptation in zebrafish. Nature 485, 471–477 (2012).

  32. 32.

    et al. Optical physiology and locomotor behaviors of wild-type and nacre zebrafish. Methods Cell Biol. 76, 261–284 (2004).

  33. 33.

    & A crystal-clear zebrafish for in vivo imaging. Sci. Rep. 6, 29490 (2016).

  34. 34.

    et al. Inter-individual and inter-strain variations in zebrafish locomotor ontogeny. PLoS One 8, e70172 (2013).

  35. 35.

    et al. Statistical analysis of zebrafish locomotor response. PLoS One 10, e0139521 (2015).

  36. 36.

    & Sensorimotor decision making in the zebrafish tectum. Curr. Biol. 25, 2804–2814 (2015).

  37. 37.

    , & Characterization of spatial aliasing and contrast sensitivity in peripheral vision. Vision Res. 36, 249–258 (1996).

  38. 38.

    Flocks, herds and schools: a distributed behavioral model. Computer Graphics 21, 25–34 (1987).

  39. 39.

    , , & Effective leadership and decision-making in animal groups on the move. Nature 433, 513–516 (2005).

  40. 40.

    et al. Uninformed individuals promote democratic consensus in animal groups. Science 334, 1578–1580 (2011).

  41. 41.

    , & Predatory fish select for coordinated collective motion in virtual prey. Science 337, 1212–1215 (2012).

  42. 42.

    , , , & Monitoring neural activity with bioluminescence during natural behavior. Nat. Neurosci. 13, 513–520 (2010).

  43. 43.

    et al. Whole-brain activity mapping onto a zebrafish brain atlas. Nat. Methods 12, 1039–1046 (2015).

  44. 44.

    et al. A wireless multi-channel neural amplifier for freely moving animals. Nat. Neurosci. 14, 263–269 (2011).

  45. 45.

    et al. Long-term dynamics of CA1 hippocampal place codes. Nat. Neurosci. 16, 264–266 (2013).

  46. 46.

    et al. KymoRod: a method for automated kinematic analysis of rod-shaped plant organs. Plant J. 88, 468–475 (2016).

  47. 47.

    Linear Control System Analysis and Design: Conventional and Modern (McGraw-Hill, 1995).

  48. 48.

    , & Asymmetric processing of visual motion for simultaneous object and background responses. Curr. Biol. 24, 2913–2919 (2014).

  49. 49.

    , & A convenient multicamera self-calibration for virtual environments. PRESENCE Teleoperators Virtual Environ. 14, 407–422 (2005).

Download references

Acknowledgements

We thank M. Colombini, A. Fuhrmann, L. Fenk, E. Campione, S. Villalba, and the IMP/IMBA Workshop for help constructing FreemoVR hardware and software. We thank M. Dickinson and T. Klausberger for helpful discussions, V. Böhm for help with experiments, and the MFPL fish facility for fish care. The manual mouse behavior annotation was performed by the Preclinical Phenotyping Facility at Vienna Biocenter Core Facilities. This work was supported by European Research Council (ERC) starting grants 281884 to A.D.S., 311701 to W.H., 337011 to K.T.-R.; Wiener Wissenschafts-, Forschungs- und Technologiefonds (WWTF) grant CS2011-029 to A.D.S.; FWF (http://www.fwf.ac.at/) research project grants P28970 to K.T.-R. and P29077 to K.N.; NSF grants PHY-0848755 to I.D.C., IOS-1355061 to I.D.C., EAGER-IOS-1251585 to I.D.C.; ONR grants N00014-09-1-1074 to I.D.C., N00014-14-1-0635 to I.D.C; ARO grants W911NG-11-1-0385 to I.D.C., W911NF-14-1-0431 to I.D.C. A.D.S and W.H. were further supported by the IMP, Boehringer Ingelheim and the Austrian Research Promotion Agency (FFG). K.T.-R. is supported by grants from the University of Vienna (research platform “Rhythms of Life”). IDC acknowledges further support from the “Struktur- und Innovationsfonds für die Forschung (SI-BW)” of the State of Baden-Württemberg and from the Max Planck Society. I.D.C. and R.B. gratefully acknowledge fish care and technical support from C. Bauer, J. Weglarski, A. Bruttel, and G. Mazué.

Author information

Affiliations

  1. Research Institute of Molecular Pathology, Vienna Biocenter, Vienna, Austria.

    • John R Stowers
    • , Maximilian Hofbauer
    • , Johannes Griessner
    • , Peter Higgins
    • , Wulf Haubensak
    •  & Andrew D Straw
  2. loopbio gmbh, Kritzendorf, Austria.

    • John R Stowers
    •  & Maximilian Hofbauer
  3. Max F. Perutz Laboratories, University of Vienna, Vienna, Austria.

    • Maximilian Hofbauer
    • , Sarfarazhussain Farooqui
    • , Ruth M Fischer
    •  & Kristin Tessmar-Raible
  4. Research Platform “Rhythms of Life,” University of Vienna, Vienna, Austria.

    • Maximilian Hofbauer
    • , Sarfarazhussain Farooqui
    •  & Kristin Tessmar-Raible
  5. Department of Collective Behaviour, Max Planck Institute for Ornithology, Konstanz, Germany.

    • Renaud Bastien
    •  & Iain D Couzin
  6. Chair of Biodiversity and Collective Behaviour, Department of Biology, University of Konstanz, Konstanz, Germany.

    • Renaud Bastien
    •  & Iain D Couzin
  7. Medizinische Universität Wien, Dept. for Internal Medicine I, Wien, Austria.

    • Sarfarazhussain Farooqui
    •  & Karin Nowikovsky
  8. Institute of Biology I and Bernstein Center Freiburg, Faculty of Biology, Albert-Ludwigs-University Freiburg, Freiburg, Germany.

    • Andrew D Straw

Authors

  1. Search for John R Stowers in:

  2. Search for Maximilian Hofbauer in:

  3. Search for Renaud Bastien in:

  4. Search for Johannes Griessner in:

  5. Search for Peter Higgins in:

  6. Search for Sarfarazhussain Farooqui in:

  7. Search for Ruth M Fischer in:

  8. Search for Karin Nowikovsky in:

  9. Search for Wulf Haubensak in:

  10. Search for Iain D Couzin in:

  11. Search for Kristin Tessmar-Raible in:

  12. Search for Andrew D Straw in:

Contributions

A.D.S., K.T.-R., W.H., and I.D.C. conceived the projects. J.R.S., M.H., R.M.F., R.B., and A.D.S. developed the hardware and software and built the apparatus. J.R.S., M.H., R.B., J.G., P.H., S.F., and A.D.S. performed experiments. J.R.S., M.H., R.B., J.G., S.F., W.H., I.D.C., K.T.-R. and A.D.S. performed data analyses. A.D.S., K.T.-R., I.D.C., J.R.S., M.H., and J.G. wrote the manuscript. A.D.S., K.T.-R., W.H., I.D.C., and K.N. funded the work. J.R.S. and M.H. contributed equally to this work. J.G., P.H., and S.F. contributed equally to this work.

Competing interests

J.R.S. and M.H. are executives with loopbio, gmbh, a company offering virtual reality services. The other authors declare no competing financial interests.

Corresponding authors

Correspondence to Kristin Tessmar-Raible or Andrew D Straw.

Integrated supplementary information

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–15 and Supplementary Tables 1–4.

  2. 2.

    Life Sciences Reporting Summary

Zip files

  1. 1.

    Supplementary Software

    Zip file containing software source code and documentation.

Excel files

  1. 1.

    Supplementary Data 1

    Metadata associated with figures and videos

Videos

  1. 1.

    Demonstration of VR from the perspective of a freely moving observer

    (left) Video taken from a camera (GoPro) showing the view from the perspective of a freely moving observer. (right) The colored L-shaped box virtual world FreemoVR is simulating and the position of the camera in the virtual world (red dot). Once the camera enters the ‘FlyCave’ VR arena its' position is estimated from the tracking software (right, red dot) and the perspective correct VR is projected onto the walls of the arena. As the camera moves, the projection is updated in real-time to maintain a perspective correct display. Reproduced with permission from Stowers et al. 2014.

  2. 2.

    Demonstration of multiple-display perspective correct VR

    Related to Video 1. (left) Video taken from above, looking into the ‘FlyCave’ arena, showing the 3D position of the camera (red dot) and the projection onto the arena walls as it moves in space. (right) The virtual world being simulated, and the estimated position of the camera in the virtual world (red dot). Reproduced with permission from Stowers et al. 2014.

  3. 3.

    Photo realistic and naturalistic VR for freely swimming fish

    (left) Swimming behavior of a zebrafish, its position highlighted in red, as it navigates a virtual world. The fish swims in a hemispherical bowl filled with water. (right) The virtual world being simulated. The world consists of a cyan sphere and a magenta pyramid in a naturalistic environment. As the fish approaches the pyramid the rendering is updated to display a perspective correct view of the world.

  4. 4.

    Photo realistic and naturalistic VR for freely flying Drosophila

    (left) An Drosophila flies inside the cylindrical ‘FlyCave’ VR arena. Its' position is tracked and highlighted in red. (right). The virtual world simulated consists of a cyan sphere and a magenta pyramid in a naturalistic environment. As the fly explores the arena the virtual world is updated in real-time to maintain a perspective correct display for the subject.

  5. 5.

    Simulation of a virtual post for freely flying Drosophila

    (left) A flying Drosophila (position highlighted in red) interacts with a virtual vertical gray post. (right) The virtual world being simulated. On the arena walls a checkerboard texture is moved vertically to control the fly's altitude and to prevent it flying into the walls.

  6. 6.

    Interaction of a Drosophila with a real post

    A flying Drosophila (position highlighted in red) interacts with real post. On the arena walls a checkerboard texture is moved vertically to control the fly's altitude and to prevent it flying into the walls.

  7. 7.

    Simulation of a virtual post for freely swimming Zebrafish

    (left) A juvenile Zebrafish (position highlighted in red) interacts with a virtual post. (right) The virtual world being simulated contains a black upright post placed at the center of a sphere covered in a checkerboard pattern.

  8. 8.

    A virtual elevated maze paradigm for freely moving mice

    An unrestrained mouse explores an elevated platform placed above a 75'' consumer television. FreemoVR simulates a virtual world consisting of two platforms placed virtually 20cm and 40cm below the physical platform. By tracking the mouse head position, a perspective correct virtual reality can be displayed to the mouse, retaining naturalistic parallax queues and thus the percept of height to the mouse.

  9. 9.

    Mouse head tracking

    Illuminated and filmed from above, the software detects the position of the mouse head in real-time (indicated in green) and uses this to create a perspective correct virtual reality. The detected mouse contour and center are shown in magenta.

  10. 10.

    Remote control flies – controlling the behavior of freely flying Drosophila by exploiting the optomotor response

    (left) A Drosophila flies in the ‘FlyCave’ VR arena (position highlighted in red). As the fly flies, the virtual world is modified; rotated about its center, eliciting the optomotor response in the subject and causing it to turn. Doing this continuously causes the fly to follow a path of our design, an infinity-symbol (8) shaped path (right).

  11. 11.

    Zebrafish swims among a cloud of 3D dots

    (left) A zebrafish swims (position highlighted in red) among a cloud of dots. (right) The simulated virtual world containing the 3D cloud of dots. The dots all move with the same velocity. The velocity of the dots is controlled to cause the fish to swim along an infinity-symbol (∞) shaped path. Dot size is 6.2°, double the size of the “large dot” stimulus in Fig. 4 to increase visibility in the video recording.

  12. 12.

    Zebrafish in 2AFC teleportation experiment

    (left) Wide-angle camera footage of zebrafish swimming in 2AFC teleportation experiment. (right) The simulated virtual world containing either a checkerboard floor or virtual plants with a gravel floor. When a fish makes a decision, operationally defined as entering a teleportation portal (black and white or magenta shape), the fish is virtually teleported to the environment coupled to the portal. Depending on the particular experiment, the specific coupling between portal and destination varies, but remains fixed for each individual fish.

  13. 13.

    Zebrafish in 2AFC swarm teleportation experiment

    (left) Wide-angle camera footage of zebrafish swimming in 2AFC swarm experiment. (right) The simulated virtual world containing the either a swarm of space invaders or a scene without swarm. When a fish makes a decision, operationally defined as entering a teleportation portal (black and white or magenta shape), the fish is virtually teleported to the environment coupled to the portal. Depending on the particular experiment, the specific coupling between portal and destination varies, but remains fixed for each individual fish.

  14. 14.

    Social feedback experiment with real and virtual fish

    Camera footage of zebrafish swimming with a virtual fish. The virtual fish is reacting to the position of the real fish. Here, ω=1, the virtual fish equally balances social and goal-oriented behavior.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nmeth.4399

Further reading