Standard animal behavior paradigms incompletely mimic nature and thus limit our understanding of behavior and brain function. Virtual reality (VR) can help, but it poses challenges. Typical VR systems require movement restrictions but disrupt sensorimotor experience, causing neuronal and behavioral alterations. We report the development of FreemoVR, a VR system for freely moving animals. We validate immersive VR for mice, flies, and zebrafish. FreemoVR allows instant, disruption-free environmental reconfigurations and interactions between real organisms and computer-controlled agents. Using the FreemoVR platform, we established a height-aversion assay in mice and studied visuomotor effects in Drosophila and zebrafish. Furthermore, by photorealistically mimicking zebrafish we discovered that effective social influence depends on a prospective leader balancing its internally preferred directional choice with social interaction. FreemoVR technology facilitates detailed investigations into neural function and behavior through the precise manipulation of sensorimotor feedback loops in unrestrained animals.
Access optionsAccess options
Subscribe to Journal
Get full journal access for 1 year
only $20.17 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
We thank M. Colombini, A. Fuhrmann, L. Fenk, E. Campione, S. Villalba, and the IMP/IMBA Workshop for help constructing FreemoVR hardware and software. We thank M. Dickinson and T. Klausberger for helpful discussions, V. Böhm for help with experiments, and the MFPL fish facility for fish care. The manual mouse behavior annotation was performed by the Preclinical Phenotyping Facility at Vienna Biocenter Core Facilities. This work was supported by European Research Council (ERC) starting grants 281884 to A.D.S., 311701 to W.H., 337011 to K.T.-R.; Wiener Wissenschafts-, Forschungs- und Technologiefonds (WWTF) grant CS2011-029 to A.D.S.; FWF (http://www.fwf.ac.at/) research project grants P28970 to K.T.-R. and P29077 to K.N.; NSF grants PHY-0848755 to I.D.C., IOS-1355061 to I.D.C., EAGER-IOS-1251585 to I.D.C.; ONR grants N00014-09-1-1074 to I.D.C., N00014-14-1-0635 to I.D.C; ARO grants W911NG-11-1-0385 to I.D.C., W911NF-14-1-0431 to I.D.C. A.D.S and W.H. were further supported by the IMP, Boehringer Ingelheim and the Austrian Research Promotion Agency (FFG). K.T.-R. is supported by grants from the University of Vienna (research platform “Rhythms of Life”). IDC acknowledges further support from the “Struktur- und Innovationsfonds für die Forschung (SI-BW)” of the State of Baden-Württemberg and from the Max Planck Society. I.D.C. and R.B. gratefully acknowledge fish care and technical support from C. Bauer, J. Weglarski, A. Bruttel, and G. Mazué.
Integrated supplementary information
(left) Video taken from a camera (GoPro) showing the view from the perspective of a freely moving observer. (right) The colored L-shaped box virtual world FreemoVR is simulating and the position of the camera in the virtual world (red dot). Once the camera enters the ‘FlyCave’ VR arena its' position is estimated from the tracking software (right, red dot) and the perspective correct VR is projected onto the walls of the arena. As the camera moves, the projection is updated in real-time to maintain a perspective correct display. Reproduced with permission from Stowers et al. 2014.
Related to Video 1. (left) Video taken from above, looking into the ‘FlyCave’ arena, showing the 3D position of the camera (red dot) and the projection onto the arena walls as it moves in space. (right) The virtual world being simulated, and the estimated position of the camera in the virtual world (red dot). Reproduced with permission from Stowers et al. 2014.
(left) Swimming behavior of a zebrafish, its position highlighted in red, as it navigates a virtual world. The fish swims in a hemispherical bowl filled with water. (right) The virtual world being simulated. The world consists of a cyan sphere and a magenta pyramid in a naturalistic environment. As the fish approaches the pyramid the rendering is updated to display a perspective correct view of the world.
(left) An Drosophila flies inside the cylindrical ‘FlyCave’ VR arena. Its' position is tracked and highlighted in red. (right). The virtual world simulated consists of a cyan sphere and a magenta pyramid in a naturalistic environment. As the fly explores the arena the virtual world is updated in real-time to maintain a perspective correct display for the subject.
(left) A flying Drosophila (position highlighted in red) interacts with a virtual vertical gray post. (right) The virtual world being simulated. On the arena walls a checkerboard texture is moved vertically to control the fly's altitude and to prevent it flying into the walls.
A flying Drosophila (position highlighted in red) interacts with real post. On the arena walls a checkerboard texture is moved vertically to control the fly's altitude and to prevent it flying into the walls.
(left) A juvenile Zebrafish (position highlighted in red) interacts with a virtual post. (right) The virtual world being simulated contains a black upright post placed at the center of a sphere covered in a checkerboard pattern.
An unrestrained mouse explores an elevated platform placed above a 75'' consumer television. FreemoVR simulates a virtual world consisting of two platforms placed virtually 20cm and 40cm below the physical platform. By tracking the mouse head position, a perspective correct virtual reality can be displayed to the mouse, retaining naturalistic parallax queues and thus the percept of height to the mouse.
Illuminated and filmed from above, the software detects the position of the mouse head in real-time (indicated in green) and uses this to create a perspective correct virtual reality. The detected mouse contour and center are shown in magenta.
Remote control flies – controlling the behavior of freely flying Drosophila by exploiting the optomotor response
(left) A Drosophila flies in the ‘FlyCave’ VR arena (position highlighted in red). As the fly flies, the virtual world is modified; rotated about its center, eliciting the optomotor response in the subject and causing it to turn. Doing this continuously causes the fly to follow a path of our design, an infinity-symbol (8) shaped path (right).
(left) A zebrafish swims (position highlighted in red) among a cloud of dots. (right) The simulated virtual world containing the 3D cloud of dots. The dots all move with the same velocity. The velocity of the dots is controlled to cause the fish to swim along an infinity-symbol (∞) shaped path. Dot size is 6.2°, double the size of the “large dot” stimulus in Fig. 4 to increase visibility in the video recording.
(left) Wide-angle camera footage of zebrafish swimming in 2AFC teleportation experiment. (right) The simulated virtual world containing either a checkerboard floor or virtual plants with a gravel floor. When a fish makes a decision, operationally defined as entering a teleportation portal (black and white or magenta shape), the fish is virtually teleported to the environment coupled to the portal. Depending on the particular experiment, the specific coupling between portal and destination varies, but remains fixed for each individual fish.
(left) Wide-angle camera footage of zebrafish swimming in 2AFC swarm experiment. (right) The simulated virtual world containing the either a swarm of space invaders or a scene without swarm. When a fish makes a decision, operationally defined as entering a teleportation portal (black and white or magenta shape), the fish is virtually teleported to the environment coupled to the portal. Depending on the particular experiment, the specific coupling between portal and destination varies, but remains fixed for each individual fish.
Camera footage of zebrafish swimming with a virtual fish. The virtual fish is reacting to the position of the real fish. Here, ω=1, the virtual fish equally balances social and goal-oriented behavior.
About this article
The Science of Nature (2018)