Long-duration animal tracking in difficult lighting conditions

High-throughput analysis of animal behavior requires software to analyze videos. Such software typically depends on the experiments’ being performed in good lighting conditions, but this ideal is difficult or impossible to achieve for certain classes of experiments. Here, we describe techniques that allow long-duration positional tracking in difficult lighting conditions with strong shadows or recurring “on”/“off” changes in lighting. The latter condition will likely become increasingly common, e.g., for Drosophila due to the advent of red-shifted channelrhodopsins. The techniques enabled tracking with good accuracy in three types of experiments with difficult lighting conditions in our lab. Our technique handling shadows relies on single-animal tracking and on shadows’ and flies’ being accurately distinguishable by distance to the center of the arena (or a similar geometric rule); the other techniques should be broadly applicable. We implemented the techniques as extensions of the widely-used tracking software Ctrax; however, they are relatively simple, not specific to Drosophila, and could be added to other trackers as well.

Unmodified Ctrax cannot handle tracking with recurring changes between two different lighting states ("on"/"off") since it uses a single background.We hence extended Ctrax with a simple on/off detector.The detector randomly picks 100 frames, calculates the mean (average brightness) for each, and classifies the means into two clusters ("on"/"off") via k-means.If the cluster centroids differ by more than 3% in brightness, the detector assumes there are two lighting states, a separate background is calculated for each state (Supplementary Fig. 2b), and the right background is chosen for each frame during tracking.The detector also extends the Ctrax trajectory output file with the information about the timing of the lighting state changes it learned from the video.Our suspicious jump detector regularly detected tracking errors coinciding with lighting state changes, so we extended it to automatically fix such errors (Supplementary Fig. 3c-d).Combining our extensions enabled tracking with good accuracy in both our "UV on/off" (Fig. 1j) and "strong red light on/off" experiments.
Our Ctrax extensions (based on Ctrax 0.3.1)are available as project yanglab-ctrax on Google Code (https://code.google.com/p/yanglab-ctrax/).The extensions are relatively simple and worked well for our experiments.Our shadow detector relies on having a single animal "per chamber."Recent more advanced techniques 5,6 may enable reliable shadow detection in multiple animal tracking at additional implementation cost.Our other extensions -background recalculation, on/off detector, and auto-fixing jump detector -should be widely applicable.(a) Sample frame from our "UV on" experiments, showing two chambers.For the left chamber, the top edge of the chamber sidewall is outlined in yellow, and the two egg-laying sites at the bottom of the chamber are outlined in white.The blue arrow points to a UV LED (below the chamber in our setup).Note that there is one fly per chamber (white arrows).A red "light pad" provides additional lightingthat is invisible to Drosophila -for tracking.
(b) Sample frame from our "UV on/off" experiments at a time when the UV is off.With UV on, the frame would look similar to (a).The small dark spots on the egg-laying sites are eggs (arrows).(e-h) Sample frame with strong shadows that led to false positives, with white and green arrows pointing to flies.(e) Frame in grayscale, which Ctrax uses for tracking.(f) Difference between frame (e) and background ("frame without flies"), with darkness proportional to the absolute value of the difference.The shadows (blue arrows) of the right fly have a larger difference than the fly itself.(We used Ctrax's "Background Brightness" normalization, which performed best for our chambers.)(g) The same difference as in (f) is now shown in green and superimposed onto the background.(h) Flies detected by Ctrax shown as ellipses.For each chamber, our shadow detector picks only one ellipse (fly) -the one closest to the center (yellow arrow) of the chamber, which eliminates all false positives in this frame.
(i) Results of tracking using unmodified Ctrax and Ctrax with extensions on four "UV on" sample videos (8h each).Ctrax with extensions correctly detected just two flies for each video, while unmodified Ctrax detected hundreds of flies.We manually examined the 12 jumps our suspicious jump detector reported for the four videos (Supplementary Fig. 3b), and tracking was correct in all cases (i.e., the flies did jump in these cases).Note that neither table (i) nor (j) lists "minor" tracking errors that were below the detection threshold of the suspicious jump detector (Supplementary Fig. 3e).
(j) Results of tracking using Ctrax with extensions on three "UV on/off" sample videos (8h each).(We did not run unmodified Ctrax since it was not designed to handle "on"/"off" changes.)"1 min on/off" was 1 min "on," 1 min "off," 1 min "on," etc.The correct number of flies was detected for each video.We manually examined all jumps our suspicious jump detector either automatically fixed (Supplementary Fig. 3c-d) or just reported.Of the 44 jumps it automatically fixed for the three videos, it made one error but was correct in the remaining 43 cases.So the "auto fix" feature strongly reduced tracking errors here.For the 9 jumps the detector just reported, tracking was correct.

Figure 1
Figure 1 Tracking in difficult lighting conditions and performance of the Ctrax extensions.

(c- d )
Sample frames from our experiments using strong red light to optogenetically activate neurons, with arrows pointing to flies.(c) Two long chambers (one outlined in yellow) illuminated with strong red light; there is no red visible in the image since -to reduce light intensity for the camera -a filter that lets only blue light (400-500nm) pass (LEE Filters, 713 J.Winter Blue) was placed in front of the camera.(d) Same chambers as in (c) with red light turned off.

F
p l e me n t a r y F i g u r e 1 b a S u p p l e me n t a r y F i g u r e