Natural events present multiple types of sensory cues, each detected by a specialized sensory modality. Combining information from several modalities is essential for the selection of appropriate actions. Key to understanding multimodal computations is determining the structural patterns of multimodal convergence and how these patterns contribute to behaviour. Modalities could converge early, late or at multiple levels in the sensory processing hierarchy. Here we show that combining mechanosensory and nociceptive cues synergistically enhances the selection of the fastest mode of escape locomotion in Drosophila larvae. In an electron microscopy volume that spans the entire insect nervous system, we reconstructed the multisensory circuit supporting the synergy, spanning multiple levels of the sensory processing hierarchy. The wiring diagram revealed a complex multilevel multimodal convergence architecture. Using behavioural and physiological studies, we identified functionally connected circuit nodes that trigger the fastest locomotor mode, and others that facilitate it, and we provide evidence that multiple levels of multimodal integration contribute to escape mode selection. We propose that the multilevel multimodal convergence architecture may be a general feature of multisensory circuits enabling complex input–output functions and selective tuning to ecologically relevant combinations of cues.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.


  1. 1.

    & The Merging of the Senses (MIT press, 1993)

  2. 2.

    , & Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nature Rev. Neurosci.. 14, 429–442 (2013)

  3. 3.

    , , & Multisensory integration: flexible use of general operations. Neuron 81, 1240–1253 (2014)

  4. 4.

    , , & Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. J. Cogn. Neurosci.. 1, 12–24 (1989)

  5. 5.

    , , , & Multimodal integration of carbon dioxide and other sensory cues drives mosquito attraction to humans. Cell 156, 1060–1071 (2014)

  6. 6.

    & Multisensory integration: current issues from the perspective of the single neuron. Nature Rev. Neurosci.. 9, 255–266 (2008)

  7. 7.

    Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)

  8. 8.

    H. P. &. Stork, D. G. Pattern Classification (John Wiley and Sons, 2001)

  9. 9.

    , & The Elements of Statistical Learning Vol. 2 (Springer, 2009)

  10. 10.

    et al. The importance of mixed selectivity in complex cognitive tasks. Nature 497, 585–590 (2013)

  11. 11.

    , , & Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146–154 (2012)

  12. 12.

    & Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983)

  13. 13.

    & Descending efferents from the superior colliculus relay integrated multisensory information. Science 227, 657–659 (1985)

  14. 14.

    , & Cellular and synaptic architecture of multisensory integration in the mouse neocortex. Neuron 79, 579–593 (2013)

  15. 15.

    , & Structure and function of the deutocerebrum in insects. Annu. Rev. Entomol.. 34, 477–501 (1989)

  16. 16.

    & Cluster organization and response characteristics of the giant fiber pathway of the blowfly Calliphora erythrocephala. J. Comp. Neurol.. 294, 59–75 (1990)

  17. 17.

    , , & Responses of Drosophila giant descending neurons to visual and mechanical stimuli. J. Exp. Biol.. 217, 2121–2129 (2014)

  18. 18.

    , & Multi-sensory integration in brainstem and auditory cortex. Brain Res. 1485, 95–107 (2012)

  19. 19.

    & External inferior colliculus integrates trigeminal and acoustic information: unit responses to trigeminal nucleus and acoustic stimulation in the guinea pig. Neurosci. Lett.. 395, 71–75 (2006)

  20. 20.

    , , , & Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007)

  21. 21.

    Multisensory integration in the dorsal cochlear nucleus: unit responses to acoustic and trigeminal ganglion stimulation. Eur. J. Neurosci.. 21, 3334–3348 (2005)

  22. 22.

    et al. An integrated micro- and macroarchitectural analysis of the Drosophila brain by computer-assisted serial section electron microscopy. PLoS Biol. 8, e1000502 (2010)

  23. 23.

    , , & Elastic volume reconstruction from series of ultra-thin microscopy sections. Nature Methods 9, 717–720 (2012)

  24. 24.

    et al. Tools for neuroanatomy and neurogenetics in Drosophila. Proc. Natl Acad. Sci.. USA 105, 9715–9720 (2008)

  25. 25.

    et al. Refinement of tools for targeted gene expression in Drosophila. Genetics 186, 735–755 (2010)

  26. 26.

    et al. A GAL4 driver resource for developmental and behavioral studies on the larval CNS of Drosophila. Cell Rep. 8, 897–908 (2014)

  27. 27.

    et al. Nociceptive neurons protect Drosophila larvae from parasitoid wasps. Curr. Biol. 17, 2105–2116 (2007)

  28. 28.

    et al. High-throughput analysis of stimulus-evoked behaviors in Drosophila larva reveals multiple modality-specific escape strategies. PLoS ONE 8, e71706 (2013)

  29. 29.

    , & Dendritic filopodia, Ripped Pocket, NOMPC, and NMDARs contribute to the sense of touch in Drosophila larvae. Curr. Biol. 22, 2124–2134 (2012)

  30. 30.

    et al. Drosophila NOMPC is a mechanotransduction channel subunit for gentle-touch sensation. Nature 493, 221–225 (2013)

  31. 31.

    , , & Sound response mediated by the TRP channels NOMPC, NANCHUNG, and INACTIVE in chordotonal organs of Drosophila larvae. Proc. Natl Acad. Sci.. USA 110, 13612–13617 (2013)

  32. 32.

    , & Larval defense against attack from parasitoid wasps requires nociceptive neurons. PLoS ONE 8, e78704 (2013)

  33. 33.

    , , & painless, a Drosophila gene essential for nociception. Cell 113, 261–273 (2003)

  34. 34.

    et al. Discovery of brainwide neural–behavioral maps via multiscale unsupervised structure learning. Science 344, 386–392 (2014)

  35. 35.

    et al. Light-avoidance-mediating photoreceptors tile the Drosophila larval body wall. Nature 468, 921–926 (2010)

  36. 36.

    et al. Thermosensory and nonthermosensory isoforms of Drosophila melanogaster TRPA1 reveal heat-sensor domains of a thermoTRP Channel. Cell Rep. 1, 43–55 (2012)

  37. 37.

    & Central projections of Drosophila sensory neurons in the transition from embryo to larva. J. Comp. Neurol. 425, 34–44 (2000)

  38. 38.

    , , , & Positional cues in the Drosophila nerve cord: semaphorins pattern the dorso-ventral axis. PLoS Biol. 7, e1000135 (2009)

  39. 39.

    et al. Projections of Drosophila multidendritic neurons in the central nervous system: links with peripheral dendrite morphology. Development 134, 55–64 (2007)

  40. 40.

    et al. Independent optical excitation of distinct neural populations. Nature Methods 11, 338–346 (2014)

  41. 41.

    & Merging the senses into a robust percept. Trends Cogn. Sci.. 8, 162–169 (2004)

  42. 42.

    , , & Response properties and organization of nociceptive neurons in area 1 of monkey primary somatosensory cortex. J. Neurophysiol. 84, 719–729 (2000)

  43. 43.

    Physiological properties of unmyelinated fiber projection to the spinal cord. Exp. Neurol. 16, 316–332 (1966)

  44. 44.

    , & A normalization model of multisensory integration. Nature Neurosci. 14, 775–782 (2011)

  45. 45.

    & From the connectome to brain function. Nature Methods 10, 483–490 (2013)

  46. 46.

    , , & The structure of the nervous system of the nematode Caenorhabditis elegans. Phil. Trans. R. Soc. Lond. B 314, 1–340 (1986)

  47. 47.

    et al. Network anatomy and in vivo physiology of visual cortical neurons. Nature 471, 177–182 (2011)

  48. 48.

    , & Wiring specificity in the direction-selectivity circuit of the retina. Nature 471, 183–188 (2011)

  49. 49.

    et al. Space-time wiring specificity supports direction selectivity in the retina. Nature 509, 331–336 (2014)

  50. 50.

    et al. A visual motion detection circuit suggested by Drosophila connectomics. Nature 500, 175–181 (2013)

  51. 51.

    et al. A GAL4-driver line resource for Drosophila neurobiology. Cell Rep. 2, 991–1001 (2012)

  52. 52.

    et al. An internal thermal sensor controlling temperature preference in Drosophila. Nature 454, 217–220 (2008)

  53. 53.

    et al. Modulation of TRPA1 thermal sensitivity enables sensory discrimination in Drosophila. Nature 481, 76–80 (2012)

  54. 54.

    , & Using translational enhancers to increase transgene expression in Drosophila. Proc. Natl Acad. Sci. USA 109, 6626–6631 (2012)

  55. 55.

    et al. Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499, 295–300 (2013)

  56. 56.

    et al. Imaging neural activity in worms, flies and mice with improved GCaMP calcium indicators. Nature Methods 6, 875–881 (2009)

  57. 57.

    et al. Enhanced locomotion caused by loss of the Drosophila DEG/ENaC protein Pick-pocket1. Curr. Biol. 13, 1557–1563 (2003)

  58. 58.

    , , , & Altered electrical properties in Drosophila neurons developing without synaptic transmission. J. Neurosci. 21, 1523–1531 (2001)

  59. 59.

    & Remote control of behavior through genetically targeted photostimulation of neurons. Cell 121, 141–152 (2005)

  60. 60.

    et al. Neuronal control of Drosophila courtship song. Neuron 69, 509–522 (2011)

  61. 61.

    , , & Visualization of gene expression in living adult Drosophila. Science 274, 252–255 (1996)

  62. 62.

    , , & Refined spatial manipulation of neuronal function by combinatorial restriction of transgene expression. Neuron 52, 425–436 (2006)

  63. 63.

    & Organizing activity of wingless protein in Drosophila. Cell 72, 527–540 (1993)

  64. 64.

    , , & High-throughput behavioral analysis in C. elegans. Nature Methods 8, 592–598 (2011)

  65. 65.

    , , , & JAABA: interactive machine learning for automatic annotation of animal behavior. Nature Methods 10, 64–67 (2013)

  66. 66.

    et al. NIH Image to ImageJ: 25 years of image analysis. Nature Methods 9, 671–675 (2012)

  67. 67.

    , & in Exocytosis and Endocytosis 349–369 (Springer, 2008)

  68. 68.

    An analysis of histogram-based thresholding algorithms. CVGIP. Graph. Models Image Proc. 55, 532–537 (1993)

  69. 69.

    A modified method for lead staining of thin sections. J. Electron Microsc. (Tokyo) 17, 158–159 (1968)

  70. 70.

    et al. Automated molecular microscopy: the new Leginon system. J. Struct. Biol. 151, 41–60 (2005)

  71. 71.

    , , & CATMAID: collaborative annotation toolkit for massive amounts of image data. Bioinformatics 25, 1984–1986 (2009)

  72. 72.

    & Development and structure of synaptic contacts in Drosophila. Semin. Cell Dev. Biol. 17, 20–30 (2006)

  73. 73.

    , , , & Charting the Drosophila neuropile: a strategy for the standardised characterization of genetically amenable neurites. Dev. Biol. 260, 207–225 (2003)

  74. 74.

    , & Tiling of the Drosophila epidermis by multidendritic sensory neurons. Development 129, 2867–2878 (2002)

  75. 75.

    et al. Gephi: an open source software for exploring and manipulating networks (International AAAI Conference on Weblogs and Social Media, 2009)

  76. 76.

    , & Embryonic development of identified neurons: Segment-specific differences in the H cell homologues. J. Neurosci. 1, 103–106 (1981)

  77. 77.

    et al. Comprehensive maps of Drosophila higher olfactory centers: spatially segregated fruit and pheromone representation. Cell 128, 1187–1203 (2007)

  78. 78.

    & Theoretical Neuroscience (MIT Press, 2001)

Download references


We thank G. Rubin and the Janelia Fly EM Project for the gift of the comprehensive image dataset of the larval nervous system; S. Lauchie and A. Brothers for assistance with EM imaging; B. Arruda and T. Dang for assistance with behavioural screens; L. Herren, I. Andrade, K. Floria and A. Berthold van der Bourg for assistance with neuronal reconstruction; G. Rubin, H. Dionne and B. Pfeiffer for GAL4 and Split GAL4 lines; A. Nern and G. Rubin for the single cell FLP-out stocks; J.-M. Knapp for Tsh-LexA stock; H.-H. Li and Janelia Fly Light Project Team for images of neuronal lines; K. Hibbard, M. Mercer, T. Laverty and the rest of Janelia Fly Core for stock construction and fly crosses; G. Denisov for the roll and crawl detection LARA software; E. Trautman, R. Svirskas, C. Weaver and D. Olbris for data analysis pipelines; Y. Park, C. Priebe, D. Naiman and J.-B. Masson for advice on statistical analysis; V. Jayaraman for input on calcium imaging; and W. Denk, B. Dickson, S. Druckmann, B. Gerber, K. Svoboda and C. Zuker for helpful comments on the manuscript. We thank Janelia HHMI for funding. A.C. and C.M.S.-M. were also funded by the Institute of Neuroinformatics of the University of Zurich and ETH Zurich, the SNSF grant 31003A_132969, the Universität Zürich Forschungskredit, and the HHMI Visiting Scientist program at Janelia. The EM image data is available via the Open Connectome Project (

Author information

Author notes

    • Tomoko Ohyama
    •  & Casey M. Schneider-Mizell

    These authors contributed equally to this work.

    • Albert Cardona
    •  & Marta Zlatic

    These authors jointly supervised this work.


  1. Howard Hughes Medical Institute Janelia Research Campus, 19700 Helix Drive, Ashburn, Virginia 20147, USA

    • Tomoko Ohyama
    • , Casey M. Schneider-Mizell
    • , Richard D. Fetter
    • , Javier Valdes Aleman
    • , Romain Franconville
    • , Marta Rivera-Alba
    • , Brett D. Mensh
    • , Kristin M. Branson
    • , Julie H. Simpson
    • , James W. Truman
    • , Albert Cardona
    •  & Marta Zlatic


  1. Search for Tomoko Ohyama in:

  2. Search for Casey M. Schneider-Mizell in:

  3. Search for Richard D. Fetter in:

  4. Search for Javier Valdes Aleman in:

  5. Search for Romain Franconville in:

  6. Search for Marta Rivera-Alba in:

  7. Search for Brett D. Mensh in:

  8. Search for Kristin M. Branson in:

  9. Search for Julie H. Simpson in:

  10. Search for James W. Truman in:

  11. Search for Albert Cardona in:

  12. Search for Marta Zlatic in:


T.O., C.M.S.-M., A.C. and M.Z. conceived the project, analysed the data and wrote the manuscript. T.O. performed and analysed behavioural and functional imaging experiments. C.M.S.-M., J.V.A. and A.C. performed neuronal reconstructions. A.C. registered the L1 volume. J.W.T. analysed the expression patterns of all the GAL4 lines and intersections for targeting of single cell types. R.D.F. generated the EM image data. M.R.A. and K.B. wrote the JAABA roll detection pipeline and performed statistical analysis. J.H.S. supported the generation of the abd1.5 dataset. R.F. provided critical suggestions for functional imaging experiments. C.M.S.-M. built the model. B.D.M. provided critical input and helped with writing the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to Albert Cardona or Marta Zlatic.

Extended data

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    This file contains the Supplementary Atlas.

Zip files

  1. 1.

    Supplementary Information

    This zipped file contains Supplementary Tables 1-6 and a Supplementary Table Guide.


  1. 1.

    MD IV activation alone

    Contours of larvae automatically tracked with the MWT software in the presence of continuous MD IV activation alone in MD IV>dTRPA1 larvae (32oC). The activation onset was at 0 seconds, the video shows the period from 5 seconds to 20 seconds. Automatic detection of rolling using JAABA is shown. In all videos of larval behavior: pink contours indicate JAABA detected larva is rolling at the time; green contours, LARA software detected larva is crawling at that time; blue contours, larvae are neither rolling, nor crawling – they may be pausing or turning. Note some false negatives – blue contours in larvae that are rolling or crawling. Some larvae roll in response to MD IV activation. The majority of larvae are crawling forward.

  2. 2.

    Vibration alone

    Contours of larvae in the presence of continuous 1000 Hz, 6.7 m/s2 vibration alone in MD IV>+ larvae (32oC). The activation onset was at 0 sec, this video shows the period from 5 sec to 20 sec. Almost no rolling is observed in response to vibration. Instead most larvae are crawling forward or turning. Most larvae have blue contours indicating JAABA did not detect rolling events (correctly). Note an occasional pink contour in a non-rolling larva indicating a false positive.

  3. 3.

    MD IV activation combined with vibration

    Contours of larvae automatically tracked with the MWT software in the presence of continuous vibration (1000 Hz, 6.7 m/s2) and MD IV activation in MD IV>dTRPA1 larvae (32oC). The activation onset was at 0 sec, the video shows the period from 5 sec to 20 sec. A large number of larvae are rolling and the rolling events are correctly detected by JAABA (pink contours).

  4. 4.

    Basin activation

    Contours of larvae automatically tracked with the MWT software in the presence of continuous thermogenetic activation of Basins in R72F11>dTRPA1 larvae (32oC). The activation onset was at 0 sec, the video shows the period from 5 sec to 20 sec. A large number of larvae are rolling and the rolling events are correctly detected by JAABA (pink contours).

  5. 5.

    Effector control

    Contours of larvae automatically tracked with the MWT software in control attp2>dTRPA1 larvae (32oC). The temperature was raised at 0 sec and this video shows the period from 5 sec to 20 sec. No larvae are rolling and no contours are pink.

  6. 6.

    Goro activation

    Contours of larvae automatically tracked with the MWT software in the presence of continuous thermogenetic activation of Goro in R69F06>dTRPA1 larvae (32oC). The activation onset was at 0 sec, this video shows the period from 5 sec to 20 sec. A large number of larvae are rolling and the rolling events are correctly detected by JAABA (pink contours).

  7. 7.

    EM volume spanning the entire nervous system of Drosophila larva.

    This video illustrates the complete central nervous system of the first instar larva of Drosophila melanogaster, imaged with transmission electron microscopy. The volume consists of 4840 serial sections of 50 nm in thickness, prepared by Rick D. Fetter and imaged at a resolution of 3.8 x 3.8 nanometers/pixel with the semi-automatic EM imaging software Leginon by Rick D. Fetter and two assistants, Shirley Launchie and Andrea Brothers. The resulting 144,953 image tiles of 4096x4096 pixels each were registered by Albert Cardona and Stephan Saalfeld using the elastic serial section registration methods available in the software TrakEM2.

  8. 8.

    Rotation of the EM reconstruction of the Goro cell shown in Figure 5b

    Black lines indicate arbor, cyan marks show input synapses, red marks show presynaptic sites and the ball indicates the cell body. Dorsal is up.

About this article

Publication history





Further reading


By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.