Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.
Subscribe to Journal
Get full journal access for 1 year
only $21.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Iber, C., Ancoli-Israel, S., Chesson, A. & Quan, S.F. AASM Manual for the Scoring of Sleep and Associated Events 1st edn. (American Academy of Sleep Medicine, 2007).
Silverstein, L.D. & Levy, C.M. The stability of the sigma sleep spindle. Electroencephalogr. Clin. Neurophysiol. 40, 666–670 (1976).
Tan, X., Campbell, I.G. & Feinberg, I. Internight reliability and benchmark values for computer analyses of non-rapid eye movement (NREM) and REM EEG in normal young adult and elderly subjects. Clin. Neurophysiol. 112, 1540–1552 (2001).
Werth, E., Achermann, P., Dijk, D.J. & Borbély, A.A. Spindle frequency activity in the sleep EEG: individual differences and topographic distribution. Electroencephalogr. Clin. Neurophysiol. 103, 535–542 (1997).
De Gennaro, L., Ferrara, M., Vecchio, F., Curcio, G. & Bertini, M. An electroencephalographic fingerprint of human sleep. Neuroimage 26, 114–122 (2005).
De Gennaro, L. & Ferrara, M. Sleep spindles: an overview. Sleep Med. Rev. 7, 423–440 (2003).
Shibagaki, M., Kiyono, S. & Watanabe, K. Spindle evolution in normal and mentally retarded children: a review. Sleep 5, 47–57 (1982).
Crowley, K., Trinder, J., Kim, Y., Carrington, M. & Colrain, I.M. The effects of normal aging on sleep spindle and K-complex production. Clin. Neurophysiol. 113, 1615–1622 (2002).
Nicolas, A., Petit, D., Rompré, S. & Montplaisir, J. Sleep spindle characteristics in healthy subjects of different age groups. Clin. Neurophysiol. 112, 521–527 (2001).
Martin, N. et al. Topography of age-related changes in sleep spindles. Neurobiol. Aging 34, 468–476 (2013).
De Gennaro, L. et al. The electroencephalographic fingerprint of sleep is genetically determined: a twin study. Ann. Neurol. 64, 455–460 (2008).
Ambrosius, U. et al. Heritability of sleep electroencephalogram. Biol. Psychiatry 64, 344–348 (2008).
Fogel, S.M. & Smith, C.T. The function of the sleep spindle: a physiological index of intelligence and a mechanism for sleep-dependent memory consolidation. Neurosci. Biobehav. Rev. 35, 1154–1165 (2011).
Walker, M.P. The role of sleep in cognition and emotion. Ann. NY Acad. Sci. 1156, 168–197 (2009).
Diekelmann, S. & Born, J. The memory function of sleep. Nat. Rev. Neurosci. 11, 114–126 (2010).
Barakat, M. et al. Fast and slow spindle involvement in the consolidation of a new motor sequence. Behav. Brain Res. 217, 117–121 (2011).
Ferrarelli, F. et al. Reduced sleep spindle activity in schizophrenia patients. Am. J. Psychiatry 164, 483–492 (2007).
Wamsley, E.J. et al. Reduced sleep spindles and spindle coherence in schizophrenia: mechanisms of impaired memory consolidation? Biol. Psychiatry 71, 154–161 (2012).
Limoges, E., Mottron, L., Bolduc, C., Berthiaume, C. & Godbout, R. Atypical sleep architecture and the autism phenotype. Brain 128, 1049–1061 (2005).
Myatchin, I. & Lagae, L. Sleep spindle abnormalities in children with generalized spike-wave discharges. Pediatr. Neurol. 36, 106–111 (2007).
Montagna, P., Gambetti, P., Cortelli, P. & Lugaresi, E. Familial and sporadic fatal insomnia. Lancet Neurol. 2, 167–176 (2003).
Espa, F., Ondze, B., Deglise, P., Billiard, M. & Besset, A. Sleep architecture, slow wave activity, and sleep spindles in adult patients with sleepwalking and sleep terrors. Clin. Neurophysiol. 111, 929–939 (2000).
Himanen, S.-L., Virkkala, J., Huupponen, E. & Hasan, J. Spindle frequency remains slow in sleep apnea patients throughout the night. Sleep Med. 4, 229–234 (2003).
Petit, D., Gagnon, J.-F., Fantini, M.L., Ferini-Strambi, L. & Montplaisir, J. Sleep and quantitative EEG in neurodegenerative disorders. J. Psychosom. Res. 56, 487–496 (2004).
Ferrara, M., Moroni, F., De Gennaro, L. & Nobili, L. Hippocampal sleep features: relations to human memory function. Front. Neurol. 3, 57 (2012).
Steriade, M. Grouping of brain rhythms in corticothalamic systems. Neuroscience 137, 1087–1106 (2006).
Vukadinovic, Z. Sleep abnormalities in schizophrenia may suggest impaired trans-thalamic cortico-cortical communication: towards a dynamic model of the illness. Eur. J. Neurosci. 34, 1031–1039 (2011).
Schimicek, P., Zeitlhofer, J., Anderer, P. & Saletu, B. Automatic sleep-spindle detection procedure: aspects of reliability and validity. Clin. Electroencephalogr. 25, 26–29 (1994).
Huupponen, E. et al. Optimization of sigma amplitude threshold in sleep spindle detection. J. Sleep Res. 9, 327–334 (2000).
Gais, S., Mölle, M., Helms, K. & Born, J. Learning-dependent increases in sleep spindle density. J. Neurosci. 22, 6830–6834 (2002).
Mölle, M., Marshall, L., Gais, S. & Born, J. Grouping of spindle activity during slow oscillations in human non-rapid eye movement sleep. J. Neurosci. 22, 10941–10947 (2002).
Anderer, P. et al. An E-health solution for automatic sleep classification according to Rechtschaffen and Kales: validation study of the Somnolyzer 24 x 7 utilizing the Siesta database. Neuropsychobiology 51, 115–133 (2005).
Schabus, M. et al. Sleep spindle-related activity in the human EEG and its relation to general cognitive and learning abilities. Eur. J. Neurosci. 23, 1738–1746 (2006).
Huupponen, E. et al. Development and comparison of four sleep spindle detection methods. Artif. Intell. Med. 40, 157–170 (2007).
Devuyst, S. et al. Automatic sleep spindle detection in patients with sleep disorders. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 3883–3886 (2006).
Barakat, M. et al. Sleep spindles predict neural and behavioral changes in motor sequence consolidation. Hum. Brain Mapp. 34, 2918–2928 (2013).
Bergmann, T.O., Molle, M., Diedrichs, J., Born, J. & Siebner, H.R. Sleep spindle-related reactivation of category-specific cortical regions after learning face-scene associations. Neuroimage 59, 2733–2742 (2012).
Ayoub, A. et al. Differential effects on fast and slow spindle activity, and the sleep slow oscillation in humans with carbamazepine and flunarizine to antagonize voltage-dependent Na+ and Ca2+ channel activity. Sleep 36, 905–911 (2013).
Ray, L.B., Fogel, S.M., Smith, C.T. & Peters, K.R. Validating an automated sleep spindle detection algorithm using an individualized approach. J. Sleep Res. 19, 374–378 (2010).
Schabus, M. et al. Hemodynamic cerebral correlates of sleep spindles during human non-rapid eye movement sleep. Proc. Natl. Acad. Sci. USA 104, 13164–13169 (2007).
Bódizs, R., Körmendi, J., Rigó, P. & Lázár, A.S. The individual adjustment method of sleep spindle analysis: methodological improvements and roots in the fingerprint paradigm. J. Neurosci. Methods 178, 205–213 (2009).
Ruch, S. et al. Sleep stage II contributes to the consolidation of declarative memories. Neuropsychologia 50, 2389–2396 (2012).
Bódizs, R., Gombos, F. & Kovács, I. Sleep EEG fingerprints reveal accelerated thalamocortical oscillatory dynamics in Williams syndrome. Res. Dev. Disabil. 33, 153–164 (2012).
Sitnikova, E., Hramov, A.E., Koronovsky, A.A. & van Luijtelaar, G. Sleep spindles and spike-wave discharges in EEG: their generic features, similarities and distinctions disclosed with Fourier transform and continuous wavelet analysis. J. Neurosci. Methods 180, 304–316 (2009).
Wendt, S.L. et al. Validation of a novel automatic sleep spindle detector with high performance during sleep in middle aged subjects. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 4250–4253 (2012).
Plante, D.T. et al. Topographic and sex-related differences in sleep spindles in major depressive disorder: a high-density EEG investigation. J. Affect. Disord. 146, 120–125 (2013).
Peppard, P.E. et al. Increased prevalence of sleep-disordered breathing in adults. Am. J. Epidemiol. 177, 1006–1014 (2013).
Feinberg, I., Koresko, R.L. & Heller, N. EEG sleep patterns as a function of normal and pathological aging in man. J. Psychiatr. Res. 5, 107–144 (1967).
Nir, Y. et al. Regional slow waves and spindles in human sleep. Neuron 70, 153–169 (2011).
McCormick, L., Nielsen, T., Nicolas, A., Ptito, M. & Montplaisir, J. Topographical distribution of spindles and K-complexes in normal subjects. Sleep 20, 939–941 (1997).
Donoho, D.L. An invitation to reproducible computational research. Biostatistics 11, 385–388 (2010).
The authors thank the Registered Polysomnographic Technologist (RPSGT) experts who participated in the spindle identification task as well as the participants and organizers of the Wisconsin Sleep Cohort who provided the polysomnography data. We also thank C. Liang, E. Leary, H. Ollila and H. Kraemer for their helpful discussions and E.Þ. Ágústsson and H. Moore for their input in the pilot study for this project. We are grateful to the authors of the previously published algorithms who generously shared their code and knowledge about spindle detectors. S.C.W. is supported by the Brain and Behavior Research Foundation and as a Canadian Institutes of Health Research Banting Fellow. E.M. is supported by US National Institutes of Health (NIH) grant NS23724. P.P. received funding from the Caltech SURF program and the NAVY (ONR-MURI N00014-06-1-0734 and UCLA-MURI N00014-10-1-0933). EEG data collection was supported by grants from the National Heart, Lung, and Blood Institute (grant R01HL62252) and the National Center for Research Resources (grant 1UL1RR025011) at the NIH.
The authors declare no competing financial interests.
About this article
Cite this article
Warby, S., Wendt, S., Welinder, P. et al. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nat Methods 11, 385–392 (2014). https://doi.org/10.1038/nmeth.2855
Physiological Reviews (2020)
Adverse Cardiovascular Effects of Traffic Noise with a Focus on Nighttime Noise and the New WHO Noise Guidelines
Annual Review of Public Health (2020)
Sleep spindle and slow wave abnormalities in schizophrenia and other psychotic disorders: Recent findings and future directions
Schizophrenia Research (2020)
Current Opinion in Behavioral Sciences (2020)
IEEE Transactions on Neural Systems and Rehabilitation Engineering (2020)