Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods

Nature Methods volume 11, pages 385–392 (2014)Cite this article

Subjects

Abstract

Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Generation of the gold standard and spindle-detection performance of individual experts.
Figure 2: By-event and by-subject characteristics of 1,987 spindles in the gold-standard data set.
Figure 3: Consensus and performance of the non-expert group for spindle detection.
Figure 4: Automated spindle-detector performance.
Figure 5: Performance of experts, non-experts and automated spindle detection algorithms.

Similar content being viewed by others

References

  1. Iber, C., Ancoli-Israel, S., Chesson, A. & Quan, S.F. AASM Manual for the Scoring of Sleep and Associated Events 1st edn. (American Academy of Sleep Medicine, 2007).

  2. Silverstein, L.D. & Levy, C.M. The stability of the sigma sleep spindle. Electroencephalogr. Clin. Neurophysiol. 40, 666–670 (1976).

    Article  CAS  Google Scholar 

  3. Tan, X., Campbell, I.G. & Feinberg, I. Internight reliability and benchmark values for computer analyses of non-rapid eye movement (NREM) and REM EEG in normal young adult and elderly subjects. Clin. Neurophysiol. 112, 1540–1552 (2001).

    Article  CAS  Google Scholar 

  4. Werth, E., Achermann, P., Dijk, D.J. & Borbély, A.A. Spindle frequency activity in the sleep EEG: individual differences and topographic distribution. Electroencephalogr. Clin. Neurophysiol. 103, 535–542 (1997).

    Article  CAS  Google Scholar 

  5. De Gennaro, L., Ferrara, M., Vecchio, F., Curcio, G. & Bertini, M. An electroencephalographic fingerprint of human sleep. Neuroimage 26, 114–122 (2005).

    Article  Google Scholar 

  6. De Gennaro, L. & Ferrara, M. Sleep spindles: an overview. Sleep Med. Rev. 7, 423–440 (2003).

    Article  Google Scholar 

  7. Shibagaki, M., Kiyono, S. & Watanabe, K. Spindle evolution in normal and mentally retarded children: a review. Sleep 5, 47–57 (1982).

    Article  CAS  Google Scholar 

  8. Crowley, K., Trinder, J., Kim, Y., Carrington, M. & Colrain, I.M. The effects of normal aging on sleep spindle and K-complex production. Clin. Neurophysiol. 113, 1615–1622 (2002).

    Article  Google Scholar 

  9. Nicolas, A., Petit, D., Rompré, S. & Montplaisir, J. Sleep spindle characteristics in healthy subjects of different age groups. Clin. Neurophysiol. 112, 521–527 (2001).

    Article  CAS  Google Scholar 

  10. Martin, N. et al. Topography of age-related changes in sleep spindles. Neurobiol. Aging 34, 468–476 (2013).

    Article  Google Scholar 

  11. De Gennaro, L. et al. The electroencephalographic fingerprint of sleep is genetically determined: a twin study. Ann. Neurol. 64, 455–460 (2008).

    Article  Google Scholar 

  12. Ambrosius, U. et al. Heritability of sleep electroencephalogram. Biol. Psychiatry 64, 344–348 (2008).

    Article  Google Scholar 

  13. Fogel, S.M. & Smith, C.T. The function of the sleep spindle: a physiological index of intelligence and a mechanism for sleep-dependent memory consolidation. Neurosci. Biobehav. Rev. 35, 1154–1165 (2011).

    Article  Google Scholar 

  14. Walker, M.P. The role of sleep in cognition and emotion. Ann. NY Acad. Sci. 1156, 168–197 (2009).

    Article  Google Scholar 

  15. Diekelmann, S. & Born, J. The memory function of sleep. Nat. Rev. Neurosci. 11, 114–126 (2010).

    CAS  PubMed  Google Scholar 

  16. Barakat, M. et al. Fast and slow spindle involvement in the consolidation of a new motor sequence. Behav. Brain Res. 217, 117–121 (2011).

    Article  CAS  Google Scholar 

  17. Ferrarelli, F. et al. Reduced sleep spindle activity in schizophrenia patients. Am. J. Psychiatry 164, 483–492 (2007).

    Article  Google Scholar 

  18. Wamsley, E.J. et al. Reduced sleep spindles and spindle coherence in schizophrenia: mechanisms of impaired memory consolidation? Biol. Psychiatry 71, 154–161 (2012).

    Article  Google Scholar 

  19. Limoges, E., Mottron, L., Bolduc, C., Berthiaume, C. & Godbout, R. Atypical sleep architecture and the autism phenotype. Brain 128, 1049–1061 (2005).

    Article  Google Scholar 

  20. Myatchin, I. & Lagae, L. Sleep spindle abnormalities in children with generalized spike-wave discharges. Pediatr. Neurol. 36, 106–111 (2007).

    Article  Google Scholar 

  21. Montagna, P., Gambetti, P., Cortelli, P. & Lugaresi, E. Familial and sporadic fatal insomnia. Lancet Neurol. 2, 167–176 (2003).

    Article  CAS  Google Scholar 

  22. Espa, F., Ondze, B., Deglise, P., Billiard, M. & Besset, A. Sleep architecture, slow wave activity, and sleep spindles in adult patients with sleepwalking and sleep terrors. Clin. Neurophysiol. 111, 929–939 (2000).

    Article  CAS  Google Scholar 

  23. Himanen, S.-L., Virkkala, J., Huupponen, E. & Hasan, J. Spindle frequency remains slow in sleep apnea patients throughout the night. Sleep Med. 4, 229–234 (2003).

    Article  Google Scholar 

  24. Petit, D., Gagnon, J.-F., Fantini, M.L., Ferini-Strambi, L. & Montplaisir, J. Sleep and quantitative EEG in neurodegenerative disorders. J. Psychosom. Res. 56, 487–496 (2004).

    Article  Google Scholar 

  25. Ferrara, M., Moroni, F., De Gennaro, L. & Nobili, L. Hippocampal sleep features: relations to human memory function. Front. Neurol. 3, 57 (2012).

    Article  Google Scholar 

  26. Steriade, M. Grouping of brain rhythms in corticothalamic systems. Neuroscience 137, 1087–1106 (2006).

    Article  CAS  Google Scholar 

  27. Vukadinovic, Z. Sleep abnormalities in schizophrenia may suggest impaired trans-thalamic cortico-cortical communication: towards a dynamic model of the illness. Eur. J. Neurosci. 34, 1031–1039 (2011).

    Article  Google Scholar 

  28. Schimicek, P., Zeitlhofer, J., Anderer, P. & Saletu, B. Automatic sleep-spindle detection procedure: aspects of reliability and validity. Clin. Electroencephalogr. 25, 26–29 (1994).

    Article  CAS  Google Scholar 

  29. Huupponen, E. et al. Optimization of sigma amplitude threshold in sleep spindle detection. J. Sleep Res. 9, 327–334 (2000).

    Article  CAS  Google Scholar 

  30. Gais, S., Mölle, M., Helms, K. & Born, J. Learning-dependent increases in sleep spindle density. J. Neurosci. 22, 6830–6834 (2002).

    Article  CAS  Google Scholar 

  31. Mölle, M., Marshall, L., Gais, S. & Born, J. Grouping of spindle activity during slow oscillations in human non-rapid eye movement sleep. J. Neurosci. 22, 10941–10947 (2002).

    Article  Google Scholar 

  32. Anderer, P. et al. An E-health solution for automatic sleep classification according to Rechtschaffen and Kales: validation study of the Somnolyzer 24 x 7 utilizing the Siesta database. Neuropsychobiology 51, 115–133 (2005).

    Article  Google Scholar 

  33. Schabus, M. et al. Sleep spindle-related activity in the human EEG and its relation to general cognitive and learning abilities. Eur. J. Neurosci. 23, 1738–1746 (2006).

    Article  CAS  Google Scholar 

  34. Huupponen, E. et al. Development and comparison of four sleep spindle detection methods. Artif. Intell. Med. 40, 157–170 (2007).

    Article  Google Scholar 

  35. Devuyst, S. et al. Automatic sleep spindle detection in patients with sleep disorders. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 3883–3886 (2006).

    Article  CAS  Google Scholar 

  36. Barakat, M. et al. Sleep spindles predict neural and behavioral changes in motor sequence consolidation. Hum. Brain Mapp. 34, 2918–2928 (2013).

    Article  Google Scholar 

  37. Bergmann, T.O., Molle, M., Diedrichs, J., Born, J. & Siebner, H.R. Sleep spindle-related reactivation of category-specific cortical regions after learning face-scene associations. Neuroimage 59, 2733–2742 (2012).

    Article  Google Scholar 

  38. Ayoub, A. et al. Differential effects on fast and slow spindle activity, and the sleep slow oscillation in humans with carbamazepine and flunarizine to antagonize voltage-dependent Na+ and Ca2+ channel activity. Sleep 36, 905–911 (2013).

    Article  Google Scholar 

  39. Ray, L.B., Fogel, S.M., Smith, C.T. & Peters, K.R. Validating an automated sleep spindle detection algorithm using an individualized approach. J. Sleep Res. 19, 374–378 (2010).

    Article  Google Scholar 

  40. Schabus, M. et al. Hemodynamic cerebral correlates of sleep spindles during human non-rapid eye movement sleep. Proc. Natl. Acad. Sci. USA 104, 13164–13169 (2007).

    Article  CAS  Google Scholar 

  41. Bódizs, R., Körmendi, J., Rigó, P. & Lázár, A.S. The individual adjustment method of sleep spindle analysis: methodological improvements and roots in the fingerprint paradigm. J. Neurosci. Methods 178, 205–213 (2009).

    Article  Google Scholar 

  42. Ruch, S. et al. Sleep stage II contributes to the consolidation of declarative memories. Neuropsychologia 50, 2389–2396 (2012).

    Article  Google Scholar 

  43. Bódizs, R., Gombos, F. & Kovács, I. Sleep EEG fingerprints reveal accelerated thalamocortical oscillatory dynamics in Williams syndrome. Res. Dev. Disabil. 33, 153–164 (2012).

    Article  Google Scholar 

  44. Sitnikova, E., Hramov, A.E., Koronovsky, A.A. & van Luijtelaar, G. Sleep spindles and spike-wave discharges in EEG: their generic features, similarities and distinctions disclosed with Fourier transform and continuous wavelet analysis. J. Neurosci. Methods 180, 304–316 (2009).

    Article  Google Scholar 

  45. Wendt, S.L. et al. Validation of a novel automatic sleep spindle detector with high performance during sleep in middle aged subjects. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 4250–4253 (2012).

    PubMed  Google Scholar 

  46. Plante, D.T. et al. Topographic and sex-related differences in sleep spindles in major depressive disorder: a high-density EEG investigation. J. Affect. Disord. 146, 120–125 (2013).

    Article  CAS  Google Scholar 

  47. Peppard, P.E. et al. Increased prevalence of sleep-disordered breathing in adults. Am. J. Epidemiol. 177, 1006–1014 (2013).

    Article  Google Scholar 

  48. Feinberg, I., Koresko, R.L. & Heller, N. EEG sleep patterns as a function of normal and pathological aging in man. J. Psychiatr. Res. 5, 107–144 (1967).

    Article  CAS  Google Scholar 

  49. Nir, Y. et al. Regional slow waves and spindles in human sleep. Neuron 70, 153–169 (2011).

    Article  CAS  Google Scholar 

  50. McCormick, L., Nielsen, T., Nicolas, A., Ptito, M. & Montplaisir, J. Topographical distribution of spindles and K-complexes in normal subjects. Sleep 20, 939–941 (1997).

    Article  CAS  Google Scholar 

  51. Donoho, D.L. An invitation to reproducible computational research. Biostatistics 11, 385–388 (2010).

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank the Registered Polysomnographic Technologist (RPSGT) experts who participated in the spindle identification task as well as the participants and organizers of the Wisconsin Sleep Cohort who provided the polysomnography data. We also thank C. Liang, E. Leary, H. Ollila and H. Kraemer for their helpful discussions and E.Þ. Ágústsson and H. Moore for their input in the pilot study for this project. We are grateful to the authors of the previously published algorithms who generously shared their code and knowledge about spindle detectors. S.C.W. is supported by the Brain and Behavior Research Foundation and as a Canadian Institutes of Health Research Banting Fellow. E.M. is supported by US National Institutes of Health (NIH) grant NS23724. P.P. received funding from the Caltech SURF program and the NAVY (ONR-MURI N00014-06-1-0734 and UCLA-MURI N00014-10-1-0933). EEG data collection was supported by grants from the National Heart, Lung, and Blood Institute (grant R01HL62252) and the National Center for Research Resources (grant 1UL1RR025011) at the NIH.

Author information

Authors and Affiliations

  1. Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA

    Simon C Warby, Sabrina L Wendt, Emil G S Munk, Oscar Carrillo & Emmanuel Mignot

  2. Danish Center for Sleep Medicine, Glostrup University Hospital, Glostrup, Denmark

    Sabrina L Wendt, Emil G S Munk & Poul Jennum

  3. Computational Vision Laboratory, California Institute of Technology, Pasadena, California, USA

    Peter Welinder & Pietro Perona

  4. Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, Denmark

    Helge B D Sorensen

  5. Department of Population Health Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA

    Paul E Peppard

Authors
  1. Simon C Warby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sabrina L Wendt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Welinder
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emil G S Munk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Oscar Carrillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Helge B D Sorensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Poul Jennum
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Paul E Peppard
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Pietro Perona
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Emmanuel Mignot
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.C.W., E.M. and P.P. designed the research. P.W. and P.P. designed and coded the Internet interface. S.C.W. and P.W. collected the spindle scoring data. S.C.W. and S.L.W. performed the data analysis. S.L.W. wrote the code to implement the automated spindle detectors. All authors provided input on data analysis and interpretation. P.E.P. also provided source EEG data. H.B.D.S., P.J., E.M. and P.P. also provided financial support. S.C.W., S.L.W. and E.M. wrote the manuscript, which was discussed and edited by all authors.

Corresponding authors

Correspondence to Pietro Perona or Emmanuel Mignot.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–13, Supplementary Tables 1–8 and Supplementary Note (PDF 2846 kb)

Supplementary Software

MATLAB code (text-based .m files) for automatic spindle detectors a1-a6. (ZIP 17 kb)

Source data

Rights and permissions

Reprints and permissions

About this article

Cite this article

Warby, S., Wendt, S., Welinder, P. et al. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nat Methods 11, 385–392 (2014). https://doi.org/10.1038/nmeth.2855

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.2855

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing