Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods

Abstract

Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Generation of the gold standard and spindle-detection performance of individual experts.
Figure 2: By-event and by-subject characteristics of 1,987 spindles in the gold-standard data set.
Figure 3: Consensus and performance of the non-expert group for spindle detection.
Figure 4: Automated spindle-detector performance.
Figure 5: Performance of experts, non-experts and automated spindle detection algorithms.

References

  1. 1

    Iber, C., Ancoli-Israel, S., Chesson, A. & Quan, S.F. AASM Manual for the Scoring of Sleep and Associated Events 1st edn. (American Academy of Sleep Medicine, 2007).

  2. 2

    Silverstein, L.D. & Levy, C.M. The stability of the sigma sleep spindle. Electroencephalogr. Clin. Neurophysiol. 40, 666–670 (1976).

    CAS  Article  Google Scholar 

  3. 3

    Tan, X., Campbell, I.G. & Feinberg, I. Internight reliability and benchmark values for computer analyses of non-rapid eye movement (NREM) and REM EEG in normal young adult and elderly subjects. Clin. Neurophysiol. 112, 1540–1552 (2001).

    CAS  Article  Google Scholar 

  4. 4

    Werth, E., Achermann, P., Dijk, D.J. & Borbély, A.A. Spindle frequency activity in the sleep EEG: individual differences and topographic distribution. Electroencephalogr. Clin. Neurophysiol. 103, 535–542 (1997).

    CAS  Article  Google Scholar 

  5. 5

    De Gennaro, L., Ferrara, M., Vecchio, F., Curcio, G. & Bertini, M. An electroencephalographic fingerprint of human sleep. Neuroimage 26, 114–122 (2005).

    Article  Google Scholar 

  6. 6

    De Gennaro, L. & Ferrara, M. Sleep spindles: an overview. Sleep Med. Rev. 7, 423–440 (2003).

    Article  Google Scholar 

  7. 7

    Shibagaki, M., Kiyono, S. & Watanabe, K. Spindle evolution in normal and mentally retarded children: a review. Sleep 5, 47–57 (1982).

    CAS  Article  Google Scholar 

  8. 8

    Crowley, K., Trinder, J., Kim, Y., Carrington, M. & Colrain, I.M. The effects of normal aging on sleep spindle and K-complex production. Clin. Neurophysiol. 113, 1615–1622 (2002).

    Article  Google Scholar 

  9. 9

    Nicolas, A., Petit, D., Rompré, S. & Montplaisir, J. Sleep spindle characteristics in healthy subjects of different age groups. Clin. Neurophysiol. 112, 521–527 (2001).

    CAS  Article  Google Scholar 

  10. 10

    Martin, N. et al. Topography of age-related changes in sleep spindles. Neurobiol. Aging 34, 468–476 (2013).

    Article  Google Scholar 

  11. 11

    De Gennaro, L. et al. The electroencephalographic fingerprint of sleep is genetically determined: a twin study. Ann. Neurol. 64, 455–460 (2008).

    Article  Google Scholar 

  12. 12

    Ambrosius, U. et al. Heritability of sleep electroencephalogram. Biol. Psychiatry 64, 344–348 (2008).

    Article  Google Scholar 

  13. 13

    Fogel, S.M. & Smith, C.T. The function of the sleep spindle: a physiological index of intelligence and a mechanism for sleep-dependent memory consolidation. Neurosci. Biobehav. Rev. 35, 1154–1165 (2011).

    Article  Google Scholar 

  14. 14

    Walker, M.P. The role of sleep in cognition and emotion. Ann. NY Acad. Sci. 1156, 168–197 (2009).

    Article  Google Scholar 

  15. 15

    Diekelmann, S. & Born, J. The memory function of sleep. Nat. Rev. Neurosci. 11, 114–126 (2010).

    CAS  PubMed  Google Scholar 

  16. 16

    Barakat, M. et al. Fast and slow spindle involvement in the consolidation of a new motor sequence. Behav. Brain Res. 217, 117–121 (2011).

    CAS  Article  Google Scholar 

  17. 17

    Ferrarelli, F. et al. Reduced sleep spindle activity in schizophrenia patients. Am. J. Psychiatry 164, 483–492 (2007).

    Article  Google Scholar 

  18. 18

    Wamsley, E.J. et al. Reduced sleep spindles and spindle coherence in schizophrenia: mechanisms of impaired memory consolidation? Biol. Psychiatry 71, 154–161 (2012).

    Article  Google Scholar 

  19. 19

    Limoges, E., Mottron, L., Bolduc, C., Berthiaume, C. & Godbout, R. Atypical sleep architecture and the autism phenotype. Brain 128, 1049–1061 (2005).

    Article  Google Scholar 

  20. 20

    Myatchin, I. & Lagae, L. Sleep spindle abnormalities in children with generalized spike-wave discharges. Pediatr. Neurol. 36, 106–111 (2007).

    Article  Google Scholar 

  21. 21

    Montagna, P., Gambetti, P., Cortelli, P. & Lugaresi, E. Familial and sporadic fatal insomnia. Lancet Neurol. 2, 167–176 (2003).

    CAS  Article  Google Scholar 

  22. 22

    Espa, F., Ondze, B., Deglise, P., Billiard, M. & Besset, A. Sleep architecture, slow wave activity, and sleep spindles in adult patients with sleepwalking and sleep terrors. Clin. Neurophysiol. 111, 929–939 (2000).

    CAS  Article  Google Scholar 

  23. 23

    Himanen, S.-L., Virkkala, J., Huupponen, E. & Hasan, J. Spindle frequency remains slow in sleep apnea patients throughout the night. Sleep Med. 4, 229–234 (2003).

    Article  Google Scholar 

  24. 24

    Petit, D., Gagnon, J.-F., Fantini, M.L., Ferini-Strambi, L. & Montplaisir, J. Sleep and quantitative EEG in neurodegenerative disorders. J. Psychosom. Res. 56, 487–496 (2004).

    Article  Google Scholar 

  25. 25

    Ferrara, M., Moroni, F., De Gennaro, L. & Nobili, L. Hippocampal sleep features: relations to human memory function. Front. Neurol. 3, 57 (2012).

    Article  Google Scholar 

  26. 26

    Steriade, M. Grouping of brain rhythms in corticothalamic systems. Neuroscience 137, 1087–1106 (2006).

    CAS  Article  Google Scholar 

  27. 27

    Vukadinovic, Z. Sleep abnormalities in schizophrenia may suggest impaired trans-thalamic cortico-cortical communication: towards a dynamic model of the illness. Eur. J. Neurosci. 34, 1031–1039 (2011).

    Article  Google Scholar 

  28. 28

    Schimicek, P., Zeitlhofer, J., Anderer, P. & Saletu, B. Automatic sleep-spindle detection procedure: aspects of reliability and validity. Clin. Electroencephalogr. 25, 26–29 (1994).

    CAS  Article  Google Scholar 

  29. 29

    Huupponen, E. et al. Optimization of sigma amplitude threshold in sleep spindle detection. J. Sleep Res. 9, 327–334 (2000).

    CAS  Article  Google Scholar 

  30. 30

    Gais, S., Mölle, M., Helms, K. & Born, J. Learning-dependent increases in sleep spindle density. J. Neurosci. 22, 6830–6834 (2002).

    CAS  Article  Google Scholar 

  31. 31

    Mölle, M., Marshall, L., Gais, S. & Born, J. Grouping of spindle activity during slow oscillations in human non-rapid eye movement sleep. J. Neurosci. 22, 10941–10947 (2002).

    Article  Google Scholar 

  32. 32

    Anderer, P. et al. An E-health solution for automatic sleep classification according to Rechtschaffen and Kales: validation study of the Somnolyzer 24 x 7 utilizing the Siesta database. Neuropsychobiology 51, 115–133 (2005).

    Article  Google Scholar 

  33. 33

    Schabus, M. et al. Sleep spindle-related activity in the human EEG and its relation to general cognitive and learning abilities. Eur. J. Neurosci. 23, 1738–1746 (2006).

    CAS  Article  Google Scholar 

  34. 34

    Huupponen, E. et al. Development and comparison of four sleep spindle detection methods. Artif. Intell. Med. 40, 157–170 (2007).

    Article  Google Scholar 

  35. 35

    Devuyst, S. et al. Automatic sleep spindle detection in patients with sleep disorders. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 3883–3886 (2006).

    CAS  Article  Google Scholar 

  36. 36

    Barakat, M. et al. Sleep spindles predict neural and behavioral changes in motor sequence consolidation. Hum. Brain Mapp. 34, 2918–2928 (2013).

    Article  Google Scholar 

  37. 37

    Bergmann, T.O., Molle, M., Diedrichs, J., Born, J. & Siebner, H.R. Sleep spindle-related reactivation of category-specific cortical regions after learning face-scene associations. Neuroimage 59, 2733–2742 (2012).

    Article  Google Scholar 

  38. 38

    Ayoub, A. et al. Differential effects on fast and slow spindle activity, and the sleep slow oscillation in humans with carbamazepine and flunarizine to antagonize voltage-dependent Na+ and Ca2+ channel activity. Sleep 36, 905–911 (2013).

    Article  Google Scholar 

  39. 39

    Ray, L.B., Fogel, S.M., Smith, C.T. & Peters, K.R. Validating an automated sleep spindle detection algorithm using an individualized approach. J. Sleep Res. 19, 374–378 (2010).

    Article  Google Scholar 

  40. 40

    Schabus, M. et al. Hemodynamic cerebral correlates of sleep spindles during human non-rapid eye movement sleep. Proc. Natl. Acad. Sci. USA 104, 13164–13169 (2007).

    CAS  Article  Google Scholar 

  41. 41

    Bódizs, R., Körmendi, J., Rigó, P. & Lázár, A.S. The individual adjustment method of sleep spindle analysis: methodological improvements and roots in the fingerprint paradigm. J. Neurosci. Methods 178, 205–213 (2009).

    Article  Google Scholar 

  42. 42

    Ruch, S. et al. Sleep stage II contributes to the consolidation of declarative memories. Neuropsychologia 50, 2389–2396 (2012).

    Article  Google Scholar 

  43. 43

    Bódizs, R., Gombos, F. & Kovács, I. Sleep EEG fingerprints reveal accelerated thalamocortical oscillatory dynamics in Williams syndrome. Res. Dev. Disabil. 33, 153–164 (2012).

    Article  Google Scholar 

  44. 44

    Sitnikova, E., Hramov, A.E., Koronovsky, A.A. & van Luijtelaar, G. Sleep spindles and spike-wave discharges in EEG: their generic features, similarities and distinctions disclosed with Fourier transform and continuous wavelet analysis. J. Neurosci. Methods 180, 304–316 (2009).

    Article  Google Scholar 

  45. 45

    Wendt, S.L. et al. Validation of a novel automatic sleep spindle detector with high performance during sleep in middle aged subjects. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 4250–4253 (2012).

    PubMed  Google Scholar 

  46. 46

    Plante, D.T. et al. Topographic and sex-related differences in sleep spindles in major depressive disorder: a high-density EEG investigation. J. Affect. Disord. 146, 120–125 (2013).

    CAS  Article  Google Scholar 

  47. 47

    Peppard, P.E. et al. Increased prevalence of sleep-disordered breathing in adults. Am. J. Epidemiol. 177, 1006–1014 (2013).

    Article  Google Scholar 

  48. 48

    Feinberg, I., Koresko, R.L. & Heller, N. EEG sleep patterns as a function of normal and pathological aging in man. J. Psychiatr. Res. 5, 107–144 (1967).

    CAS  Article  Google Scholar 

  49. 49

    Nir, Y. et al. Regional slow waves and spindles in human sleep. Neuron 70, 153–169 (2011).

    CAS  Article  Google Scholar 

  50. 50

    McCormick, L., Nielsen, T., Nicolas, A., Ptito, M. & Montplaisir, J. Topographical distribution of spindles and K-complexes in normal subjects. Sleep 20, 939–941 (1997).

    CAS  Article  Google Scholar 

  51. 51

    Donoho, D.L. An invitation to reproducible computational research. Biostatistics 11, 385–388 (2010).

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank the Registered Polysomnographic Technologist (RPSGT) experts who participated in the spindle identification task as well as the participants and organizers of the Wisconsin Sleep Cohort who provided the polysomnography data. We also thank C. Liang, E. Leary, H. Ollila and H. Kraemer for their helpful discussions and E.Þ. Ágústsson and H. Moore for their input in the pilot study for this project. We are grateful to the authors of the previously published algorithms who generously shared their code and knowledge about spindle detectors. S.C.W. is supported by the Brain and Behavior Research Foundation and as a Canadian Institutes of Health Research Banting Fellow. E.M. is supported by US National Institutes of Health (NIH) grant NS23724. P.P. received funding from the Caltech SURF program and the NAVY (ONR-MURI N00014-06-1-0734 and UCLA-MURI N00014-10-1-0933). EEG data collection was supported by grants from the National Heart, Lung, and Blood Institute (grant R01HL62252) and the National Center for Research Resources (grant 1UL1RR025011) at the NIH.

Author information

Affiliations

Authors

Contributions

S.C.W., E.M. and P.P. designed the research. P.W. and P.P. designed and coded the Internet interface. S.C.W. and P.W. collected the spindle scoring data. S.C.W. and S.L.W. performed the data analysis. S.L.W. wrote the code to implement the automated spindle detectors. All authors provided input on data analysis and interpretation. P.E.P. also provided source EEG data. H.B.D.S., P.J., E.M. and P.P. also provided financial support. S.C.W., S.L.W. and E.M. wrote the manuscript, which was discussed and edited by all authors.

Corresponding authors

Correspondence to Pietro Perona or Emmanuel Mignot.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–13, Supplementary Tables 1–8 and Supplementary Note (PDF 2846 kb)

Supplementary Software

MATLAB code (text-based .m files) for automatic spindle detectors a1-a6. (ZIP 17 kb)

Source data

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Warby, S., Wendt, S., Welinder, P. et al. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nat Methods 11, 385–392 (2014). https://doi.org/10.1038/nmeth.2855

Download citation

Further reading

Search

Sign up for the Nature Briefing newsletter for a daily update on COVID-19 science.
Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing