Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Discrimination of the behavioural dynamics of visually impaired infants via deep learning

Nature Biomedical Engineering volume 3, pages 860–869 (2019)Cite this article

Subjects

Abstract

Sensory loss is associated with behavioural changes, but how behavioural dynamics change when a sensory modality is impaired remains unclear. Here, by recording under a designed standardized scenario, the behavioural phenotypes of 4,196 infants who experienced varying degrees of visual loss but retained high behavioural plasticity, we show that behaviours with significantly higher occurrence in visually impaired infants can be identified, and that correlations between the frequency of specific behavioural patterns and visual-impairment severity, as well as variations in behavioural dynamics with age, can be quantified. We also show that a deep-learning algorithm (a temporal segment network) trained with the full-length videos can discriminate, for an independent dataset from 400 infants, mild visual impairment from healthy behaviour (area under the curve (AUC) of 85.2%), severe visual impairment from mild impairment (AUC of 81.9%), and various ophthalmological conditions from healthy vision (with AUCs ranging from 81.6% to 93.0%). The video dataset of behavioural phenotypes in response to visual loss and the trained machine-learning algorithm should help the study of visual function and behavioural plasticity in infants.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overall study pipeline.
Fig. 2: Occurrence and magnitude of behaviours in various visual conditions.
Fig. 3: Quantification of vision-behaviour correlations for various visual conditions.
Fig. 4: Self-learning phenotypic features by deep learning.
Fig. 5: Deep-learning-based detection of ophthalmological conditions.
Fig. 6: Deep-learning-based detection of ophthalmological conditions not overlapping with the training set.
Fig. 7: Diagnosis of each ophthalmological condition by using deep learning.
Fig. 8: Characteristics of signals for known behaviours in the trained network.

Similar content being viewed by others

Data availability

The authors declare that the main data supporting the results in this study are available within the paper and its Supplementary Information. The datasets generated during the study and representative videos for each behaviour are available for research purposes from the corresponding authors on reasonable request. The raw videos are not publicly available due to restrictions of portrait rights and patient privacy.

Code availability

We used the temporal segment network framework (https://github.com/yjxiong/temporal-segment-networks). Our experimental protocol made use of proprietary libraries and we are unable to publicly release this code. However, we detail the experiments and implementation details in the Methods and Supplementary Information.

References

  1. Merabet, L. B. & Pascual-Leone, A. Neural reorganization following sensory loss: the opportunity of change. Nat. Rev. Neurosci. 11, 44–52 (2010).

    Article  CAS  Google Scholar 

  2. Akre, K. L. & Johnsen, S. Psychophysics and the evolution of behavior. Trends Ecol. Evolution. 29, 291–300 (2014).

    Article  Google Scholar 

  3. Klein, M. et al. Sensory determinants of behavioral dynamics in drosophila thermotaxis. Proc. Natl Acad. Sci. USA 112, E220–E229 (2015).

    Article  CAS  Google Scholar 

  4. Dey, S. et al. Cyclic regulation of sensory perception by a female hormone alters behavior. Cell 161, 1334–1344 (2015).

    Article  CAS  Google Scholar 

  5. Bridgeman, B. & Tseng, P. Embodied cognition and the perception-action link. Phys. Life Rev. 8, 73–85 (2011).

    Article  Google Scholar 

  6. Pamplona, D., Triesch, J. & Rothkopf, C. A. Power spectra of the natural input to the visual system. Vis. Res. 83, 66–75 (2013).

    Article  CAS  Google Scholar 

  7. Prasad, S. & Galetta, S. L. Anatomy and physiology of the afferent visual system. Handb. Clin. Neurol. 102, 3–19 (2011).

    Article  Google Scholar 

  8. Lea, A. J. et al. Developmental plasticity: bridging research in evolution and human health. Evol. Med Public Health 2017, 162–175 (2018).

    Article  Google Scholar 

  9. Williams, B. A., Mandrekar J. N., Mandrekar, S. J., Cha, S. S. & Furth, A. F. Finding Optimal Cut Points for Continous Covariates with Binary and Time to Event Outcomes Technical Reports Series 79 (Department of Health Science Research, Mayo Clinic, 2006).

  10. Online Mendelian Inheritance in Man (Johns Hopkins Univ., accessed 9 January 2019); https://omim.org/

  11. Bavelier, D. & Neville, H. J. Cross-modal plasticity: where and how? Nat. Rev. Neurosci. 3, 443–452 (2002).

    Article  CAS  Google Scholar 

  12. Hoyt, C. & Taylor, D. Pediatric ophthalmology and strabismus 4th edn (Elsevier, 2013).

  13. Katz, L. N., Hennig, J. A., Cormack, L. K. & Huk, A. C. A distinct mechanism of temporal integration for motion through depth. J. Neurosci. 35, 10212–10216 (2015).

    Article  CAS  Google Scholar 

  14. Long, E. et al. Dynamic response to initial stage blindness in visual system development. Clin. Sci. 131, 1515–1527 (2017).

    Article  Google Scholar 

  15. Tirosh, E., Shnitzer, M. R., Davidovitch, M. & Cohen, A. Behavioral problems among visually impaired between 6 months and 5 years. Int J. Rehabil. Res. 21, 63–70 (1998).

    Article  CAS  Google Scholar 

  16. Tröster, H., Brambring, M. & Beelmann, A. The age dependence of stereotyped behaviors in blind infants and preschoolers. Child Care Health Dev. 17, 137–157 (1991).

    Article  Google Scholar 

  17. Molloy, A. & Rowe, F. J. Manneristic behaviors of visually impaired children. Strabismus 19, 77–84 (2011).

    Article  Google Scholar 

  18. Mariotti, S. P. Global Data on Visual Impairment 2010 (World Health Organization, 2012).

  19. Gilbert, C. & Foster, A. Childhood blindness in the context of VISION 2020—the right to sight. Bull. World Health Organ 79, 277–232 (2001).

    Google Scholar 

  20. Frisén, L. Vanishing optotypes. new type of acuity test letters. Arch. Ophthalmol. 104, 1194–1198 (1986).

    Article  Google Scholar 

  21. Hilton, A. F. & Stanley, J. C. Pitfalls in testing children’s vision by the sheridan gardiner single optotype method. Br. J. Ophthalmol. 56, 135–139 (1972).

    Article  CAS  Google Scholar 

  22. Brémond-Gignac, D. et al. Visual development in infants: physiological and pathological mechanisms. Curr. Opin. Ophthalmol. 22, S1–S8 (2011).

    Article  Google Scholar 

  23. Li, Y. & Lin, H. Progress in screening and treatment of common congenital eye diseases. Eye Sci. 28, 157–162 (2013).

    PubMed  Google Scholar 

  24. Romeo, D. M. et al. Visual function assessment in late-preterm newborns. Early Hum. Dev. 88, 301–305 (2012).

    Article  Google Scholar 

  25. Livingstone, I. A., Lok, A. S. & Tarbert, C. New mobile technologies and visual acuity. Conf. Proc. IEEE Eng. Med Biol. Soc. 2014, 2189–2192 (2014).

    CAS  PubMed  Google Scholar 

  26. Blaikie, A. J. & Dutton, G. N. How to assess eyes and vision in infants and preschool children. BMJ 350, h1716 (2015).

    Article  Google Scholar 

  27. Mayer, D. L. et al. Monocular acuity norms for the Teller Acuity Cards between ages one month and four years. Invest. Ophthalmol. Vis. Sci. 36, 671 (1995).

    CAS  PubMed  Google Scholar 

  28. Wang, L. et al. Temporal segment networks: towards good practices for deep action recognition. ACM Trans. Inf. Syst. 22, 20–36 (2016).

    Google Scholar 

  29. Azen, R. & Budescu, D. V. The dominance analysis approach for comparing predictors in multiple regression. Psychol. Methods 8, 129 (2003).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the National Supercomputer Center in Guangzhou for computing resources support. This study was funded by the National Key R&D Program of China (2018YFC0116500), the Key Research Plan for the National Natural Science Foundation of China in Cultivation Project (91846109), the Natural Science Foundation of China (81822010), Guangdong Science and Technology Innovation Leading Talents (2017TX04R031), and the Pearl River Scholar Program (2016).

Author information

Author notes

  1. These authors contributed equally: Erping Long, Zhenzhen Liu, Yifan Xiang.

Authors and Affiliations

  1. State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-Sen University, Guangzhou, China

    Erping Long, Zhenzhen Liu, Yifan Xiang, Xiaoyan Li, Zhuoling Lin, Jing Li, Jingjing Chen, Yi Zhu, Chuan Chen, Xiaohang Wu, Wangting Li, Hui Chen, Ruiyang Li, Yahan Yang, Weirong Chen, Haotian Lin & Yizhi Liu

  2. Zhongshan School of Medicine, Sun Yat-Sen University, Guangzhou, China

    Andi Xu, Xiucheng Huang & Yan Zhang

  3. School of Public Health, Sun Yat-Sen University, Guangzhou, China

    Jialing Huang

  4. Department of Molecular and Cellular Pharmacology, University of Miami Miller School of Medicine, Miami, FL, USA

    Yi Zhu & Chuan Chen

  5. VoxelCloud, Los Angeles, CA, USA

    Ziheng Zhou & Xiaowei Ding

  6. School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  7. Center for Precision Medicine, Sun Yat-sen University, Guangzhou, China

    Haotian Lin

Authors
  1. Erping Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenzhen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yifan Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jialing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiucheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiaoyan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhuoling Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jingjing Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yi Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Chuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Ziheng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Xiaowei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Xiaohang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Wangting Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Ruiyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Yahan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Weishi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Weirong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Haotian Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Yizhi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.L., E.L. and Z.Liu designed the research; H.L., X.L., Z.Lin, Z.Liu, J.L., J.C., X.W., W.L., E.L., H.C., R.L., Y.Y., W.C. and Y.L. collected the data; Y.X., A.X., X.H., Y.Zhang, Z.Z., X.D. and E.L. conducted the study; E.L. and J.H. analysed the data; H.L. and E.L. co-wrote the manuscript; Z.Liu, Y.Zhu and C.C. critically revised the manuscript; W.Z. did the technical review; and all authors discussed the results and provided comments regarding the manuscript.

Corresponding authors

Correspondence to Haotian Lin or Yizhi Liu.

Ethics declarations

Competing interests

The authors declare no competing interests

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, figures, tables and references.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Long, E., Liu, Z., Xiang, Y. et al. Discrimination of the behavioural dynamics of visually impaired infants via deep learning. Nat Biomed Eng 3, 860–869 (2019). https://doi.org/10.1038/s41551-019-0461-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-019-0461-9

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics