Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Discrimination of the behavioural dynamics of visually impaired infants via deep learning

Abstract

Sensory loss is associated with behavioural changes, but how behavioural dynamics change when a sensory modality is impaired remains unclear. Here, by recording under a designed standardized scenario, the behavioural phenotypes of 4,196 infants who experienced varying degrees of visual loss but retained high behavioural plasticity, we show that behaviours with significantly higher occurrence in visually impaired infants can be identified, and that correlations between the frequency of specific behavioural patterns and visual-impairment severity, as well as variations in behavioural dynamics with age, can be quantified. We also show that a deep-learning algorithm (a temporal segment network) trained with the full-length videos can discriminate, for an independent dataset from 400 infants, mild visual impairment from healthy behaviour (area under the curve (AUC) of 85.2%), severe visual impairment from mild impairment (AUC of 81.9%), and various ophthalmological conditions from healthy vision (with AUCs ranging from 81.6% to 93.0%). The video dataset of behavioural phenotypes in response to visual loss and the trained machine-learning algorithm should help the study of visual function and behavioural plasticity in infants.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Fig. 1: Overall study pipeline.
Fig. 2: Occurrence and magnitude of behaviours in various visual conditions.
Fig. 3: Quantification of vision-behaviour correlations for various visual conditions.
Fig. 4: Self-learning phenotypic features by deep learning.
Fig. 5: Deep-learning-based detection of ophthalmological conditions.
Fig. 6: Deep-learning-based detection of ophthalmological conditions not overlapping with the training set.
Fig. 7: Diagnosis of each ophthalmological condition by using deep learning.
Fig. 8: Characteristics of signals for known behaviours in the trained network.

Data availability

The authors declare that the main data supporting the results in this study are available within the paper and its Supplementary Information. The datasets generated during the study and representative videos for each behaviour are available for research purposes from the corresponding authors on reasonable request. The raw videos are not publicly available due to restrictions of portrait rights and patient privacy.

Code availability

We used the temporal segment network framework (https://github.com/yjxiong/temporal-segment-networks). Our experimental protocol made use of proprietary libraries and we are unable to publicly release this code. However, we detail the experiments and implementation details in the Methods and Supplementary Information.

References

  1. Merabet, L. B. & Pascual-Leone, A. Neural reorganization following sensory loss: the opportunity of change. Nat. Rev. Neurosci. 11, 44–52 (2010).

    CAS  Article  Google Scholar 

  2. Akre, K. L. & Johnsen, S. Psychophysics and the evolution of behavior. Trends Ecol. Evolution. 29, 291–300 (2014).

    Article  Google Scholar 

  3. Klein, M. et al. Sensory determinants of behavioral dynamics in drosophila thermotaxis. Proc. Natl Acad. Sci. USA 112, E220–E229 (2015).

    CAS  Article  Google Scholar 

  4. Dey, S. et al. Cyclic regulation of sensory perception by a female hormone alters behavior. Cell 161, 1334–1344 (2015).

    CAS  Article  Google Scholar 

  5. Bridgeman, B. & Tseng, P. Embodied cognition and the perception-action link. Phys. Life Rev. 8, 73–85 (2011).

    Article  Google Scholar 

  6. Pamplona, D., Triesch, J. & Rothkopf, C. A. Power spectra of the natural input to the visual system. Vis. Res. 83, 66–75 (2013).

    CAS  Article  Google Scholar 

  7. Prasad, S. & Galetta, S. L. Anatomy and physiology of the afferent visual system. Handb. Clin. Neurol. 102, 3–19 (2011).

    Article  Google Scholar 

  8. Lea, A. J. et al. Developmental plasticity: bridging research in evolution and human health. Evol. Med Public Health 2017, 162–175 (2018).

    Article  Google Scholar 

  9. Williams, B. A., Mandrekar J. N., Mandrekar, S. J., Cha, S. S. & Furth, A. F. Finding Optimal Cut Points for Continous Covariates with Binary and Time to Event Outcomes Technical Reports Series 79 (Department of Health Science Research, Mayo Clinic, 2006).

  10. Online Mendelian Inheritance in Man (Johns Hopkins Univ., accessed 9 January 2019); https://omim.org/

  11. Bavelier, D. & Neville, H. J. Cross-modal plasticity: where and how? Nat. Rev. Neurosci. 3, 443–452 (2002).

    CAS  Article  Google Scholar 

  12. Hoyt, C. & Taylor, D. Pediatric ophthalmology and strabismus 4th edn (Elsevier, 2013).

  13. Katz, L. N., Hennig, J. A., Cormack, L. K. & Huk, A. C. A distinct mechanism of temporal integration for motion through depth. J. Neurosci. 35, 10212–10216 (2015).

    CAS  Article  Google Scholar 

  14. Long, E. et al. Dynamic response to initial stage blindness in visual system development. Clin. Sci. 131, 1515–1527 (2017).

    Article  Google Scholar 

  15. Tirosh, E., Shnitzer, M. R., Davidovitch, M. & Cohen, A. Behavioral problems among visually impaired between 6 months and 5 years. Int J. Rehabil. Res. 21, 63–70 (1998).

    CAS  Article  Google Scholar 

  16. Tröster, H., Brambring, M. & Beelmann, A. The age dependence of stereotyped behaviors in blind infants and preschoolers. Child Care Health Dev. 17, 137–157 (1991).

    Article  Google Scholar 

  17. Molloy, A. & Rowe, F. J. Manneristic behaviors of visually impaired children. Strabismus 19, 77–84 (2011).

    Article  Google Scholar 

  18. Mariotti, S. P. Global Data on Visual Impairment 2010 (World Health Organization, 2012).

  19. Gilbert, C. & Foster, A. Childhood blindness in the context of VISION 2020—the right to sight. Bull. World Health Organ 79, 277–232 (2001).

    Google Scholar 

  20. Frisén, L. Vanishing optotypes. new type of acuity test letters. Arch. Ophthalmol. 104, 1194–1198 (1986).

    Article  Google Scholar 

  21. Hilton, A. F. & Stanley, J. C. Pitfalls in testing children’s vision by the sheridan gardiner single optotype method. Br. J. Ophthalmol. 56, 135–139 (1972).

    CAS  Article  Google Scholar 

  22. Brémond-Gignac, D. et al. Visual development in infants: physiological and pathological mechanisms. Curr. Opin. Ophthalmol. 22, S1–S8 (2011).

    Article  Google Scholar 

  23. Li, Y. & Lin, H. Progress in screening and treatment of common congenital eye diseases. Eye Sci. 28, 157–162 (2013).

    PubMed  Google Scholar 

  24. Romeo, D. M. et al. Visual function assessment in late-preterm newborns. Early Hum. Dev. 88, 301–305 (2012).

    Article  Google Scholar 

  25. Livingstone, I. A., Lok, A. S. & Tarbert, C. New mobile technologies and visual acuity. Conf. Proc. IEEE Eng. Med Biol. Soc. 2014, 2189–2192 (2014).

    CAS  PubMed  Google Scholar 

  26. Blaikie, A. J. & Dutton, G. N. How to assess eyes and vision in infants and preschool children. BMJ 350, h1716 (2015).

    Article  Google Scholar 

  27. Mayer, D. L. et al. Monocular acuity norms for the Teller Acuity Cards between ages one month and four years. Invest. Ophthalmol. Vis. Sci. 36, 671 (1995).

    CAS  PubMed  Google Scholar 

  28. Wang, L. et al. Temporal segment networks: towards good practices for deep action recognition. ACM Trans. Inf. Syst. 22, 20–36 (2016).

    Google Scholar 

  29. Azen, R. & Budescu, D. V. The dominance analysis approach for comparing predictors in multiple regression. Psychol. Methods 8, 129 (2003).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the National Supercomputer Center in Guangzhou for computing resources support. This study was funded by the National Key R&D Program of China (2018YFC0116500), the Key Research Plan for the National Natural Science Foundation of China in Cultivation Project (91846109), the Natural Science Foundation of China (81822010), Guangdong Science and Technology Innovation Leading Talents (2017TX04R031), and the Pearl River Scholar Program (2016).

Author information

Authors and Affiliations

Authors

Contributions

H.L., E.L. and Z.Liu designed the research; H.L., X.L., Z.Lin, Z.Liu, J.L., J.C., X.W., W.L., E.L., H.C., R.L., Y.Y., W.C. and Y.L. collected the data; Y.X., A.X., X.H., Y.Zhang, Z.Z., X.D. and E.L. conducted the study; E.L. and J.H. analysed the data; H.L. and E.L. co-wrote the manuscript; Z.Liu, Y.Zhu and C.C. critically revised the manuscript; W.Z. did the technical review; and all authors discussed the results and provided comments regarding the manuscript.

Corresponding authors

Correspondence to Haotian Lin or Yizhi Liu.

Ethics declarations

Competing interests

The authors declare no competing interests

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, figures, tables and references.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Long, E., Liu, Z., Xiang, Y. et al. Discrimination of the behavioural dynamics of visually impaired infants via deep learning. Nat Biomed Eng 3, 860–869 (2019). https://doi.org/10.1038/s41551-019-0461-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-019-0461-9

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing