Inferring latent learning factors in large-scale cognitive training data

Subjects

Abstract

The flexibility to learn diverse tasks is a hallmark of human cognition. To improve our understanding of individual differences and dynamics of learning across tasks, we analyse the latent structure of learning trajectories from 36,297 individuals as they learned 51 different tasks on the Lumosity online cognitive training platform. Through a data-driven modelling approach using probabilistic dimensionality reduction, we investigate covariation across learning trajectories with few assumptions about learning curve form or relationships between tasks. Modelling results show substantial covariation across tasks, such that an entirely unobserved learning trajectory can be predicted by observing trajectories on other tasks. The latent learning factors from the model include a general ability factor that is expressed mostly at later stages of practice and additional task-specific factors that carry information capable of accounting for manually defined task features and task domains such as attention, spatial processing, language and math.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Illustration of different learning curve prediction problems for one user.
Fig. 2: Predictive model performance.
Fig. 3: Inferred latent learning factors as a function of practice.
Fig. 4: Analysis of the first learning factor.
Fig. 5: Correlations between latent learning factors and manually derived task features.
Fig. 6: Changes in predicted learning curves based on changes in individual scores for individual factors.
Fig. 7: Distribution of individual scores for latent learning factors across demographic categories.

Data availability

The original and preprocessed versions of the data can be accessed at https://osf.io/g9zkf. Source data are provided with this paper.

Code availability

The code used to analyse the data, run the Bayesian PCA model and create figures and tables can be accessed at https://osf.io/g9zkf.

References

  1. 1.

    Spearman, C. ‘General intelligence’ objectively determined and measured. Am. J. Psychol. 15, 201–293 (1904).

    Article  Google Scholar 

  2. 2.

    Carroll, J. B. et al. Human Cognitive Abilities: A Survey of Factor-Analytic Studies (Cambridge Univ. Press, 1993).

  3. 3.

    Conway, A. R., Cowan, N., Bunting, M. F., Therriault, D. J. & Minkoff, S. R. A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence 30, 163–183 (2002).

    Article  Google Scholar 

  4. 4.

    Kovacs, K. & Conway, A. R. Process overlap theory: a unified account of the general factor of intelligence. Psychol. Inq. 27, 151–177 (2016).

    Article  Google Scholar 

  5. 5.

    Jensen, A. R. Clocking the Mind: Mental Chronometry and Individual Differences (Elsevier, 2006).

  6. 6.

    Schubert, A.-L., Hagemann, D. & Frischkorn, G. T. Is general intelligence little more than the speed of higher-order processing?. J. Exp. Psychol. 146, 1498–1512 (2017).

    Article  Google Scholar 

  7. 7.

    Kaufman, S. B., DeYoung, C. G., Gray, J. R., Brown, J. & Mackintosh, N. Associative learning predicts intelligence above and beyond working memory and processing speed. Intelligence 37, 374–382 (2009).

    Article  Google Scholar 

  8. 8.

    Van Der Maas, H. L. et al. A dynamical model of general intelligence: the positive manifold of intelligence by mutualism. Psychol. Rev. 113, 842–861 (2006).

    PubMed  Article  Google Scholar 

  9. 9.

    Barbey, A. K. Network neuroscience theory of human intelligence. Trends Cogn. Sci. 22, 8–20 (2018).

    PubMed  Article  Google Scholar 

  10. 10.

    Steyvers, M., Hawkins, G. E., Karayanidis, F. & Brown, S. D. A large-scale analysis of task switching practice effects across the lifespan. Proc. Natl Acad. Sci. USA 116, 17735–17740 (2019).

    CAS  PubMed  Article  Google Scholar 

  11. 11.

    Steyvers, M. & Benjamin, A. S. The joint contribution of participation and performance to learning functions: exploring the effects of age in large-scale data sets. Behav. Res. Methods 51, 1531–1543 (2019).

    PubMed  Article  Google Scholar 

  12. 12.

    Donner, Y. & Hardy, J. L. Piecewise power laws in individual learning curves. Psychon. Bull. Rev. 22, 1308–1319 (2015).

    PubMed  PubMed Central  Article  Google Scholar 

  13. 13.

    Ilin, A. & Raiko, T. Practical approaches to principal component analysis in the presence of missing values. J. Machine Learning Res. 11, 1957–2000 (2010).

    Google Scholar 

  14. 14.

    Tipping, M. E. & Bishop, C. M. Probabilistic principal component analysis. J. R. Stat. Soc. B 61, 611–622 (1999).

    Article  Google Scholar 

  15. 15.

    Lim, Y.J. & Teh, Y.W. Variational Bayesian approach to movie rating prediction. In Proc. International Conference on Knowledge Discovery and Data Mining 15–21 (ACM, 2007).

  16. 16.

    Bell, R. M. & Koren, Y. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proc. Seventh IEEE International Conference on Data Mining 43–52 (IEEE Computer Society, 2007).

  17. 17.

    Driver, C. C. & Voelkle, M. C. Hierarchical Bayesian continuous time dynamic modeling. Psychol. Methods 23, 774–799 (2018).

    PubMed  Article  Google Scholar 

  18. 18.

    Kievit, R. A. et al. Developmental cognitive neuroscience using latent change score models: a tutorial and applications. Dev. Cogn. Neurosci. 33, 99–117 (2018).

    PubMed  Article  Google Scholar 

  19. 19.

    Isiordia, M. & Ferrer, E. Curve of factors model: a latent growth modeling approach for educational research. Educ. Psychol. Meas. 78, 203–231 (2018).

    PubMed  Article  Google Scholar 

  20. 20.

    Ram, N. & Grimm, K. J. in Handbook of Child Psychology and Developmental Science (ed. Lerner, R. M.) 1–31 (Wiley, 2015).

  21. 21.

    McArdle, J. J., Ferrer-Caja, E., Hamagami, F. & Woodcock, R. W. Comparative longitudinal structural analyses of the growth and decline of multiple intellectual abilities over the life span. Dev. Psychol. 38, 115–142 (2002).

    PubMed  Article  Google Scholar 

  22. 22.

    Preacher, K. J., Wichman, A. L., MacCallum, R. C. & Briggs, N. E. Latent Growth Curve Modeling (Sage, 2008).

  23. 23.

    McNeish, D., Dumas, D. G. & Grimm, K. J. Estimating new quantities from longitudinal test scores to improve forecasts of future performance. Multivariate Behav. Res. https://doi.org/10.1080/00273171.2019.1691484 (2019).

  24. 24.

    Rosenberg, M. D., Casey, B. & Holmes, A. J. Prediction complements explanation in understanding the developing brain. Nat. Commun. 9, 589 (2018).

    PubMed  PubMed Central  Article  Google Scholar 

  25. 25.

    Settles B., Brust, C., Gustafson, E., Hagiwara, M. & Madnani, N. Second language acquisition modeling. In Proc. Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications (eds Tetreault, J., Burstein, J., Kochmar, E., Leacock, C. & Yannakoudakis, H.) 56–65 (ACL, 2018).

  26. 26.

    Luttinen, J. & Ilin, A. Transformations in variational Bayesian factor analysis to speed up learning. Neurocomputing 73, 1093–1102 (2010).

    Article  Google Scholar 

  27. 27.

    Fabrigar, L. R., Wegener, D. T., MacCallum, R. C. & Strahan, E. J. Evaluating the use of exploratory factor analysis in psychological research. Psychol. Methods 4, 272–299 (1999).

    Article  Google Scholar 

  28. 28.

    Abdi, H. in Encyclopedia for Research Methods for the Social Sciences (ed. Lewis-Beck, M. S. et al.) 792–795 (Sage, 2004).

  29. 29.

    Widaman, K. F., Ferrer, E. & Conger, R. D. Factorial invariance within longitudinal structural equation models: measuring the same construct across time. Child Dev. Perspect. 4, 10–18 (2010).

    PubMed  PubMed Central  Article  Google Scholar 

  30. 30.

    Salthouse, T. A. The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103, 403–428 (1996).

    CAS  PubMed  Article  Google Scholar 

  31. 31.

    Jensen, A. R. Regularities in Spearman’s law of diminishing returns. Intelligence 31, 95–105 (2003).

    Article  Google Scholar 

  32. 32.

    Griffiths, T. L. Manifesto for a new cognitive revolution. Cognition 135, 21–23 (2015).

    PubMed  Article  Google Scholar 

  33. 33.

    Goldstone, R. L. & Lupyan, G. Discovering psychological principles by mining naturally occurring data sets. Topics Cogn. Sci. 8, 548–568 (2016).

    Article  Google Scholar 

  34. 34.

    Molenaar, D., Dolan, C. V., Wicherts, J. M. & van der Maas, H. L. Modeling differentiation of cognitive abilities within the higher-order factor model using moderated factor analysis. Intelligence 38, 611–624 (2010).

    Article  Google Scholar 

  35. 35.

    Tucker-Drob, E. M. Differentiation of cognitive abilities across the life span. Dev. Psychol. 45, 1097–1118 (2009).

    PubMed  PubMed Central  Article  Google Scholar 

  36. 36.

    Kievit, R. A. et al. Mutualistic coupling between vocabulary and reasoning supports cognitive development during late adolescence and early adulthood. Psychol. Sci. 28, 1419–1431 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  37. 37.

    Kievit, R. A., Hofman, A. D. & Nation, K. Mutualistic coupling between vocabulary and reasoning in young children: a replication and extension of the study by Kievit et al.(2017). Psychol. Sci. 30, 1245–1252 (2019).

    PubMed  PubMed Central  Article  Google Scholar 

  38. 38.

    Evans, N. J., Brown, S. D., Mewhort, D. J. & Heathcote, A. Refining the law of practice. Psychol. Rev. 125, 592–605 (2018).

    PubMed  Article  Google Scholar 

  39. 39.

    Frischkorn, G. & Schubert, A.-L. Cognitive models in intelligence research: advantages and recommendations for their application. J. Intell. 6, 34 (2018).

    PubMed Central  Article  Google Scholar 

  40. 40.

    Melby-LervÅg, M., Redick, T. S. & Hulme, C. Working memory training does not improve performance on measures of intelligence or other measures of ‘far transfer’ evidence from a meta-analytic review. Perspect. Psychol. Sci. 11, 512–534 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  41. 41.

    Simons, D. J. et al. Do ‘brain-training’ programs work? Psychol. Sci. Public Interest 17, 103–186 (2016).

    PubMed  Article  Google Scholar 

Download references

Acknowledgements

Feedback on the task feature categorization framework was provided by A. Kaluszka, O. Claflin, A. Osman and N. Ng. The authors received no specific funding for this work.

Author information

Affiliations

Authors

Contributions

M.S. and R.J.S. planned the research. R.J.S. provided the data. M.S. analysed the data. M.S. and R.J.S. interpreted the results and wrote the paper.

Corresponding authors

Correspondence to Mark Steyvers or Robert J. Schafer.

Ethics declarations

Competing interests

R.J.S. is an employee of Lumos Labs and owns stock in the company. M.S. has no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Task features for cognitive tasks grouped by process, stimulus, input method, and game design.

Solid circles indicate the presence of the task feature. The primary cognitive area associated with each task is given between parentheses. Source data

Extended Data Fig. 2 Glossary of task features.

Definitions are provided for each process, stimulus, input method and game design feature used to describe the cognitive tasks.

Extended Data Fig. 3 Inferred latent learning factors with five factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process, stimulus, and game design features. Source data

Extended Data Fig. 4 Inferred latent learning factors with seven factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process, stimulus, and game design features. Source data

Extended Data Fig. 5 Inferred latent learning factors after oblique rotation (promax) that results in non-orthogonal factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. The matrix shows the correlations between factors. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process and stimulus features. Source data

Extended Data Fig. 6 Changes in predicted learning curves based on changes in individual scores for individual factors.

For each factor (row), the blue, black, and red lines show the predicted learning curves for 10, 50, and 90 percentile scores on that particular factor and median scores on all other factors. Therefore, the red (blue) lines show the predicted learning curves for individuals who score high (low) on that factor while holding constant the contribution of other factors. Intercept differences are highlighted with circles at the start of the learning curve. To facilitate comparison, the vertical axis is the same for each particular task (column) but is different across tasks. Tasks are ordered by primary domain (colour).

Supplementary information

Supplementary Information

Supplementary Methods, Supplementary Results and Supplementary Figs. 1–4.

Reporting Summary

Peer Review Information

Source data

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Steyvers, M., Schafer, R.J. Inferring latent learning factors in large-scale cognitive training data. Nat Hum Behav (2020). https://doi.org/10.1038/s41562-020-00935-3

Download citation

Search

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing