Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Inferring latent learning factors in large-scale cognitive training data

Nature Human Behaviour volume 4, pages 1145–1155 (2020)Cite this article

Subjects

Abstract

The flexibility to learn diverse tasks is a hallmark of human cognition. To improve our understanding of individual differences and dynamics of learning across tasks, we analyse the latent structure of learning trajectories from 36,297 individuals as they learned 51 different tasks on the Lumosity online cognitive training platform. Through a data-driven modelling approach using probabilistic dimensionality reduction, we investigate covariation across learning trajectories with few assumptions about learning curve form or relationships between tasks. Modelling results show substantial covariation across tasks, such that an entirely unobserved learning trajectory can be predicted by observing trajectories on other tasks. The latent learning factors from the model include a general ability factor that is expressed mostly at later stages of practice and additional task-specific factors that carry information capable of accounting for manually defined task features and task domains such as attention, spatial processing, language and math.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Illustration of different learning curve prediction problems for one user.
Fig. 2: Predictive model performance.
Fig. 3: Inferred latent learning factors as a function of practice.
Fig. 4: Analysis of the first learning factor.
Fig. 5: Correlations between latent learning factors and manually derived task features.
Fig. 6: Changes in predicted learning curves based on changes in individual scores for individual factors.
Fig. 7: Distribution of individual scores for latent learning factors across demographic categories.

Similar content being viewed by others

Data availability

The original and preprocessed versions of the data can be accessed at https://osf.io/g9zkf. Source data are provided with this paper.

Code availability

The code used to analyse the data, run the Bayesian PCA model and create figures and tables can be accessed at https://osf.io/g9zkf.

References

  1. Spearman, C. ‘General intelligence’ objectively determined and measured. Am. J. Psychol. 15, 201–293 (1904).

    Article  Google Scholar 

  2. Carroll, J. B. et al. Human Cognitive Abilities: A Survey of Factor-Analytic Studies (Cambridge Univ. Press, 1993).

  3. Conway, A. R., Cowan, N., Bunting, M. F., Therriault, D. J. & Minkoff, S. R. A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence 30, 163–183 (2002).

    Article  Google Scholar 

  4. Kovacs, K. & Conway, A. R. Process overlap theory: a unified account of the general factor of intelligence. Psychol. Inq. 27, 151–177 (2016).

    Article  Google Scholar 

  5. Jensen, A. R. Clocking the Mind: Mental Chronometry and Individual Differences (Elsevier, 2006).

  6. Schubert, A.-L., Hagemann, D. & Frischkorn, G. T. Is general intelligence little more than the speed of higher-order processing?. J. Exp. Psychol. 146, 1498–1512 (2017).

    Article  Google Scholar 

  7. Kaufman, S. B., DeYoung, C. G., Gray, J. R., Brown, J. & Mackintosh, N. Associative learning predicts intelligence above and beyond working memory and processing speed. Intelligence 37, 374–382 (2009).

    Article  Google Scholar 

  8. Van Der Maas, H. L. et al. A dynamical model of general intelligence: the positive manifold of intelligence by mutualism. Psychol. Rev. 113, 842–861 (2006).

    Article  PubMed  Google Scholar 

  9. Barbey, A. K. Network neuroscience theory of human intelligence. Trends Cogn. Sci. 22, 8–20 (2018).

    Article  PubMed  Google Scholar 

  10. Steyvers, M., Hawkins, G. E., Karayanidis, F. & Brown, S. D. A large-scale analysis of task switching practice effects across the lifespan. Proc. Natl Acad. Sci. USA 116, 17735–17740 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Steyvers, M. & Benjamin, A. S. The joint contribution of participation and performance to learning functions: exploring the effects of age in large-scale data sets. Behav. Res. Methods 51, 1531–1543 (2019).

    Article  PubMed  Google Scholar 

  12. Donner, Y. & Hardy, J. L. Piecewise power laws in individual learning curves. Psychon. Bull. Rev. 22, 1308–1319 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  13. Ilin, A. & Raiko, T. Practical approaches to principal component analysis in the presence of missing values. J. Machine Learning Res. 11, 1957–2000 (2010).

    Google Scholar 

  14. Tipping, M. E. & Bishop, C. M. Probabilistic principal component analysis. J. R. Stat. Soc. B 61, 611–622 (1999).

    Article  Google Scholar 

  15. Lim, Y.J. & Teh, Y.W. Variational Bayesian approach to movie rating prediction. In Proc. International Conference on Knowledge Discovery and Data Mining 15–21 (ACM, 2007).

  16. Bell, R. M. & Koren, Y. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proc. Seventh IEEE International Conference on Data Mining 43–52 (IEEE Computer Society, 2007).

  17. Driver, C. C. & Voelkle, M. C. Hierarchical Bayesian continuous time dynamic modeling. Psychol. Methods 23, 774–799 (2018).

    Article  PubMed  Google Scholar 

  18. Kievit, R. A. et al. Developmental cognitive neuroscience using latent change score models: a tutorial and applications. Dev. Cogn. Neurosci. 33, 99–117 (2018).

    Article  PubMed  Google Scholar 

  19. Isiordia, M. & Ferrer, E. Curve of factors model: a latent growth modeling approach for educational research. Educ. Psychol. Meas. 78, 203–231 (2018).

    Article  PubMed  Google Scholar 

  20. Ram, N. & Grimm, K. J. in Handbook of Child Psychology and Developmental Science (ed. Lerner, R. M.) 1–31 (Wiley, 2015).

  21. McArdle, J. J., Ferrer-Caja, E., Hamagami, F. & Woodcock, R. W. Comparative longitudinal structural analyses of the growth and decline of multiple intellectual abilities over the life span. Dev. Psychol. 38, 115–142 (2002).

    Article  PubMed  Google Scholar 

  22. Preacher, K. J., Wichman, A. L., MacCallum, R. C. & Briggs, N. E. Latent Growth Curve Modeling (Sage, 2008).

  23. McNeish, D., Dumas, D. G. & Grimm, K. J. Estimating new quantities from longitudinal test scores to improve forecasts of future performance. Multivariate Behav. Res. https://doi.org/10.1080/00273171.2019.1691484 (2019).

  24. Rosenberg, M. D., Casey, B. & Holmes, A. J. Prediction complements explanation in understanding the developing brain. Nat. Commun. 9, 589 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  25. Settles B., Brust, C., Gustafson, E., Hagiwara, M. & Madnani, N. Second language acquisition modeling. In Proc. Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications (eds Tetreault, J., Burstein, J., Kochmar, E., Leacock, C. & Yannakoudakis, H.) 56–65 (ACL, 2018).

  26. Luttinen, J. & Ilin, A. Transformations in variational Bayesian factor analysis to speed up learning. Neurocomputing 73, 1093–1102 (2010).

    Article  Google Scholar 

  27. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C. & Strahan, E. J. Evaluating the use of exploratory factor analysis in psychological research. Psychol. Methods 4, 272–299 (1999).

    Article  Google Scholar 

  28. Abdi, H. in Encyclopedia for Research Methods for the Social Sciences (ed. Lewis-Beck, M. S. et al.) 792–795 (Sage, 2004).

  29. Widaman, K. F., Ferrer, E. & Conger, R. D. Factorial invariance within longitudinal structural equation models: measuring the same construct across time. Child Dev. Perspect. 4, 10–18 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Salthouse, T. A. The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103, 403–428 (1996).

    Article  CAS  PubMed  Google Scholar 

  31. Jensen, A. R. Regularities in Spearman’s law of diminishing returns. Intelligence 31, 95–105 (2003).

    Article  Google Scholar 

  32. Griffiths, T. L. Manifesto for a new cognitive revolution. Cognition 135, 21–23 (2015).

    Article  PubMed  Google Scholar 

  33. Goldstone, R. L. & Lupyan, G. Discovering psychological principles by mining naturally occurring data sets. Topics Cogn. Sci. 8, 548–568 (2016).

    Article  Google Scholar 

  34. Molenaar, D., Dolan, C. V., Wicherts, J. M. & van der Maas, H. L. Modeling differentiation of cognitive abilities within the higher-order factor model using moderated factor analysis. Intelligence 38, 611–624 (2010).

    Article  Google Scholar 

  35. Tucker-Drob, E. M. Differentiation of cognitive abilities across the life span. Dev. Psychol. 45, 1097–1118 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kievit, R. A. et al. Mutualistic coupling between vocabulary and reasoning supports cognitive development during late adolescence and early adulthood. Psychol. Sci. 28, 1419–1431 (2017).

    Article  PubMed  Google Scholar 

  37. Kievit, R. A., Hofman, A. D. & Nation, K. Mutualistic coupling between vocabulary and reasoning in young children: a replication and extension of the study by Kievit et al.(2017). Psychol. Sci. 30, 1245–1252 (2019).

    Article  PubMed  Google Scholar 

  38. Evans, N. J., Brown, S. D., Mewhort, D. J. & Heathcote, A. Refining the law of practice. Psychol. Rev. 125, 592–605 (2018).

    Article  PubMed  Google Scholar 

  39. Frischkorn, G. & Schubert, A.-L. Cognitive models in intelligence research: advantages and recommendations for their application. J. Intell. 6, 34 (2018).

    Article  PubMed Central  Google Scholar 

  40. Melby-LervÅg, M., Redick, T. S. & Hulme, C. Working memory training does not improve performance on measures of intelligence or other measures of ‘far transfer’ evidence from a meta-analytic review. Perspect. Psychol. Sci. 11, 512–534 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  41. Simons, D. J. et al. Do ‘brain-training’ programs work? Psychol. Sci. Public Interest 17, 103–186 (2016).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Feedback on the task feature categorization framework was provided by A. Kaluszka, O. Claflin, A. Osman and N. Ng. The authors received no specific funding for this work.

Author information

Authors and Affiliations

  1. Department of Cognitive Sciences, University of California, Irvine, CA, USA

    Mark Steyvers

  2. Lumos Labs, San Francisco, CA, USA

    Robert J. Schafer

Authors
  1. Mark Steyvers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert J. Schafer
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.S. and R.J.S. planned the research. R.J.S. provided the data. M.S. analysed the data. M.S. and R.J.S. interpreted the results and wrote the paper.

Corresponding authors

Correspondence to Mark Steyvers or Robert J. Schafer.

Ethics declarations

Competing interests

R.J.S. is an employee of Lumos Labs and owns stock in the company. M.S. has no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Task features for cognitive tasks grouped by process, stimulus, input method, and game design.

Solid circles indicate the presence of the task feature. The primary cognitive area associated with each task is given between parentheses.

Source data

Extended Data Fig. 2 Glossary of task features.

Definitions are provided for each process, stimulus, input method and game design feature used to describe the cognitive tasks.

Extended Data Fig. 3 Inferred latent learning factors with five factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process, stimulus, and game design features.

Source data

Extended Data Fig. 4 Inferred latent learning factors with seven factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process, stimulus, and game design features.

Source data

Extended Data Fig. 5 Inferred latent learning factors after oblique rotation (promax) that results in non-orthogonal factors.

Top panel: the heatmap visualizes the latent learning factors (columns) across games (rows). Positive (negative) values are visualized by brown (blue) colours. Each latent learning factor corresponds to a group of 8 columns, where the columns within a group correspond to different stages of practice (practice increases from left to right). Cognitive tasks are coloured according to primary task domain. The matrix shows the correlations between factors. Bottom panel: correlations between latent learning factors and manually derived task features. Positive (negative) correlations are illustrated by red (blue) colours. The hierarchical tree visualizes the similarity between the task features on the basis of the pattern of correlations between task feature and latent factors. Task features are grouped by process and stimulus features.

Source data

Extended Data Fig. 6 Changes in predicted learning curves based on changes in individual scores for individual factors.

For each factor (row), the blue, black, and red lines show the predicted learning curves for 10, 50, and 90 percentile scores on that particular factor and median scores on all other factors. Therefore, the red (blue) lines show the predicted learning curves for individuals who score high (low) on that factor while holding constant the contribution of other factors. Intercept differences are highlighted with circles at the start of the learning curve. To facilitate comparison, the vertical axis is the same for each particular task (column) but is different across tasks. Tasks are ordered by primary domain (colour).

Supplementary information

Supplementary Information

Supplementary Methods, Supplementary Results and Supplementary Figs. 1–4.

Reporting Summary

Peer Review Information

Source data

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steyvers, M., Schafer, R.J. Inferring latent learning factors in large-scale cognitive training data. Nat Hum Behav 4, 1145–1155 (2020). https://doi.org/10.1038/s41562-020-00935-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-020-00935-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing