Neglecting students’ socio-emotional skills magnified learning losses during the pandemic

Did the dramatic learning losses from remote learning in the context of COVID-19 stem at least partly from schools having overlooked students’ socio-emotional skills—such as their ability to self-regulate emotions, their mental models, motivation, and grit—during the emergency transition to remote learning? We study this question using a cluster-randomized control trial with 18,256 high-school students across 87 schools in the State of Goiás, Brazil. The intervention sent behavioral nudges through text messages to students or their caregivers, targeting their socio-emotional skills during remote learning. Here we show that these messages significantly increased standardized test scores relative to the control group, preventing 7.5% of learning losses in math and 24% in Portuguese, consistent with the hypothesis that neglecting students’ socio-emotional skills magnified learning losses during the pandemic.


A.1 Additional Information on the Experiment
Participants.Participants consist of public school students enrolled in grades 10-12; typical age is 15-18 years old.All contacts were provided to Movva by the Goiás State Secretariat of Education.The total number of contacts in the database correspond to 18,256 students, 12,056 of which randomly assigned, across 57 schools, to receive SMS nudges between June 9th and December 31st, and 6,200 across 30 schools assigned not to receive nudges or any other SMS communication from their schools.
Power calculations before the onset of the intervention pointed out this sample size was large enough to detect relevant minimum effects on the outcomes of interest.
Data collection.Before the start of the intervention, the contacts' database was shared with the authors to complete the randomization at the school level, stratified by gender, grade and phone ownership.Schools were randomly assigned to either a treatment or a control group, following the group sizes above and using the statistical software Stata.The database including a treatment assignment indicator was then returned to Movva such that SMS nudges could be sent accordingly to the treatment group, but not to the control group.Data on online access to the platform and participation in offline school activities was shared by the Secretariat of Education with Movva, while data on motivation to return to regular classes once they resume was collected by Movva directly over SMS surveys, from rotating sub-samples of approximately 280 students in the treatment and control groups every week -from the week after the intervention started until 3 weeks after it ended.Weekly subsamples were also randomly drawn from the subject pool.Data on quarterly grades and attendance was shared by the Secretariat of Education in March 2021.Data on actual test scores in Portuguese and math from an assessment test conducted in person between April and May 2021 was shared by the Secretariat of Education in July 2021.Balance tests using Wald tests of simple and composite linear hypotheses were conducted before and after propensity score re-weighting to ensure that each treatment group is comparable with respect to students' and school characteristics.
These tests and results are detailed in Appendix B.2. Outcome data was shared with the authors, and analyzed following a pre-analysis plan pre-registered as trial 5986 at the AEA RCT Registry (Appendix A.3).We did not pre-register that we would analyze treatment effects on the high dropout risk proxy rather than official enrolment status because we did not anticipate at the time that the State would automatically enroll all students in 2021.We also did not pre-register the additional experiments, varying content within treated students, that we report in a separate section.All analyses were conducted by the authors using the statistical software Stata.Finally, collecting information on human participants over time is subject to attrition.Participants were free to leave the study at any time, which creates a risk of biasing the results if such attrition is correlated with treatment assignment.In this context, we tested whether the probability of students responding to the SMS surveys was affected by the treatment.The results, which are reported in appendix B.2, indicate that the probability of responding to SMS surveys is not systematically affected by the treatment.
Intervention.Movva, the start-up that powered the intervention evaluated in this study, specializes in promoting behavior change by sending nudges -frequent reminders and encouragement messages -directly to users' cell phones.Eduq+, the intervention evaluated in this study, has been shown to improve educational outcomes in an environment of regular classes across different settings (Lichand et al., 2022;Lichand & Wolf, 2021).In the context of this study, two nudges per week were sent over text messages (SMS) to high-school students or their primary caregivers, depending on phone ownership, in the treatment group.Nudges were organized in 2-week sequences of 4 messages, as follows (translated from Portuguese): Measures.School-level proficiency levels in State-wide exams: Publicly available data on school-level proficiency in the State-widge standardized exam for 2018, 2019 and 2021.This dataset contains average proficiency by school, grade (5th, 9th and 12th), subject (math and Portuguese), and year.Proficiency ranges from 0 to 500 within each subject.An average test score of 250 or above indicates basic proficiency or higher.As such, we define a sub-proficient indicator equal to 1 if a school's average score in that subject is below 250 in that year, and 0 otherwise.
Student-level standardized test scores: Instituto Sonho Grande shared administrative records on standardized test scores with the authors.This dataset contains information on actual test scores for 2,044 of our study participants from an assessment test conducted in person between April and May (in the second school quarter) 2021.Math and Portuguese test scores were available for 1,336 out of those students.We standardized scores withing each subject relative to the control group (such that test scores in our analysis have a mean of 0 and standard deviation of 1).We also generated a summary measure defined by the arithmetic mean of these two scores to deal with family-wise error rates in multiple hypotheses testing following Kling et

al. (2007).
Student's prolonged absenteeism right before the winter break: The Secretariat of Education shared administrative records with Movva, which then shared it with the authors.This dataset contained information on access to the online platform and participation in offline school activities at the student level, an indicator variable equal to 0 if the student logged in to the platform or participated in offline school activities on a given day and 1 in case they did not.Based on this information, we created a measure of prolonged absenteeism which was equal to 1 if a student had no attendance on record for two weeks in a row right before the winter break (between June 15th and June 26th), and 0 otherwise.See Appendix B.3.
Student's motivation to return to school once they reopen: Each week, students assigned to be surveyed by text message reported their motivation to return to school once regular classes resume by answering the following question: "Do you plan on returning to school once regular classes resume?".Movva coded lack of motivation to return as a binary indicator based on SMS replies, equal to 1 if the reply was "No" or similar, and zero otherwise, and shared that information with the authors.Student dropout risk.We define high dropout risk equal to 1 if a student had no math and no Portuguese grades on record in that school quarter, and 0 otherwise.See Appendix B.3.

Analysis method.
All results presented in the paper use intention-to-treat analyses by linking student identification numbers to the treatment condition they were assigned to before the start of the intervention.The Education Secretariat did not share information on siblings, so we could not drop pairs assigned to different treatment arms as we had specified in the pre-analysis plan.This is not an identification threat since within-household spillovers would only lead us to under-estimate treatment effects.Throughout the paper, we report intention-to-treat effects obtained from Ordinary Least Squares (OLS) regressions, by regressing each outcome on a binary indicator equal to 1 if the student was assigned to treatment and 0 otherwise.We restrict our analyses to intention-to-treat (ITT) effects because that we had no means of verifying whether students effectively received messages as intended.All p-values comparing treatment and control groups are obtained from t-tests of equality of coefficients between the treatment and control groups, with standard errors clustered at the school level in each case.All p-values comparing groups assigned to the content experiments within the treated group are obtained from t-tests of equality of coefficients between these treated groups, with classroom fixed-effects and standard errors clustered at the individual level in each case.

A.2 Literature review
This section outlines alternative mechanisms through which behavioral nudges might affect learning outcomes amongst children and adolescents.

Information
A number of studies have shown parents to have inaccurate information about their children's school effort and performance, as well as about the future returns to education in terms of expected employability and wages.In particular, parents tend to overestimate their children's school attendance (e.g., Bergman, 2021;Lichand et al., 2022 ) and school ranking (e.g., Dizon-Ross, 2019), and underestimate the returns to schooling (e.g., Jensen, 2010).
In face of informational constraints, providing parents with accurate information on their children's performance relative to their peers can lead to higher monitoring (Bergman, 2021), improving educational outcomes (e.g., Siebert et al., 2018;Barrera-Osorio et al., 2020;Dizon-Ross, 2019).
Naturally, students themselves can be systematically inaccurate in their self-appraisal of academic performance, or in their expectations about returns to effort (e.g., Yeager et al., 2019).As such, nudges to students may also help them to overcome informational constraints, increase school effort and improve educational outcomes.

Attention
Perhaps surprisingly, recent evidence suggests that nudges can lead to the same improvements even without child-specific information, by inducing parents to acquire information independently about their children's school performance (Lichand et al., 2022).
This evidence lines up with the hypothesis that executive functions (including attention, working memory and self-control) are constrained by the availability of mental bandwidth, which is taxed by financial worries linked to poverty (Mullainathan and Shafir, 2013).A number of studies have found poverty to reduce cognitive capacity precisely because of the drain that financial instability imposes on mental resources, leaving less bandwidth available for other decisions and tasks (e.g., Mani et al. 2013;Gennetian and Shafir 2015).Ultimately, under such stressors, sub-optimal decisions are likely to emerge, particularly when it comes to investments in children's education (Bergman, 2019).
In this context, nudges can reallocate parents' attention to children's school life.Lichand et al. (2022) show that even though nudged parents do not become more accurate about their children's attendance, they do become more accurate about the extent to which their children's GPA changes over the course of the school year, albeit only coarsely -consistent with higher monitoring effort.
When students are nudged directly, similar mechanisms might be at play: nudges might encourage them to acquire information about their own standing or about returns to effort even if the intervention does not convey information directly relevant to the specific circumstances of the recipient.

Future Orientation
These arguments are tied closely with another strand of the literature which argues that such behavioral nudges are effective in improving educational outcomes by helping participants overcome cognitive biases, in particular when it comes to time discounting.This problem has been well explored in decision-making scenarios in education and investments in human capital (e.g., Sutter et al. 2013;Castillo et al. 2011, Castleman andPage 2014).The tendency to overvalue immediate outcomes relative to future ones can be associated with self-control problems and lead to under-investments in general, in particular when it comes to children's education.
The literature provides a great deal of support for the hypothesis that behavioral interventions such as reminders, goal setting and social rewards can help individuals to overcome present-bias.Mayer et al. ( 2019) documents that behavioral nudges aimed at increasing parental engagement with their children's education have larger effects among present-oriented participants, consistent with the idea that nudges might also increase future orientation.
It is easy to imagine that such cognitive biases also plague the decision making of children hyperlinkalan2019(e.g., Alan, Boneva and Ertac, 2019).As such, by re-focusing students' attention to long-term goals, nudges might help them to overcome procrastination and other behaviors detrimental to learning.

From Constraints to Behavior
If nudges help students and their families overcome constraints arising from informational and psychological barriers, then they are expected to help engage students more effectively with the educational process (e.g., through higher attendance and homework completion; e.g., Lichand et al., 2022 ), resulting in improved learning outcomes (captured through higher test scores and lower grade repetition rates; e.g., Lichand et al., 2022) and increased motivation to stay in school (captured through lower dropout rates; e.g.Lichand & Wolf, 2021).
Whether parent-teacher interactions are a key mediator for translating less binding constraints into behavior change is an open question.Lichand & Wolf (2021) illustrate that the nature of these interactions might be key for nudges to ultimately improve educational outcomes.As such, without these interactions in the context of remote learning (especially in our study setting, where connectivity is very limited, such that, for most families, in-person interactions are not expected to be replaced by virtual interactions), it is not clear that nudges would effectively mitigate learning losses.This is the research question that this paper sets out to answer.

Introduction
The COVID-19 pandemic has forced 1.5 billion schoolchildren in 160 countries to stay at home while schools were shut down on sanitary grounds.Brazil is no exception.The nationwide decision to shut down schools for almost the entirety of the 2020 school year in order to limit the spread of the COVID-19 pandemic has forced all schools to switch to remote learning.Such rapid transition, combined with a mismatch between delivery channels and access conditions -as several State Sec-retariats of Education switched to online, while nearly 70 million households have no or only precarious access to internet -, are expected to severely impact learning, and potentially lead to a spike in school dropouts (Vegas, 2020); World Bank, 2020).
Schools have been trying to keep contact with their students by sending personal letters via post or by creating an online platform with tools that students can use.However, the attendance of the students, whether on the platform with the online tools or at school to pick up printed class material, is reported to be remarkably low.São Paulo State has reported that only 50% of its 3.5 million students are accessing the online learning platform daily as expected.The State Secretariat also broadcasts content on television.It is much harder to gather data on the share of students following classes on this format daily.
With the goal of increasing engagement in remote learning -and, particularly, online attendance and assignment completion -during the pandemic, as well as limiting its effects on learning gaps and school dropouts once schools are back, the Goiás State Secretariat of Education is testing various strategies in partnership with Instituto Sonho Grande.Goiás a relatively poor state located in the Center-West region of Brazil.Instituto Sonho Grande is a non-profit organization committed to improving high-school educational outcomes in Brazilian public school.As part of those strategies, they are interested in evaluating nudges (reminders and encouragement messages) sent twice a week to high school students, directly on their mobile phones via text messages (SMS).Towards that goal, they have hired Eduq+, an educational nudgebot that has been shown to improve educational outcomes (during normal times) in Brazil and Ivory Coast.
Eduq+ nudges users twice a week with motivating facts and suggested activities to engage them in the daily school life.It also allows schools to broadcast messages to all users weekly.The intervention has been evaluated in the context of regular schooling, targeted at parents of primary school children.The nudgebot has been shown to promote large impacts on school attendance, test scores and grade promotion rates (Bettinger et al., 2020), and to decrease school dropouts by 50% across multiple primary grades (Lichand and Wolf, 2020).
The version of Eduq+ to be evaluated in this study is, however, different from that in those studies, since nudges will be sent directly to students themselves.In case they do not have their own phone, messages will be sent to the mobile phone of their primary caregivers.Moreover, the context of remote learning is also much more challenging.Whether the intervention is still able to improve educational outcomes under those conditions is an empirical question.
This pre-analysis plan summarizes the design of a field experiment to test the following primary hypotheses: • Does nudging students increase usage of online learning tools by high school students?-Hypothesis: SMS nudges increase the share of students who access the online platform daily, and the share of students who hand in assignments (online or not).
• Does nudging students mitigate the negative effects of school closures on learning outcomes?-Hypothesis: SMS nudges improve attendance and grades, and decrease grade repetition and dropouts once in-person classes resume.and Linos, 2020), we conclude that the design is well powered to detect relevant short-term effect sizes.

Outcomes
We will document the effects of the treatments on the following categories of outcomes for students enrolled in high school (age 15 to 18): • Short-term outcomes: probability of logging into the online platform or picking up the material in school, probability of handed in of assignments, as measured by administrative records; • Long-term outcomes: attendance, grades, probability of grade repetition and probability of dropout, as measured by administrative records.
Since some students will receive messages on their own mobile phones, while for others it is their caregivers who will be nudged by Eduq+, we will estimate treatment effects within those two subgroups.Power calculations indicate that we could detect treatment effects of at least 1 and 0.9 percentage point for these two subsamples, respectively.
Since there are siblings in the data, we will remove from the main analysis cases when not all siblings are assigned to the same treatment conditions.Depending on how many siblings there are, we also plan to estimate within-family's externalities of the nudges, taking advantage of that sub-sample.

Empirical analysis
Since the intervention is randomly assigned, comparing treatment and control groups yields treatment effects of the SMS nudges on the outcomes of interest (Section III).
Using ordinary least squares regressions, we will estimate: Y

B.3 Short-Run Effects on students' Motivation and Attendance
This Section complements the analyses in the main text by estimating treatment effects on students' motivation to engage with school activities shortly after the onset of the intervention.These additional results help us provide direct evidence that the treatment effects on learning outcomes that we document are caused by the impacts of the intervention on students' socio-emotional skills and motivation targeted by the text messages.
Figure 4 estimates average treatment effects of the intervention on an indicator variable equal to 1 if the student attended no classes in the two weeks before the winter break, and 0 otherwise, based on administrative data.Despite quality issues for quarterly attendance data, we were able to obtain daily administrative data on attendance for each student in our sample during June 2020.Such data was in fact made available for the vast majority of students (see the Supplementary Materials).
Prolonged absenteeism is a well-known predictor of student dropouts.The figure showcases that while 7.21% of students in the control group had not followed remote learning activities right before the winter break, that figure was only 0.33% in the treatment group -an over 95% reduction (p-value = 0.00).All in all, these patterns confirm that students react to the messages both when it comes to their motivation to return to in-person classes and when it comes to attendance in remote learning activities right before the winter break.
Figures 2 and 3 plot the distribution of actual and standardized Portuguese and Math test scores.Last, Table 5 compiles intraclass correlations for different outcome

Figure 5
Figure5estimates average treatment effects on an indicator variable equal to 1 if the student states that they do not want to go back to school once in-person classes return, and 0 otherwise, using self-reported data.Panel A displays weekly averages for the treatment and control groups, and Panel B estimates week-by-week treatment effects.Panel 5a documents a striking pattern for lack of motivation to return to inperson classes in the control group, which increased more than 2-fold in little over a month (starting from 15% by the 2nd week of June and reaching 39% by the 3rd week of July).Panel 5a also shows that lack of motivation to return to in-person

table Panel (a): Summary std. test scores
Table 5 compiles intraclass correlations for different outcome variables, relevant for the computation of statistical power discussed in the main text.
Note: Pairwise correlations between standardised test scores and low (below median) baseline grades in math and Portuguese as well as the presence of online activities pre-pandemic for the control group.Panel (a) considers our summary measure (arithmetic mean), while panels (b) and (c) consider Portuguese and math test scores respectively (standardised).

Table 7
Balance tests: sub-sample with valid standardized test scores Students and schools considered are only the ones that had students that made the exam in Q2/2021.Panel A gives statistics for student variables.Panel B is based on 2019 Brazilian School Census and SAEGO dataset at the school level.P-values in Panel A were computed with standard errors clustered at the school level. Notes: