A novel approach to anxiety level prediction using small sets of judgment and survey variables

Anxiety, a condition characterized by intense fear and persistent worry, affects millions each year and, when severe, is distressing and functionally impairing. Numerous machine learning frameworks have been developed and tested to predict features of anxiety and anxiety traits. This study extended these approaches by using a small set of interpretable judgment variables (n = 15) and contextual variables (demographics, perceived loneliness, COVID-19 history) to (1) understand the relationships between these variables and (2) develop a framework to predict anxiety levels [derived from the State Trait Anxiety Inventory (STAI)]. This set of 15 judgment variables, including loss aversion and risk aversion, models biases in reward/aversion judgments extracted from an unsupervised, short (2–3 min) picture rating task (using the International Affective Picture System) that can be completed on a smartphone. The study cohort consisted of 3476 de-identified adult participants from across the United States who were recruited using an email survey database. Using a balanced Random Forest approach with these judgment and contextual variables, STAI-derived anxiety levels were predicted with up to 81% accuracy and 0.71 AUC ROC. Normalized Gini scores showed that the most important predictors (age, loneliness, household income, employment status) contributed a total of 29–31% of the cumulative relative importance and up to 61% was contributed by judgment variables. Mediation/moderation statistics revealed that the interactions between judgment and contextual variables appears to be important for accurately predicting anxiety levels. Median shifts in judgment variables described a behavioral profile for individuals with higher anxiety levels that was characterized by less resilience, more avoidance, and more indifference behavior. This study supports the hypothesis that distinct constellations of 15 interpretable judgment variables, along with contextual variables, could yield an efficient and highly scalable system for mental health assessment. These results contribute to our understanding of underlying psychological processes that are necessary to characterize what causes variance in anxiety conditions and its behaviors, which can impact treatment development and efficacy.

>50% did not fully complete the questionnaire.Of the ≥48,000 participants who completed the survey, those who did not clear data integrity assessments were omitted.Participants meeting quality assurance procedures (including survey completion as described in General Data Exclusion) were selected, with a limit of 4000-4050 total participants.Eligible participants were required to be between 18-70 years of age at the time of the questionnaire, could comprehend the English language, and had access to an electronic device (e.g., laptop, cell phone).Data from 4019 participants (mean age ± std = 51.4 ± 14.9 years) passed Gold Research's initial data integrity assessments, were anonymized, and then sent to the research team.Participants provided informed consent as described in Ethical Statement.

Survey Questions
For initial recruitment, participants received the following communication:

Gold Research Inc., a national market research firm and its client, Northwestern
University, request your participation in this study of emotional health.We will be evaluating how different emotions and experiences are connected and may relate to our emotional health.The information you provide will be kept confidential, coded to be anonymous so it cannot be connected back to you and will be used only for research purposes.Researchers will not be able to contact you or restudy you after this survey.We will not share your information with any other third party.

We will also not use your information to identify you individually or use your responses to market
or sell other services or products to you.As part of this effort, you will not be asked to provide any personal identifiers such as your name, email, phone number, address or social media handles.A unique identifier will be generated for you and each survey participant to enhance privacy.As part of the survey process, we will be able to tell if you completed the survey, but we will not be able to tell which answers were yours.For this study, we are going to ask you some questions about yourself and how much you like or dislike a set of pictures.You may discontinue this study at any time.We appreciate your help with this study, given the serious challenges facing many people regarding emotional health at this time.We thank you in advance.

Decline
If participants responded with "Accept", they were sent a further communication with the following: Thank you for participating in our survey.All responses during this survey are anonymous and confidential.We will be able to tell if you completed the survey, but we will not be able to tell which answers were yours.In this study, we aim to understand how different emotions and experiences relate to visual processing.
We are going to:

*Ask you some questions about yourself *Have you rate how much you like or dislike a set of pictures
For this study, your identity is protected and your answers are anonymous and confidential.Press "Next" to proceed.
The survey would then begin if participants pressed "Next".
Out of the larger survey, the questions regarding self-reported perceived loneliness, demographics, COVID-19 history are listed below using the abbreviations described in the main text for this study.Given the state part of State-Trait Anxiety Inventory (STAI) is a published and validated instrument (4), it is not detailed below.

Loneliness:
Please rate how often you have experienced each of the following on a scale of 1-5.
You avoid spending time with others or you spend most of your time alone (in the past one month): There are two pieces of information asked for in the following question.

Select the highest level of education you have achieved, and
Enter the total number of the years you completed below.

Edu:
Please select the highest level of education you have achieved: Participants were then shown two sets of 24 pictures with survey questions in between the two sets.

General Data Exclusion
A general set of quality assurance procedures were implemented as first used in other publications (1)(2)(3).Following this framework, participants meeting at least one of six criteria were removed from the cohort: (1) participants whose relative preference analysis yielded extreme outliers > 3 IQRS or incomplete measurements and met three other criteria for relative preference computation (please see next section), ( 2) participants that had mis-matching responses to years of education and education level in the survey, (3) participants with ten or more clinician-diagnosed illnesses, (4) participants that selected the same response for at least one section of the survey, (5) participants that rated all images in the behavioral task the same or with a variance of 1 (meaning only two of 7 Likert points were used), (6) participants that completed the questionnaire in less than 800 seconds (1-3).
For each participant, ratings of images in each of the six image categories were split into two setspositive and negative ratings.These positive and negative ratings for all six categories were then used to compute the mean, Shannon entropy and variance, yielding a tuple denoted as ( + ,  + ,  + ) for the positive ratings and ( − ,  − ,  − )for the negative ratings within each picture category.It should be noted that for computation of H, data was screened for cases where K=0 for a given category (i.e., cases where the subject made all neutral ratings to neither approach nor to avoid any picture in a given category).Computation of H for a given picture category requires that be K>0 given that H computation results in an undefinable log10(0/0) when K=0.In such cases, H was set to 0 for categories in which the subject rated "0" for all the stimuli.These computations resulted in 12 (, , ) tuples, or a total of 36 variables.For each participant, the mean across six categories was also computed for both positive and negative ratings which resulted in additional six variables: ( + ,  + ,  + ) representing reward behavior, and ( − ,  − ,  − ) representing aversion behavior.
For fitting models to participants' ratings, data was further screened for inclusion/exclusion criteria was as follows: 1. Valid entropy (H) calculations (as described above), 2. Exclusion of extreme outliers in the variables extracted from the graphs as followsloss aversion values > 200 (resulting in N=42 exclusions) and positive quadratic area > 100 (resulting in N=5 exclusions).
3. At least three points for a non-linear fit, or sufficient data points to fit the model with a computable R 2 , and 4. Consistency of individual and group models.This last criterion meant that the curve concavity for individual subject fits be consistent with the curve concavity of the grouplevel fits (and boundary envelopes).
Criteria ( 3) and ( 4) are important operational definitions since there is a potential for convergence failures with curve fitting.Overall, 3476 participants out of 4019 met all quality assurance criteria for picture rating data and survey data.
Next, for each participant, three separate curves for value function, limit function and tradeoff function (see Figure 2(B-D) were plotted with this data, detailed below, using MATLAB and the library polyfit.These curves from 500 randomly selected participants are shown in Figure S1.Subsequently, judgment variables were extracted from each of the participants' curves.The curves are briefly detailed below, followed by a description of the judgment variables extracted from each curve (for more details, see Azcona et al. 2022 (14)).
An array of judgment variables was extracted from each of the three curves (Figure 2(B)-(D) and Table 1).These variables are described below in psychological terms, and organized relative to the curve from which they were extracted: (1) The (K, H) curve, which consists of (K + , H + ) on the positive x-axis and (K -, H -) on the negative x-axis is analogous to the value function from Kahneman's Prospect Theory.The features that were extracted from this curve include: Loss Aversion (LA), Risk Aversion (RA), Loss Resilience (LR), Ante and Insurance.
Loss Aversion (LA): Loss aversion was computed as the absolute value of the ratio of the linear regression slope of (log K -, log H -) to the linear regression slope of (log K + , log H + ).It intuitively measures the degree to which one overweighs losses relative to gains.It is a fundamental measure in Kahneman and Tversky's Prospect Theory.

Risk Aversion (RA):
Risk aversion is extracted as the ratio of the second derivative of the (K + , H + ) curve to its first derivative, which itself produces a curve.To produce a unitary value for prediction, we calculated Risk Aversion at K + = 0.5.Informally, RA measures the degree to which an individual prefers a likely reward in comparison to a better but more uncertain reward.RA is a common notion in economics that studies decision-making under uncertainty.

Loss Resilience (LR):
Loss resilience is defined to be the absolute value of the ratio of the second derivative of the (K -, H -) curve to its first derivative, which also produces a curve.For prediction, we calculated LR at K -= 1.5.Informally, loss resilience is the degree to which an individual prefers to lose a small defined amount in comparison to losing a greater amount with more uncertainty associated with this loss.
Ante: Ante is the value of K + when setting H + = 0.It intuitively measures the stake one needs to engage in a game of chance and models the amount of a bid an individual is willing to make to enter a game of chance like poker.
Insurance: Insurance is the value of K -when setting H -= 0. It intuitively measures how much insurance an individual might need against bad outcomes.It mirrors the ante, but in the framework of potential losses.
(2) The (K,) curve, which consists of (K + ,  + ) on the positive x-axis and (K -,  -) on the negative x-axis is analogous to the mean-variance curve derived from portfolio theory (15) (15).

Reward Tipping Point (Reward TP):
The reward tipping point is the value of K + where the derivative d  + /d K + is equal to zero.Intuitively, this represents the rating intensity with maximum variance for approach behavior, and signals the amount of reward needed before an individual potentially decides to approach a goal-object.

Aversion Tipping Point (Aversion TP):
The aversion tipping point is the value of K -where the derivative d -/dK -is equal to zero.Intuitively, this represents the rating intensity with maximum variance for avoidance behavior, and signals the amount of aversive value needed before an individual potentially decides to avoid a goal-object.

Total Reward Risk (Total RR):
The total reward risk is the area under the curve on the positive quadrant of the graph of (K + ,  + ).Intuitively, this variable represents the relationship between K + and  + and can be thought of as a quantity that measures the amount of value one overall associates to positive stimuli.

Total Aversion Risk (Total AR):
The total aversion risk is the area under the curve on the negative quadrant of the graph of (K -,  -).Intuitively, this variable represents the relationship between K - and  -and can be thought of as a quantity that measures the amount of value one overall associates to negative stimulus.
(3) The (H + , H -) curve allows the comparison of patterns in approach and avoidance judgments.
The RPT Features that are extracted from this curve include: Reward-Aversion tradeoff, Tradeoff Range, Reward-Aversion Consistency and Consistency Range.

Reward-Aversion tradeoff (RA tradeoff):
The reward-aversion tradeoff is the mean of the polar angles of the points in the (H + , H -) plane.Intuitively, this measures the mean ratio of entropies or patterns in approach to avoidance behavior.
Tradeoff range: This is the standard deviation of the polar angles of the points in the (H + , H -) plane.Intuitively, this measures the standard deviation in the patterns of approach and avoidance behavior.This variance represents the spread for positive preferences and negative preferences across a set of potential goal-objects and is one measure of the breadth of an individual's (or group's) preferences.

Reward-Aversion consistency (RA consistency):
The reward-aversion consistency measures the mean of the distances of the data points in the (H + , H -) curve to the origin.Intuitively, this defines how individuals can have strong preferences (i.e., biases) for the same thing, reflecting conflict, or having low preferences for something, reflecting indifference.This gets at the consistency or compatibility of approach and avoidance, and how you can both like and dislike something, or be indifferent to both its positive and negative features.
Consistency Range: The consistency range measures the standard deviation of the distances of the data points in the (H + , H -) plane to the origin.Intuitively, this measures how the points in the H + H - plane vary about radial distance from the origin.The variance in this radial distance will reflect how much an individual goes between having conflicting preferences and having indifferent ones.

Supplementary
Peak Positive and Negative Risk, Reward and Aversion TippingPoint, Total Reward and Aversion Risk.The peak positive risk is value of  + where the derivative d  + /d K + . It calibrates preference magnitude (i.e., K) relative to the variance in rewards and sanctions.The (K, ) curve models the following question: Would an individual prefer a dollar with probability one, or value drawn from a normal distribution with mean of two and variance of two?The features that are extracted from this curve include: Peak Negative Risk (Peak NR): The peak negative risk is value of  -where the derivative d -/d K -is equal to zero.Intuitively, this represents the maximum variance for avoidance behavior.Like with the peak positive risk, this transition point is important to consider for avoidance decisions in the context of Markowitz's decision utility

Table 1 .
Summary of contextual variables for 3476 participants.S.D. = standard deviation.

Table 2 .
Summary fits of judgment variables for 3476 participants (A) Goodness of fit statistics are displayed for individual logarithmic fits (log K, H) and quadratic (, ) fits of the picture rating groups of the IAPS images.For both the positive (approach) and negative (avoidance) curves, goodness of fit statistics are assessed through coefficient of determination ( 2 ) values, coefficient of determination values adjusted for degrees of freedom (Radj 2 ) and F-test values (F-value).Mean and Standard Deviation (SD) of the goodness of fit statistics are reported.Participants with less than two of the six picture rating categories were not included in fits.Values are rounded off at two decimal places.(B) Fifteen judgment features, which are derived from the (K,H), (K, ) and (H + ,H -) curves of the IAPS picture rating groups.Five features are derived from the value function (K,H), six from the quadratic function, and four from the trade-off function (H + ,H -).Mean, standard deviation (SD), standard error of mean (SEM), and 95% confidence intervals of the values of the fifteen judgment features are reported for the cohort.Means are rounded off at two decimal places.Standard deviations (SD) are rounded off at three decimal places.Standard errors of means (SEM) are rounded off at four decimal places.Finally, the 95% confidence interval is reported to two decimal places.