Introduction

In psychiatric practice, the most common mental illnesses include anxiety, depression, bipolar disorder, schizophrenia, and obsessive-compulsive disorder in Chinese population [1]. The rating scales, which are incorporated into clinical practice, offer psychiatrists important clinical information that may have been missed or not addressed in the clinical interview. The information can also be used to follow progression of symptoms and effectiveness of treatment.

However, the number of competing scales can make choosing a measure difficult [2]. Most of the self-rating scales target specific symptoms such as the Patient Health Questionnaire (PHQ-9) for depression [3], Generalized Anxiety Disorder Questionnaire 7-item scale (GAD-7) for anxiety [4], and Mood Disorder Questionnaire (MDQ) for hypo-mania [5]. These scales help quantify disease severity after a brief interview with a doctor. Symptom Checklist-90-Revised (SCL-90-R) [6] is a widely used self-rating scale which covers a relatively broad range of psychological problems and symptoms including anxiety, depression, psychosis, and obsession. However, the SCL-90-R contains 90 questions and takes around twenty minutes to complete. Besides, the SCL-90-R does not evaluate the manic or cognitive symptoms. Therefore, there is a need for developing a time-saving and comprehensive self-rating symptom screening scale. Moreover, the fast-screening questionnaire would also provide useful information for recommending of illness-specific scales following the initial clinical interview.

The COVID-19 pandemic has caused a heavy burden on mental health [7]. Home quarantine and other social distancing rules are making people feel isolated, stressed, anxious or even panicky. Anxious and depressive moods have been reported as a main health concerns in this pandemic [8, 9]. In addition, because of the quarantine rules, those who have mental problems are less likely to leave home for help. Therefore, an online fast-screening questionnaire that can access easily may help them to evaluate their current mental status and decide whether to visit a doctor.

The popularity of smartphones and cloud services makes it possible for quickly transfer, analyze, and store data. Through the internet, the doctors can receive the initial general screening report from the patients, quickly assess the main symptoms, and offer more precise interviews and diagnoses when patient comes to the clinic. In addition, a fast online screen questionnaire will also benefit large-scale epidemiological surveys.

In the present study, we propose a fast online screen questionnaire for symptom identification of mental illness symptoms (FSQ-MIS). To the best of our knowledge, there is the first online questionnaire that covers most of the major mental symptoms. We tested the reliability and validity on a large cohort of Chinese participants. The proposed FSQ-MIS was also used to predict the trend of anxiety and depressive events during the pandemic, and the parallel validity with DSM-5 diagnosis were also obtained.

Materials and methods

Participants and procedure

A total of 3828 young adult mental disorder patients (mean age = 25.4, SD = 4.0; male/female = 1803/2025) were recruited in the outpatient psychiatric department at Second Affiliated Hospital, Zhejiang University School of Medicine from December 2019 to November 2021. The normal controls were 984 medical staff and students (mean age = 25.0, SD = 4.7; male/female=476/507). There exists no significant age and gender difference between patients and controls (t = 0.356, p = 0.722; x2 = 0.539, p = 0.463). All the 4812 participants filled out the FSQ-MIS.

The patients were also evaluated using other clinical classic scales, including Hamilton Anxiety Rating Scale (HARS, n = 1703), Hamilton Depression Rating Scale (HDRS, n = 1849), Pittsburgh Sleep Quality Index (PSQI, n = 2363), Yale-Brown obsessive-compulsive scale (YBOCS, n = 544), Positive and Negative Syndrome Scale (PANSS, n = 145) and Bech-Rafaelsen Mania Scale (BRMS, n = 40).

According to DSM-5, 2940 subjects were diagnosed with depressive/anxiety-related disorder; 1814 subjects were diagnosed with sleep disturbance; 525 subjects were diagnosed with psychosis; 463 subjects were diagnosed with obsessive-compulsive disorders, and 456 subjects were diagnosed with bipolar or excitation status.

Each patient received a face-to-face interview with the examiner and completed the questionnaires online using their smartphones. FSQ-MIS and PSQI are self-rating scales, which were fulfilled by the participants using their own smartphones. HARS, HDRS, PANSS, YBOCS, and BMRS were examiner-rating scales, which were completed by the professional senior psychological assessors. The information was encrypted and stored in the hospital information system. The study protocol was approved by the Ethics Review Committee of the Second Affiliated Hospital of Zhejiang University and the informed consent was obtained from all subjects.

Measurements

The FSQ-MIS is a 19-item self-rating scale for assessing symptoms within the last week. It is a force-choice questionnaire with YES/NO responses (YES = 1, NO = 0). The specific items, which reference the SCl-90 [6] and MDQ [5], are listed in Table 1.

Table 1 Items of FSQ-MIS and the comparison with six scales.
Full size table

The HARS is a widely used classical anxiety symptom rating scale consisting of 20 items for rating psychotic and somatic anxiety. The HDRS is a classical depressive symptom rating scale. In present study, we used the 24-item version. The PSQI is a self-rating scale for the estimation of sleep disturbance. The YBOCS is a 10-item clinical examiner-rating scale for assessing the severity of obsessive and compulsive symptoms. The PANSS is a 30-item examiner-rating scale for assessing psychotic symptoms, including positive, negative, aggressive and general pathological symptoms. The BRMS is a scale for assessing manic/hypomanic symptoms. The above six measurements have been verified high reliability and validity both in the Chinese norm [10,11,12,13,14,15] and the present study (Table 1). The total scores of these six scales were used for verifying the criterion-related validity of FSQ-MIS.

The FSQ-MIS covers the core symptoms of anxiety, depression, bipolar disorder, psychosis, and obsessive-compulsive disorder. The comparison between FSQ-MIS and the six classical rating scales was shown in the lower half of the Table 1 [10,11,12,13,14,15].

Data analysis

We first used the Kaiser–Meyer–Olkin (KMO) Measure of Sampling Adequacy and Bartlett’s Test of Sphericity value to determine the suitability of the data for component analyses. Principal component analysis (PCA) with varimax rotation was then conducted for analyzing the construct validity of the FSQ-MIS. The receiver operating characteristic (ROC) curve and area under curve (AUC) were calculated to evaluate the discriminant validity of the FSQ-MIS in screening mental disorders according to DSM-5 criteria. Pearson correlation coefficient was used to evaluate the criterion validity between FSQ-MIS and other scales, including HARS, HDRS, PSQI, PANSS, PSQI, and BRMS. We defined the factor which had the equal weight in anxious and depressive symptoms as the FSQ-MIS-F1. Subsequently, the general log-linear analysis was conducted to estimate the main effect of time stages in predicting of having anxiety and depression in mental disorder population of this study, which was screened by FSQ-MIS-F1 or diagnosed by DSM-5 respectively. All the analyses were conducted using SPSS 26.0 (IBM Corp, Armonk, NY, USA).

Results

Reliability

A subset of the patients (n = 134) received the FSQ-MIS retest, the one-week test-retest reliability in patients was 0.852 (p < 0.001). The split-half reliability (odd items vs. even items) for the 4812 participants was 0.844 (p < 0.001).

Validity

Construct validity

The KMO value for the FSQ-MIS was 0.890 and the result of Bartlett’s test of Sphericity was above the satisfactory level (x2 = 13011.027, p < 0.001), indicating that the sampling was adequate for factor analysis.

The principal component analysis showed the 6-factor model explained 54.28% of the variance. The varimax rotated matrix is listed in Table 2. We define the 6 factors as anxiety/depression symptom (F1), cognitive deficit (F2), sleep disturbance (F3), psychotic symptom (F4), obsessive-compulsive symptom (F5), and excitation symptom (F6). The F1 consisted of item 2, 3, 4, 6, 14, and 15. The F2 consisted of item 5,9, and 17. The F3 consisted of item 1, 10, and 16. The F4 consisted of item 12, 7 and 8. The F5 consisted of item 11 and 13. The F6 consisted of item 18 and 19.

Table 2 Factor analysis matrix of FSQ-MIS.
Full size table

Anxiety and depression are two mental disorders having a high comorbidity rate [16]. The F1 factor was consistent with this clinical conclusion. The factor analysis exhibited that item 2, 3, 4, 6, and 14 has high weights in the component F1 (Table 2). Moreover, the F1 had significant correlations with both HDRS and HARS (Table 3), indicating that F1 reflects both anxiety and depressive symptoms. Therefore, we define this factor as FSQ-MIS-F1 and used it in the following general log-linear analysis for prediction.

Table 3 Criterion validity of factors in FSQ-MIS.
Full size table

Criterion-related validity

In this study, we used HDRS, HARS, PSQI, PANSS, YBOCS and BRMS for testing the criterion-related validity of FSQ-MIS. We calculated the correlation between the PCA-derived 6 factors and the 6 popular criterion scales. As shown in Table 3, F1 is highly correlated with both HDRS and HARS. F2 is highly correlated with all the popular criterion scales, indicating its potential in evaluating cognitive status. F3, F4, F5, and F6 were significantly correlated with PSQI, PANSS, YSOCS, and BRMS respectively. These results indicated the 5 factors (F1, 3, 4, 5, 6) had high discriminative power in identifying anxiety/depression, insomnia, psychotic, obsessive, and excitation symptoms.

Receiver operating characteristic analysis

To estimate the validity of FSQ-MIS in discriminating patients with mental disorders from healthy controls, ROC curve analysis was conducted for quantification. The FSQ-MIS total score and each factor score were used for discrimination. As shown in Fig. 1a, the AUC ranges from 0.716 to 0.983, indicating satisfactory discrimination validity in distinguishing patients from healthy individuals.

Fig. 1: ROC analysis.
figure 1

a ROC curves of FSQ-MIS total score and factor scores for discriminate patients and healthy controls. Total score for discrimination, AUC = 0.983 ± 0.002, p < 0.001, 95%CI [0.980, 0.986]. F1 for discrimination, AUC = 0.972 ± 0.002, p < 0.001, 95%CI [0.968, 0.977]. F2 for discrimination, AUC = 0.944 ± 0.004, p < 0.001, 95%CI [0.937, 0.952]. F3 for discrimination, AUC = 0.902 ± 0.004, p < 0.001, 95%CI [0.893, 0.911]. F4 for discrimination, AUC = 0.716 ± 0.007, p < 0.001, 95%CI [0.702, 0.731]. F5 for discrimination, AUC = 0.748 ± 0.008, p < 0.001, 95%CI [0.733, 0.763]. F6 for discrimination, AUC = 0.734 ± 0.008, p < 0.001, 95%CI [0.719, 0.750]. b ROC curves for measuring the performance of FSQ-MIS-Factors (F1, F3, F4, F5, F6) in identifying the mental disorders. F1 predicts anxiety and depression related diagnosis, AUC = 0.700 ± 0.010, p < 0.001, 95%CI [0.681, 0.720]; F3 predicts sleep disturbance related diagnosis, AUC = 0.791 ± 0.007, p < 0.001, 95%CI [0.777, 0.806]; F4 predicts psychosis related diagnosis, AUC = 0.820 ± 0.011, p < 0.001, 95%CI [0.799, 0.842]; F5 predicts obsessive-compulsive related diagnosis, AUC = 0.692 ± 0.012, p < 0.001, 95%CI [0.667, 0.716]; F6 predicts excitation symptom related diagnosis, AUC = 0.692 ± 0.013, p < 0.001, 95%CI [0.667, 0.717].

Full size image

As shown in Fig. 1b, to further evaluate the diagnostic validity of FSQ-MIS in screening different mental illness symptoms, we plotted the ROC curves of F1, 3, 4, 5, 6 in predicting anxiety and depression-related diagnosis (e.g., anxiety, depression, anxiety/depressive state), sleep disturbance related diagnosis (e.g., insomnia, narcolepsy), psychosis related diagnosis (e.g., brief psychotic disorder, delusion disorder, schizophrenia), obsessive-compulsive related diagnosis (e.g., obsessive-compulsive disorder, body dysmorphic disorder, eating disorder) and excitation symptom-related diagnosis (e.g., cyclothymic disorder, bipolar disorder, excitation state). The AUCs ranged from 0.692 to 0.820, indicating satisfactory discrimination validity of the FSQ-MIS.

Application in evaluating anxiety and depression situation during COVID-19

Anxiety and depression were the most common symptoms during the pandemic. our proposed FSQ-MIS showed advanced performance in predicting the anxiety and depression proportion during the COVID-19 pandemic. The two years were manually divided into four stages: from December 2019 to May 2020 (stage I), from June 2020 to November 2020 (stage II), from December 2020 to May 2021 (stage III), and from June 2021 to November 2021(stage IV). Stage I was the outbreak and most serious period of the COVID-19 pandemic in China. After the period, the pandemic gradually faded away. The proportion of having anxiety and depression screened by FSQ-MIS-F1 or DSM-5 diagnosis (e.g., anxiety, depression, anxious and depressive status) were used as structure variables in a general loglinear analysis to predict the COVID-19 effects. The result revealed stage I contributed most to anxiety/depression occurrence. The subsequent stage II and stage III had less influence on anxiety/depression occurrence, and stage IV had almost no effect on the occurrence (Table 4 and Fig. 2). The results demonstrated that anxiety and depression decreased with the relief of the pandemic. In addition, the FSQ-MIS also exhibited high parallel validity with DSM-5.

Table 4 Parameter estimation of General Loglinear Analysis.
Full size table
Fig. 2: General loglinear analysis of anxiety and depression in the four COVID-19 stages.
figure 2

FSQ-MIS-F1 = Factor1 of the fast-screening questionnaire for mental illness symptoms. The results show that the anxiety and depression decreased with the relief of the pandemic in China mainland, and the FSQ-MIS exhibits high parallel validity with DSM-5.

Full size image

Discussion

In this study, we developed a new fast online screening questionnaire, FSQ-MIS, for identifying symptoms of different mental disorders during the COVID-19 pandemic. The FSQ-MIS exhibited high test-retest and split-half reliability, as well as satisfied construct, criterion-related, and discrimination validity. When applied to the COVID-19 dataset, the FSQ-MIS-F1 exhibited high parallel validity with DSM-5 in predicting the anxiety and depression trends during different pandemic stages.

The PCA analysis found 6 factors that could explain 54.28% variance. The depression and anxiety symptoms are mixed because the factor F1 consists of both depression and anxiety items. This coincided with the previous findings that the comorbidity of anxiety and depression is as high as 67% [17]. Hence, the FSO-MIS is not used to completely distinguish anxiety and depression. It is more appropriate to combine anxiety and depression into one factor F1 in this scale. The cognition factor (F2) is a general factor, which showed significant correlation with all other 6 criteria scales including HARS, HDRS, PSQI, PANSS, YBOCS, and BRMS. In clinical practice, the phenomenon of cognitive deficits is observed in almost all kinds of mental illness [18,19,20,21,22]. Even though the cognitive factor showed high power in identifying patients with normal controls (Fig. 1), the power in identifying disease categories was low. This was the reason why F2 was excluded from the ROC curves in factor prediction analysis (Fig. 2).

As for the performance in identifying mental disorders, the factors of FSQ-MIS show high discrimination and predictability. The factors derived from FSQ-MIS exhibit satisfying performance in identifying mental disorders from healthy controls (AUC from 0.73 to 0.98), indicating the high parallel validity with DSM-5. In addition, the factors could also identify specific mental disorder symptoms with high accuracy (AUC from 0.69 to 0.82).

In the present study, data collection spanned around two years after the COVID-19 outbreak. We divided the two years into four stages to investigate the trend of anxiety and depression proportion (including anxious and depressive status) during COVID- 19. The result presented two similar curves, which were discriminated by FSQ-MIS-F1 and DSM-5 respectively. The anxiety and depression proportions gradually declined during the two years, coinciding with previous work on regression of stress response [23]. Moreover, similar patterns of FSQ-MIS-F1 and DSM-5 prove the parallel validity of FSQ-MIS with DSM-5 criteria.

There were several limitations to this study that should be addressed: (1) Even though the FSQ-MIS is validated using a large-cohort dataset (n = 4812), all the participants were collected in one hospital in China, therefore, the efficiency of FSQ-MIS still needs to be further validated in multi-center studies; (2) The participants in this study are mainly young adults. More children and aged people should be included in the future study; (3) The questionnaire should be further tested in a larger sample of the general population to exam its sensitivity in screening mental disorder prevalence.

In summary, the present study developed a new fast online questionnaire that is efficient in screening mental illness symptoms during the COVID-19 pandemic. The reliability and validity test demonstrated its advantages in identifying and evaluating psychiatric symptoms. The combination of the proposed self-rating questionnaire with a smartphone application makes it easily accessible for users anytime. We believe the scale to be a useful tool for research and clinical practice.