Introduction

Agreement on patient symptoms across data sources is relevant to primary care practice and research. This information is often obtained directly from research participants by self-administered surveys or interviews,1 which can be relatively cost-efficient and organisationally straightforward to implement.2,3 However, the limitations of self-report relate to accuracy, recall, interviewer skills, and willingness to report.4 Another common method for assessing symptoms is medical record review, which can be costly, labour-intensive, and time-consuming,5 especially for large province-wide or nationwide studies where study participants access different medical services across diverse geographical areas. Medical record abstraction is further limited by illegibility, varying levels of completeness, and inaccuracies resulting from delayed documentation by busy physicians.6,7

Agreement between clinicians and patients regarding the assessment of presenting clinical symptoms is important for patient satisfaction8–12 and symptom resolution.13–15 In research, differences in estimations of the prevalence or incidence of symptoms that are dependent on data source can lead to substantial differences in estimated disease parameters.16 An understanding of the relative agreement and disagreement between the occurrence of symptoms by self-report compared with medical records can be useful in the interpretation of the clinical and research literature.

Some studies have assessed the agreement between patient self-report and medical record data and found that agreement differs depending on the medical issue.17–22 Symptom research has looked at patient-clinician agreement of symptoms related to angina pectoris,23 myocardial infarction,24,25 psychological and somatic disorders,8,26–29 HIV infection,30 and cancer.31,32 One study found fair to substantial agreement between adult self-assessment and clinician assessment on the symptoms related to ‘strep throat’.33 However, there is a relative paucity of primary care reports for infectious diseases, particularly for respiratory infection.

A large clinical trial on influenza among Hutterite community members used both self-report and medical records to collect data on symptoms, which allowed for the assessment of agreement between sources. The objective of the current study was to compare research participants' self-report of 10 signs and symptoms related to respiratory infection with primary care records.

Methods

Study design and population

The present study is a cross-sectional analysis of data collected for a randomised controlled trial (RCT) of vaccinating children in Hutterite communities against influenza.34 The Hutterites are an Anabaptist religious group who live in communal technologically-advanced farming colonies of about 100 people. There are approximately 347 colonies in Canada: 179 in Alberta, 61 in Saskatchewan, 105 in Manitoba, and two colonies in British Columbia.35 Individuals do not have a personal income; all earnings are held in common and the funds for essentials are distributed according to need.36 Typically, at the age of 15 years, adolescents finish their grade eight education and become apprentices in the duties of the colony. Work is assigned along the lines of traditional gender roles: farming and agricultural jobs for men; gardening, cleaning, and kitchen duties for women.37 Most Hutterites are married by the age of 23 years, following baptism as adult colony members.36

Research nurses enrolled and followed people from 46 Hutterite colonies in the RCT (22 in Alberta, 22 in Saskatchewan, and two colonies in Manitoba). Colonies were eligible for participation based on geography (within 150 km radius of designated cities or towns) and membership (at least 10 members at high risk of influenza complications). Children and adolescents were vaccinated with either a standard dose of inactivated trivalent influenza vaccine or hepatitis A vaccine. All participants (the vaccinated children and other colony members) were then monitored for influenza-related symptoms during the influenza season. The design, methods, and results of that trial have been described elsewhere.34

Patient reports of signs and symptoms

Study surveillance took place from 28 December 2008 to 23 June 2009. Research nurses handed out packages of family diaries of daily checklists for 10 common signs and symptoms of influenza: fever, cough, runny nose, sore throat, headache, sinus problems, muscle aches, fatigue, earache, and chills. Oral and written instructions including examples were provided. If someone in the family was symptomatic, then the appropriate symptom box would be checked and the name of the sick person(s) would be filled in beside it. If all the family members were asymptomatic, the box ‘No one was sick’ was checked. Fever was defined as a temperature of ≥38°C. Participating families were given thermometers to take oral temperatures for this purpose.

Self-report data were collected using a two-step process: (1) a family representative filled out the study diaries for participating family members on a daily basis; and (2) the research nurses visited the Hutterite colonies twice per week to check diary entries and interview individual participants (or mothers in the case of infants) to confirm the reported symptoms and assess other symptoms. For each symptomatic person the nurses filled out an individual symptom checklist, including those elicited at the interview but not reported on the study diary. They also collected information about outpatient healthcare visits made for reported symptoms, including physician name or healthcare facility, location, and date of medical visit. This surveillance method ensured a limited amount of time between medical visits and data collection by the research nurses (e.g. 1–3 days).

Requests for information from medical records

For each reported medical visit a one-page Patient Information Request Form was faxed to the medical facility asking for patient record data regarding presenting symptoms, with an equivalent list of symptoms as in the study diaries (Figure 1). The institutional review boards at McMaster University, the University of Calgary, the University of Saskatchewan, and the University of Manitoba approved the study. The analysis was restricted to an individual's first confirmed medical visit to maintain independence of observations.

Figure 1
figure 1

Content of Patient Information Request Form faxed to medical offices

Full size image

Statistical analyses

We calculated individual two-by-two contingency tables for each symptom. For self-report we included the symptoms reported by the research nurse on the day of the medical consultation. To test for differences in the mean number of reports per source we used the paired Student t test. Significance levels were set at p<0.05.

For symptom agreement we calculated total agreement (number of concordant pairs/total sample) and kappa coefficient. Kappa measures the strength of agreement beyond that expected solely by chance (observed agreement — chance agreement/1 — chance agreement) where 0=chance agreement and 1=perfect agreement.38 Due to the challenges associated with interpreting kappa values,39,40 we also calculated positive agreement (agreement about the presence of a symptom by both raters) and negative agreement (agreement about the absence of a symptom by both raters).41,42

For infants and young children (<5 years of age), the research nurses interviewed the mothers (in rare cases, the father or other guardian) about symptoms and medical visits. Therefore, agreement statistics were also calculated by age group to explore potential differences between subjective reporting by parents (for children aged <5 years) and self-reporting by older children and adults (aged ≥5 years). Cross-tabulations and kappa estimates were computed using SPSS 16.0 (SPSS, Chicago, IL, USA).

Results

Availability of data for both medical records and self-reports

Of the 3,274 participants in the RCT, 252 (8%) individuals in 37 of the 46 (76%) colonies reported at least one outpatient medical visit during the study influenza season. The medical visit was confirmed by the care source used for 176 (70%) individuals (Figure 2). This was the sample included for analysis.

Figure 2
figure 2

Flow diagram of participants included in the sample for analysis

Full size image

Sample characteristics

The mean age of the study participants was 24 years (34 (19%) aged <3 years, 56 (32%) aged 3–15 years, 13 (7%) aged 16–22 years, 46 (26%) aged 23–49 years, and 27 (15%) aged >50 years); 63% were female. Medical visits were made between January and June 2009. Because we used data from participants' first confirmed medical visits reported during the influenza season, 141 (80%) were made prior to the introduction of the novel H1N1 pandemic influenza in Canada on 23 April 2009.43 At least one of the 10 symptoms was self-reported by 142 (81%) persons. Of the 142, 48% were symptomatic for <4 days at the time of the medical visit; the mean (SD) number of sick days was 3.7 (4.5). According to the medical records, 162 (92%) individuals were diagnosed with a respiratory illness: otitis media (24%), upper respiratory tract infection (17%), sinusitis, pharyngitis, bronchitis (12% each), pneumonia (4%), and influenza (3%). Most received care from a family physician or general practitioner (95%) at a family physician office (80%), while 17% visited a hospital emergency department.

Symptom reporting by data source

Table 1 shows the numbers and proportions of symptoms recorded in the medical records and self-reported by participants. Self-reports for fever, sore throat, and earache were significantly underestimated by 19%, 9%, and 8% compared with medical records. A significantly higher number of the 10 symptoms was recorded in the medical records than in self-reports (mean (SD) 2.5 (1.4) vs. 2.1 (1.4); paired t=2.2, p=0.03).

Table 1 Symptom reporting according to each data source and total agreement, kappa estimate, positive agreement, and negative agreement of symptoms between medical record and self-report
Full size table

Twenty-three participants (13%) self-reported ‘other’ symptoms while 48 (27%) had ‘other’ symptoms according to the completed Patient Information Request Forms. There were 12 self-reported other symptoms: vomiting (n=4), diarrhoea (n=3), eye problems (n=3), nausea (n=3), chest pain (n=2), dizziness, nasal congestion, neck pain, hoarseness, sneezing, crying, and nose bleed (n=1 each). From the medical records there were 15 other symptoms: vomiting (n=7), dizziness (n=5), wheeze (n=5), diarrhoea (n=4), chest pain (n=4), nasal congestion (n=4), tracheal pain (n=4), postnasal drip (n=2), rash (n=2), shortness of breath (n=2), abdominal pain (n=2), irritability (n=2), eye pain (n=2), constipation (n=1), and brochospasm (n=1).

Symptom agreement between patient self-report and medical report

Total agreements between self-report and medical record report ranged from 61% for sore throat to 88% for muscle aches and earache (Table 1). The highest kappa values were for earache (0.51) and cough (0.41). Other kappa values ranged from 0.38 (sinus problems) to 0.05 (chills). Negative agreement was considerably higher (68% for sore throat to 93% for muscle aches and earache) than positive agreement (13% for chills to 58% for earache) for each symptom except cough, where positive agreement (77%) was higher than negative agreement (64%).

The stratified analyses by age group are presented in Table 2. Thirty percent of the sample (n=52) were aged <5 years. A significantly higher number of symptoms was recorded in the medical records than in the parental report for this younger age group (mean (SD) 2.5 (1.2) vs. 2.0 (1.4); paired t=2.1, p=0.04). There was no difference in the older age group (mean (SD) 2.4 (1.4) vs. 2.2 (1.9)). There was a significantly higher proportion of earache documented in the medical records compared with parental reports; this difference was not found in the older age group. Also, there was a significant difference in the reporting of sore throat by data source in the older age group only. Participants aged <5 years had infrequent reports and very high negative agreements for five symptoms (headache, sinus problems, muscle aches, fatigue, and chills). Agreements and kappa were better for earache in participants aged ≥5 years.

Table 2 Symptom reporting according to each data source and total agreement, kappa estimate, positive agreement, and negative agreement of symptoms between medical records and self (or parental) report, stratified by age group
Full size table

Discussion

Main findings

This study compared data collected from self-reports and from medical records. Total agreements were good for fever, cough, headache, sinus problems, muscle aches, fatigue, earache, and chills (72–88%) but less adequate for sore throat (61%) and runny nose (64%). There were lower positive agreements (13–50%) and higher negative agreements (74–93%), indicating poorer agreement regarding the presence of symptoms with an imbalance weighted towards the absence of symptoms. The exception was cough, which had a positive agreement of 77% and a negative agreement of 64%. Our findings varied by age group, with better agreement for earache and lower agreements for headache, sinus problems, muscle aches, fatigue, and chills in older children and adults.

Interpretation of findings in relation to previously published work

Symptom disagreement may have been influenced by the perspectives of the person experiencing them, the observer, the symptoms themselves, measurement error, and context.44 The difference between directly experiencing a symptom and externally observing something indicative of a symptom should lead to different evaluations. The situational or contextual basis of the judgement also differs,45 suggesting that both perspectives are important. The patient experiences the totality of symptoms as troublesome, uncomfortable, and concerning. The patient's goal in identifying the symptoms may be to cease symptom interference in their life by treatment or alleviation. For clinicians, the key concern is to identify those signs and symptoms that confirm a diagnosis, exclude the presence of more serious disease, and identify the appropriate prognosis and plan of action. In our study, fatigue and muscle aches — which may be frustrating symptoms for patients but have very low diagnostic value in respiratory disease — were documented infrequently in the medical records. Clinicians may have focused on symptoms that they perceived to have a higher diagnostic value such as fever and cough, whose co-occurrence is predictive of laboratory-confirmed influenza during a seasonal epidemic.46–48 Physicians may also reliably record data about their patients' main complaints or classic symptoms but not the less typical symptoms.49

Other studies have found better agreement for concrete objective symptoms that require less interpretation.20,50–53 Reporting of fatigue, a ‘subjective’ symptom,25,54 in the medical records was extremely low (8%), suggesting that clinicians are less likely to recognise or document this symptom. Symptoms which required subjective assessment by parents or clinicians in younger children were reported infrequently, and their high negative agreements reflected the high concordance regarding the absence of those symptoms.

The moderate total agreement for fever (74%) may have been affected by differences in measurement. Fever was explicitly and objectively defined for the RCT as a temperature ≥38°C and consistently measured by thermometer. Temperatures documented in the health records may not have been collected or documented in a consistent manner (e.g. on the basis of patient or parent reports and not measured or measured by different methods or techniques). Fever might also have been influenced by the phase of infection or the use of antipyretics at the time of the medical visit.55

We must also consider limitations or errors at each source. Self-reported information can be imprecise due to better understanding of some symptoms than others, underreporting, lack of motivation to report accurately, and poor compliance. Medical records can also be problematic. Several studies have found non-reporting and misreporting in medical records.56,57 Busier physicians may record less in the medical record or delay recording, leading to errors in recall.7 The process of abstracting information from the medical chart itself is also subject to imprecision.23,58 Furthermore, medical records were not written or kept for the purposes of this study and were guided by institutional policy, provider training, and provider preference.58,59

Discordances may also be attributed to the differences between settings such as the nature of the patient-physician (or participant-researcher) interaction, differential elicitation of symptoms, variation in reporting styles (specifics of the symptoms and diseases classification or documentation system23 versus the research protocol for data collection), environment (community or medical facility), and different motivations for reporting symptoms in each context. In the clinical setting, symptom information is often collected passively during the patient visit and then documented in the medical chart.31 The clinicians were then asked to translate their clinical notes into the symptom checklist on our research form. For the RCT, checklists were used followed by face-to-face interviews. Checklists have been shown to capture more symptom complaints than open-ended and passive reports.60 We used a comprehensive approach to data collection, including open-ended questions, condition-specific prompts, and follow-up with participants to minimise any missing data. In contrast, we were unable to assess the level to which medical records were complete.6

Limitations of this study

By using a homogeneous population and focusing on a specific set of symptoms, generalisability is limited. Overall, medical records captured more symptoms than self-reports. This is contrary to symptom research in other populations which suggests that clinicians report fewer symptoms than patients.31,33,61,62 The Hutterites are known as being ‘stoic’ and bearing pain and physical ailments without complaint.63 Participants may have underestimated their symptoms or hesitated to report them to the research nurse to avoid being perceived as complainers. Epidemiological studies of unique communities are important. Because we live in a multicultural pluralistic society, clinicians should consider the symptom experience within the broader community, cultural and psychosocial contexts of their patients.64

Missing self-reported data about outpatient medical services were not considered in the analyses as we could only follow up on self-reported medical visits. Also, we limited our analyses to participants whose physician or hospital had provided medical record information. It is possible that agreement between sources would be different for participants with missing medical record data.

We did not collect information regarding severity, which may have influenced reporting. Although our symptom list included simplified terms that were meant to be clear and unambiguous, there may have been discrepancies related to diverse definitions of particular symptoms.

We relied on clinic or hospital personnel to abstract the medical record data, but we cannot assume that data was abstracted in a methodologically consistent manner. By using a checklist, we considered an unchecked symptom to be absent. However, we cannot distinguish if the symptom was negatively reported as ‘not present’ in the medical record (e.g. no sore throat) or if there was an absence of reporting. We do not know how an independent researcher would have filled out the symptom checklist compared with the attending physician.

Conclusions

Information from patient medical records might be a valuable supplement to self-reports, enhancing the probability that symptoms are fully captured by research investigators. Deciding which data source to use depends on the population and outcome of interest and whether the results will be used for clinical decision-making, research, or surveillance.7