Main

For more than 20 years there has been a variety of monitors used at home to warn caregives of potential life-threatening events among infants at perceived risk for sudden death. More recently, monitors have been developed that record waveforms of cardiorespiratory data before, during, and after such events; these data potentially permit identification of factors associated with serious morbidity and mortality. It is commonplace for researchers and clinicians to assess infants based on a review of these waveforms and there have been a number of reports describing cardiorespiratory patterns among infants placed on home monitors (1–6). However, none of these reports provides a sufficient description of methods of assessment of waveforms to assure reproducibility of results, nor do any describe efforts to assure acceptable agreement between raters. To interpret these data and have a common basis for comparisons between studies, it is essential to establish reliable methods that describe and assess cardiorespiratory variables based on waveforms recorded from home monitors. Although several studies have reported a high level of agreement between observers when summarizing the total amount of apnea during a continuous overnight recording (7–11), studies have not assessed agreement between observers for categorization of specific waveforms as having apnea or bradycardia present or determined if observers agree on the specific characteristics of apnea or bradycardia events.

Early in the design and development of a multicenter study of the utility of home infant monitoring (CHIME) we found marked disparity in interpretation of such data among senior clinical investigators at the various study centers. Using 80 physiologic waveforms of beat-to-beat heart rate (derived from two lead ECGs) and respiration (thoracic impedance signal and inductance signal from thorax, abdomen, and sum of thorax and abdomen), seven senior investigators frequently disagreed when classifying the data as obstructive, central, or mixed apnea or no apnea (there was complete agreement only 4% of the time). From these preliminary data two issues became apparent: 1) each senior investigator used different criteria to define categories of apnea and 2) because of the uncertainty in identifying obstructed breathing, assignment to a type of apnea was often problematic.

Accordingly, the purposes of the present study, after the establishment of definitions, were: 1) to determine the level of agreement among observers in describing the features of cardiorespiratory waveforms recorded during home monitoring, 2) to identify the factors that interfere with agreement, and 3) to determine whether, with training focused on the sources of disagreement, CHIME technicians could achieve a high level of agreement for categorizing events (κ > 0.8) and for measuring event duration (ICC >0.8). Demonstration of this level of agreement in the assessment of home cardiorespiratory recordings is of paramount importance both for the conduct of our multicenter study and for the comparison of data among different studies.

Some important factors can contribute to the lack of agreement among investigators. The cardiorespiratory data obtained with a home monitor, compared with that obtained in a laboratory, may be limited by fewer channels of physiologic data, lack of corroboration of an event by an observer, and signal artifact, e.g. the distinction between movement artifact and airflow obstruction may be difficult without an end-tidal CO2 tracing, an airflow channel or a trained observer. Some events, e.g. a "normal" breath is inherently difficult to define no matter how much information is available. Even if algorithms were developed to enhance consistency of detecting or judging cardiorespiratory events, there needs to be clear definition of the physiologic behavior to develop the rules.

Thus, to conduct our multicenter study we needed to discern the factors that interfere with agreement and develop strategies to reach an acceptable level of agreement. To attain these goals we chose to develop stringent rules to describe and assess cardiorespiratory data, we trained a group of scorers to use these rule, we determined the reliability of their scoring, and, using an iterative approach, we refined the rules to enhance interrater reliability. In this report we describe our efforts in achieving reliability of the description and assessment of cardiorespiratory events obtained with a home infant monitor. Additionally, we tested the concordance between events detected on the home monitor and those observed with a multichannel recorder.

METHODS

The reliability assessment included two groups of raters: four CHIME investigators (J.M.S., L.J.B., S.L.D.W., C.E.H.) with considerable experience (from 10 to 20 y each) scoring physiologic signals, and five technicians trained to perform the CHIME centralized scoring at the Data Coordinating and Analysis Center.

Our approach to the assessment of interrater reliability involved: development of an event classification protocol and training of personnel in the use of this protocol; initial testing of interrater reliability among four CHIME investigators and five trained scoring technicians; review of the sources of disagreement between raters; retraining of the technicians; and finally, subsequent reassessment of interrater reliability.

Overview of approach. First, from a large array of physiologic waveforms recorded at home that met thresholds (i.e. events detected by the monitor) for either apnea or bradycardia, we selected representative samples for use in assessment of interrater agreement.

Second, for a given tracing each rater assessed the following: apnea recordings, 1) the longest interbreath interval based on the sum channel, and 2) occurrence of out-of-phase efforts on the abdominal and rib cage channels; bradycardia recordings, 1) duration of heart rate below 80 beats/min, and 2) heart rate nadir.

Third, we used two types of statistical analyses: 1) κ coefficient for categorical variables (i.e. presence or absence of events: apnea, bradycardia, and out-of-phase breath) and 2) ICC coefficient for the magnitude of differences in measurement of continuous variables (i.e. event duration and heart rate nadir).

Fourth, we recognized that disagreement in measurement of the duration of events had the potential to exaggerate disagreements in categorizing events as present or absent (e.g. raters might only differ in measurement by 1 or 2 s but fall on opposite sides of the event threshold). To address this issue we adopted a strategy where raters measured all interbreath intervals ≥10 s and any duration of a heart rate <80 beats/min, even though the monitor thresholds for recording waveforms had been set at apnea ≥16 s or bradycardia ≥5 s. The result of this approach was that: 1) we were able to assess the impact on agreement of varying the threshold for categorization of apnea or bradycardia as present (i.e. to what extent does a tight cutoff exaggerate disagreement), and 2) we were able to evaluate quantitatively (with ICC) the reliability in measurement of duration using a wide span of data.

Physiologic recordings. Infants were enrolled in one of four groups: SIDS siblings (infants who had a prior sibling who died of SIDS as documented at autopsy); ALTE (infants who had experienced an idiopathic apparent life threatening event that met specified severity criteria); preterm infants (born at <34 wk of gestation and <1750-g birth weight); and healthy term infants (born at term who had no problems requiring treatment during the newborn period). Written informed consent was obtained form the parent(s) or guardian(s) of all infants, and the study was approved by the institutional review board at each CHIME study site.

Each infant had cardiorespiratory waveforms recorded in the home using the CHIME study monitor (NonInvasive Monitoring Systems, Miami, FL). Ribcage and abdominal respiratory effort were recorded by respiratory inductance plethysmography bands and a third channel (proportional to tidal volume) was calculated based on their weighted algebraic sum (sum channel) (12,13). Heart rate was determined using an R-wave detection algorithm from the ECG taken using standard disposable infant ECG electrodes.

The monitor was set to record the physiologic waveforms whenever it detected apnea ≥16 s in duration or a heart rate that fell below 80 beats/min for ≥5 s. When the monitor identified an event as having met the recording criteria, physiologic waveforms were recorded for 75 s preceding onset of the event and for 30 s after resolution of the event.

Selection of waveforms for reliability studies from physiologic recordings. To compile a representative set of waveforms for evaluation of reliability, a member of the study team who did not participate in the reliability assessment screened a series of recordings. This screening verified the presence of apnea and/or bradycardia as well as false events. Two data sets were created such that each set contained waveforms from each of the four study groups and each subject contributed a similar number of waveforms. Ten infants (two ALTE, four preterm, two SIDS siblings, and two healthy term) contributed waveforms to data set no. 1, which included 62 waveforms recorded as having met the monitor's apnea threshold (i.e. potential apnea) and 26 waveforms recorded as having met the monitor's bradycardia threshold (i.e. potential bradycardia). In four of the above waveforms the potential apnea and bradycardia overlapped. In these cases the waveforms were assessed twice, once to assess the potential apnea and once to assess the potential bradycardia. Eight infants (two ALTE, three preterm, two SIDS siblings, and one healthy term) contributed to data set no. 2, which included 73 waveforms recorded as potential apnea and 40 recorded as potential bradycardia, none of which overlapped. Both the CHIME investigators and the technicians each assessed data set no. 1. After a period of additional training, the technicians each assessed only data set no. 2.

Evaluation of waveforms. A protocol was developed that described in detail the criteria for assessment of waveforms. Figure 1 provides a summary of these guidelines.

Figure 1
figure 1

Procedures for assessment of events. (A) Assessment of events recorded as potential apnea. Step 1: Determination of the adjusted baseline breath amplitude. The two cursor lines show demarcation of a region on the sum channel that contains deflections of "typical" size. The criteria for this region are: it precedes the event, contains at least three deflections, and is the longest segment that contains neither artifact nor obvious sighs. The software calculates, displays and stores the adjusted baseline breath amplitude. Step 2: Measurement of apnea duration. The cursors are placed at the peak of the two deflections that represent the longest interval between deflections that are ≥25% of the adjusted baseline breath amplitude. The software calculates and displays the apnea duration. If the duration is ≥10 s, the duration is stored, otherwise the event is classified as false. Step 3: Counting the number of out-of-phase deflections. Each of 14 out-of-phase deflections is denoted by a number. Out-of-phase deflections are defined as abdominal and ribcage signals that occur in opposite directions (i.e. 180° out of phase) at the same or very close to the same time and that do not result in a sum channel deflection of ≥25% of the adjusted baseline breath amplitude. If the abdominal and ribcage signals are shifted in time, to be considered out-of-phase, this shift can be no more than 25% of the overall duration of the breath (i.e. 90-270° out-of-phase with respect to each other). The total number of out-of-phase defections that are within the defined apnea are counted and manually entered into the system. (B) Assessment of events recorded as potential bradycardia. Step 1: Measurement of bradycardia duration. The cursors are placed at the point where the heart rate first declines below 80 beats/min and at the point that it returns to above 80 beats/min. The software tool determines and displays the bradycardia duration. The duration is recorded by the software tool as long as there is at least 1 s below 80 beats/min. If there is not at least 1 s below 80 beats/min, the event is classified as false. Step 2: Measurement of heart rate nadir. The cursors are placed on either side of the bradycardia (taking care to exclude periods including artifact). The software tool calculates, displays and records the heart rate nadir.

Full size image

For each waveform recorded as potential apnea by the monitor, each rater initially determined if the longest interbreath interval was ≥10 s and, if so, used the software tool to measure and store 1) duration of apnea and 2) number of out-of-phase deflections within the apnea where abdominal and rib cage channels had minimal or no deflection on the sum channel. Waveforms with interbreath intervals of <10 s were noted as having apnea absent and were not assessed further. The purpose of recording the number of out-of-phase efforts is that they may have value in identifying some periods of obstructed breathing. We recognize that out-of phase deflections are not equivalent to obstructed breathing and that ultimately we shall need additional data (e.g. airflow) to understand better the relationship between out-of-phase efforts and obstructed breathing. However, it is important here to assess whether we can reliably identify out-of-phase breathing.

For each waveform recorded as potential bradycardia (i.e. met monitor's threshold of <80 beats/min for ≥5 s) each rater initially determined if there was any period where the heart rate fell below 80 beats/min and, if so, recorded the duration of the bradycardia (i.e. the number of seconds heart rate <80 beats/min) and the heart rate nadir. Waveforms judged to have no period of time below the heart rate threshold were noted as having bradycardia absent and were not assessed further.

Assessment of waveforms was performed on a computer with a 21-inch video display using a software tool developed for this project. The physiologic montage included the following five channels: abdominal inductance, thoracic inductance, sum channel, ECG, and heart rate. The assessment procedure involved transfer of the contents of a single removable hard drive (i.e. all of the waveforms recorded from an individual subject during a 2-4-wk time interval) into the software tool on the base station computer. The software tool was used to display and assess each waveform in a consistent manner according to our protocol. After the rater assessed all of the waveforms recorded on a particular removable hard drive, a new removable hard drive, obtained from another subject, was transferred to the base station computer and evaluated.

Training. The four investigators received their training as part of the process of developing the criteria to be followed in the assessment of waveforms. This process involved approximately 30 h during which the investigators reviewed a large number of waveforms, reached a consensus regarding the assessment criteria for the CHIME study, and verified that they each understood and were able to apply these criteria. At the time of this formal assessment of reliability (approximately 2 mo later) each of the investigators was provided with a written summary of the criteria that they developed.

The five technicians were all newly hired for the centralized scoring for the CHIME study, located at the Data Coordinating and Analysis Center in Boston, MA. All of these individuals had some prior experience assessing physiologic signals, but not in the format used for the CHIME study and not in the laboratories of any of the CHIME investigators. Before the assessment of data set no. 1, the technicians received approximately 120 h of formal training and practice scoring according to the previously described criteria. This training was under the supervision of two individuals at the Data Coordinating and Analysis Center (M.J.C. and M.P.) who both participated in the development of the assessment criteria and had over 10 y of experience in the assessment of physiologic signals.

After the initial assessment of agreement (data set no. 1) the technicians received an additional training session. This consisted of approximately 40 h during which the technicians reviewed the specific reasons for disagreement and were instructed in strategies designed to improve the consistency in interpretation of the most difficult types of signals.

Data analysis. Agreement among raters was assessed with κ for categorical variables including: apnea present or absent, bradycardia present or absent, and number of out-of-phase deflections exceeding or not exceeding a specific threshold; and ICC for continuous variables including: duration of apnea, duration of bradycardia, and heart rate nadir (14). κ and ICC properties are as follows: +1 for perfect agreement, <0 for poorer than chance agreement, and ≥0 for better than chance agreement. There are published "arbitrary benchmarks" when using κ for describing the relative strength of agreement: <0.00 poor, 0.00-0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, and 0.81-1.00 almost perfect (14).

Agreement was initially assessed for data set no. 1 followed by a review of the results in an attempt to identify sources of major disagreements among the raters. After additional training to reduce these sources of disagreement, data set no. 2 was scored by the five technicians and κ and ICC values were calculated. The goal of this second assessment was to determine whether it was possible to train individuals to improve agreement. For this purpose it was not necessary to reassess both groups of observers. There is ample reason to believe that if we achieved success with additional training of one group of individuals that similar training among other groups of individuals would likewise meet with success. We chose to perform this second assessment among the technicians because these were the individuals designated to perform the scoring in the data analysis phase of the entire CHIME study. We elected to use centralized scoring for this large multicenter study because we recognized that this would provide the best opportunity to achieve and sustain a high level of agreement among raters.

RESULTS

Categorizing waveforms as true or false events. Because each rater measured the duration of any apnea ≥10 s and any bradycardia ≥1 s (i.e. even when the rater's measurement was at a duration lower than the threshold set on the monitor), we could assess the impact on agreement of the use of more versus less stringent thresholds to categorize the presence of apnea or bradycardia. Figure 2A provides the agreement in categorizing apnea as present as the threshold varies from 10 (i.e. most liberal) to 16 s (i.e. most stringent). Two effects are demonstrated: 1) agreement improved as the threshold decreased below the monitor's recording threshold, and 2) with training, the technicians achieved a high level of agreement (κ = 0.83), even with use of the most stringent threshold.

Figure 2
figure 2

Agreement (κ) in assessment of categorical variables. (A) Agreement in classifying apnea as present or absent as the threshold varies from 10 to 16 s. (B) Agreement in classifying a bradycardia as present or absent as the threshold varies from 1 to 5 s. (C) Agreement in classifying an apnea as including out-of-phase efforts as the number required for such categorization varies from 1 to 4.

Full size image

Figure 2B provides the agreement in categorizing bradycardia as present as the threshold varies from 1 (i.e. most liberal) to 5 s (i.e. most stringent). For bradycardia events the trends are the same, although the impact of the changing the cutoff value is less apparent than that for apnea. The agreement among technicians was always near perfect.

Agreement in the characteristics of apnea and bradycardia events. Figure 2C illustrates the agreement in identifying out-of-phase efforts during an apnea using four different thresholds (i.e. ≥1, ≥2, ≥3, or ≥4 out-of-phase efforts). Agreement improved with training and also improved as the threshold value increased. The technicians achieved our goal of κ > 0.8 after the final training only after use of the threshold of ≥4 out-of-phase efforts. Figure 3 provides an example of a waveform where there was considerable rater disagreement regarding the number of out-of-phase breaths. The additional training received by the technicians focused on standardizing the use of the software's zoom and gain adjustment features.

Figure 3
figure 3

Example of an apnea event with disagreement in the counts of the number of out-of-phase efforts.

Full size image

Figures 4 and 5 illustrate agreement in measurement of the duration of apnea and bradycardia with Bland-Altman plots (15). These plots show the duration measured for each event and the extent to which each rater's measurement differed from the median. For data set no. 1, at least two investigators, in 23 waveforms, and at least two technicians, in 25 waveforms, noted an apnea duration of ≥10 s. For data set no. 2, at least two technicians, in 51 waveforms, noted an apnea duration of ≥10 s. The median apnea duration measurements were similar for the two data sets (10-28 s).

Figure 4
figure 4

Bland-Altman plots for apnea events. (A) All 23 events from data set no. 1 for which at least two of the four investigators measured an apnea duration of ≥10 s (16 events were four of four, 5 were three of four, and 2 were two of four). (B) All 25 events from data set no. 1 for which at least two of the five technicians measured an apnea duration of ≥10 s (22 events were five of five, 1 was four of five, 1 was three of five, and 1 was two of five). (C) All 51 events from data set no. 2 for which at least two of the five technicians measured an apnea duration of ≥10 s (44 events were five of five, 5 were four of five, and 2 were three of five).

Full size image
Figure 5
figure 5

Bland-Altman plots for bradycardia events. (A) All 10 events from data set no. 1 for which at least two of the four investigators measured a bradycardia (i.e. heart rate <80 beats/min) duration of ≥1 s (2 events were four of four and 8 were three of four). (B) All 11 events from data set no. 1 for which at least two of the five technicians measured a bradycardia (i.e. heart rate <80 beats/min) duration of ≥1 s (all 11 were five of five). (C) All 13 events from data set no. 2 for which at least two of the five technicians measured a bradycardia (i.e. heart rate <80 beats/min) duration of ≥1 s (all 13 were five of five).

Full size image

As described before, we assessed agreement in measuring apnea duration by calculating the ICC coefficient. We included in this analysis all apnea events for which two or more observers found a longest interbreath interval of at least 10 s. This approach permits assessment of agreement over the widest possible range of measurement. Agreement in measurement of duration was initially relatively low for both investigators and technicians (ICC 0.33 and 0.36, respectively). However, the technicians improved markedly after additional training (ICC = 0.85). One can appreciate the practical impact of this improvement by contrasting the wide dispersion on the Bland-Altman plots shown in Figure 4, A and B, with the relatively tight fit found in Figure 4C. Figure 6 provides an example of a waveform with rater disagreement in measurement of apnea duration. The additional training received by technicians focused on strategies to differentiate various types of artifact (especially due to movement and poor signal quality) from genuine breaths.

Figure 6
figure 6

Example of an apnea event with disagreement in measurement of duration. Four different measurements of duration are illustrated; each was obtained by at least two of the nine raters who assessed this waveform.

Full size image

Figure 5, A-C, gives Bland-Altman plots for bradycardia duration. For data set no. 1, at least two investigators, in 10 waveforms, and at least two technicians, in 11 waveforms, noted a bradycardia duration of ≥1 s. For data set no. 2 at least two technicians, in 13 waveforms, noted a bradycardia duration of ≥1 s. The bradycardia duration measurements were similar for the two data sets (4-24 s). Agreement in measurement of bradycardia duration was higher for technicians than for investigators (ICC 0.83 versus 0.99), but was quite high in all cases. This suggests that it is easier to achieve consistency in measurement of bradycardia duration than in apnea duration.

The nadir in heart rate during bradycardia events was a mean of 62 beats/min in both data sets. Agreement in measurement of heart rate nadir was better for the technicians than investigators (ICC 0.77 versus 0.65). With additional training focused on measurement of heart rate in artifact free regions, technicians achieved total agreement (ICC = 1.0).

DISCUSSION

The purpose of this study was to quantify the level of agreement between raters when describing the features of cardiorespiratory waveforms recorded from infants on home monitors, to identify factors that impede interrater agreement and to determine whether CHIME study technicians could achieve a high (κ and ICC coefficients >0.8) level of agreement. We have shown that despite considerable initial training we were unable to achieve a high level of agreement among both investigators and technicians. We have shown, however, that additional training of technicians, directed at specific sources of disagreement, did result in improved agreement to levels considered generally sufficient for research purposes. Thus, from these data, the benefits of rigorous scoring algorithms, and the need for intensive training are quite clear. Our assessment before establishing clear guidelines indicated that experienced investigators disagreed in their categorization of cardiorespiratory events. Published descriptions of cardiorespiratory patterns in infants, except for assessment of agreement in the total amount of apnea during overnight studies, have not included formal attempts to assess interrater agreement. We can reasonably assume that experts would exhibit considerable disagreement in their judging of certain published data. Conflicting findings, within and across studies, may result from such disagreement, even with use of similar methods. In this form of clinical research, one must define events carefully and meticulously instruct and train those responsible for assessment of waveforms.

We have also demonstrated the impact on agreement of varying the threshold for defining presence of apnea and bradycardia. As expected, when there is not a high level in agreement in measuring event duration, there is exaggeration in disagreement in judging categorically the presence of apnea or bradycardia. This exaggeration is minimized by using more liberal thresholds. A practical strategy, often used in home monitoring, that reduces the impact of this problem is the common practice of setting the record threshold on a monitor at 5 s below the alarm threshold.

There are several factors that may contribute to disagreement between observers. First, the signals provided do not represent a "gold standard" for assessing cardiorespiratory events and, therefore, may be insufficient for an observer to define events accurately. Second, raters may differ in their understanding of the definition of variables and the criteria they employ when describing the waveforms. This may result from insufficient definition of a variable or insufficient training of raters. Hunt et al. (16) have described how varying definitions for apnea-onset (i.e. peak of inspiration versus end of expiration) may have a substantial impact on the assessment of apnea during overnight pneumogram recordings. Finally, inherent ambiguities in the signals make it difficult for raters to agree even when they use similar criteria in making judgments. An example of a physiologic pattern inherently difficult to interpret is the identification of out-of-phase breathing efforts during apnea (defined as efforts in opposite directions on of the abdominal and rib cage channels with no substantial deflection of the sum channel). Single out-of-phase breathing efforts often have low amplitude excursions and may be viewed variably as obstructive, central, or mixed apnea. Only with four or more out-of-phase breaths during a period of apnea for ≥16 s did we find high agreement between raters. The relationship between out-of-phase efforts and obstructed breathing remains to be determined, as does the biologic relevance of repetitive out-of-phase efforts compared with single out-of-phase breaths during the same time frame. Even "breaths" with no thoraco-abdominal asynchrony (i.e. in-phase), but low amplitude, are judged variably. Events with low amplitude excursions contribute substantially to disagreement among observers. Only with computer-based scoring that follows established rules will one likely reduce this type of disagreement. Computer-based automated scoring offers the potential to eliminate inconsistencies in scoring. The development of such a scoring system, however, is difficult and first requires consensus of the criteria used by the computer.

When interpreting the level of agreement observed in this study there are several factors that need consideration. The waveforms displayed continuous physiologic data beginning 75 s before an event and continuing until 30 s after resolution of the event (typically a total of 2 min). This is roughly twice the duration usually displayed by commercially available monitors. The increased ability to assess each epoch within its overall context may have favorably influenced reliability. Alternatively, in some instances, depending on the quality of the recorded signals, even a 2-min display time may be insufficient to understand the context of the displayed signals. Second, we note that the criteria developed for assessment of waveforms, although based on commonly used standards, were developed uniquely for this project. Our reliability may be strongly influenced by the specific criteria used. In addition, waveforms were displayed on a computer screen using unique software for display of waveforms and measurement of the variables. The display and measurement techniques clearly affect the level of agreement achieved. One must also consider the selection process for the waveforms used in the data sets. Although we selected waveforms to provide a representative sample of CHIME data, it is possible that we would have achieved different levels of reliability with waveforms that were either harder or easier to interpret than those we had selected. Finally, the impact of assessment of breathing by respiratory inductance plethysmography needs consideration. Most home monitors use transthoracic impedance to assess breathing movements. Although the levels of interrater agreement observed in this study might not seem applicable to assessment of waveforms of transthoracic impedance, in our judgment interrater agreement in assessment of waveforms of transthoracic impedance is as difficult to achieve as that for respiratory inductance plethysmography. Each of the factors contributing to disagreement apply similarly to both methods, except that one cannot assess out-of-phase breathing by transthoracic impedance.

In summary, although researchers and clinicians commonly describe events based on cardiorespiratory recordings, agreement among experienced individuals may be poor. This lack of agreement may result in conflicting findings, within and across studies, and mandates careful definition of variables, intensive training, and demonstration of reliability in the conduct of studies. However, when clear guidelines are developed, and sufficient training is provided, raters can attain a high level of agreement in judging the type and duration of cardiorespiratory events. The CHIME study, using a respiratory inductance plethysmography-based home monitor for breath detection, has developed guidelines and tools for assessment of cardiorespiratory waveforms and implemented training strategies that have achieved a high level of agreement in the assessment of cardiorespiratory waveforms.