Introduction

Infant resuscitation events are rare emergencies that require critical thinking, teamwork, clinical skills, and effective communication. High-quality team performance during resuscitations is key for successful clinical care of infants.1 Professional organizations have developed clear algorithms to direct teams in the proper steps of resuscitation with guides, including cardiopulmonary resuscitation (CPR), Neonatal Resuscitation Program (NRP),2 and Pediatric Acute Life Support (PALS).3 While resuscitation algorithms give specific recommendations related to practical steps during resuscitation, they do not fully address the complexities of executing those steps in the context of a team with individuals from different training backgrounds and with varying skill levels. Team training focused on crisis resource management and team communication are critical components of successful team performance during resuscitation.4,5,6 Team training in high reliability principles and culture of safety can improve consistency of practice and collaboration across disciplines to improve quality of care.7 Conceptual frameworks, including the Systems Engineering Initiative for Patient Safety model, can help teams understand the complex interactions between human factors including interactions between systems, processes, and outcomes.8,9 Decision support tools have been shown to improve compliance with NRP algorithms.10,11 Use of Read-Do checklists have also been shown to be effective tools for improving team performance.12,13 Other tools have been developed focused on adult emergency response with inclusion of behavioral components and practical steps of resuscitation,14,15,16,17 but to our knowledge, none currently exist that are efficient, facile, infant-focused, and applicable to both in-unit and simulation settings. Tools to reliably track overall infant code team performance are lacking, thus limiting the ability to evaluate metrics critical for optimal outcomes.

Measurement is a key component of effective quality improvement as it allows a team to assess the current state of practice, set goals, and monitor effects of improvement interventions.18 Tracking baseline team performance and identifying recurring themes can allow for focused educational efforts targeted at individualized, specific factors. Monitoring progress over time allows teams to assess effectiveness of interventions. In conjunction with tools that help track performance, simulation practice for rare events can be used to both evaluate and improve performance. Simulation practice and tool use have been shown to improve team resuscitation performance.19,20 Mock code programs allow for team practice in a controlled, consequence-free setting. Multidisciplinary team simulation practice allows for targeted educational efforts to be created and practiced.

Our project goals were to (1) gain consensus on the essential elements of infant team resuscitation and create a simple tool that would include both clinical resuscitation steps and crisis resource management factors, (2) use the tool to facilitate feedback during debriefs for immediate impact, and (3) use the tool to document performance over time, follow trends, and identify specific data-driven learning opportunities and test targeted quality improvement interventions.

Methods

Context

Boston Children’s Hospital (BCH) neonatal intensive care unit (NICU) is an urban 24-bed tertiary/quaternary care unit in Boston, MA, USA with approximately 650 annual admissions, of whom 80% are admitted to the medical service and 20% to the surgical service. All patients are out-born. PALS is used as the default resuscitation algorithm. Resuscitation events are typically attended by multidisciplinary staff, including attending neonatologists, neonatology fellows, neonatal nurse practitioners, respiratory therapists, clinical nurse specialists, nurses, and occasionally subspecialist physicians, pharmacists, and families. Prior to this study, the NICU had a well-established Crisis Resource Management (CRM) program that occurred off-site in our simulation center. The NICU CRM program offers monthly, 4-h courses, focused on team communication and understanding the human and systems factors that influence patient safety. We also completed debriefs after each simulation and in-unit resuscitation event.

Tool development

Our study used a group consensus method to identify key, observable elements critical for effective resuscitation. The modified Delphi method provides a framework to use data, literature review, and expert opinions to reach consensus agreement.21,22 The insight and opinion from a group of experts provides improved results compared to any single opinion. Given the complexity of infant resuscitation and limited evidence of key elements indicating successful team performance, expert opinion was chosen as the most accurate way to gain collective expertise and reduce bias to develop a comprehensive tool. We performed a literature search about key elements of code team performance and a review of 2 years of post-resuscitation debriefing documentation to generate ideas of key elements for inclusion. The multidisciplinary expert team included two institutional leaders (NICU Medical Director and Nurse Leadership), two neonatal simulation-trained expert facilitators (MD and RN), two neonatal fellows, two nurse practitioners, two neonatal nurses, and one respiratory therapist. For each cycle, respondents were asked if they agreed or disagreed with inclusion of each element on the checklist. An element was included if >80% of experts agreed it should be included.

After definition of essential elements, the modified Delphi process was used again to define an importance score and assign a weighted value to each element. The multidisciplinary expert panel was asked to define the importance of each checklist element, with 1 point for “major” and 0.5 points for “moderate.” Mean score and standard deviation were calculated. The importance score for the element was finalized after >80% of experts agreed.

Quality improvement interventions and study of the interventions

After consensus was obtained and the infant resuscitation team performance tool (Infa-RePT) was finalized (Fig. 1), we developed a multidisciplinary study team to use quality improvement methodology to improve team performance and created a key driver diagram (Fig. 2). In the baseline period (January 2019–December 2019), study team members completed the Infa-RePT score based on their individual observations during neonatal resuscitations in the simulation or in-unit environment. Study team members were observers and not part of the resuscitation efforts. The first ten resuscitation events were scored independently by a minimum of two expert reviewers and these scores were assessed for inter-rater agreement. We also tabulated the frequency of omission of key elements to identify unit trends in the baseline period and created a priority matrix using this data to inform Plan–Do–Study–Act cycles and opportunities for improvements through targeted interventions.

Fig. 1: NICU resuscitation team performance tool (Infa-RePT).
figure 1

Scoring tool developed via the modified Delphi method to be used in both simulation and in-unit settings to monitor team resuscitation performance.

Full size image
Fig. 2: Key driver diagram.
figure 2

Visual display of the quality improvement team’s theory of contributors to achieving the project aim, defining drivers, and change concepts.

Full size image

In the intervention period (January 2020–March 2021), the team used quality improvement methodology to perform tests of change and monitor team performance over time using the Infa-RePT tool. Key interventions included (1) creation of role responsibility read-do checklists for leaders and recorders, (2) enhancement of a mock code program, and (3) creation and dissemination of a mock code video highlighting key learning points. Other interventions included formalization of a code review process and staff tool training via in-person staff education, email education, and in real time prior to simulations.

The team reviewed pre-intervention resuscitation debriefs and identified “clarification of role responsibilities” as an opportunity for focused improvement. In response to this, the first intervention included creation of two role-responsibility checklists that included reminders for the “leader” and “recorder” roles. The tools were designed as “read-do” checklists that could be used in real time with key reminders. After creation, the tools were piloted, staff were educated, and then the checklists were placed on all code carts in the NICU. The second intervention was the creation of a monthly multidisciplinary, high-fidelity simulation mock code program, which was 30 min in length, held in proximity to the NICU, and included participants working that day in the NICU. Three expert facilitators led each simulation and debrief. Lessons learned from the sessions were disseminated broadly in a unit-wide educational email to broaden impact. Infa-RePT scores were completed at the end of each mock code with team input. Staff were also educated on the use of Infa-RePT tool to ensure appropriate use during in-unit resuscitation events. After review of all Infa-RePT scores, as a third intervention our multidisciplinary QI team created an infant resuscitation video with educational pause points to highlight key considerations during the code event. All staff were invited to view the video and reminders were circulated prior to mock code participation. This new mock code program enhancement was in addition to the ongoing crisis resource management simulation sessions. An additional project intervention included formalization of the code review process during multidisciplinary leadership meetings to ensure thorough review, systematically track issues, and fast-track system changes.

All staff members are required to maintain PALS certification and some may have participated in PALS training during the study period. No specific additional resuscitation training was performed during the study period, but lessons learned from debriefs were shared across the unit through safety briefs, email educational updates, and practice changes as per routine unit practice.

Measures

Infa-RePT performance scores were collected over the baseline and intervention periods as an outcome measure. Given the important benefit of having a full team present to participate in the mock code practicing in their trained role, we also tracked the multi-disciplinary attendance as a process measure. Ten roles were selected as essential for each mock code and the percentage of roles filled was tracked. Roles included attending, fellow, neonatal nurse practitioner, nurse leader/charge nurse, bedside nurse, two code cart nurses, recorder, nurse assistant, and respiratory therapist. To study tool generalizability, we also compared scores obtained from simulation events to in-unit events.

Analysis

To determine whether special cause variation was found after interventions, we developed statistical process charts (SPC) using the Chartrunner software to make X bar S charts. Excel was used to make run charts for analysis. Statistics are presented as percentage or mean ± standard deviation and t test was used to compare groups.

Results

Modified Delphi process for tool development

The initial search and review process yielded 27 candidate elements that were grouped into key categories for the modified Delphi process. Twelve multidisciplinary experts participated in the modified Delphi process. Two cycles were performed. After the second cycle, >80% agreement among the team determined the final 25 elements to include in the checklist. The group also collectively decided whether each element should be evaluated with options of “yes/no” or “yes/somewhat/no.” The team placed elements under main categories crucial for effective team resuscitation to be CPR, PALS, Role Responsibilities, Communication and Other Elements (Fig. 1).

The next step included ranking the elements based on importance as “moderate” or “major.” After 4 modified Delphi cycles, a >80% consensus was reached on each element. The elements on the tool were assigned a point value, with a lower score indicating a better performance. Elements of moderate and major importance were given a 0.5 and 1 point score, respectively, if not observed (Fig. 1). The highest (worst) possible score is 42.

Tool testing and baseline data collection

To assess inter-rater variability, a subset of the expert team consisting of at least 2 (range 2–4) observers scored 10 simulation resuscitations with an average agreement of 86% concordance (range 76–96%, SD 7.3) for checklist elements. Fifteen elements had ≥90% concordance and only 4 elements had ≤70% concordance (Table 1). The element with the lowest concordant response was “Appropriate switch off of compressors,” which only was agreed upon 50% of the time (Table 1). Overall, the scoring agreement was very high indicating low inter-rater variability and high reliability of the tool.

Table 1 Inter-rater agreement across all essential elements.
Full size table

During the pre-intervention period, 20 resuscitations were observed and scored. Infa-RePT scores were monitored over time with an average total score of 7.4 (range 2–12, SD 2.0). This included 16 crisis resource management simulations and 8 in-unit resuscitations. For 16 simulation scenarios, the average performance score was 7.6 (range 3.5–12, SD 2.8) and for 8 in-unit events, the average performance score was 5.7 (range 2–10.5, SD 3). Seventeen elements were observed >75% of the time. The most frequently observed elements were “RN code cart role clearly identified,” “other team member role clearly identified,” “proper specialists were called in a timely way if needed,” “family notified and updated promptly”, and “debrief occurred,” all of which each occurred >95% of the time. The least frequently observed element was “Reversible causes reviewed (Hs and Ts),” which were observed in 21% of resuscitations. Other elements that were not frequently observed were “appropriate pulse checks” 37% of the time, “code team used clear communication” 46% of the time, and “appropriate switch of the compressors” only 50% of the time (Table 2). Team compliance with key elements was lacking most frequently in the CPR category, followed by the PALS category.

Table 2 Least frequently observed elements during team resuscitations.
Full size table

Outcome and process measures

The Infa-RePT performance score significantly improved (decreased) and demonstrated special cause variation on SPC chart analysis with a decrease from 7.4 baseline (or “pre”) to 4.2 during the project (Fig. 3). Center line shift occurred after the second main intervention, the enhancement of the mock code program. Variability also decreased and improved as shown in S analysis on SPC chart. The multidisciplinary team attendance at the mock code sessions remained high throughout the project period with a median of 95% on run chart analysis (Fig. 4). No difference was seen between Infa-RePT score based on setting of resuscitation; in-unit (n = 10; average score = 5) and simulation (n = 26; average score = 6.2) were not significantly different (p = 0.3). (Supplemental Fig. 1). The educational video had 93 individual views (as of 11 March 2021) with 110 mock code participants attending simulations after the video was made available.

Fig. 3: Code performance improved over time.
figure 3

SPC control chart X bar S analysis shows Infa-RePT scores over time and a significant decrease in scores from 7.4 to 4.2 with special cause variation noted indicating improved performance after initiation of a mock code program. After the project start, a significant decrease in variability was also noted.

Full size image
Fig. 4: Attendance rate at mock code sessions remained high.
figure 4

The team documented the participation of each role during every simulation session with a goal of 100% of roles being filled by staff. Run chart analysis shows a high rate of attendance and participation with a median of 95%.

Full size image

Team confidence scores

Staff confidence in performing their role during resuscitation scenarios improved after the project started. We conducted a pre-project survey (n = 72) and a post-mock code survey (n = 103) of multidisciplinary staff in which they reported their confidence in different code scenarios. For resuscitations requiring intubation and those requiring chest compressions and medication delivery respectively, 68 and 63% of pre-survey respondents “strongly agreed” or “agreed” that they felt confident that they could perform their role well, compared to 91 and 96% in the post-survey. Pre-survey results also identified additional opportunities for focused education; only 41 and 22% “strongly agreed” or “agreed” that they felt confident to perform their role well in resuscitations requiring delivery of electrical shocks and emergent pericardiocentesis, respectively. These procedures will be used for future mock code scenarios to improve confidence and provide opportunity for practice.

Multidisciplinary staff reported that expansion of simulation program to include monthly mock codes in addition to the monthly CRM simulations was beneficial. Pre-project survey respondents reported that 68% felt that practice during simulation could help them perform their role better during in-unit patient resuscitations. Post-mock code survey results showed that 96% of respondents agreed that mock code increased their confidence to perform their role, 92% agreed that the mock code increased their medical knowledge, and 96% agreed that the mock code increased their communication skills.

Discussion

Collaborative team performance is critical to provide the best care possible to critically ill infant patients. When multidisciplinary teams work together effectively, patient outcomes can be optimized, medical errors can be minimized, and efficiency can be improved. While some tools have been successfully used in pediatric settings such as simulation-only, pediatric intensive care unit, or emergency rooms, no current measurement tool was available to capture the critical code performance tailored to the in-patient NICU setting, including such items as knowledge and communication. Given no clear evidence of the key factors to measure team performance, expert consensus was used to develop a novel tool to fill this gap in ability to measure performance in a standardized manner.

Use of this tool to follow team performance over time showed areas for growth and improvement. To target these areas, simulation scenarios through an enhanced mock code program allowed for the multi-disciplinary team to gain more practice with rare in-unit events. The implementation of this program led to a significant improvement in overall performance as measured by the Infa-RePT. Checklists have been shown to improve team performance. The tool itself is a checklist that the team can use during the debrief to assess performance of providers at varying skill levels simultaneously. It allows the ability for the team to reflect on opportunities for improvement immediately after an event. Over time, the tool can be used to collect composite data and monitor for trends or key vulnerabilities that can direct education, system changes, or focused interventions using quality improvement methodology. Sustainability of the tool usage is critical to ensure continued improvements. Embedding usage into workflow and regularly scheduled unit feedback are two methods currently employed.

Our study had several limitations. The modified Delphi method is a consensus generating methodology and focuses on expert agreement and not on direct evidence to show validity. This tool was studied at one site with no delivery room and patients up to 6 months corrected age, where we use PALS for resuscitation, not NRP. Therefore, we could not directly test delivery room resuscitations. Because this tool is focused on PALs resuscitations for in-unit code events, it would likely need to be modified for use in delivery room resuscitations, when teams may be smaller and role responsibilities different. To gain additional reliability, validity, and generalizability data, the tool could be tested in delivery room settings, and in NICUs more broadly. During the study, different team members participated in each resuscitation. Therefore, we were unable to assess serial individual performance but instead focused on group data over time. Additionally, there were several potentials for observer bias affecting the resuscitation scores because the team was not blinded, observations were performed in real-time, and observers had external knowledge of the participants. Also, time-based improvements may have contributed to score differences. Analysis also did not account for possible clustering effects caused by individual team members participating in variable numbers of resuscitations. Event complexity was not recorded and may affect team performance. Despite this, in general, simulation events were more complex than in-unit resuscitations and no difference in scores was found between these two environments (Supplemental Fig. 1).

Evaluating resuscitation performance and providing meaningful team feedback is essential to improve skills. To our knowledge, no current tool for use both in simulation and in-unit resuscitations incorporates both the resuscitation algorithm steps and key communication and team training crisis resource management elements that are essential for effective team performance. This tool can be used for immediate feedback to teams after resuscitation events in both the simulation and in-unit environment. It is facile and efficient and can be integrated into a debriefing session. The tool results are valuable not only to the individual team members but also to NICU leadership and educators who can use the specifics of the resuscitation performance data over time to target education and track effectiveness of the educational interventions. We showed how targeted interventions can improve team performance over time.

Conclusions

Through the modified Delphi method, we developed an in-hospital infant team resuscitation performance scoring tool (Infa-RePT) based on expert consensus. Healthcare teams can utilized the Infa-RePT to capture scores across varied settings (simulation or clinical) to target educational efforts to improve team performance and infant resuscitation skills. Interventions including enhancement of a mock code program, role responsibility tool development, and addition of an educational video improved team performance.