408 - Analysis of Novice and Expert Rater Reliability for a Neonatal Resuscitation Program Adherence Assessment Tool

Publication Number: 408.3972

Sophia Martens, Tufts University School of Medicine, Scarborough, ME, United States; Misty Melendi, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Anya Cutler, MaineHealth Institute for Research, Montpelier, VT, United States; Lauren Weller, Tufts University School of Medicine, Portland, ME, United States; Leah Marie Seften, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Tara Palnitkar, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Allison Zanno, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Leah Mallory, Barbara Bush Children’s Hospital, Portland, ME, United States

Background: Simulation plays an important role for healthcare teams practicing neonatal resuscitation, a high acuity low occurrence event.1 This is especially important at rural hospitals with low birth volumes, where teams feel underprepared for neonatal resuscitation.2,3 In a pilot study at a rural hospital, we implemented a monthly neonatal resuscitation telesimulation (TS) program and found improved neonatal resuscitation program (NRP® ) guideline adherence.4 NRP® adherence was scored using a previously published scoring tool adapted for the 8th NRP® edition. We created an anchoring tool to standardize NRP® adherence scoring.

Objective: Analyze the interrater reliability of the adapted NRP® adherence scoring tool using the anchoring tool to evaluate simulated interprofessional resuscitation teams, including novice and expert raters.

Design/Methods: Two board-certified neonatologists scored 15 NRP® TS videos using the adapted NRP® scoring tool and anchoring tool. The neonatologists then independently trained two medical students (novice to NRP®) to score with the anchoring tool as a guide, using two different recorded NRP® TS scenarios. Each student then scored the 15 NRP® TS videos individually. Intraclass correlation coefficient was calculated between the two neonatologists, and between one neonatologist and two medical students with a two-way random effects model and an “average rater” unit using the psych package in R.

Results: The median cumulative score for neonatologists was 66.8% (range 39.0%, 86.2%) and was 70.3% (range 50.0%, 94.5%) for the students. ICC between the neonatologists was near perfect (0.99, p=0.000), very good between the students (0.778, p=0.001) and good between one neonatologist and the students (0.727, p=0.017.) See Table.

Conclusion(s): Using an anchoring tool with a modified previously published NRP® adherence assessment tool to rate recorded simulated neonatal resuscitations by practicing interprofessional teams, we were able to achieve good rater reliability across two expert and two novice raters. Expert raters demonstrated nearly perfect agreement and novices achieved very good agreement, with opportunity to improve in the equipment category. The ability to reliably incorporate a larger pool of less experienced raters is important as our center seeks to expand a telesimulation-based NRP® training program across a rural health system. This tool is also a part of the NeoImPACTS training collaborative.6 Next steps are to assess tool performance with recorded live delivery and correlate scores with neonatal outcomes.

Table- Intraclass correlation among and between rater groups by scoring section
For fifteen telesimulation videos of rural interprofessional delivery room teams doing a resuscitation as part of the Maine Ongoing Outreach Simulation Education (MOOSE) Program (Melendi M, Zanno AE, Holmes JA, Chipman M, Cutler A, Stoddard H, Seften LM, Gilbert A, Ottolini M, Craig A, Mallory LA. Development and Evaluation of a Rural Longitudinal Neonatal Resuscitation Program Telesimulation Program (MOOSE: Maine Ongoing Outreach Simulation Education). Am J Perinatol. 2024 Oct 22. doi: 10.1055/a-2421-8486. Epub ahead of print. PMID: 39326455.)