399 - Performance Metrics of Real-Time Observation Versus Video Recording of Newborn Resuscitations in the Democratic Republic of the Congo

Publication Number: 399.6129

Marcus J. Donnelly, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, NC, United States; Øyvind Meinich-Bache, Univeristy of Stavanger, KVERNALAND, Rogaland, Norway; Daniel Ishoso, Kinshasa School of Public Health, Kinshasa, Kinshasa, Congo, (Congo – Kinshasa); Siren Rettedal, Stavanger University Hospital, Stavanger, Rogaland, Norway; Amy Mackay, University of Alabama at Birmingham School of Medicine, Birmingham, AL, United States; Melissa Bauserman, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, NC, United States; Eric Musalu. Mafuta, Ecole de Santé Publique de Kinshasa/UNIKIN, Kinshasa, Kinshasa, Congo, (Congo – Kinshasa); Jackie Patterson, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, NC, United States

Background: Newborn resuscitation care is typically recorded based on recall which is subject to bias. Audio-video (AV) recording is a gold standard for data collection but is resource-intensive due to issues of informed consent and time required to annotate videos. Real-time observation may be a simpler alternative, but its accuracy is unknown. In a cohort study in the Democratic Republic of the Congo, we are gathering data on resuscitation care by direct observation using a mobile health application and via AV recording.

Objective: To evaluate the performance metrics of newborn resuscitation data collected via real-time observation compared to AV recording.

Design/Methods: In our cohort study, at the time of a newborn’s birth, an observer uses the Liveborn mobile health application to record the timing of stimulation, suction, and ventilation (Figure 1). AV recording is also initiated once a baby is moved to the warmer for resuscitation. A medical student (M. Donnelly) annotates the timing of video-captured events using ELAN 6.4 software; annotations are then reviewed by a neonatologist (J. Patterson). We filtered AV annotation data to eliminate events lasting < 3 seconds and then analyzed the proportion of events missed by the observer. We evaluated the accuracy, precision, sensitivity, and specificity of real-time observation compared to AV recording (Figure 2).

Results: We collected data on 38 resuscitations by both real-time observation and AV recordings; 95% of cases included stimulation, 84% included suction, and 82% included ventilation. Real-time observation captured nearly all occurrences of each action (median missed events=0%; Table 1). Accuracy of real-time observation was at least 84% and specificity at least 90% (Table 1). Precision of real-time observations was highest for ventilation (71%) and lowest for stimulation (35%); similarly, sensitivity was highest for ventilation (80%) and lowest for stimulation (48%).

Conclusion(s): While the accuracy and specificity of real-time observation versus AV recording of newborn resuscitations are reasonably high, precision and sensitivity varied by action type. The adequacy of these performance metrics should be evaluated considering the purpose of the data being collected. Implementation strategies to increase the feasibility and acceptability of AV recording of care are needed. Future studies should evaluate the performance of real-time data collection compared to recall.

Figure 1: Data collection strategies for newborn resuscitation.
A) Event registration screen of the Liveborn app: In real-time, an observer uses Liveborn to record the start and stop times of key resuscitation actions including stimulation, suction, and ventilation. B) Resuscitation station with AV recording: When a newborn is moved to the warmer for resuscitation, the observer initiates AV recording via a mounted tablet.

Table 1: Performance metrics comparing real-time observation with Liveborn to AV recording.


Figure 2: Example plots comparing real-time observation data to AV-recorded data.
The plots depict the frequency of stimulation, suctioning and ventilation for one case of resuscitation with annotations of AV-recorded data depicted in blue and real-time observation with Liveborn depicted in orange. Accuracy represents the proportion of time the observer documented what was actually happening. Precision represents the proportion of time the observer’s documentation of suctioning was actually true, i.e., avoiding false detection of suctioning. Sensitivity represents the proportion of actual time spent suctioning that was documented by the observer, i.e., not missing suctioning. Specificity represents the proportion of actual time not suctioning that was documented by the observer, i.e., documenting the absence of suctioning. Abbreviations: AV=audio video; FN=false negative; FP=false positive; TN=true negative; TP=true positive.

Figure 1: Data collection strategies for newborn resuscitation.
A) Event registration screen of the Liveborn app: In real-time, an observer uses Liveborn to record the start and stop times of key resuscitation actions including stimulation, suction, and ventilation. B) Resuscitation station with AV recording: When a newborn is moved to the warmer for resuscitation, the observer initiates AV recording via a mounted tablet.

Table 1: Performance metrics comparing real-time observation with Liveborn to AV recording.


Figure 2: Example plots comparing real-time observation data to AV-recorded data.
The plots depict the frequency of stimulation, suctioning and ventilation for one case of resuscitation with annotations of AV-recorded data depicted in blue and real-time observation with Liveborn depicted in orange. Accuracy represents the proportion of time the observer documented what was actually happening. Precision represents the proportion of time the observer’s documentation of suctioning was actually true, i.e., avoiding false detection of suctioning. Sensitivity represents the proportion of actual time spent suctioning that was documented by the observer, i.e., not missing suctioning. Specificity represents the proportion of actual time not suctioning that was documented by the observer, i.e., documenting the absence of suctioning. Abbreviations: AV=audio video; FN=false negative; FP=false positive; TN=true negative; TP=true positive.