173 - Harnessing Large Language Models for Safety Event Detection in Pediatric Emergency Medicine: The RescueGPT Approach

Publication Number: 173.5335

Tina Yi Jin Hsieh, Harvard Medical School, Boston, MA, United States; Carl Eriksson, Doernbecher Children's Hospital at Oregon Health & Science University, Portland, OR, United States; Garth D. Meckler, University of British Columbia Faculty of Medicine, Vancouver, BC, Canada; Matthew l. Hansen, Oregon Health & Science University School of Medicine, Portland, OR, United States; Nathan Bahr, OHSU, Portland, OR, United States; Steven Bedrick, Oregon Health & Science University School of Medicine, Portland, OR, United States; Diego Trujillo, Beth Israel Deaconess Medical Center, Boston, MA, United States; Kyu Seo Kim, Oregon Health & Science University School of Medicine, Portland, OR, United States; Byeongyeon Cho, Harvard Medical School, Brookline, MA, United States; Xiaoqian Jiang, The University of Texas Health Science Center at Houston, Houston, TX, United States; Jeanne-Marie Guise, Mass Institute for Equity-Focused Learning Health Systems, Boston, MA, United States

Background: Pediatric out-of-hospital cardiac arrest (OHCA) poses a significant risk of adverse safety events (ASEs). Traditional ASE identification relies on manual emergency medical service (EMS) record review, a labor-intensive process ill-suited to high-volume EMS data. This study introduces RescueGPT, a novel framework that combines Large Language Models (LLMs) with a specialized clinical knowledge base to automate ASE detection in EMS care records for pediatric OHCA cases.

Objective: RescueGPT aims to streamline the recognition, extraction, and interpretation of ASE-related information, providing a scalable solution to improve prehospital care quality and safety.

Design/Methods: We collected pediatric OHCA records between 2017and 2020 from a national EMS provider. We used the Pediatric Prehospital Adverse Safety Event Detection System (PEDS) as a knowledge base, and the LinkML framework to build an ontology that defines ASEs across six key EMS care domains, including patient assessment, airway management, and procedural interventions. We then implemented the Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES) method to transform unstructured EMS notes into structured, schema-driven prompts, tailored to guide the GPT-3.5 model in recognizing ASEs. By parsing data into structured concepts based on PEDS rules, the model generated targeted prompts that facilitated accurate entity extraction while maintaining HIPAA compliance through the secure Azure AI platform.

Results: Accuracy, recall, precision, F1 score, and specificity, were calculated across a sample of 42 cases across 9 ASE-related entities. Results showed high accuracy in detecting common ASEs (e.g., “Patient Rhythm” - 0.98; “Patient Age” “Weight & Length”-0.95) but revealed challenges in rare events, like “Failure to Establish IV Access,” with an accuracy of 0.74, mainly due to more complex medical narratives. Performance metric scores were lower for infrequent ASEs including “Incorrect Airway Equipment Size” or “Failure to Ventilate Patient”, highlighting areas for improvement through rebalancing techniques or cost-sensitive learning methods.

Conclusion(s): While RescueGPT shows potential for ASE detection at scale, improvements are needed in ontology refinement, interpretation of complex text, and the addition of a “non-Applicable” classification for cases with missing details. Future work will validate RescueGPT across diverse EMS datasets and ensure up-to-date integration of knowledge baseRescueGPTn. RescueGPT can potentially advance pediatric OHCA safety, offering the clinical community faster, more effective ASE detection.

Figure 1. Workflow of development and utilization of RescueGPT algorithm
To transform unstructured clinical text data into structured, meaningful concepts pertinent to pediatric OHCA, we employed the LinkML framework to construct our ontology schema, informed by the PEDS tool. This structure lays the groundwork for systematically organizing and understanding the data

Figure 2. Confusion Matrices across the 9 entities


Figure 3. Heatmap of Performance metrics