WIP 70 - An Open-Source, Interoperable, Population Mortality Risk Model for Neonatology

Publication Number: WIP 70.7474

Chelsea K. Bitler, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Jesse Klug, UPMC, Pittsburgh, PA, United States; Sarah K. Nutman, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Jesse D. Raffa, Massachusetts Institute of Technology, Cambridge, MA, United States; Leo Anthony Celi, Massachusetts Institute of Technology; Harvard Medical School, CAMBRIDGE, MA, United States; Harry Hochheiser, University of Pittsburgh, Pittsburgh, PA, United States; C Briana Bertoni, Levine Children's Hospital, Charlotte, NC, United States; Brian King, Harvard Medical School, Boston, MA, United States; Thomas A. Hooven, University of Pittsburgh School of Medicine, Pittsburgh, PA, United States; Christopher Horvat, UPMC/University of Pittsburgh, Pittsburgh, PA, United States

Background: Population risk adjustment models are commonly used for benchmarking in the field of critical care medicine to facilitate meaningful comparisons of patient populations and care quality by accounting for differences in patient characteristics. Despite widespread use in critically ill adult and non-neonatal pediatric patient populations, such models have not been widely adopted by the field of neonatology. Modern informatics best practices combined with 21st Century Cures Act interoperability requirements afford an opportunity to develop statistically robust mortality risk models for the field of neonatology.

Objective: Create a free, open-source, in-hospital neonatal mortality risk adjustment model by adapting a pre-existing Global Open-Source Severity of Illness Score (GOSSIS) IT pipeline to address the distinct needs of the neonatal population.

Design/Methods: The University of Pittsburgh Institutional Review Board approved this work. A multicenter, retrospective cohort is being derived from extraction of electronic record data of 12,500+ neonates admitted to level 3 or 4 NICUs in the UPMC health system between 2015 – 2024. Data include vital signs, lab results, diagnoses, and demographics collected within 6-24 hours of admission. Model predictor selection is guided by the United States Core Data for Interoperability (USCDI) and USCDI+ mapping efforts and value sets are being uploaded to the National Library of Medicine Value Set Repository. Linear and nonlinear machine learning classification models are being developed using a 70/30 training/assessment split, with a final contemporary test holdout dataset from 2023-24. Model performance is being assessed with c-statistics, areas under the precision-recall curves, calibration plots, and Brier scores and compared with existing risk adjustment models, including GOSSIS. Work is ongoing with model performance assessments expected in March 2025. This work will support the development of an optimal model for incorporation into an interoperable software application to support quality improvement benchmarking in neonatology.