246 - Precision Medical Education in Pediatric Emergency Medicine: Mapping Fellows’ Clinical Experiences to the American Board of Pediatrics Content Domains

Publication Number: 246.6096

Masrur A. Khan, New York University Grossman School of Medicine, Astoria, NY, United States; Nicholas Genes, NYU Grossman School of Medicine, New York, NY, United States; Jesse Burk-Rafel, New York University Grossman School of Medicine, New York, NY, United States; Nivedha Satyamoorthi, New York University Grossman School of Medicine, Ridgewood, NJ, United States; Marina Marin, NYU Grossman School of Medicine, New York, NY, United States; Jung G. Kim, NYU Grossman School of Medicine, New York, NY, United States; Selin T.. Sagalowsky, New York University Grossman School of Medicine, Scarsdale, NY, NY, United States

Background: Pediatric Emergency Medicine (PEM) fellowship programs rely on experiential learning. However, mechanisms to categorize, quantify, and understand fellows’ clinical exposures are lacking. Manually developed crosswalk tools have mapped electronic health record (EHR) diagnoses to disease categories. The American Board of Pediatrics (ABP) PEM content outline defines the breadth and relative importance of clinical encounters for PEM physicians.

Objective: Our aim was to map PEM fellows’ visit diagnoses using international classification of disease (ICD-10) codes to ABP content domains, with a goal to develop educational dashboards that operationalize competency-based and precision medical education.

Design/Methods: A deep-learning analytical model (the open-source sentence transformer all-mpnet-base-v2) and embedding software (Chroma) were used to transform the library of ICD-10 code text descriptors to a vector database. ABP core content domains were annotated to clarify a tiered hierarchy of domains (Level 1), subdomains (Level 2), and sub-subdomains (Level 3). ICD-10 codes from emergency department (ED) visits staffed by PEM fellows in single institution from 01/2018 to 03/2024 were mapped to the ABP content levels. Clinical face validity was evaluated by manual clinician review.

Results: Of 5912 mapped clinical encounters, the median vector distance (MVD) for Level 1 domains was 1.018 (IQR 0.24) with 55% (n=3267) of MVD falling between 0.9 and 1.2. Manual clinician review determined face validity of 84% of Level 1, 75% of Level 2, and 40% of Level 3 mappings. Within Level 1 domains, diagnoses with the lowest MVD were: child abuse (0.818, IQR 0.44), trauma (0.885, IQR 0.20), and behavioral disorders (0.949, IQR 0.29). For Level 2 sub-domains, the majority of mapped encounters were musculoskeletal/orthopedic (n = 1585, 26.1%; MVD 1.07, IQR 0.16); gastrointestinal (n = 346, 5.7%; MVD 0.96, IQR 0.21); and otolaryngologic (n = 307, 5.1%; MVD 0.97, IQR 0.23). Only 33.5% (n=2034) of encounters were automatically mapped to Level 3 sub-sub-domains.

Conclusion(s): In this preliminary analysis, the majority of Level 1 and Level 2 mappings demonstrated clinical face validity, though most encounters had poor Level 3 mappings using our crosswalk tool. Next steps include repeating our analysis and improving mapping using alternate, more accurate transformer models; improving validity and reliability analysis with multi-physician review; and translating this tool into dashboard interfaces for trainees and educators to engage in curricular innovation, individualized learning, and targeted feedback and coaching.

Figure 1.
Example of Vector Mapping

Figure 2.
Frequency of ABP Level 1 Content Domains Mapped to PEM Fellow Clinical Encounters

Figure 3.
Frequency of ABP Level 2 Content Domains Mapped to PEM Fellow Clinical Encounters (minimum 10 encounters)


Figure 1.
Example of Vector Mapping

Figure 2.
Frequency of ABP Level 1 Content Domains Mapped to PEM Fellow Clinical Encounters

Figure 3.
Frequency of ABP Level 2 Content Domains Mapped to PEM Fellow Clinical Encounters (minimum 10 encounters)