220 - GENERATIVE ARTIFICIAL INTELLIGENCE MEETS PEDIATRICS: CHATGPT’S ABILITY TO GENERATE PEDIATRIC CERTIFYING EXAM-STYLE MULTIPLE CHOICE QUESTIONS

Publication Number: 220.5001

Maeve A. Reidy, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Maeve Reidy, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Aaron Altman, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Catherine Nettesheim, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; David Roth, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Yonairy Collado Puello, Children's National Health System, Silver Spring, MD, United States; Aditi Jain, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; martha Stewart, Children's Healthcare of Atlanta, Atlanta, GA, United States; Krishna Trivedi, Northwestern University - Lurie Children's Hospital, Chicago, IL, United States; rhett Lieberman, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States

Background: Generative artificial intelligence (AI) is increasingly being utilized as an aid in the field of medical education and assessment. Its creation of multiple-choice questions (MCQs) is more rapid and efficient, but limitations exist, prompting the need for careful expert review. Even with meticulously designed prompts, question quality variability exists, particularly among sub-specialties. Generative AI‘s potential in creating sub-specialty certifying exam-style MCQs has been studied, but not within pediatrics.

Objective: This study aims to evaluate the ability of generative AI, specifically ChatGPT, to design American Board of Pediatrics (ABP) certifying exam-style MCQs.

Design/Methods: From May to June 2024, we developed ABP certifying exam-style, single-best-answer, second order MCQs using ChatGPT 3.5, applying tailored prompts centered on content from American Academy of Pediatrics’ (AAP) Pediatric in Review articles. Articles for each subspecialty domain were identified using the AAP’s ABP Content Specifications Map for the pediatric certifying exam. Board certified sub-specialists from a quaternary academic institution reviewed the MCQs, blinded to the question source and their counterparts’ opinions. Only topics reviewed by 2 or more sub-specialists were included in the study. Sub-specialists’ comments were analyzed and grouped into 4 standardized categories: wrong question (WQ), wrong answer (WA), methodologically flawed (MF), or no/minor edits (NME). Results were analyzed by topic.

Results: Originally 273 MCQs were created, with 210 across 16 core topics being included in the study. 63 were excluded as they were not reviewed by ≥2 sub-specialists. Per topic, the MCQs were distributed amongst the four categories as illustrated in Table 1.

Conclusion(s): With the rise of generative AI and its expanding application in medical education, its potential in creating ABP certifying exam-style MCQs had not been explored. We found that >50% of MCQs within 10 of 16 topics, most notably Oncology and Cardiology, were categorized as NME, reflecting that it can generate appropriate ABP certifying exam-style MCQs in these domains. With other subspecialties, its ability to generate well designed MCQs varied, with limitations being highlighted specifically in Endocrinology, Rheumatology and Adolescent Care with higher WQ and WA percentages. This study highlights generative AI’s promising potential in ABP certifying exam-style MCQs creation, with the need for continued exploration and awareness of the essential role that sub-specialist experts play in ensuring question quality.

Table 1. Categorical Distribution of MCQs For Included ABP Topics