WIP 29 - ChatPPC: AI-Driven Support for Resident Education in Primary Care

Publication Number: WIP 29.7497

Aydin Zahedivash, Lucile Packard Children's Hospital Stanford, Palo Alto, CA, United States; Prachi M. Keni, Lucile Packard Children's Hospital Stanford, Menlo Park, CA, United States; Chethan Sarabu, Stanford University School of Medicine, New York, NY, United States; Elizabeth Stuart, Stanford University School of Medicine, Stanford, CA, United States

Background: Primary care practice requires integrating extensive knowledge of clinical guidelines, community resources, institutional policies, and systems of care. Keeping providers current with evolving clinical practices and workflows is particularly challenging in academic settings, where residents and faculty have competing demands on their attention and time. While many practices, like ours, store resources and guidance in websites or shared documents, finding up-to-date information can be time-consuming. To streamline this process and improve quality of care, we developed ChatPPC, an artificial intelligence (AI)-powered large language model (LLM) that ingests clinic and community information, answers workflow and procedure questions, and serves as a knowledge assistant for providers.

Objective: (1) Explore how residents are using ChatPPC and how they are interacting with the information.
(2) Evaluate ChatPPC’s effectiveness in providing clinic workflow and community resource information during visits.
(3) Assess ChatPPC’s role in helping providers stay current with evolving practice information.
(4) Offer PAS 2025 attendees a framework for designing a similar tool for their institutions.

Design/Methods: (1) Using link tracing software, we will track clicks to our clinic web resources from ChatPPC to identify frequently searched topics. With OpenAI usage analytics, we’ll measure time to reach answers and quantify how often users use ChatPPC output versus clicking for details. This data will also allow us to follow the uptake of new clinic policies and procedures as they are released.
(2) In January 2025, after residents have become familiar with ChatPPC, we will conduct three focus groups with 12 residents each to assess its use as a stand-alone tool, a website access point, and a resource for staying updated on clinic protocols.
(3) From November 2024 to January 2025, three senior clinic staff will assess ChatPPC’s responses to key workflow questions, informing LLM training refinements. We’ll create a guide for PAS attendees on replicating these steps at their institutions.