551 - Diagnosing Kawasaki Disease: A Machine Learning Algorithm Compared to Pediatric Emergency Medicine Physicians

Publication Number: 551.4224

Michael A. Gardiner, University of California, San Diego School of Medicine, Santee, CA, United States; Jonathan Y. Lam, University of California San Diego, La Jolla, CA, United States; Shamim Nemati, University of California, San Diego School of Medicine, San Diego, CA, United States; Adriana Tremoulet, University of California, San Diego School of Medicine, La Jolla, CA, United States; Jane C. Burns, UCSD, La Jolla, CA, United States

Background: Kawasaki Disease (KD) is the most common cause of acquired heart disease in children in developed countries. KD is often missed early in the disease course as it shares many clinical signs and symptoms with other benign childhood febrile illnesses. The diagnosis of KD is made through provider synthesis of patient symptoms, physical exam, and laboratory findings, as there is no diagnostic test for KD. In response to challenges in diagnosis of KD, our group developed and validated a machine learning (ML) algorithm named Kawasaki Match, a clinical decision support tool.

Objective: To compare the accuracy of Kawasaki Match against Pediatric Emergency Medicine (PEM) physicians in diagnosing KD.

Design/Methods: We recruited PEM physicians from a single center to evaluate 50 randomized case vignettes (25 KD, 25 febrile controls (FC)). Vignettes were created by medical record review of subjects from an existing KD and FC database. Diagnoses for FC were adjudicated by KD experts after chart review. All KD cases met American Heart Association criteria for complete or incomplete KD. Vignettes included history, vital signs, physical exam, and laboratory results. We asked physicians to assess the likelihood of KD and disposition. Patient data including the 5 clinical signs of KD and laboratory results were provided to Kawasaki Match and a prediction score for KD determined. A previously validated cut-point of ≥40% likelihood was used to diagnose KD. Accuracy of diagnosis was compared between Kawasaki Match, and physician assessment.

Results: 14 PEM physicians completed the case vignettes (Table 1). Overall physician accuracy was 67.5% (95% CI 64% - 71%; range 60% - 78%). Kawasaki Match outperformed all individual physicians with an accuracy of 80%. Table 2 details the performance characteristics for Kawasaki Match and physician assessment. Kawasaki Match demonstrated higher sensitivity and negative predictive value than physician assessment. Kawasaki Match identified KD in 80% (12/15) of KD cases for which 50% or fewer providers predicted KD. There was no significant correlation between provider experience or confidence in diagnosing KD and accuracy.

Conclusion(s): Kawasaki Match exhibited greater accuracy, sensitivity, and negative predictive value than PEM physicians in the diagnosis of KD. Kawasaki Match shows promise as a screening tool for KD with a prospective study planned to assess the impact of this decision support tool on physician behavior and patient outcomes.

Table 1
Characteristics of Physicians Completing Kawasaki Disease and Febrile Control Patient Vignettes

Table 2
Performance Characteristics of Kawasaki Match vs. PEM Physicians (95% CI).

Figure
Shapley Additive Explanations (SHAP) summary plot for Kawasaki Match analysis of case vignettes. A feature for a subject with a SHAP value < 0 decreases the risk score, value > 0 increases the risk score. A higher risk score indicates a higher probability of Kawasaki Disease. Features ranked in order of importance from top to bottom.