398 - Association between Instruction Understandability and Quality of AI-Based Translation for Pediatric Discharge Instructions

Publication Number: 398.3732

Taylor Salaguinto, Seattle Children's, Seattle, WA, United States; Nguyen Tran, Seattle Children's, Seattle, WA, United States; Blanca Fields, Seattle Children's, Bellevue, WA, United States; Yu-Hsiang C. Lin, Seattle Children's, Seattle, WA, United States; K. Casey Lion, University of Washington/ Seattle Children's Hospital, Seattle, WA, United States

Background: Patient understanding of discharge instructions is essential for successful hospital to home transition, particularly for families with languages other than English for care. Artificial Intelligence (AI) has the potential to decrease translation barriers. While overall safety of AI translation for clinical care must still be proven, understanding factors associated with better AI performance can inform plans for future implementation.

Objective: To test whether understandability of English discharge instructions is associated with the quality of resulting AI-based translations.

Design/Methods: This cross-sectional study assessed a sample of English instructions given to families discharged from the medical or surgical units of a pediatric hospital. Instructions were translated into Spanish, Vietnamese, Simplified Chinese, and Somali using a cloud-based neural machine translation system. Two reviewers scored the understandability of English instructions using the Patient Education Materials Assessment Tool (PEMAT; score 0-100%); inter-rater differences were discussed to reach consensus. Scores >70% were deemed “high understandability.” Two professional translators per language evaluated translations for fluency, adequacy, meaning, and error severity on a 5-point Likert scale. We used linear regression, clustered by discharge document, to test for an association between understandability (high vs low) and translation quality for AI-based discharge instructions.

Results: Among 34 complete discharge instruction sets, mean word count was 323 (range 77-1177 words), with 91% containing undefined medical jargon (range 0-13 words/document). Mean PEMAT score was 67% (range 20-85%); over half (62%) had low understandability (scored < 70%). Instructions were broken into 148 segments for translation quality scoring. The understandability of English instructions was significantly associated with AI translation quality in Spanish and Vietnamese: high understandability was associated with translations that scored better for fluency, meaning, and error severity in Spanish, and better adequacy and error severity in Vietnamese (Table 1 & Figure 1). We found no significant associations between English understandability and AI translation quality for Simplified Chinese or Somali.

Conclusion(s): Over half of the English discharge instructions scored low on understandability, which was associated with poorer AI translation quality in Spanish and Vietnamese. Future AI translation success may require improving English discharge instructions first.

Table 1
Table 1: Linear regression results, examining association between PEMAT score of discharge instruction understandability (high vs. low, dichotomized at 70%) and translator-scored quality of AI-translated discharge instructions, with clustering by document.

Figure 1
Figure 1: Unadjusted average Likert score for aspects of translation quality in 4 languages, comparing documents with low understandability to those with high understandability.