Research Assistant Cohen Children's Medical Center
Background: The growing influence of artificial intelligence (AI) chatbots like Chat Generative Pre-trained Transformer (ChatGPT) in healthcare offers a novel avenue of accessing pediatric health information for caregivers. However, little is known about the suitability of such AI-generated content. Objective: This study aims to compare the readability and accuracy of pediatrician and ChatGPT-generated responses to commonly asked immunization questions. Design/Methods: Immunization questions and pediatrician responses (n = 21) were extracted from HealthyChildren.org, the official parenting website of the American Academy of Pediatrics (AAP). Questions were entered into ChatGPT 3.5 with the prompt “How would you respond to a parent asking the following question: [insert question]?” For accuracy, two Board-Certified pediatricians rated the medical correctness and completeness of randomized ChatGPT and pediatrician answers on a 5-point Likert scale. Readability was assessed using average word count, Flesch Reading Ease (FRE) score, and Flesch-Kincaid Grade Level (FKGL) (Table 1). Raters were also asked which response they preferred. Results: A total of 42 responses (21 from pediatricians and 21 from ChatGPT) were included and analyzed. While pediatrician (M = 4.91, SD = 0.30) responses had significantly higher correctness scores than ChatGPT (M = 4.62, SD = 0.50), there were no significant differences for completeness. ChatGPT-generated responses had a significantly lower FRE score and higher FKGL (p < 0.001) but similar word count compared to pediatrician responses. Fifteen questions were about COVID; of these, 7 (46%) questions required up to date information to accurately answer.
Conclusion(s): Both readability metrics revealed that pediatrician responses were less complex and more readable – without compromising medical accuracy – compared to ChatGPT-generated responses. The lower correctness in ChatGPT 3.5’s responses could be attributed to its last update in 2021, rendering its responses potentially outdated. As caregivers increasingly turn to AI chatbots to seek health information, pediatricians should advise them to take caution when using these tools, especially for swiftly evolving fields such as pediatric immunizations and COVID. While ChatGPT has its limitations, the AAP should consider its user-friendliness by using medically-trained AI-chatbots on their website to ease the process of finding information for parents. Further research is warranted on how to refine AI chatbots so that they may increase the accessibility of medically accurate health information.
