Sema Bozkaya YILMAZ, Gunce BASARIR
The Medical Bulletin of Haseki - 2026;64(4):229-236
Aim: Caregiver-facing large language model (LLM) responses to pediatric seizure questions must provide accurate, complete, and safety-critical guidance because omissions may affect urgent decision-making. This study aimed to evaluate the clinical accuracy, completeness, readability, empathy, and reproducibility of responses generated by three LLMs to standardized caregiver questions about pediatric seizures. Methods: This cross-sectional comparative study evaluated GPT-5.5, Gemini 3 Flash, and DeepSeek V3. Ten standardized caregiver questions addressing concerns about childhood seizures were submitted to each model in two runs during a 48-hour testing window in May 2026. Model identities were removed, and responses were evaluated by two pediatric neurologists using a 5-point rubric covering clinical accuracy and safety, completeness, readability and understandability, and empathy and tone. Inter-run consistency was assessed using Spearman's rank correlation coefficient. Results: A total of 60 artificial intelligence-generated responses were evaluated. Expert-rated performance differed among the models in this exploratory analysis. Gemini received the highest overall scores within this prompt set and the May 2026 testing window, with a median score of 5.00; it was followed by GPT-5.5 and DeepSeek V3, which had median scores of 4.50 and 4.00, respectively. Although all models produced fluent and understandable responses, some outputs omitted clinically important safety information, including the five-minute emergency threshold. Inter-run consistency was moderate (Spearman's rho=0.517). These findings are exploratory and may differ across model updates, testing periods, and clinical scenarios. Conclusion: In this targeted expert evaluation of caregiver-oriented responses to pediatric seizures, model performance varied across LLMs, and repeated responses demonstrated only moderate consistency. Although Gemini achieved the highest scores within the prompt set and the testing window, the occasional omission of safety-critical advice, including the five-minute emergency threshold, indicates that LLMs should be used as supplementary sources of general information rather than as stand-alone tools for urgent seizure-related decisions. Relative model performance may change with model updates and when different clinical scenarios are evaluated.