Bilge TÜREDİ, Ali SEZER
The Journal of Pediatric Research - 2026;13(3):294-302
Aim: Large language models (LLMs) are increasingly being used by patients and caregivers in order to obtain medical information. This study evaluated the guideline concordance and public readability of two contemporary LLMs in answering frequently asked questions regarding common pediatric penile conditions. Materials and Methods: Five common pediatric penile conditions were evaluated: phimosis, hypospadias, congenital penile curvature, paraphimosis, and buried penis. Ten standardized questions were developed for each condition, covering diagnosis, treatment, surgical considerations, complications, and long-term outcomes. A total of 50 questions were submitted to two widely used LLMs in both English and Turkish. The responses were assessed according to the European Association of Urology Paediatric Urology Guidelines and graded as fully concordant (Grade 1), partially concordant (Grade 2), partially concordant with inaccuracies (Grade 3), or non-concordant (Grade 4). Public readability was evaluated using a five-point scale. Results: ChatGPT (OpenAI, GPT-5.5 version; accessed June 2026) achieved Grade 1 concordance in 40 out of 50 responses (80%), with 49 responses (98%) considered acceptable (Grade 1 or 2). Gemini (Google DeepMind, Gemini 3.5 Pro version; accessed June 2026) achieved Grade 1 concordance in 41 of 50 responses (82%), and all responses (100%) were classified as acceptable. No Grade 4 responses were identified, and only one Grade 3 response was observed overall. Concordance rates did not differ significantly between the 2 models (p>0.05). Public readability was high for both systems (4.60+/-0.58 vs. 4.67+/-0.49; p>0.05). No significant differences were observed between the English and Turkish responses. Reduced concordance occurred in different questions, with only two overlapping areas identified. Conclusion: Both LLMs demonstrated high guideline concordance and excellent public readability. No clinically significant misinformation was identified, and most deficiencies reflected omissions of specific guideline details. LLMs may serve as reliable educational resources but should complement, rather than replace, professional medical consultation.
