Erkan DUMAN, Aydın MAÇİN, Ayşe ERGİN
Cyprus Journal of Medical Sciences - 2026;11(5):369-377
BACKGROUND/AIMS: Large language models (LLMs) such as ChatGPT-4o mini, ChatGPT-4o, Gemini 1.5 Pro and Microsoft Copilot play an important role in improving health literacy and supporting patient decision making. This study aims to evaluate the clinical appropriateness and readability of responses provided by these LLMs to frequently asked questions about retinal detachment (RD), epiretinal membrane (ERM), and macular hole (MH). MATERIALS AND METHODS: This comparative cross-sectional study did not involve human participants. A total of 121 questions related to RD, ERM and MH were compiled. These questions were presented to LLM interfaces and were categorised into different themes. Inter-rater agreement between two retinal specialists who scored the responses based on their clinical expertise was assessed using the intraclass correlation coefficient (ICC). In addition, the readability of the responses was analysed using five readability indices. RESULTS: Response appropriateness rates were similar for ChatGPT-4o mini (100%), ChatGPT-4o (99.2%) and Gemini 1.5 Pro (99.2%), while Microsoft Copilot (90.9%) showed comparatively lower performance. A high level of inter-rater agreement was observed among clinicians. Regarding readability, ChatGPT-4o performed best, whereas ChatGPT-4o mini performed worst. Gemini 1.5 Pro and Microsoft Copilot were intermediate between these two models. The excellent inter-rater agreement (ICC: 0.891-0.938) further supports the reliability of the expert evaluation process. CONCLUSION: Our study showed that the evaluated LLMs provided largely appropriate and comprehensive answers to questions related to vitreoretinal surgery. ChatGPT-4o showed the highest performance in both appropriateness and readability, while ChatGPT-4o mini, despite its high appropriateness, had the lowest readability score.
