Atakan KÜSKÜN, Edip Serdar GÜNER, H. Harun ÖZTÜRK, Fatma ÖZTORA, Enes ESEN, Ömer KOSTAK, M. Savaş TEPE
Eurasian Journal of Family Medicine - 2026;15(2):154-163
Aim: To evaluate the performance of Generative Pre-trained Transformer-4o (GPT-4o) in generating mammography reports from complete four-image mammograms. Methods: This retrospective comparative study used previously reported 510 anonymized mammograms from Kırklareli Training and Research Hospital. Four JPEG images per patient were combined into a single composite input for GPT-4o, which generated structured reports in Turkish. Three radiologists independently evaluated the AI-generated reports by using the original mammography reports as reference. A custom interface and a Likert scale (0-5) for each section were used. Breast Imaging-Reporting and Data System (BI-RADS) classification performance was measured via accuracy, Cohen's Kappa, confusion matrix, and class-based metrics. Intraclass correlation coefficients are calculated for inter-rater reliability assessment. Results: GPT-4o demonstrated high performance in detecting normal findings (Lymph Node Evaluation: 4.97/5; Skin/Subcutaneous Tissues: 4.93/5) and some pathological features (Suspicious Calcifications: 4.53/5; Structural Distortion: 4.44/5). However, it achieved critically low performance in clinical decision-making: BI-RADS Category classification scored 2.48/5 with 9.6% overall accuracy and Cohen's Kappa of 0.017 (near-chance agreement), and Recommendations scored 2.50/5. The model systematically misclassified cases as BI-RADS 1 while failing to recognize BI-RADS 3 (1% recall), creating both false reassurance and inappropriate clinical escalation. Conclusion: GPT-4o, despite showing some success in identifying specific mammographic features, currently lacks the reliability required for independent BI-RADS classification and recommendation generation. The model's tendency towards misclassification and inappropriate escalation raises significant concerns regarding its clinical safety when used autonomously.
