Concordance Between the Multidisciplinary Team and ChatGPT-4o Decisions - A Blinded, Cross-Sectional Concordance Study in Systemic Autoimmune Rheumatic Diseases

Sep 30, 2026

Artificial intelligence is increasingly used for diagnostic decision-making, yet its agreement with clinical teams remains uncertain. This cross-sectional concordance study compared multidisciplinary team (rheumatology, pulmonology, thoracic radiology) decisions with single-session plans generated by ChatGPT-4o in adults with systemic autoimmune rheumatic diseases. Evaluations covered clinical and radiological diagnoses, treatment and drug-free follow-up needs, and additional investigations, using a blinded standardized prompt and Cohen’s Kappa for agreement.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Abstract

Background/Objective

Methods

Results

Conclusions

Introduction

Article Concordance Between the Multidisciplinary Team and ChatGPT-4o Decisions: A Blinded, Cross-Sectional Concordance Study in Systemic Autoimmune Rheumatic Diseases Firdevs Uluta¸s 1, *, Göksel Altını¸sık 2, Gülay Güngör 3, Vefa Çakmak 3, Nilüfer Yi ˘git 2, Duygu Herek 3, Murat Yi ˘git 1, U ˘gur Karasu 1 and Veli Çobankara 1 Academic Editor: Ayman El-Baz Received: 2 December 2025 Revised: 16 December 2025 Accepted: 25 December 2025 Published: 30 December 2025 Copyright: © 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. 1 Division of Rheumatology, Department of Internal Medicine, Pamukkale University Faculty of Medicine, 20000 Denizli, Türkiye 2 Department of Pulmonology, Pamukkale University Faculty of Medicine, 20000 Denizli, Türkiye 3 Department of Radiology, Pamukkale University Faculty of Medicine, 20000 Denizli, Türkiye * [Correspondence: firdevsulutas1014@gmail.com](Correspondence: firdevsulutas1014@gmail.com); Tel.: +90-5300944632 Abstract Background/Objective: In recent years, artificial intelligence (AI) has gained increasing prominence in the fields of diagnostic decision-making in medicine. The aim of this study was to compare multidisciplinary team (MDT: rheumatology, pulmonology, thoracic radiology) decisions with single-session plans generated by ChatGPT-4o. Methods: In this cross-sectional concordance study, adults (≥18 years) with confirmed systemic autoimmune rheumatic disease (SARD) and having MDT decisions within the last 6 months were included. The study documented diagnostic, treatment, and monitoring decisions in cases of SARDs by recording answers to six essential questions: (1) What is the most likely clinical diagnosis? (2) What is the most likely radiological diagnosis? (3) Is there a need for anti-inflammatory treatment? (4) Is there a need for antifibrotic treatment? (5) Is drug-free follow-up appropriate? and (6) Are additional investigations required? Consequently, all evaluations were performed with ChatGPT-4o in a single-session format using a standardized single-prompt template, with the system blinded to MDT decisions. All data analyses in this study were conducted using the R programming language (version 4.3.2) . An agreement between AI-generated and MDT decisions was assessed using Cohen’s Kappa (κ) statistic where κ (kappa) values represent the level of agreement: <0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, >0.80 = almost perfect agreement. These analyses were performed using the irr and psych packages in R. Statistical significance of the models was evaluated through p-values, while overall model fit was assessed using the Likelihood Ratio Test. Results: A total of 47 patients were involved in this study, with a predominance of female patients (61.70%, n = 29) . The mean age was 61.74 ± 10.40 years. The most frequently observed diagnosis was rheumatoid arthritis (RA), accounting for 31.91% of cases (n = 15) . This was followed by cases of anti-neutrophil cytoplasmic antibody (ANCA)-associated vasculitis, interstitial pneumonia with autoimmune features (IPAF), and sarcoidosis. The analyses indicate a statistically significant level of agreement across all decision types. For clinical diagnosis decisions, agreement was moderate (κ = 0.52), suggesting that the AI system can reach partially consistent conclusionsin diagnostic processes. The need for an immunosuppressive treatment and follow-up without medication decisions demonstrated a higher level of concordance, reaching the moderate-to-high range (κ = 0.64 and κ = 0.67, respectively) . For antifibrotic treatment decisions, agreement was moderate (κ = 0.49), while radiological diagnosis decisions also fell within the moderate range (κ = 0.55) . The lowest agreement—though still moderate—was observed in further investigation required

FAQs

What was the objective of this study?

To compare multidisciplinary team decisions with single-session plans generated by ChatGPT-4o for adults with confirmed systemic autoimmune rheumatic diseases.

What decisions were evaluated in the concordance assessment?

Clinical diagnosis, radiological diagnosis, need for anti-inflammatory treatment, need for antifibrotic treatment, appropriateness of drug-free follow-up, and whether additional investigations were required.

How was agreement between AI and the multidisciplinary team measured?

Using Cohen’s Kappa (κ), with κ values interpreted as slight, fair, moderate, substantial, or almost perfect agreement ranges.

Sep 30, 2026

Upload by Callum