Evaluating the Impact of Large Language Model AI on Acute Pancreatitis Management - A ChatGPT-Based Investigation

Aug 28, 2026

Uploaded by Aurora

Evidence-based management of acute pancreatitis (AP) is critical for patient outcomes, yet practical decision support can be constrained by access and time. This study evaluated an AI chatbot system, ChatGPT, by testing guideline alignment for acute pancreatitis management and its usefulness in clinical decision-making. Six curated management questions were posed, responses were assessed against internationally recognized guidelines and reviewed for accuracy, comprehensiveness, grammar, spelling, and reference quality. Results showed partial adherence with generally nonspecific depth and notable risks from fabricated or erroneous references, raising academic-integrity and ethical concerns for research use.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Abstract

Background

Methods

Results

Conclusion

Introduction

Scan to know paper details and author's profile Evaluating the Impact of Large Language Model AI on Acute Pancreatitis Management: A Chatgpt-based Investigation Dylan Gracias, BBiomed, MD, Adrian Siu, BPharm (Hons), MD, MS, Ishith Seth, BBiomed (Hons), MD, MS, Dilshaad Dooreemeah, MBBS, FRACS & Angus Lee, MBBS, FRACS ABSTRACT Background: Evidence-based management of acute pancreatitis (AP) is important for patient outcomes. The present study evaluated suggestions by artificial intelligence (AI) chatbot system, ChatGPT, for the management of acute pancreatitis, its alignment with clinical guidelines, and assistance in clinical decision-making. Methods: Six questions on pancreatitis management were curated by experienced RACS-qualified general surgeons and were put forth to ChatGPT. The chatbot was also asked to provide five high-level evidence references to support each of its responses. Each response was analyzed for its accuracy and comprehensiveness with respect to current internationally recognized guidelines and by two BoardCertified General Surgeons for acute pancreatitis management, as well as for its spelling, grammar, and reference quality. A five-point Likert Scale was utilized to analyze ChatGPT's responses, with scores ranging from 1 (strongly disagree) to 5 (strongly agree) . Ten questions were designed to assess accuracy, consistency, informativeness, reliability, and coherence. Keywords: chatGPT, acute pancreatitis, management, artificial intelligence, large language model. Classiﬁcation: NLM Code:W 26.5, WI 753, Q 171 Language: English LJP Copyright ID: 392886 | London Journal of Medical & Health Research
Volume 2024 | Issue 10 | Compilation 1.0 | 449U | | --- | --- | © 2024 . Dylan Gracias, BBiomed, MD, Adrian Siu, BPharm (Hons), MD, MS, Ishith Seth, BBiomed (Hons), MD, MS, Dilshaad Dooreemeah, MBBS, FRACS & Angus Lee, MBBS, FRACS This is a research/review paper, distributed under the terms of the Creative Commons Attribution-Non-commercial 4 . 0 Unported License [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)), permitting all noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited. Evaluating the Impact of Large Language Model AI on Acute Pancreatitis Management: A Chatgpt-Based Investigation Dylan Gracias, BBiomed, MDα , Adrian Siu, BPharm (Hons), MD, MSσ , Ishith Seth, BBiomed (Hons), MD, MSρ , Dilshaad Dooreemeah, MBBS, FRACSѠ & Angus Lee, MBBS, FRACS¥ ABSTRACT Background: Evidence-based management of acute pancreatitis (AP) is important for patient outcomes. The present study evaluated suggestions by artificial intelligence (AI) chatbot system, ChatGPT, for the management of acute pancreatitis, its alignment with clinical guidelines, and assistance in clinical decisionmaking. Methods: Six questions on pancreatitis management were curated by experienced RACS-qualified general surgeons and were put forth to ChatGPT. The chatbot was also asked to provide five high-level evidence references to support each of its responses. Each response was analyzed for its accuracy and comprehensiveness with respect to current internationally recognized guidelines and by two BoardCertified General Surgeons for acute pancreatitis management, as well as for its spelling, grammar, and reference quality. A five-point Likert Scale was utilized to analyze ChatGPT's responses, with scores ranging from 1 (strongly disagree) to 5 (strongly agree) . Ten questions were designed to assess accuracy, consistency, informativeness, reliability, and coherence. These were independently rated by three junior doctors and two General Surgeons, with any scoring discrepancies resolved through consensus. Results: ChatGPT successfully adhered to clinical guidelines when generating recommendations for the management of acute pancreatitis. The depth of information remained general and non-specific but was pres

FAQs

What was the purpose of evaluating ChatGPT in acute pancreatitis management?

The study assessed whether ChatGPT’s suggestions align with clinical guidelines for acute pancreatitis management and whether it can assist clinical decision-making.

How were ChatGPT’s answers evaluated in the study?

Six curated pancreatitis management questions were posed to ChatGPT, which also provided high-level evidence references. Responses were evaluated for accuracy, comprehensiveness, guideline conformity, and language quality, with rating using a five-point Likert scale.

What key limitations were found in ChatGPT’s recommendations and references?

While ChatGPT adhered to clinical guidelines, the information depth remained general and nonspecific. The study also found missing references and fabricated or erroneous references, posing challenges for researchers and clinicians.

Aug 28, 2026

Uploaded by Aurora