The effectiveness of large language models in medical AI research for physicians - A randomized controlled trial

Q: How was the randomized controlled trial designed and who participated?

The study is a superiority, open-label randomized controlled trial with 64 junior ophthalmologists. Participants completed a 2-week automated cataract identification project either with ChatGPT-3.5 assistance or without assistance.

Q: What effect did ChatGPT-3.5 have on project completion?

The intervention group achieved higher overall completion (87.5% vs 25.0%) and higher unassisted completion (68.7% vs 3.1%). The intervention group also showed better project planning and faster completion times.

Q: What risks and dependency concerns were reported after washout?

After a 2-week washout, 41.2% of successful intervention participants completed a new project without LLM support. A survey indicated that 42.6% feared regurgitating information without understanding and 40.4% worried about fostering lazy thinking, suggesting potential dependency risks.

Sep 29, 2026

Uploaded by Levi

Large language models can reduce the technical burden of conducting medical AI projects, while also raising concerns about hallucination and overreliance. This superiority, open-label randomized controlled trial assigns 64 junior ophthalmologists to complete a 2-week automated cataract identification project with or without ChatGPT-3.5 assistance. Completion is higher with LLM support (87.5% vs 25.0%) and unassisted success is also improved (68.7% vs 3.1%). After washout, 41.2% can complete a new task unaided, yet survey results indicate fear of regurgitation and concern about fostering lazy thinking.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Summary

Study Design and Participants

Trial registration

Intervention and Task

Results

Completion rates and unassisted outcomes

Planning and completion time

Washout and follow-up completion

Survey on risks and dependency

Discussion and Implications

Introduction

Article The effectiveness of large language models in medical AI research for physicians: A randomized controlled trial Authors Yuanjun Shang, Yuanfan Lin, Ruiyang Li, ..., Yunxi Lai, Wenben Chen, Haotian Lin Correspondence [chenwb56@mail.sysu.edu.cn](chenwb56@mail.sysu.edu.cn) (W.C.), linht5@mai[l.sysu.edu.cn](l.sysu.edu.cn) (H.L.) In brief Shang et al. show that LLMs effectively assist physicians in overcoming the technical barriers related to the design, execution, and reporting of medical AI projects, raising completion from 25.0% to 87.5% . The findings suggest that these tools democratize medical research but highlight AI hallucination and dependency risks that require further study. • After washout, 41% successful intervention physicians complete a new project unaided • LLMs democratize medical AI research but pose AI hallucination and dependency risks • Preliminary prompt guidelines are proposed for effective interaction with LLMs Shang et al., 2025, Cell Reports Medicine 6, 102469 December 16, 2025 © 2025 The Author(s) . Published by Elsevier Inc. [https://doi.org/10.1016/j.xcrm.2025.102469](https://doi.org/10.1016/j.xcrm.2025.102469) ll ll OPEN ACCESS Article The effectiveness of large language models in medical AI research for physicians: A randomized controlled trial Yuanjun Shang,1,6 Yuanfan Lin,1,6 Ruiyang Li,1,6 Yuanrui Shang,1 Mingyuan Li,1 Lanqin Zhao,1 Ling Jin,1 Andi Xu,1 Dong Liu,1 Qianni Wu,1 Mingjie Luo,1 Jianyu Pang,1 Shaowei Bi,1 Yuchun He,2 Miaohong Xu,1 Xinwei Chen,1 Zizheng Cao,1 Sijia Yu,3 Jiaman Zhao,1 Yunxi Lai,1 Wenben Chen,1,* and Haotian Lin1,4,5,7,* 1Zhongshan Ophthalmic Center, Sun Yat-sen University, WHO Collaborating Centre for Eye Care and Vision, State Key Laboratory of Ophthalmology, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China 2The Second Affiliated Hospital of Fujian Medical University, Quanzhou, China 3Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou, China 4Hainan Eye Hospital and Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Haikou, Hainan, China 5Department of Genetics and Biomedical Informatics, Zhongshan School of Medicine, Institute for Frontier Interdisciplinary Research in Health Sciences and Technology, Sun Yat-sen University, Guangzhou, China 6These authors contributed equally 7Lead contact *Correspondence: [chenwb56@mail.sysu.edu.cn](chenwb56@mail.sysu.edu.cn) (W.C.), [linht5@mail.sysu.edu.cn](linht5@mail.sysu.edu.cn) (H. L.) [https://doi.org/10.1016/j.xcrm.2025.102469](https://doi.org/10.1016/j.xcrm.2025.102469) SUMMARY Physicians offer invaluable clinical insights, but their involvement in medical AI research is hindered by limited technical expertise. We conduct a superiority, open-label, randomized controlled trial involving 64 junior ophthalmologists to undertake a 2-week project on ‘‘automated cataract identification’’ under minimal engineering assistance, with (intervention, n = 32) or without (control, n = 32) ChatGPT-3.5. The overall project completion rate is higher in intervention group than controls (87.5% vs. 25.0%; difference 62.5%, p = 9.42e − 7), and the unassisted completion rate likewise (68.7% vs. 3.1%; difference 65.6%, p = 5.70e − 8). The intervention group demonstrates better project planning and faster completion times (p < 0.01). After a 2-week washout, 41.2% of successful intervention participants complete a new project without the support of large language models (LLMs). A survey shows that 42.6% of participants fear regurgitating information without understanding and 40.4% worry about fostering lazy thinking, indicating potential dependency. Therefore, LLMs can help physicians overcome technical barriers, although long-term risks require further study. Trial registration: This study was registered [at ClinicalTrials.gov]

FAQs

How was the randomized controlled trial designed and who participated?

The study is a superiority, open-label randomized controlled trial with 64 junior ophthalmologists. Participants completed a 2-week automated cataract identification project either with ChatGPT-3.5 assistance or without assistance.

What effect did ChatGPT-3.5 have on project completion?

The intervention group achieved higher overall completion (87.5% vs 25.0%) and higher unassisted completion (68.7% vs 3.1%). The intervention group also showed better project planning and faster completion times.

What risks and dependency concerns were reported after washout?

After a 2-week washout, 41.2% of successful intervention participants completed a new project without LLM support. A survey indicated that 42.6% feared regurgitating information without understanding and 40.4% worried about fostering lazy thinking, suggesting potential dependency risks.

Sep 29, 2026

Uploaded by Levi