While GPT-3.5 is unable to pass the Physician Licensing Exam in Taiwan, GPT-4 successfully meets the criteria

Sep 29, 2026

Upload by Eliana

Background: This study evaluates ChatGPT-3.5 and ChatGPT-4 when answering medical questions from Taiwan’s Physician Licensing Exam, covering basic medical sciences through specialized clinical topics. It focuses on AI capability in a non-English setting, specifically traditional Chinese. Methods: Exam 2022 questions were manually entered into ChatGPT and responses were compared with official answers; accuracy differences by specialty and question type were analyzed. Results: GPT-4 outperformed GPT-3.5 across areas. Conclusion: Performance varied by specialty; clinical reliability was stronger for GPT-4, with limited effect from question type.

LanguageEnglish

CategoryExam

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Abstract

Background

Methods

Results

Conclusion

Introduction

J Chin MedAssoc Original article While GPT-3.5 is unable to pass the Physician Licensing Exam in Taiwan, GPT-4 successfully meets the criteria Tsung-An Chena, Kuan-Chen Lina, Ming-Hwai Lina,b, Hsiao-Ting Changa,b, Yu-Chun Chena,b,c,d,e,*, Tzeng-Ji Chenf,g,* aDepartment of Family Medicine, Taipei Veterans General Hospital, Taipei, Taiwan, ROC; bSchool of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan, ROC; cBig Data Center, Taipei Veterans General Hospital, Taipei, Taiwan, ROC; dInstitute of Hospital and Health Care Administration, National Yang Ming Chiao Tung University, Taipei, Taiwan, ROC; eDepartment of Family Medicine, Taipei Veterans General Hospital Yuli Branch, Hualien, Taiwan, ROC; fDepartment of Family Medicine, Taipei Veterans General Hospital Hsinchu Branch, Hsinchu, Taiwan, ROC; gDepartment of Post-Baccalaureate Medicine, National Chung Hsing University, Taichung, Taiwan, ROC Abstract Background: This study investigates the performance of ChatGPT-3 .5 and ChatGPT-4 in answering medical questions from Taiwan’s Physician Licensing Exam, ranging from basic medical knowledge to specialized clinical topics. It aims to understand these artificial intelligence (AI) models’ capabilities in a non-English context, specifically traditional Chinese. Methods: The study incorporated questions from the Taiwan Physician Licensing Exam in 2022, excluding image-based queries. Each question was manually input into ChatGPT, and responses were compared with official answers from Taiwan’s Ministry of Examination. Differences across specialties and question types were assessed using the Kruskal–Wallis and Fisher’s exact tests. Results: ChatGPT-3 .5 achieved an average accuracy of 67 .7% in basic medical sciences and 53 .2% in clinical medicine. Meanwhile, ChatGPT-4 significantly outperformed ChatGPT-3 .5, with average accuracies of 91.9% and 90.7%, respectively. ChatGPT-3 .5 scored above 60.0% in seven out of 10 basic medical science subjects and three of 14 clinical subjects, while ChatGPT-4 scored above 60.0% in every subject. The type of question did not significantly affect accuracy rates. Conclusion: ChatGPT-3 .5 showed proficiency in basic medical sciences but was less reliable in clinical medicine, whereas ChatGPT-4 demonstrated strong capabilities in both areas. However, their proficiency varied across different specialties. The typeof question had minimal impact on performance. This study highlights the potential of AI models in medical education and nonEnglish languages examination and the need for cautious and informed implementation in educational settings due to variability across specialties. Keywords: Accuracy; Artificial intelligence; ChatGPT; Medical education; Taiwan’s Physician Licensing Exam 1. INTRODUCTION In the rapidly evolving field of artificial intelligence (AI), emergence of sophisticated language models such as Generative Pretrained Transformer (ChatGPT-3.5 and ChatGPT-4), developed *Address correspondence. Dr. Tzeng-Ji Chen, Department of Family Medicine, Taipei Veterans General Hospital Hsinchu Branch, 81, Section 1, Zhongfeng Road, Zhudong Township, Hsinchu 310, Taiwan, ROC. E-mail address: tjchen@ [vhct.gov.tw](vhct.gov.tw) (T.-J. Chen), Dr. Yu-Chun Chen, Department of Family Medicine, Taipei Veterans General Hospital Yuli Branch, 91, Xinxing Street, Yuli Township, Hualien 981, Taiwan, [ROC. E-mail address: yuchn.chen@gmail.com](ROC. E-mail address: yuchn.chen@gmail.com) (Y.-C. Chen). Author contributions: Dr. Yu-Chun Chen and Dr. Tzeng-Ji Chen contributed equally to this work. Conflicts of interest: Dr. Tzeng-Ji Chen and Dr. Yu-Chun Chen, editorial board members at the Journal of the Chinese Medical Association, have no roles in the peer review process of or decision to publish this article. The other authors declare that they have no conflicts of interest related to the subject matter or materials discussed in this article. Journal of Chinese Medical Assoc

FAQs

What is the purpose of evaluating ChatGPT-3.5 and ChatGPT-4 in this study?

To assess how well ChatGPT-3.5 and ChatGPT-4 answer medical questions from Taiwan’s Physician Licensing Exam across different knowledge areas and specialties in a traditional Chinese context.

How were the exam questions used and how were answers assessed?

Questions from the 2022 Taiwan Physician Licensing Exam (excluding image-based queries) were manually input into ChatGPT, and responses were compared with official answers from Taiwan’s Ministry of Examination.

What were the main accuracy findings comparing ChatGPT-3.5 and ChatGPT-4?

ChatGPT-3.5 averaged 67.7% in basic medical sciences and 53.2% in clinical medicine, while ChatGPT-4 significantly outperformed it with averages of 91.9% and 90.7% respectively, scoring above 60% in every subject.

Sep 29, 2026

Upload by Eliana