Analyzing the Accuracy of Large Language Models in United States Medical Licensing Exam Social-Science Preparation Question Banks - original open access article

Q: Which AI platforms were evaluated for answering USMLE social-science preparation questions?

The study tested ChatGPT, Gemini, and Perplexity by inputting multiple-choice questions and recording correct versus incorrect responses.

Q: What were the main accuracy findings for Step 1 and Step 2?

For Step 1, Gemini showed the highest accuracy, while ChatGPT achieved the highest accuracy for Step 2. All platforms outperformed the student averages for both steps, with reported statistical significance in specific comparisons.

Q: What conclusion did the study draw about AI and ethical reasoning?

The findings emphasize the importance of understanding whether empathetic and ethical thinking can be replicated or used to prepare students, highlighting AI’s ability to solve some ethical dilemmas while focusing on accuracy rather than direct measures of empathetic reasoning.

Sep 4, 2026

Uploaded by Aria Callaghan

Artificial intelligence is reshaping medical education as the United States Medical Licensing Examinations increasingly include Social Sciences (Ethics/Legal/Professional) items that demand reasoning through complex scenarios. This study assesses three AI platforms’ performance on preparation question banks by entering multiple-choice items into ChatGPT, Gemini, and Perplexity and comparing accuracy against step-specific benchmarks and student averages. Statistical testing via ANOVA and t-tests evaluates significance, with results discussed in relation to ethical dilemma handling and accuracy limits.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

AI and LLMs in medical education

Rationale for evaluating social-science exam items

Methods

Data sources and question bank collection

AI platforms and response scoring

Statistical analysis

Results

Step 1 accuracy comparison

Step 2 accuracy comparison

Comparison with student averages

Conclusion

Implications for ethical reasoning and education

Cureus Journal of Computer Science Open Access Original Article Received 07/09/2025 Review began 07/30/2025 Review ended 08/22/2025 Published 09/01/2025 © Copyright 2025 Bourmand et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CCBY 4.0. , which permits unrestricted use , distribution , and reproduction in any medium , provided the original author and source are credited. DOI: [https://doi.org/10.7759/s44389-025-09114-9](https://doi.org/10.7759/s44389-025-09114-9) Analyzing the Accuracy of Large Language Models in United States Medical Licensing Exam Social-Science Preparation Question Banks Raika Bourmand 1 , Sofia E. Olsson 2 , Fiza Z. Baloch 1 , Akhila Sonti 1 , Arman Fijany 3 1. Medicine, Burnett School of Medicine at TCU, Fort Worth, USA 2. Otolaryngology, UC Davis Medical Center, Sacramento, USA 3. Plastic Surgery, UC Davis Medical Center, Sacramento, USA Corresponding author: Raika Bourmand, [raika.bourmand@tcu.edu](raika.bourmand@tcu.edu) Abstract Introduction: Artificial intelligence (AI) emergence has changed the medical education landscape. The United States Medical Licensing Exams increasingly include questions in the “Social Sciences (Ethics/Legal/Professional)” domain, which often require reasoning through complex scenarios. This study evaluates three AI platforms in answering such preparation questions. Methods: Social-science questions from UWorld and Amboss were accumulated for Steps 1 and 2. Multiple-choice questions were entered into ChatGPT, Gemini, and Perplexity, yielding a correct/incorrect response. Percentages of correct responses were compared between platforms and to student averages. Analysis of variance and t-tests conducted determined statistical significance, the upper threshold being P = 0 .05. As this study focused on quantitative performance, outcomes were limited to accuracy on multiplechoice items, rather than direct measures of empathetic reasoning. Results: One hundred nine UWorld and 63 Amboss questions were available for Step 1. A total of 189 UWorld and 185 Amboss questions were available for Step 2. Google Gemini had the highest accuracy for Step 1 (86 .6%), while ChatGPT had the highest accuracy for Step 2 (83 .6%) . All platforms outperformed the student average for Step 1 (68 .5%) and Step 2 (68 .9%), with ChatGPT and Gemini doing so significantly for Step 1 (p < 0 .01), and ChatGPT doing so significantly for Step 2 (p < 0 .01) . Conclusion: It is critical to understand whether empathetic thinking can be replicated by technology or prepare students. This study highlights the ability for AI to solve ethical dilemmas that students may struggle with. Categories: AI applications, AI Ethics and Fairness, AI/ML-based decision support systems Keywords: artificial intelligence, medical ethics, medical education, large language model(llm), exam preparation Introduction The rapid advancement and widespread adoption of artificial intelligence (AI) and large language models (LLMs) is transforming multiple industries including various aspects of medicine. The AI market, valued at $50 billion in 2023, is projected to grow remarkably, reaching $826 billion by 2030 [1] . With the democratization of AI tools, these emerging technologies may offer new ways to support students navigating medical training, including preparation for pivotal exams such as the United States Medical Licensing Examination (USMLE) . As medicine adapts to this technological shift, the role of LLMs in shaping how future physicians learn, reason, and make decisions is becoming a central question. Despite the promise of AI in medical education, its integration has been met with mixed responses [2-4] . A 2023 study by Alkhaaldi et al. [2] found that only 20.4% of surveyed residency applicants endorsed using AI to complete written assessments, and 9 .4% used AI for clinical work. However, the majority of applicants expressed

FAQs

Which AI platforms were evaluated for answering USMLE social-science preparation questions?

The study tested ChatGPT, Gemini, and Perplexity by inputting multiple-choice questions and recording correct versus incorrect responses.

How were the question banks collected and scored?

Social-science questions from UWorld and Amboss were accumulated for USMLE Steps 1 and 2, and each question was scored by whether the AI selected the correct option.

What were the main accuracy findings for Step 1 and Step 2?

For Step 1, Gemini showed the highest accuracy, while ChatGPT achieved the highest accuracy for Step 2. All platforms outperformed the student averages for both steps, with reported statistical significance in specific comparisons.

What conclusion did the study draw about AI and ethical reasoning?

The findings emphasize the importance of understanding whether empathetic and ethical thinking can be replicated or used to prepare students, highlighting AI’s ability to solve some ethical dilemmas while focusing on accuracy rather than direct measures of empathetic reasoning.

Sep 4, 2026

Uploaded by Aria Callaghan