Performance of the Artificial Intelligence large language models ChatGPT 3.5, Gemini (Google Bard), ChatGPT 4.0, and Gemini 2.5 flash in surgical subspecialty questions of Brazilian medical residency exams

Sep 8, 2026

Objective: With rapid advances in artificial intelligence and its growing influence on medical education, large language models such as ChatGPT and Gemini increasingly support clinical reasoning. Yet evidence remains limited regarding their performance on Brazilian medical residency entrance examinations and their value as educational tools for trainees. Therefore, this study evaluated AI performance on surgical subspecialty residency entrance exams across six programs using structured, single-best-answer multiple-choice items. Methods: Performance was assessed using 464 practice questions drawn from major institutions in São Paulo.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages6

Reading Time15 min

AccessFree to download

Abstract

Objective

Methods

Results

Conclusion

Introduction

How to cite this article

Official Publication of the Instituto Israelita de Ensino e Pesquisa Albert Einstein Performance of the Artificial Intelligence large language models ChatGPT 3.5, Gemini (Google Bard), ChatGPT 4.0, and Gemini 2.5 flash in surgical subspecialty questions of Brazilian medical residency exams ❚ Authors Maria Clara Pimenta de Figueiredo, Victor Hugo Alves Diniz, Ana Clara de Campos Granado, Gabriel Chagas Lutfala Paulino, Gabryella Rodrigues de Oliveira ❚ Correspondence E-mail: [pimenta8mariaclara@gmail.com](pimenta8mariaclara@gmail.com) ❚ DOI DOI: 10.31744/einstein_journal/2026AO1436 ❚ In Brief The application of Artificial Intelligence has been expanded to medicine and presents a promising future for medical education. Despite technological advances, it is still important to consider the role of professionals in developing the essential clinical judgments required for medical practice. ❚ Highlights ■ ChatGPT and Gemini are showing increased ability to accurately answer multiple-choice questions on medical exams. ■ There was no statistical significance in the rate of correct answers by ChatGPT 3.5 and Gemini 1.5. However, we observed that ChatGPT 4.0 performed significantly better, and so did Gemini 2.5 Flash, when comparing to the literature. ■ The question taxonomy did not appear to be a relevant factor regarding the success rate of the models. ❚ How to cite this article: Figueiredo MC, Diniz VH, Granado AC, Paulino GC, Oliveira GR. Performance of the Artificial Intelligence large language models ChatGPT 3 .5, Gemini (Google Bard), ChatGPT 4.0, and Gemini 2.5 flash in surgical subspecialty questions of Brazilian medical residency exams. einstein (São Paulo) . 2026;24:eAO1436 . einstein (São Paulo) OR IG INAL ARTICLE Official Publication of the Instituto Israelita de Ensino e Pesquisa Albert Einstein e-ISSN: 2317-6385 How to cite this article: Figueiredo MC, Diniz VH, Granado AC, Paulino GC, Oliveira GR. Performance of the Artificial Intelligence large language models ChatGPT 3 . 5, Gemini (Google Bard), ChatGPT 4 .0, and Gemini 2.5 flash in surgical subspecialty questions of Brazilian medical residency exams. einstein (São Paulo) . 2026;24:eAO1436 . Associate Editor: Helder I Nakaya Hospital Israelita Albert Einstein, São Paulo, SP, Brazil ORCID: [https://orcid.org/0000-0001-5297-9108](https://orcid.org/0000-0001-5297-9108) Corresponding Author: Maria Clara Pimenta de Figueiredo Rua Tessalia Vieira de Camargo, 126, Cidade Universitária Zeferino Vaz Zip code: 13084-971, Campinas, SP, Brazil Phone: (55 21) 99494-3537 E-mail: [pimenta8mariaclara@gmail.com](pimenta8mariaclara@gmail.com) Received on: Oct 4, 2024 Accepted on: Aug 6, 2025 Conflict of interest: none. Copyright the authors This content is licensed under a Creative Commons Attribution 4.0 International License. ORIGINAL ARTICLE Performance of the Artificial Intelligence large language models ChatGPT 3.5, Gemini (Google Bard), ChatGPT 4.0, and Gemini 2.5 flash in surgical subspecialty questions of Brazilian medical residency exams Maria Clara Pimenta de Figueiredo1, Victor Hugo Alves Diniz1, Ana Clara de Campos Granado1, Gabriel Chagas Lutfala Paulino1, Gabryella Rodrigues de Oliveira1 1 Universidade Estadual de Campinas, Campinas, SP, Brazil. DOI: 10.31744/einstein_journal/2026AO1436 ❚ ABSTRACT Objective: Given the rapid advancement of Artificial Intelligence and its significant impact on medical education, particularly with the development of large language models, such as ChatGPT and Gemini, ChatBots have shown an increasing ability to support clinical reasoning. Regarding this, there has been a growing interest in assessing the performance of ChatBots in medical examinations. However, there are insufficient data on these tools for addressing Brazilian medical exam-related inquiries, as well as their potential as educational tools for medical students. Therefore, we aimed to eva

FAQs

What question does the study aim to answer about large language models?

The study evaluates how ChatGPT and Gemini perform on surgical subspecialty residency entrance exam questions in Brazil and whether they could function as educational adjuncts.

Which models were tested and how were the questions structured?

ChatGPT 3.5, Gemini (Google Bard), ChatGPT 4.0, and Gemini 2.5 Flash were tested using 464 single-correct-answer multiple-choice practice questions from six surgical residency programs in São Paulo.

What were the main findings regarding accuracy?

All models showed substantial performance differences, with higher correctness for ChatGPT 4.0 (77.6%) and Gemini 2.5 Flash (81%) compared with ChatGPT 3.5 (55.4%) and Gemini (51.1%).

How should the tools be used in medical education, according to the conclusion?

The findings suggest advanced large language models may support medical education when used appropriately, while not replacing the clinical decision-making abilities developed through formal surgeon training.

Sep 8, 2026

Upload by 8796093062539