Class of LLMs - Benchmarking Large Language Models on the Brazilian National Medical Examination

Sep 6, 2026

Upload by Maeve

The evaluation of Large Language Models (LLMs) in medicine has largely depended on English benchmarks tied to North American guidelines, reducing transferability to other healthcare systems. This paper benchmarks twenty-two proprietary and open-weight LLMs on ENAMED 2025, Brazil’s high-stakes National Examination for the Evaluation of Medical Training. The dataset contains 90 multiple-choice questions grounded in Brazilian public health policy, clinical practice, and Portuguese terminology, assessed via accuracy and ENAMED’s official IRT framework.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

1 Introduction

2 Related Work

3 Methodology

Class of LLMs: Benchmarking Large Language Models on the Brazilian National Medical Examination João Vitor Mariano Correia 1 , Pedro Henrique Alves de Castro2 , Gabriel Lino Garcia 1 , Pedro Henrique Paiola 1 , João Paulo Papa 1 1Department of Computing, Faculty of Sciences, São Paulo State University 2Department of Medical Sciences, Nove de Julho University Abstract The evaluation of Large Language Models (LLMs) in medicine has predominantly relied on English-language benchmarks aligned with North American clinical guidelines, limiting their applicability to other healthcare systems. In this paper, we evaluate twenty-two proprietary and open-weight LLMs on the 2025 National Examination for the Evaluation of Medical Training (ENAMED), a high-stakes, government-standardized assessment used to evaluate medical graduates in Brazil. The benchmark comprises 90 multiple-choice questions grounded in Brazilian public health policy, clinical practice, and Portuguese medical terminology, and is released as an open dataset. Model performance is measured using both standard accuracy and the official Item Response Theory (IRT) framework employed by ENAMED, enabling direct comparison with human proficiency thresholds. Results reveal a clear stratification of model capabilities: proprietary frontier models achieve the highest performance, whereas many open-weight and smallerdomain-adapted models fail to meet the minimum proficiency criterion. Across comparable scales, large generalist models consistently outperform specialized medical fine-tunes, suggesting that general reasoning capacity is a stronger predictor of success than narrow domain adaptation in this setting. These findings establish ENAMED as a rigorous benchmark for evaluating medical LLMs in Portuguese and highlight both the potential and current limitations of such models for educational assessment. 1 Introduction The integration of Artificial Intelligence into clinical practice has advanced Large Language Models (LLMs) from experimental systems to evaluated tools for tasks like summarization and question answering. While frontier models now pass global, English-language benchmarks such as the USMLE (Nori et al., 2023 ; Singhal et al., 2023), these assessments ignore the organizational structure of the Brazilian Unified Health System (SUS), regional epidemiology, and the Federal Council of Medicine (CFM) professional norms. In 2025, Brazil introduced the National Examination for the Evaluation of Medical Training (ENAMED), a centralized assessment consolidating undergraduate and residency evaluations. Aligned with National Curricular Guidelines (DCNs), ENAMED uniquely prioritizes public health policies and primary care. Its inaugural administration yielded unsatisfactory institutional scores, providing a rigorous, governmentstandardized context for evaluating AI preparedness and clinical reasoning in Brazilian medical education. This work evaluates the performance of generalpurpose and domain-adapted Large Language Models on ENAMED. Our results show that recent generalist models consistently outperform several specialized medical models. The main contributions of this study are twofold: (i) a systematic, domainlevel analysis of LLM behavior in a national medical assessment setting, highlighting both their potential and the challenges of deploying such models in context-specific healthcare environments, and (ii) the release of a structured dataset derived from ENAMED 2025, enabling reproducible evaluation and future research on LLM performance in Brazilian medical education. 2 Related Work Early evaluations of medical LLMs relied primarily on English-language benchmarks, including general-purpose and biomedical questionanswering datasets such as MMLU (Hendryckset al., 2021), PubMedQA (Jin et al., 2019), and MedQA (Jin et al., 2021) . Although these datasets effectively assess biomedical knowledge and multi- 101 Proceedings of the 17th Internat

FAQs

What benchmark does the paper use to evaluate medical LLMs in Brazil?

The study evaluates 22 proprietary and open-weight LLMs on ENAMED 2025, Brazil’s National Examination for the Evaluation of Medical Training.

How is model performance measured in the study?

Performance is measured using standard accuracy and the official Item Response Theory (IRT) framework used by ENAMED, allowing comparison to human proficiency thresholds.

What does the paper find about different types of LLMs on ENAMED?

Proprietary frontier models perform best, while many open-weight and smaller domain-adapted models fail to reach the minimum proficiency criterion; generalist models outperform specialized medical fine-tunes at comparable scales.

Sep 6, 2026

Upload by Maeve