ALVORADA-BENCH - CAN LANGUAGE MODELS SOLVE BRAZILIAN UNIVERSITY ENTRANCE EXAMS? - Abstract

Sep 6, 2026

Upload by McQueen

Language models are increasingly used in Brazil, but evaluation remains largely English-centric. This paper introduces Alvorada-Bench 1, a 4,515-question text-only benchmark drawn from five Brazilian university entrance examinations. Twenty models are tested under zero-shot, role-playing, and chain-of-thought prompting, producing 270,900 responses with structured confidence, perceived difficulty, and Bloom-level self-reports. Top systems achieve 94%+ accuracy overall, with notable drops in Mathematics and IME/ITA engineering exams, indicating multi-step reasoning gaps.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Motivation and evaluation gap

Cultural specificity and benchmark limitations

Alvorada-Bench overview

Dataset and Methodology

The Alvorada-Bench Dataset

arXiv :2508 . 15835v1 [ cs .CL] 19 Aug 2025 ALVORADA-BENCH: CAN LANGUAGE MODELS SOLVE BRAZILIAN UNIVERSITY ENTRANCE EXAMS? Henrique Godoy Inteli São Paulo, Brazil [henrique.godoy@sou.inteli.edu.br](henrique.godoy@sou.inteli.edu.br) ABSTRACT Language models are increasingly used in Brazil, but most evaluation remains English-centric. This paper presents Alvorada-Bench 1 , a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations. Evaluating twenty models under zero-shot, role-playing, and chain-of-thought prompting, producing 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level. The top models exceed 94% accuracy overall, but accuracy declines on Mathematics and on the engineering oriented IME and ITA exams, indicating persistent weaknesses in multi-step reasoning. Confidence is well calibrated and correlates with perceived difficulty, revealing that models can accurately assess their own certainty capabilities. A cost-accuracy analysis shows that high accuracy is achievable at under $2 per 1K tokens. On ENEM 2024 the topmodel (O3) achieved perfect scores in Languages subject questions while even the weakest system (GPT-4.1 Nano) only underperforms humans in Mathematics. Through exams that distill decades of Brazilian educational priorities and assess millions of students yearly, Alvorada-Bench establishes whether language models can navigate the intersection of language, culture, and reasoning that defines academic readiness in Brazil. 1 Introduction Language models increasingly mediate critical decisions across diverse applications, from educational assessment to medical diagnosis, yet their evaluation remains predominantly English-centric. As these models expand into global markets serving linguistically and culturally diverse populations, this evaluation gap poses significant risks. Current evaluations demonstrate remarkable performance on standardized tests. GPT-4 scores at the 90th percentile on the SAT, passes the Bar Exam in the top 10%, and outperforms 85% of participants in coding contests [2, 5] . However, these benchmarks embed cultural assumptions that limit global applicability. Translation cannot address implicit cultural frameworks, as SAT questions about financial aid assume familiarity with concepts irrelevant in countries with free universities. This cultural specificity produces measurable degradation: performance drops from 70.9% on English educational tasks to 49.7% in Telugu [3], while Chinese outputs exhibit 41% lexical divergence from native usage despite English-like syntactic patterns [4] . These issues are evident in non-English contexts such as Brazil, where Portuguese serves a population exceeding 220 million, making it the sixth most spoken language worldwide. However, Portuguese remains underrepresented in the benchmarks. Brazilian university entrance exams offer a compelling solution that combines cultural specificity with rigorous standardization. Refined over decades through expert review, statistical validation, and millions of student responses, these exams serve as a natural experiment in knowledge assessment, capturing both cognitive demands and the cultural knowledge expected of educated Brazilians. To address this gap, this paper introduce Alvorada-Bench: a benchmark comprising 4,515 questions drawn from five Brazilian university entrance examinations—ENEM (Exame Nacional do Ensino Médio), FUVEST (São Paulo), 1Data and code available at [https://huggingface.co/datasets/HenriqueGodoy/Alvorada-bench](https://huggingface.co/datasets/HenriqueGodoy/Alvorada-bench) and [https://](https://)[ ](https://)[github.com/herniqeu/Alvorada-bench](github.com/herniqeu/Alvorada-bench) UNICAMP (Campinas), IME (Instituto Militar de Engenharia), and ITA (Instituto Tecnológico de Aeronáutica) spanning from 1981 to 2025 . Using Alvorada-Bench, we conduct a controlled evaluation of 20 models from Op

FAQs

What is Alvorada-Bench and what does it contain?

Alvorada-Bench is a text-only benchmark with 4,515 multiple-choice questions drawn from five Brazilian university entrance examinations. The dataset covers years spanning 1981 to 2025 and aligns with disciplinary areas linked to the BNCC.

How were language models evaluated on Alvorada-Bench?

Twenty models were evaluated using zero-shot, role-playing, and chain-of-thought prompting. Each interaction includes structured self-reports such as confidence, perceived difficulty, and Bloom level.

What performance patterns appear across subjects and exams?

Overall accuracy can exceed 94%, but accuracy declines on Mathematics and on engineering-oriented IME and ITA exams. This suggests persistent weaknesses in multi-step reasoning despite strong performance elsewhere.

Sep 6, 2026

Upload by McQueen