Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil

Sep 18, 2026

Uploaded by Mia

The Exame Nacional do Ensino Médio (ENEM) is a key Brazilian admission exam, with publicly released (anonymized) responses and a socioeconomic status questionnaire. The study uses the ENEM dataset to compare GPT-3.5, GPT-4, and MariTalk against human groups defined by SES, evaluating multiple-choice answers and essay outputs. Results show LLM performance is not significantly biased versus humans on the tests, with distances largely driven by human accuracy. Essay analyses find differences in word choice and text statistics, yet indicate LLM outputs do not represent distinct human groups in this ENEM setting.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Abstract

Introduction

Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil Marcelo Sartori Locatelli, Matheus Prado Miranda, Igor Joaquim da Silva Costa, Matheus Torres Prates, Victor Thom, Mateus Zaparoli Monteiro, Tomas Lacerda, Adriana Pagano, Eduardo Rios Neto, Wagner Meira Jr., Virgilio Almeida Universidade Federal de Minas Gerais, Belo Horizonte, Brazil {locatellimarcelo, matheus.prado, igor.joaquim, matheus.prates, victor-thome, mateuszaparoli, tomas.muniz, meira, [virgilio](virgilio}@dcc.ufmg.br)[}](virgilio}@dcc.ufmg.br)[@dcc.ufmg.br](virgilio}@dcc.ufmg.br), [apagano@letras.ufmg.br](apagano@letras.ufmg.br), [eduardo@cedeplar.ufmg.br](eduardo@cedeplar.ufmg.br) arXiv :2408 .05035v 1 [ cs .CL] 9 Aug 2024 Abstract The Exame Nacional do Ensino Mdio (ENEM) is a pivotal test for Brazilian students, required for admission to a significant number of universities in Brazil. The test consists of four objective high-school level tests on Math, Humanities, Natural Sciences and Languages, and one writing essay. Students’ answers to the test and to the accompanying socioeconomic status questionnaire are made public every year (albeit anonymized) due to transparency policies from the Brazilian Government. In the context of large language models (LLMs), these data lend themselves nicely to comparing different groups of humans with AI, as we can have access to human and machine answer distributions. We leverage these characteristics of the ENEM dataset and compare GPT- 3.5 and 4, and MariTalk, a model trained using Portuguese data, to humans, aiming to ascertain how their answers relate to real societal groups and what that may reveal about the model biases. We divide the human groups by using socioeconomic status (SES), and compare their answer distribution with LLMs for each question and for the essay. We find no significant biases when comparing LLM performance to humans on the multiple-choice Brazilian Portuguese tests, as the distance between model and human answers is mostly determined by the human accuracy. A similar conclusion is found by looking at the generated text as, when analyzing the essays, we observe that human and LLM essays differ in a few key factors, one being the choice of words where model essays were easily separable from human ones. The texts also differ syntactically, with LLM generated essays exhibiting, on average, smaller sentences and less thought units, among other differences. These results suggest that, for Brazilian Portuguese in the ENEM context, LLM outputs represent no group of humans, being significantly different from the answers from Brazilian students across all tests. Introduction Over the last few years, advancements in NLP and machine learning have culminated in the creation of large language models (LLMs) that excel in a variety of tasks, such as translating, question-answering, summarization, and natural language inference. As these models are getting more and more adopted by the general public, as well as the industry and academia, researchers from many different backgrounds have started evaluating how they fare on a plethora Copyright © 2024, Association for the Advancement of Artificial Intelligence ([www.aaai.org](www.aaai.org)). All rights reserved. of benchmarks for distinct subjects and tasks (Chang et al. 2024) . A significant part of this research effort focuses on benchmarks where there is a desirable outcome, or groundtruth that the model is expected to produce, for example the MMLU dataset (Hendrycks et al. 2021), question answering (Liang et al. 2022), and even some more domain specific tasks, such as standardized exams (Giannos, Delardas et al. 2023; Nunes et al. 2023) . While these types of evaluation are important, researchers have recently been shifting their attention towards measuring aspects of LLMs that are not directly related to performance. These come in the form of various fairness related concerns, with recent

FAQs

What do the results indicate about biases in LLM performance on ENEM?

The study finds no significant biases when comparing LLM performance to humans on the multiple-choice Portuguese tests, with answer distances largely determined by human accuracy. Essay differences appear in word choice and text structure, but overall LLM outputs are significantly different from students across groups.

Sep 18, 2026

Uploaded by Mia