Evaluating Large Language Models through Multidimensional Item Response Theory - A Comprehensive Case Study on ENEM

Sep 6, 2026

LLM evaluations on tasks such as high-stakes multidisciplinary tests often rely on raw accuracy, treating easy and hard items equally and overlooking the chance of correct guessing. The study repurposes the official Brazilian three-parameter logistic Item Response Theory (IRT) calibration used by INEP to score ENEM, and applies it to LLM outputs. It fits a four-dimensional 3-PL model aligned with ENEM knowledge domains, revealing domain-specific proficiency gaps hidden by similar accuracies.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

1. Introduction

2. The Multidimensional Item Response Theory

Evaluating Large Language Models through Multidimensional Item Response Theory: A Comprehensive Case Study on ENEM Leonardo Taschetto 1 , Renato Fileto 1 1 Dept. of Computer Science, Federal Univ. of Santa Catarina, Florianpolis-SC, Brazil Abstract. LLM evaluations on tasks like high-stakes multidisciplinary tests still rely on raw accuracy, a metric that weights easy and difficult questions equally and ignores guessing. To help bridge this methodological gap, we repurpose the official three-parameter logistic Item Response Theory (IRT) calibration that the Brazilian education authority (INEP) uses to score humans on the Exame Nacional do Ensino Me´dio (ENEM), and apply it to LLM responses. We then fit a four-dimensional 3-PL model aligned with ENEM’s knowledge domains. Results show that similar accuracies can mask proficiency gaps exceeding one standard deviation across domains. Mathematics remains the toughest domain for both humans and models, whereas questions on Human Sciences are system atically easier for both. 1. Introduction Large Language Models (LLMs) now reach state-of-the-art (SOTA) performance on highstakes multidisciplinary tests, notably national university admission exams such as the SAT in the United States [OpenAI 2024a], the Gaokao in China [Zong and Qiu 2024], and Brazil’s ENEM [Abonizio et al. 2024] . These tests have been used as benchmarks to compare LLM capabilities. However, LLM performance studies on these exams mostly report a single metric – accuracy – which offers only a coarse view of test-taker proficiency. It weighs easy and hard questions equally, ignores how well each question differentiates examinees who are stronger or weaker in particular abilities, and fails to adjust for the non-zero chance of correctly guessing answers. Meanwhile, in the realm of human testing, where reliable ranking of candidates is critical, exam authorities apply Item Response Theory (IRT) [Baker 2001] to calibrate each question weight on the candidate score according to how sharply it discriminates high-and low-performing examinees. Thereby, it can produce scaled proficiency scores that distinguish candidates who achieve the same number of correct answers, according to the relevance of each question answered correctly and incorrectly for scoring proficiency in particular abilities. IRT also takes into account question difficulty, based on the percentage of a population that answers it correctly, and the probability of guessing. A few recent studies applying IRT to evaluate LLMs on university-admission exams [Zhang et al. 2023, Zong and Qiu 2024] show that IRT provides finer-grained rankings than accuracy alone. However, they do not exploit IRT distinct dimensions to assess specific abilities as we propose in this work. In addition, to the best of our knowledge, no work has applied IRT to evaluate LLMs on the Portuguese-language ENEM. Motivated by this gap, we ask two linked questions. First, using uni-dimensional IRT scoring, how do SOTA LLMs of varying sizes compare with human candidates across the ENEM’s four knowledge domains? Second, once baselines are established, how do multidimensional IRT (MIRT) shift each model’s performance within those domains? The major contributions of this paper are: (i) replication of ENEM’s official unidimensional IRT models to evaluate LLM performance on this exam on the human scale; (ii) an extended analysis using a multidimensional IRT model aligned with the exam’s official four knowledge domains. Our experimental results show that identical accuracy does not imply identical proficiency across knowledge domains, and illustrate how these differences depend on model architecture. 2. The Multidimensional Item Response Theory The Item Response Theory (IRT) is a family of mathematical models used to measure individuals’ latent abilities, i.e., unobservable characteristics or proficiencies (e.g., numerical reasoning, reading comprehension, scientific knowledg

FAQs

Why is accuracy an insufficient metric for evaluating LLMs on exams like ENEM?

Accuracy weights easy and difficult questions equally, ignores how well items discriminate between stronger and weaker examinees, and does not adjust for guessing. This can mask proficiency differences across abilities.

What methodological approach does the paper use to evaluate LLMs?

It repurposes the official three-parameter logistic IRT calibration used by INEP for ENEM human scoring, then fits a four-dimensional 3-PL model aligned with ENEM’s knowledge domains.

What do the results show about similar accuracies across ENEM domains?

The study finds that identical accuracy can hide proficiency gaps exceeding one standard deviation across domains. It also reports that Mathematics is the toughest domain for both humans and models, while Human Sciences items are systemically easier.

Sep 6, 2026

Upload by Callum