Evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

Jul 16, 2026

Upload by Maeve

Language model evaluation often reports a single accuracy figure per model and treats the higher number as unquestionably better, without assessing whether the observed difference could be sampling noise. Under realistic benchmark sizes and temperature sampling, self-variation can exceed between-model gaps. The evalci Python library converts per-item result tables into publication-ready claims using confidence intervals, paired significance tests, power analysis, clustered standard errors, and multiple-comparison corrections, with validation against independent references. A case study reanalyzes public MMLU comparisons and finds several leaderboard gaps are not significant after correction.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Introduction

Library Design

Statistical Methods

Validation

Case Study

arXiv :2607 .04429v 1 [ cs .CL] 5 Jul 2026 evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations Shreyas K Chandrahas Abstract The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this – confidence intervals, paired significance tests, power analysis, clustered standard errors, multiplecomparison correction – is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a purePython library (numpy/scipy/pandas only) that turns a per-item results table into a publicationready claim – e.g. ,“Model A beats Model B, ∆ = 3 .1 pts, 95% CI [1.2, 5.0], paired permutation p = 0 .002 , n = 1 ,319” – in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we reanalyze a public comparison of nine language models’MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at [https://pypi.org/project/evalci/](https://pypi.org/project/evalci/)(source: [https://github.com/Shreyaskc/evalci](https://github.com/Shreyaskc/evalci), DOI: [https://doi.org/10.5281/zenodo](https://doi.org/10.5281/zenodo). 21201815) . 1 Introduction A language model evaluation is an experiment: a fixed set of items is scored against a metric, and the result is a noisy estimate of an underlying population quantity, not the quantity itself. The evaluation literature has largely treated it as something else – a deterministic measurement – and reported the resulting number without an error bar. This is not a cosmetic omission. On a benchmark of a few thousand items, a 2-point gap between two models’ accuracy is frequently within the noise implied by the sample size alone. Under temperature sampling, the same model evaluated twice can differ from itself by more than the reported gap between two different models; recent work quantifying this shows that properly clustered standard errors – which account for repeated sampling of the same item – can be more than 3x larger than the naive standard errors typically reported [Miller, 2024] . This is not a new observation. Dror et al. [2018] surveyed significance testing practice in ACLand TACL papers and found it routinely ignored or misapplied. Card et al. [2020] showed that many standard NLP test sets are underpowered for the effect sizes papers actually claim. Miller [2024] derived the statistical machinery needed to analyze evaluations properly – clustered standard errors, power calculations – as a methods note. What has been missing is not the statistics but a standard, tested, pip-installable implementation shaped like an evaluation: a table of items, not a pair of scalars. scipy and statsmodels provide the general-purpose primitives (a proportion confidence interval, a bootstrap, a multiple-testing correction), but none of them speak the vocabulary an evaluation actually produces – per-item, paired-by-question, clustered-by-repeated-decode, manymodels-by-many-benchmarks – and none of them read lm-evaluation-harness or HELM output directly. We present evalci, which closes that gap. Its contribution is not a new statistical method; every routine it implements is decades old. Its contribution is packaging: a small, tested, purePython library, installable with pi

FAQs

Why does evalci argue that single accuracy numbers are misleading in language model evaluation?

A noisy evaluation over a finite benchmark can produce accuracy gaps that fall within sampling noise. Temperature sampling can also make the same model differ across runs by more than the reported gap between two models.

What is the core input format that evalci uses?

evalci is built around a per-item results table containing item_id, model, score, and optionally subset and sample_idx for repeated decodes.

Which statistical outputs can evalci produce from the per-item table?

It can generate confidence intervals, paired significance tests (e.g., permutation), power analysis, and multi-model comparison tables with multiple-comparison correction, plus clustered confidence intervals.

How does the case study change conclusions about MMLU leaderboard gaps?

After accounting for the implied pairwise comparisons via multiple-testing correction, the analysis finds that some adjacent leaderboard rank gaps are not statistically significant.

Jul 16, 2026

Upload by Maeve