Stop Guessing When to Stop Testing Efficient Model Evaluation with Just Enough Data

Jul 16, 2026

Fixed-size benchmarks limit model evaluation efficiency because they cannot match different goals such as ranking, selection, or in-development testing, leading to either excessive computation or reduced statistical reliability. The work advocates sequential testing and introduces an adaptive evaluation framework that sets principled stopping criteria aligned with needs like diminishing-returns detection and minimum detectable effect size. Experiments on the Open VLM Leaderboard show large cost savings while preserving statistical significance, with publicly available code.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Abstract

Introduction

Use Case Examples

Compute-Constrained Evaluation with Statistical Guarantees

Meaningful Change Achieved

Model Development: Efficient Candidate Model Selection

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data Ofir Arviv* Kristjan Greenewald Yotam Perlitz Hadar Mulian Michal Shmueli-Scheuer Leshem Choshen* IBM Research *Corresponding authors: [ofir.arviv@ibm.com](ofir.arviv@ibm.com) , [leshem.choshen@ibm.com](leshem.choshen@ibm.com) arXiv :2607 .08522v 1 [ cs .LG] 9 Jul 2026 Abstract The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability – a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance. Code is publicly available at [github.com/OfirArviv/adaptive-eval](github.com/OfirArviv/adaptive-eval). 1 Introduction The rapid advancement of large language and vision models (VLMs and LLMs) has spurred the creation of numerous benchmarks to assess their capabilities (Li and Lu, 2024 ; Chang et al., 2023) . However, processing high-resolution images, handling large contexts, comparing performance across multiple datasets, and utilizing expensive metrics like LLM-as-Judge have drastically increased evaluation costs (Zhao et al., 2024 ; Perlitz et al., 2024) . Current evaluation practices, typically employing fixed-size benchmarks, are inherently wasteful, continuing to the predetermined sample size even Figure 1: Half-width of the confidence interval as a function of sample size for 206 models in the Open VLM Leaderboard benchmark, averaged over 10 random seeds. As sample size increases, the CI narrows, reducing uncertainty in model performance estimates. Two stopping strategies are illustrated: (1) stopping when the CI reaches ±2 .5, saving 80% of the evaluation cost, and (2) stopping when diminishing returns plateau, reducing cost by 44% while sacrificing only 0.132 points in precision. Our framework can detect and stop evaluation based on such rules, ensuring both statistical rigor and evaluation efficiency. when the outcome is statistically clear. While practitioners often reduce costs by using fewer samples (Perlitz et al., 2024 ; Polo et al., 2024a ; Fogliato et al., 2024 ; Zhao et al., 2024), these heuristic approaches lack statistical guarantees. Critically, fixed-size approaches, whether highcost and precise or low-cost and imprecise, fail to align the evaluation effort with the evaluation’s objective. Debugging a model may only require a inexpensive coarse approximation, while definitively determining a superior model among close contenders demands sufficient data to achieve statistical significance. In this work, we propose a statistically grounded solution: an adaptive evaluation framework based on sequential testing. Rather than enforcing a fixed sample size, adaptive evaluation stops when the practical and statistical needs are met. Thus, users explicitly define their needs, and the method ensures that evaluations are neither underpowered nor excessive, effectively balancing reliability and efficiency. In addition, when efficiency is prioritized, users are fully awa

FAQs

为什么固定样本评测会让模型评测变得低效且不可靠？

固定样本基准无法适配不同评测目标所需的统计能力，导致计算成本被浪费或统计结论不够可靠，从而出现效率与可靠性的错配。

这项工作提出了什么方法来决定何时停止评测？

作者提出采用顺序检验（sequential testing）的自适应评测框架，用与评测需求相匹配的停止准则来在可靠性与效率之间做权衡。

框架在哪些具体需求下设置停止准则，并取得了什么效果？

停止准则可用于识别收益递减、检测最小可检出效应大小等常见需求。在 Open VLM Leaderboard 上，框架能在保证统计显著性的同时显著降低计算成本，例如相对固定样本评测减少约 80%。

Jul 16, 2026

Upload by Valentina