LLM-as-a-Verifier: A General-Purpose Verification Framework

Jul 16, 2026

Upload by Aria

The work introduces LLM-as-a-Verifier, a general-purpose verification framework that treats correctness determination as a new scaling axis for large language models. It replaces discrete judge scores with a probabilistic expectation over scoring-token logits to produce continuous feedback and reduce tie rates. Verification can scale across score granularity, repeated evaluation, and criteria decomposition, improving calibration and accuracy via variance and complexity reduction. A cost-efficient ranking algorithm selects the best candidate using verifier preference probabilities. Results span coding, robotics, and medical domains, and fine-grained signals support progress tracking and dense-reward RL for SAC/GRPO.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages31

Reading Time1 h 18 min

Introduction

Verification as a new scaling axis

Probabilistic continuous scoring

Scaling dimensions for verification

Cost-efficient ranking and applications

arXiv :2607 .0539 1v2 [ cs .AI ] 7 Jul 2026 | LLM-as-a-Verifier: A General-Purpose Verification Framework<br>Jacky Kwok1 , Shulu Li2 , Pranav Atreya2 , Yuejiang Liu1 , Yixing Jiang1 Chelsea Finn1 , Marco Pavone1 ,3 , Ion Stoica2 , Azalia Mirhoseini1<br>1 Stanford University 2UC Berkeley 3NVIDIA Research | | --- | | | | Figure 1: Overall Performance Results. Our proposed framework, LLM-as-a-Verifier, achieves stateof-the-art performance across coding, robotics, and medical domains: Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) .<br>Abstract: Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of large language models (LLMs) . In this work, we identify verification—the ability to determine the correctness of a solution—as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides finegrained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation substantially reduces tie rates when comparing complex solutions and enables verification to scale along multiple dimensions: (1) score granularity,(2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently leads to additional gains in verification accuracy through variance and complexity reduction. To make verification scaling practical, we further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the preference probabilities derived from the verifier’s continuous scores. LLM-as-a-Verifier is effective across coding, robotics, and medical domains. It achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) . Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build extensions for Claude Code and Codex, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can be used as a dense reward signal for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks. | [llm-as-a-verifier.com](llm-as-a-verifier.com) llm-as-a-verifier claude-code-extension Corresponding author: Jacky Kwok <[jackykwok@stanford. edu](jackykwok@stanford. edu) > Figure 2: Multiple modalities, many applications, one unified verification framework. We present LLM-as-a-Verifier, a general-purpose framework that provides fine-grained feedback for any modality without requiring additional training. By leveraging the full distribution of scoring-token logits, our method captures evaluation uncertainty and enables verification to scale along three dimensions: score granularity, repeated evaluation, and criteria decomposition. The resulting fine-grained feedback can be used for test-time scaling, progress tracking, and reinforcement learning. 1. Introduction Recent advances in large language models (LLMs) have established scaling as a central paradigm for improving their capabilities. Performance has been driven by scaling along multiple axes, including pre-training data and compute, post-training optimization, and test-time inference [1–3] . However, while generation has benefited significantly from these scaling paradigms, verification—the ability to determine the quality or correct

FAQs

What problem does LLM-as-a-Verifier address?

It targets the limited scalability of verification—deciding whether a solution is correct—compared with the well-studied scaling of generation.

How does LLM-as-a-Verifier differ from standard language-model judges?

Instead of producing discrete scores, it computes the expectation over scoring-token logits to generate continuous verification scores.

Which domains and evaluation benchmarks show the framework’s effectiveness?

The framework reports state-of-the-art performance on Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench, spanning coding, robotics, and medical tasks.

Jul 16, 2026

Upload by Aria