Grading the Graders - Verification Autonomy Levels (L0–L5) for LLM Reasoning

Sep 15, 2026

Uploaded by Theodore

Large language models (LLMs) increasingly rely on verifiers such as step checkers, self-consistency filters, tool-based fact checkers, and formal proof assistants to detect model errors. The verification literature uses “level” inconsistently, covering multiple axes including granularity, abstraction, risk, system layer, and epistemic anchoring. The paper proposes Verification Autonomy Levels (VAL) as a unified meta-standard that classifies verification by the origin of the specification and the guarantees provided, from L0 self-declaration to L5’s impossibility in unrestricted settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Verification proposals and the open question

Classical verification vs. anchored specifications in LLM systems

The proposed dichotomy: formal completeness vs. open-world anchored correctness

Why “level” is conflated across multiple axes

The motivation for VAL: the epistemic anchor and trust guarantees

Grading the Graders: Verification Autonomy Levels (L0– L5) for LLM Reasoning Yajie Yin Email: [1549080929@qq.com](1549080929@qq.com) · ORCID: 0009-0001-6168-2530 Preprint: arXiv:2608.19009 ([https://arxiv.org/abs/2608.19009](https://arxiv.org/abs/2608.19009)) Code & data: [https://github.com/1549080929-debug/math_agent](https://github.com/1549080929-debug/math_agent)[ ](https://github.com/1549080929-debug/math_agent)[License: CC-BY 4.0](License: CC-BY 4.0) Abstract | Large language models (LLMs) are increasingly paired with "verifiers"—step checkers, self-consistency filters, tool-based fact checkers, and formal proof assistants—that claim to detect the model's errors. Yet the verification literature uses the word level to mean at least five different things: verification granularity, concept abstraction, risk tier, system-stack layer, and the epistemic source of the ground truth. We propose Verification Autonomy Levels (VAL), a meta-standard that classifies any verification scheme along a single axis: where does the verification spec comefrom, and what does the verdict guarantee? VAL ranges from L0 (LLM self-declaration; no deterministic anchor) through L2 (objective ground truth; correctness only) to L3/L4 (decidable systems with single-property or domainlevel completeness), with L5 shown to be impossible in the unrestricted case. Central to VAL is the completeness blind spot: substitution-and sampling-based verifiers can confirm that proposed candidates hold, but cannot prove that no candidate was missed. We further identify a dichotomy the literature has not stated: completeness is reachable only for formally specifiable properties, whereas empirical open-world verification (fact-checking, diagnosis) caps at anchored correctness (L2) . We document this gap empirically across four domains—symbolic mathematics, behavior monitoring, medical diagnosis, and code generation (the fourth a reverse validation: framework predictions stated before evidence)—and in the strongest formal-verification baseline in our survey, whose own authors note the verifier "focuses on the correctness of each step." We further show that the levels of granularity, concept hierarchy, risk, and system stack are orthogonal to VAL, resolving a systematic conflation across 17 surveyed papers. We release a runnable classifier ([val_standard.py](val_standard.py)) and the full literature assessment as supplementary material.
Keywords: LLM verification, verification autonomy, completeness, ground truth, trustworthy AI | | --- | | 1. Introduction | Large language models (LLMs) produce fluent, confident, and frequently wrong reasoning. The dominant mitigation is verification: attach a second mechanism that checks the model's claims. The 2023–2025 literature has produced a large and rapidly growing body of verification proposals—trained step verifiers [3], self-checking schemas [5], tool-augmented fact checkers [17], graph-structured verification [2], and formal proof assistants [4]—each claiming to catch the errors the model cannot self- report. This proliferation raises a question the literature has not explicitly asked: what can a given verification scheme actually guarantee, and where does that guarantee come from? Classical verification theory explains how to verify a program against a given specification—the specification is assumed exogenous and trusted. LLM-based systems introduce a prior problem that this assumption excludes: whether the specification, the evaluator, and the verification authority are themselves appropriately anchored. We argue that this prior problem has a natural graded structure (VAL, Sec. 3), and that the LLM-verification literature has been answering it implicitly, without naming it or giving it an operational decision procedure. We further identify a dichotomy the literature has not stated: completeness is achievable only for formally specifiable properties—those with a decidable fragment (mathematical, synt

FAQs

What is the main dichotomy the paper claims about verification completeness?

Completeness is reachable only for formally specifiable properties with decidable fragments. Empirical open-world verification lacks such fragments and, for open-world questions, caps at anchored correctness (L2).

Which axes does the paper argue are orthogonal to VAL?

The paper argues that granularity, concept hierarchy, risk, and system stack are orthogonal to VAL. This resolves a systematic conflation across multiple surveyed papers by separating “what to check” from “what the verdict is anchored to.”

Sep 15, 2026

Uploaded by Theodore