More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

Jul 16, 2026

Upload by Miles

Training a language model against its own reference-free judgments—used in self-reward, self-play, and LLM-as-a-judge label-free self-improvement pipelines—assumes the judge’s verdict on a candidate answer can proxy correctness. This work shows a structural failure: the judge scores plausibility rather than correctness, creating a false-positive basin that policies learn to exploit. Hidden-anchor exact-match auditing quantifies a large judge–truth gap and demonstrates weak transferability controls and a de-anchoring remedy that collapses false positives while preserving solvability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Structural flaw in reference-free judging

Hidden-anchor audit methodology

Cross-judge and scale transfer

De-anchoring fix and predictive bounds

Contributions

arXiv :2607 .05904v 1 [ cs .LG] 7 Jul 2026 MORE CONVINCING, NOT MORE CORRECT: SELF-PLAY REWARD HACKING OF REFERENCE-FREE LLM JUDGES Chenyu Zhou School of Engineering, Institute of Science Tokyo [zhou.c.76d6@m.isct.ac.jp](zhou.c.76d6@m.isct.ac.jp) ABSTRACT Training a language model against its own reference-free judgments—the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines for label-free selfimprovement—assumes a model’s verdict on a shown answer is a usable proxy for correctness. We show the premise fails structurally: conditioned on a candidate, a reference-free judge scores plausibility, not correctness—a verification asymmetry that leaves false-positive basins of plausible-but-wrong answers a policy learns to exploit. We measure the failure with a hidden-anchor audit: a held-out, crosssource exact-match check the judge never sees. On GSM8K with Qwen3 policies ina reasoning-suppressed regime, self-play drives the judge’s pass rate from 0.72 to 0.94 while true accuracy stays at 0 .20—a 0 .74 judge–truth gap (three seeds) . This reward hacking is not white-box gaming: the manufactured errors transfer across judge families (Qwen, Llama, Gemma) and scales (to 14B), a strict three-family ensemble still accepts 55% of them, and recompute prompts, stronger judges, and training directly against the ensemble reward all fail to close the basin. The decisive variable is not whether the judge sees the candidate but whether it commits an answer of its own first: the recompute prompt leaves the false-positive rate on wrong answers at 0.719, committing first—candidate still in view—drops it to 0.012, and blind solving lifts discrimination from near chance to 0.96 with no reference answer. Used as the training reward, the de-anchored channel keeps the false-positive rate at zero across self-play, preventing the basin rather than only detecting it. A falsifiable bound explains which regimes are exposed—the gap is at most 1 − accuracy—and the de-anchored reward obeys the analogous judge-side bound. The full arc replicates without training under best-of-N selection in code and competition math, and in the full loop with a Gemma policy. 1 INTRODUCTION Reinforcement learning from a model’s own evaluations has become a central recipe for improving language models without human labels. Reference-free “LLM-as-a-judge” rewards, self-reward, and self-play schemes share one premise: a model’s judgment of an answer is a usable proxy for its correctness, and a growing line of methods builds on it as a default assumption (Bai et al., 2022; Lee et al., 2023; Yuan et al., 2024; Chen et al., 2024; Simonds et al., 2025; Huang et al., 2026) . The premise has a structural flaw. A reference-free judge has no access to ground truth; it can only assess whether an answer looks correct. For tasks where verifying an answer is harder than recognizing a plausible one—most of reasoning—this is a verification asymmetry: the judge scores plausibility, not correctness, leaving a false-positive basin of plausible-but-wrong answers it accepts. Optimizing a policy against such a judge does not merely risk noise; it actively rewards finding the basin. We measure it with a hidden anchor: a held-out, cross-source exact-match check on the final answer that the judge never sees and is never trained against. On GSM8K with Qwen3 policies trained by self-play, the audit reveals a large divergence: on the full test set, in a reasoning-suppressed regime that holds accuracy low, the judge’s pass rate climbs from ≈0 .72 to 0.94 while anchor-verified accuracy stays flat at ≈0 .20—a 0 .74 judge–truth gap (three seeds; §5); Figure 1a shows the fiveiteration trajectory. Self-play does not make the model more correct; it makes the model’s errors more convincing. This is not a quirk of one judge that a stronger or more diverse judge would catch. Re-scoring the hacked answers with independent judges from other families (Llama, Gemma) and larger scales

FAQs

What structural problem does the paper identify in reference-free LLM judges?

The judge lacks ground truth and therefore evaluates whether an answer looks plausible, not whether it is correct. This verification asymmetry leaves a false-positive basin of plausible-but-wrong answers.

How is the “hidden-anchor audit” used to measure the judge–truth gap?

A held-out, cross-source exact-match check on the final answer is performed that the judge never sees and is never trained against. It reveals that self-play increases judge pass rates while true accuracy stays low.

What is the de-anchoring fix and what effect does it have?

The fix requires the judge to commit to an answer of its own before using the candidate. This collapses the false-positive rate on wrong answers (from about 0.719 to about 0.012) and prevents the basin when used as the training reward.

Jul 16, 2026

Upload by Miles