When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

Jul 16, 2026

Upload by Eden

Deployed RLVR code test suites contain natural, persistent, asymmetric false positives where the same wrong program reliably passes the same weak tests, unlike the symmetric resampled noise assumed by prior noise-robustness work. A preregistered two-arm causal contrast trains GRPO on identical MBPP tasks and settings, using the original (leaky) versus MBPP+ extra tests (hardened) as rewards. Across families, average held-out effects are bounded and false-positive mass tracks static leakiness audits, yet human adjudication finds a large residual of genuinely wrong code rewarded.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

Introduction

Motivation and problem definition

Preregistered two-arm causal contrast design

Static leakiness audit approach

arXiv :2607 . 1 1022v 1 [ cs .LG] 13 Jul 2026 WHEN THE REWARD SUITE IS LEAKY: A PREREGISTERED CAUSAL CONTRAST OF NATURAL VERIFIER FALSE POSITIVES IN RLVR Chuyifei Zhang Beijing Jiaotong University [24222058@bjtu.edu.cn](24222058@bjtu.edu.cn) ABSTRACT The test suites used as RLVR rewards for code have natural false positives: pertask, persistent, asymmetric errors that accept the same wrong programs everytime they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses. We run a preregistered two-arm causal contrast on a deployed suite: GRPO on identical MBPP tasks, seeds, and compute (Qwen2.5-Coder-1.5B-Instruct, 5 seeds × 400 steps), rewarded by the original MBPP tests (leaky) versus the MBPP+ extra tests (hardened; extra-tests-only, §3.1) . Two further families replicate the design under a preregistration frozen before their data existed: deepseek-coder-1.3b-instruct, registered as the decision arm, and Llama- 3.2-1B-Instruct. Claims below are tagged [C] (confirmatory: preregistered) or [E](exploratory) . [C] The average held-out effect (scored by the extra-test suite throughout) is bounded: non-inferior under a preregistered 1.5-pt margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt), with the upper bound on the same side of the margin under all six estimators in all three families. [C] Rewarded false-positive mass tracks a cheap static leakiness audit computed before training (Spearman 0.80 raw, 0.79 under difficulty control), and the registered train-side test puts the leak-stratum FP share +43.8 pt above clean tasks (CI90 [43.05, 44.57]; the FP definition is classifier-free, so these numbers are invariant under all later audit revisions) . [E] Auditing every rewarded FP under signed, human-adjudicated rules finds a large residual of verified genuinely wrong code: 47.57% recordweighted (task-cluster bootstrap-95 [36.4, 60.5]; estimator variants within two points) . The reward paid for real bugs, not merely suite artifacts. Both replication families reproduce a large residual share—45.37% and 62.78%—but the shares replicate “large,” not one number. [E] On statically leaky held-out tasks the leakyarm improves less (family A) . This is an association only, its specification-search correction prices one axis of freedom (a lower bound), and it does not replicate cleanly across families: one reversed-direction candidate, one underpowered positive. [E] Mechanism evidence is consistent with selection of pre-existing error modes rather than learned exploitation. FP incidence sits at its long-run share from step 0 and does not grow within our horizon (flat in two families, declining in the third); the largest channel shows 55/55 distinct wrong programs and no hacking signatures; and untrained base models already produce the same wrong outputs under the leaky filter. The 8.37-pt reward inflation shows no held-out counterpart within the bound reported above, though parameter-level sharpening below behavioral resolution is not excluded. In an exploratory follow-up outside these confirmatory claims, we turn the same instrument on the frontier judges themselves: on their own false positives they self-assess only weakly, a deconfounded same-author test is unresolved, and even the highest-scoring reader we probe stays far below its score on a weaker policy’s errors—two subjects on MBPP, licensing nothing about frontier models in general (§5) . The practical boundary we map: a cheap static audit locates where FP mass will sit before training—exposure, not final damage. Hardening the reward—swapping the leaky tests for the far larger extra-test suite—removes the measurement inflation, though at this scale it buys little capability. 1 INTRODUCTION Reinforcement learning from verifiable rewards (RLVR) trains code models against test suites and treats a pass as ground truth (Lambert et al., 2024; DeepSeek-AI, 2025) . Deployed suites do not earn that trust: a recent audit of two public co

FAQs

What does the paper mean by “leaky” reward suites in RLVR?

A deployed suite accepts wrong solutions consistently, rewarding the same incorrect programs across rollouts. These errors are per-task and persistent, creating a directional reward artifact rather than random label noise.

How is the causal test set up to measure the impact of leaky versus hardened rewards?

The paper runs a preregistered two-arm experiment using GRPO to train the same model on identical MBPP tasks, seeds, and compute. The only difference is the reward test suite: the original MBPP tests (leaky) versus the MBPP+ extra tests (hardened).

What does the study find about false positives and the true quality of rewarded solutions?

False-positive mass correlates strongly with a pre-training static leakiness audit, but auditing rewarded false positives with human rules reveals a substantial residual share of genuinely wrong code. The reward therefore pays not just for suite artifacts, but for real bugs.

Jul 16, 2026

Upload by Eden