When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

Jul 16, 2026

Reinforcement learning (RL) boosts large language model (LLM) reasoning, yet critic-free RL methods with uniform credit assignment can reinforce flawed token-level behavior. The paper identifies “Positive-Credit Contamination,” where contextually erroneous tail tokens receive the same positive credit as plausible ones, biasing policy toward poorly calibrated reasoning over time. It introduces Tail-Aware Credit calibration On (TACO), estimating per-token tail risk from local generation context and calibrating positive credit to dampen unreliable updates while allowing recurring useful rare patterns to accumulate reinforcement. Experiments on three LLMs and eight benchmarks show consistent gains over GRPO-style baselines and improved stability for long-horizon RL, with code released publicly.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Positive-Credit Contamination in critic-free RL

Limitations of existing token-level credit methods

Tail-Aware Credit Calibration (TACO) overview

arXiv :2607 .07976v 1 [ cs .CL] 8 Jul 2026 When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning Xiuyi Lou 1 ∗, Zicheng Xu 1 Yu-Neng Chuang2 , Hoang Anh Duy Le2 , Zhaozhuo Xu3 , Guanchu Wang4 , Vladimir Braverman 1† 1Johns Hopkins University, 2Rice University, 3Workato, 4University of North Carolina at Charlotte Abstract Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs) . However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: lowprobability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token’s risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertaintydriven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: [https://github.com/xiuyilou/TACO](https://github.com/xiuyilou/TACO). 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a common post-training paradigm for reasoning-oriented large language models (LLMs) . Recent models such as OpenAI’s o-series and DeepSeek-R1 demonstrate that large-scale RL post-training can substantially improve performance on automatically verifiable tasks, including mathematics and programming [14, 2] . Early RLVR methods often build on Proximal Policy Optimization (PPO), which typically relies on a learned critic for advantage estimation [18, 15] . In contrast, Group Relative Policy Optimization (GRPO) replaces the learned critic with group-relative advantages, reducing value-modeling overhead while achieving strong empirical performance [19] . The simple yet effective design has made GRPO a widely adopted backbone for reasoning RL, motivating variants such as Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) and Group Sequence Policy Optimization (GSPO) [34, 38] . However, this simplification computes a single completion-level advantage and broadcasts it uniformly to every generated token, even though not every token in a rewarded completion is equally reliable. Specifically, tokens from the low-probability tail of the policy distribution, which are unlikely under the generation context, can be sampled in rewarded completions even when they are semantically ∗ Equal contribution. †Correspondence to Vladimir Braverman. Preprint. Qwen3-1.7B-Base Qwen3-4B-Base Qwen2.5-Math-7B Figure 1: TACO consistently improves over GRPO across representative benchmarks and models. irrelevant or erroneous; we refer to such locally unreliable tokens as implausible tail tokens. Yet, these tokens’ local effects may be bypassed or overlooked by the overall correct reasoning process and final answer [22, 10, 6] . These tokens then receive the same positive credit as reliable ones, resulting in indiscriminate reinforcement of both well-formed reasoning behavior and locally flawed continuations. Consequently, the accumulation of such updates

FAQs

What problem does the paper identify in critic-free RL methods for LLMs?

It identifies “Positive-Credit Contamination,” where low-probability tail tokens that are locally unreliable receive the same positive credit as plausible tokens, leading to indiscriminate reinforcement of flawed reasoning behavior.

How does TACO determine whether a token is unreliable?

TACO computes a tail-risk score for each token using its local generation context, distinguishing unexpected rarity from exploration driven by uncertainty.

How does TACO change training compared with GRPO-style uniform credit assignment?

TACO calibrates the positive credit based on tail risk: it softly suppresses positive updates for high-risk tokens while preserving full reinforcement for low-risk tokens, improving stability and performance on long-horizon RL benchmarks.

Jul 16, 2026

Upload by Genevieve