Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning - Token-and Turn-level Policy Optimization (T2PO)

Oct 12, 2026

Upload by Fahsai

Multi-turn reinforcement learning has improved the reasoning ability of large language models on complex interactive tasks, yet training instability and collapse remain common. Existing stabilization methods often fail because the root cause lies in inefficient exploration: policies repeatedly produce low-information actions that do not reduce uncertainty or progress the task. This work proposes Token-and Turn-level Policy Optimization (T2PO) to explicitly control exploration using uncertainty dynamics at both token and turn granularities. Experiments on WebShop, ALFWorld, and Search QA show higher training stability and improved performance through uncertainty-aware resampling and thinking interventions.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages25

Reading Time9 min

AccessFree to download

Abstract

Introduction

Motivation and challenges in multi-turn RL

Prior stabilization approaches and limitations

Key insight: hesitation as insufficient exploration

Proposed method: uncertainty-guided token- and turn-level control

T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning Haixin Wang 1 Hejie Cui 2 * Chenwei Zhang 2 Xin Liu 2 Shuowei Jin 2 Shijie Geng 2 Xinyang Zhang 2 Nasser Zalmout 2 Zhenyu Shi 2 Yizhou Sun 1 arXiv :2605 .02 178v 1 [ cs .AI] 4 May 2026 Abstract Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs’ performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token-and Turn-level Policy Optimization (T2PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T2PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T2PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T2PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at: [https://github.com/](https://github.com/)[ ](https://github.com/)WillDreamer/T2PO. 1. Introduction Recent advances in self-evolving agents are deeply rooted in multi-turn reinforcement learning (RL) (Liu et al., 2024 ; Team, 2025 ; Team et al., 2025 ; Wang et al., 2026), which Part of this work was done during an internship at Amazon. 1University of California, Los Angeles 2Amazon.com Inc. Correspondence to: Hejie Cui∗ <[cuihejie331771@gmail.com](cuihejie331771@gmail.com) > . Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . provides the foundational mechanism for training agents to reason, act, and self-evolve through iterative interactions with the environments. Despite this progress, the community still lacks a stable and scalable training paradigm. Current multi-turn RL pipelines face intertwined challenges in both effectiveness and efficiency. On the one hand, longhorizon interactions combined with sparse reward signals make credit assignment inherently difficult (Zhou et al., 2024 ; Wang et al., 2025b) . On the other hand, rollout collection is computationally expensive, driving the adoption of acceleration techniques such as low-precision inference (Liu et al., 2025a) and asynchronous sampling (Fu et al., 2025a) . Yet these efficiency-oriented solutions inevitably introduce off-policy drift and stale policy effects (Zheng et al., 2025a) . Both of these issues tend to amplify training instability and frequently lead to the notorious training collapse. To mitigate training instability, prior work has explored a variety of strategies, including fine-grained credit assignment (Feng et al., 2025), internal or process-based reward modeling (Wang et al., 2025a ; Dong et al., 2025), and trajectory-level filtering of failed interactions (Yu et al., 2025 ; Xue et al., 2025) . These approaches aim to provide denser learning signals or remove void rollouts, and have shown partial success in stabilizing optimization. However, most existing solutions operate either at a coarse trajectory level or through implicit control via reward shaping. In the inherently complex multi-turn setting, such coarse or indirect interventions make the training dynamics highly sensitive to hyperparameters and rollout distributions. As a result, they often lead to training collapse, the phenomenon characterized by rapidly degrading performanc

FAQs

How does T2PO address hesitation at the turn level?

T2PO detects turns with negligible exploration progress by analyzing uncertainty patterns and dynamically resamples those turns to avoid wasted rollouts.

Oct 12, 2026

Upload by Fahsai