[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327998-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327998-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","rspo-reward-swap-policy-optimization-for-multi-turn-llm-agents","RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents","","Reinforcement learning can fine-tune large language models for multi-turn interactive tasks, yet long-horizon settings with sparse outcome rewards often converge slowly because signals are limited and feedback is not granular. Directly optimizing with outcome rewards may also miss successful trajectories that are not sampled, restricting performance. Using dense process rewards accelerates convergence, but surrogate signals can misalign with true outcome rewards, biasing learning. This work proposes Reward-Swap Policy Optimization (RSPO) to leverage dense rewards while maintaining objective consistency via a reward-swap mechanism, verified through extensive experiments on WebShop and ALFWorld across GRPO, PPO, and GiGPO.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/rspo-reward-swap-policy-optimization-for-multi-turn-llm-agents/327998/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/rspo-reward-swap-policy-optimization-for-multi-turn-llm-agents/327998.png","ImageObject",300,407,{"name":42,"@type":43},"River Wang","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-30","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",14,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why does training with only outcome rewards perform poorly in multi-turn tasks?","Question",{"text":63,"@type":64},"Outcome-only rewards are sparse and provide no fine-grained feedback, causing slow convergence. The model may also fail to learn successful trajectories that are not sampled during training.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What challenge arises when using dense process rewards?",{"text":68,"@type":64},"Dense rewards can be misaligned with ground-truth outcome rewards. This inconsistency can bias the training direction and reduce final performance.",{"name":70,"@type":61,"acceptedAnswer":71},"How does RSPO improve training while addressing reward misalignment?",{"text":72,"@type":64},"RSPO uses dense process rewards to enable exploration beyond the behavioral space constrained by outcome rewards, then employs a reward-swap mechanism to ensure consistency between the optimization objective and true outcome rewards.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327998,1790121643,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},1099514067438,"https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542","arXiv :2607 .047 13v 1 [ cs .LG] 6 Jul 2026  \nRSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents  \nQiang Liu∗ Taian Guo∗ Ruizhi Qiao† Xing Sun  \nTencent YouTu Lab  \nAbstract  \nReinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training withoutcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model’s final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.  \n1 Introduction  \nIn recent years, Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across various domains, achieving breakthrough progress particularly in complex tasks such as mathematical problem solving, code generation, and logical reasoning [1–4] . To further elicit the deep reasoning capabilities of these models, Reinforcement Learning (RL) has been widely employed to fine-tune LLMs, optimizing the model policy by maximizing the expected reward of the generated content [5, 6] . Within this mainstream paradigm, researchers typically model reasoning tasks as a single-turn interaction process: the model receives a specific question or instruction as input and generates a Chain-of-Thought (CoT) or final answer in a single pass, after which the environment provides reward feedback based on the quality of the output [7, 8] .  \nHowever, the aforementioned tasks are predominantly single-turn. In contrast, LLM agents frequently encounter multi-turn tasks characterized by outcome-only rewards [9] . When addressing such tasks, directly employing outcome rewards for reinforcement learning often leads to slow convergence due to the sparsity of reward signals and the absence of fine-grained feedback. Moreover, for algorithms  \n∗ Equal Contribution  \n† [Correspondence: Ruizhi.Qiao@tencent.com](Correspondence: Ruizhi.Qiao@tencent.com)  \nPreprint.  \nBehavioral Space Guided by Outcome Rewards  \nUse Outcome Rewards  \nUse Dense Process Rewards  \nUse Dense Process Rewards  \nFigure 1: Schematic illustration of the difference between RSPO and baselines. When training withoutcome rewards, the trajectories sampled by the model are confined within the behavioral space guided by outcome rewards. In contrast, when training with dense rewards, the interaction-generated trajectories can break out of this space, yielding more diverse trajectories to facilitate model training.  \nlike GRPO [10], the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance.  \nTo mitigate these challenges, several approaches, such as [11, 12], utilize manually crafted dense process rewards to alleviate the reward sparsity problem. By assigning an intermediate reward to ","cbCaipCUkG4rTc9A","https://ap.wps.com/l/cbCaipCUkG4rTc9A","pdf",801139,15,"English","# Introduction\n## Problem: outcome-only rewards in multi-turn tasks\n## Proposed method: Reward-Swap Policy Optimization (RSPO)\n## Experimental evaluation and results","[{\"question\":\"Why does training with only outcome rewards perform poorly in multi-turn tasks?\",\"answer\":\"Outcome-only rewards are sparse and provide no fine-grained feedback, causing slow convergence. The model may also fail to learn successful trajectories that are not sampled during training.\"},{\"question\":\"What challenge arises when using dense process rewards?\",\"answer\":\"Dense rewards can be misaligned with ground-truth outcome rewards. This inconsistency can bias the training direction and reduce final performance.\"},{\"question\":\"How does RSPO improve training while addressing reward misalignment?\",\"answer\":\"RSPO uses dense process rewards to enable exploration beyond the behavioral space constrained by outcome rewards, then employs a reward-swap mechanism to ensure consistency between the optimization objective and true outcome rewards.\"}]","RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents | PDF",1789978463,38]