[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84237-en":3,"doc-seo-84237-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84237,13056703019662,"Evangeline","https://ap-avatar.wpscdn.com/avatar/be000253a8e92610077?_k=1778726343310543188",8,"Research & Report","Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning","Reinforcement learning for post-training large language models faces inefficiency in long-horizon agentic tasks under synchronous, batch-interleaved pipelines. Asynchronous RL improves utilization by updating as rollouts arrive, yet common approaches like GRPO struggle with training stability and task effectiveness because group-wise sampling induces latency-driven off-policy behavior. The paper proposes Single-rollout Asynchronous Optimization (SAO), replacing group-wise sampling with one rollout per prompt, adding practical value-model training and strict double-side token-level clipping. SAO trains stably for 1,000 steps and outperforms GRPO variants on agentic coding and reasoning benchmarks, and adapts effectively in simulated online learning when deployed for open GLM-5.2 training.","arXiv :2607 .07508v 1 [ cs .LG] 8 Jul 2026  \nSingle-Rollout Asynchronous Optimization for Agentic  \nReinforcement Learning  \nZhenyu Hou∗ Yujiang Li∗ Jie Tang Yuxiao Dong  \nTsinghua University  \nAbstract  \nReinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs) . Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B) .  \n Baseline  GRPO  SAO (ours)  \nFigure 1: The performance of SAO on reasoning and coding benchmarks. The four reasoning benchmarks are evaluated in a reasoning-with-Python-tool setting, where the baseline is the Qwen3- 30B-A3B SFT model; SWE-Bench Verified evaluates coding with the Qwen3-30B-A3B baseline. SAO outperforms the corresponding baseline and GRPO across all five benchmarks.  \n∗Equal Contribution. Work done while ZH and YL interned at Z.AI.  \nUnder review, Feb 2026 .  \n1 Introduction  \nLarge Language Model (LLM) development is shifting from supervised pre-training toward posttraining reinforcement learning (RL) . Recent work in Reinforcement Learning has demonstrated that scaling RL compute together with test-time compute is a highly effective way to improve model intelligence [DeepSeek-AI, 2024a, OpenAI, 2024, Cobbe et al., 2021, Lightman et al., 2023] . Most LLM RL pipelines remain synchronous and interleaved: the policy generates a batch of rollouts, and optimization starts only after the entire batch is collected [Ouyang et al., 2022, Rafailov et al., 2024] .  \nFor agentic and coding workloads, rollout lengths are highly variable, so short trajectories finish quickly while long ones become stragglers; as a result, large portions of the GPU cluster idle while waiting for the slowest rollouts [DeepSeek-AI, 2024b, Kwon et al., 2023, Yu et al., 2022] . Asynchronous RL mitigates this imbalanced generation overhead by consuming rollouts continuously as they arrive, improving utilization and wall-clock efficiency [Mnih et al., 2016, Liang et al., 2018, Hoffman et al., 2020] .  \nHowever, asynchrony introduces two challenges. First, each trajectory can be generated by multiple versions of the old rollout model, which leads to more unpredictable and severe off-policy, and thus harms the training stability. Previous works [Fu et al., 2025, Noukhovitch et al., 2024] make attempts for asynchronous RL but mainly focus on efficiency optimization rather than effectiveness. Second, group-wise methods such as GRPO [Shao et al., 2024, Wang et al., 2022] are mismatched to asynchronous training. GRPO samples a group of response","cbCainb6tF0UapR7","https://ap.wps.com/l/cbCainb6tF0UapR7","pdf",664828,1,14,"English","en",105,"# Introduction\n## Problem: inefficiency and off-policy issues in asynchronous RL\n## Proposed method: Single-rollout Asynchronous Optimization (SAO)\n## Key techniques: clipping, value-model training, skip-observation GAE\n## Evaluation and benchmark results","[{\"question\":\"What problem does SAO address in asynchronous agentic reinforcement learning?\",\"answer\":\"SAO targets training instability and off-policy effects that arise when asynchronous rollouts are generated under policy lag, especially when group-wise methods like GRPO are used for advantage estimation.\"},{\"question\":\"How does SAO reduce off-policy effects compared with GRPO?\",\"answer\":\"SAO replaces group-wise sampling with single-rollout sampling, using one rollout per prompt so training does not depend on a group waiting for the slowest trajectory.\"},{\"question\":\"What mechanisms does SAO use to improve optimization stability?\",\"answer\":\"SAO introduces strict double-side token-level clipping and masking, using token-level importance sampling from rollout log-probabilities to stabilize updates under varied policy lag.\"}]",1784194270,35,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"single-rollout-asynchronous-optimization-for-agentic-reinforcement-learning","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/single-rollout-asynchronous-optimization-for-agentic-reinforcement-learning/84237/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does SAO address in asynchronous agentic reinforcement learning?","Question",{"text":75,"@type":76},"SAO targets training instability and off-policy effects that arise when asynchronous rollouts are generated under policy lag, especially when group-wise methods like GRPO are used for advantage estimation.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does SAO reduce off-policy effects compared with GRPO?",{"text":80,"@type":76},"SAO replaces group-wise sampling with single-rollout sampling, using one rollout per prompt so training does not depend on a group waiting for the slowest trajectory.",{"name":82,"@type":73,"acceptedAnswer":83},"What mechanisms does SAO use to improve optimization stability?",{"text":84,"@type":76},"SAO introduces strict double-side token-level clipping and masking, using token-level importance sampling from rollout log-probabilities to stabilize updates under varied policy lag.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]