[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82587-en":3,"doc-seo-82587-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82587,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents","Large language model (LLM) agents are widely deployed as assistants, chatbots, and research helpers, yet they remain static once released. Model weights, system prompts, tool sets, and in-context mechanisms are fixed at deployment, so improvements rely on manual cycles of data collection, offline fine-tuning, paradigm edits, and redeployment. This work argues that enterprise-scale self-evolving agents are blocked less by RL algorithms than by missing online RL system foundations: trajectory protocols, enterprise learning substrates, and unified evolution control planes. The paper proposes an agent-service architecture and demonstrates AREAL2.0 for online policy weight updates from deployed workloads.","arXiv :2607 .0 1 120v2 [ cs .DC] 2 Jul 2026  \nNext-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents  \nRan Yan1 ,2 , Wei Fu1 ,3 , Jiale Li1 , Shusheng Xu1 , Zhiyu Mei1 , Jiaxuan Gao1 ,3 , Jiarui Zhang1 ,2 ,3 , Wentai Zhang1 , Hao Dai1 , Xujie Shen1 , Chuyi He1 , Zhen Pu1 , Jun Mei1 , Zhiyao Lin1 , Haitao Wang1 , Zhiqiang Ding1 , Jiawei Zhang1 , Huaijie Wang1 ,3 , Ruida Xu1 , Honghua Dong1 , Youhe Jiang2 , Yi Wu3 , Tongkai Yang1 , Binhang Yuan1 ,2  \n1 Ant Group, 2 HKUST, 3 Tsinghua University  \nAbstract  \nLarge language model (LLM) agents are rapidly being deployed in production, including coding assistants, customer-support chatbots, and scientific research assistants, yet they remain fundamentally static in large-scale enterprise-level deployment. The LLM weights, system prompts, tool repertoires, and in-context harnesses are frozen at deployment time, and any improvement requires a manual loop of human-curated data collection, offline fine-tuning, modification of the agentic paradigm, andre-deployment. Recent work on self-evolving agents, such as OpenClaw for individual users, indicates that the next leap in agent capability will come from agents that continually learn from their own experience. In this paper, we argue that this vision for self-evolving agent deployment is being held back for enterprise-level large-scale agentic service not by reinforcement learning (RL) algorithms but by agentic online RL systems. Specifically, we argue that current agentic RL systems and the surrounding observability software stack are inadequate along three essential aspects: (i) there is no standardized agent trajectory data protocol capable of carrying RL learning signals at step granularity across heterogeneous agent paradigms; (ii) there is no enterprise-grade comprehensive data proxy that converts real workloads into governed learning substrates; and (iii) there is no unified agent evolution control plane with automatic triggering that decides, on the basis of trajectory statistics, when to update policy model weights or evolve the in-context harness by the corresponding RL algorithm. We argue that the next generation of agentic RL systems must be co-designed around these three pillars, and we sketch concrete architectural commitments, case studies, and counter-arguments to substantiate our position. We further instantiate one scoped branch of our vision through AREAL2.0, showing how existing RL infrastructure can be reorganized into an agent service-oriented online RL loop for policy LLM weight updates from deployed online agent workloads.  \n§ Code: [https://github.com/areal-project/AReaL](https://github.com/areal-project/AReaL)  \n1 Introduction  \nLarge language model (LLM) agents are moving from laboratory demonstrations into deployed systems for research assistance, software engineering, and task automation. In fact, such LLM agents should essentially change the unit of deployment for LLM services at enterprise level—the deployed object should no longer be only a LLM that maps a sequence of input tokens to a sequence of output tokens; instead it should increasingly be a long-horizon agent policy embedded in a heterogeneous complicated bussiness environment, where the agent reads files, calls tools, retrieves documents, invokes APIs, updates memory, requests human approval, and accomplishes complicated analytic jobs. This shift creates a mismatch between how agents are deployed and how agents are improved: A deployed enterprise agent may perform thousands or millions of tool-mediated interactions, yet its future behavior usually changes only through a manual engineering cycle: operators inspect traces, define new evaluation benchmarks, edit prompts, add tool descriptions, update a LLM through reinforcement learning (RL), implement a new agentic loop, and redeploy. On the other hand, we have recently  \nwitnessed that self-evolving agents are no longer hypothetical—personal agent systems, such as OpenClaw ","cbCaii0swWjtqdwI","https://ap.wps.com/l/cbCaii0swWjtqdwI","pdf",1547873,1,13,"English","en",105,"# Introduction\n## Self-evolving agents in enterprise deployment\n## Gaps in current agentic RL systems\n# Proposed next-generation RL system pillars\n## Architecture commitments and case studies\n## AREAL2.0 online RL loop","[{\"question\":\"为什么当前的 LLM 智能体在企业部署后难以持续进化？\",\"answer\":\"部署时 LLM 权重、系统提示词、工具集合和上下文机制被冻结。后续改进通常需要依赖人工数据收集、离线微调、范式调整与重新部署。\"},{\"question\":\"论文认为制约“自进化智能体”落地的关键不在 RL 算法本身，原因是什么？\",\"answer\":\"作者指出问题主要来自在线智能体强化学习系统与观测/软件栈的不足，而不是强化学习算法缺陷。\"},{\"question\":\"论文提出的下一代 agentic RL 系统需要补齐哪些核心能力？\",\"answer\":\"需要三大支柱：可标准化、步级粒度承载学习信号的轨迹数据协议；企业级数据代理以将真实工作负载转为可治理的学习底座；以及基于轨迹统计自动触发的统一演化控制平面，用于决定何时更新策略权重或演化上下文/工具。\"}]",1784181674,33,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"next-generation-agentic-reinforcement-learning-systems-enable-self-evolving-agents","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/next-generation-agentic-reinforcement-learning-systems-enable-self-evolving-agents/82587/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么当前的 LLM 智能体在企业部署后难以持续进化？","Question",{"text":75,"@type":76},"部署时 LLM 权重、系统提示词、工具集合和上下文机制被冻结。后续改进通常需要依赖人工数据收集、离线微调、范式调整与重新部署。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"论文认为制约“自进化智能体”落地的关键不在 RL 算法本身，原因是什么？",{"text":80,"@type":76},"作者指出问题主要来自在线智能体强化学习系统与观测/软件栈的不足，而不是强化学习算法缺陷。",{"name":82,"@type":73,"acceptedAnswer":83},"论文提出的下一代 agentic RL 系统需要补齐哪些核心能力？",{"text":84,"@type":76},"需要三大支柱：可标准化、步级粒度承载学习信号的轨迹数据协议；企业级数据代理以将真实工作负载转为可治理的学习底座；以及基于轨迹统计自动触发的统一演化控制平面，用于决定何时更新策略权重或演化上下文/工具。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]