[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85832-en":3,"doc-seo-85832-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85832,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization","Reinforcement learning (RL) has become a key approach to improve large language model (LLM) reasoning, yet PPO-Clip suffers from exploration collapse during training. The work identifies the root cause: PPO-Clip implicitly uses a Euclidean metric to measure policy discrepancy, which conflicts with the intrinsic Riemannian geometry on the policy manifold. This mismatch drives overly conservative updates in low-probability regions and overly aggressive ones in high-probability regions, collapsing exploration. It introduces Riemannian Isometric Policy Optimization (RIPO), ensuring isometric policy updates on the manifold, stabilizing the bias–variance trade-off, and improving performance on seven benchmarks.","Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via  \nRiemannian Isometric Policy Optimization  \nZhicheng Cai 1 2 3 Xinyuan Guo 1 3 Hanlin Wu 1 2 3 Mingxuan Wang 2 3 Wei-Ying Ma 1 3 Ya-Qin Zhang 1 3  \nHao Zhou 1 3 *  \narXiv :2607 . 10 169v 1 [ cs .LG] 11 Jul 2026  \nAbstract  \nReinforcement learning (RL) has become a dominant paradigm for enhancing LLMs’ reasoning capabilities. However, RL algorithms with PPOClip are inherently limited by exploration collapse.  \nSubsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip’s failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable biasvariance trade-off, which stabilizes optimization.  \nExtensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24) .  \n1. Introduction  \nReinforcement learning (RL) has emerged as a central paradigm for enhancing the reasoning capabilities of large language models (LLMs) (Guo et al., 2025 ; Comanici et al., 2025 ; OpenAI, 2024) . It has demonstrated remarkable success in domains requiring long-horizon decision-making, such as mathematical reasoning (Shao et al., 2024 ; Wang  \n1Institute for AI Industry Research (AIR), Tsinghua University 2ByteDance Seed 3 SIA-Lab of Tsinghua AIR and ByteDance Seed. Correspondence to: Hao Zhou \u003C[zhouhao@air.tsinghua.edu.cn](zhouhao@air.tsinghua.edu.cn) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \net al., 2024 ; Luo et al., 2025), and so on (OpenAI, 2025 ; Jin et al., 2025 ; Dou et al., 2024 ; Anthropic, 2024) .  \nDespite these advances, the PPO-Clip (Schulman et al., 2017) used in modern LLM RL algorithms (Guo et al., 2025) incurs a critical issue: exploration collapse (Yu et al., 2025) . Specifically, PPO-Clip makes the policy rapidly concentrates on a narrow set of high-probability actions during training, thus rare but vital actions are unexplored, severely suppressing model performance. This is especially devastating in long-horizon reasoning tasks, where the stateaction space is extremely vast, thus extensive exploration is indispensable to discover a feasible solution. Ultimately, exploration collapse hinders further scaling progress of RL, stifling the potential in increasingly challenging tasks.  \nSubsequent works (Yang et al., 2025b ; Zheng et al., 2025) provide symptomatic fixes without identifying the root cause of PPO-Clip’s failure. For example, DAPO (Yu et al., 2025) raises the clipping boundary to encourage exploration. While yielding empirical gains, they remain heuristic within the PPO-Clip framework, offering limited theoretical understanding and leaving exploration collapse unresolved.  \nIn this work, we discover the fundamental flaw of PPO-Clip lies in the geometric mismatch between Euclidean metric and Riemannian manifold. Specifically, PPO-Clip implicitly measures policy discrepancy using a Euclidean metric on the importance ratio, treating equal ratio deviations as equal policy changes. However, the intrinsic discrepancy between policies is governed by the Kullback-Leibler (KL) divergence (Schulman et al., 2015), which induces a Riemannian geometry on the statistical manifold of policies. This geometric mismatch leads to patholog","cbCaivYoIUAqGFTp","https://ap.wps.com/l/cbCaivYoIUAqGFTp","pdf",1870896,6,1,11,"English","en",105,"# Abstract\n# Introduction\n# Preliminary\n## Trust Region Policy Optimization","[{\"question\":\"What problem does the document address in LLM reinforcement learning?\",\"answer\":\"It addresses exploration collapse caused by PPO-Clip in modern LLM RL algorithms, where training rapidly concentrates probability mass and fails to explore rare but important actions.\"},{\"question\":\"Why does PPO-Clip fail according to the paper?\",\"answer\":\"The paper shows a geometric mismatch: PPO-Clip measures policy discrepancy with an implicit Euclidean metric, which is inconsistent with the Riemannian geometry induced by KL divergence on the policy manifold.\"},{\"question\":\"What is RIPO and how does it fix the issue?\",\"answer\":\"Riemannian Isometric Policy Optimization (RIPO) enforces isometric policy updates on the Riemannian manifold by adapting the clipping boundary to local manifold geometry, balancing exploration and exploitation and improving stability via a favorable bias–variance trade-off.\"}]",1784206575,28,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"beyond-euclidean-clipping-overcoming-exploration-collapse-in-llm-rl-via-riemannian-isometric-policy-optimization","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/beyond-euclidean-clipping-overcoming-exploration-collapse-in-llm-rl-via-riemannian-isometric-policy-optimization/85832/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does the document address in LLM reinforcement learning?","Question",{"text":76,"@type":77},"It addresses exploration collapse caused by PPO-Clip in modern LLM RL algorithms, where training rapidly concentrates probability mass and fails to explore rare but important actions.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"Why does PPO-Clip fail according to the paper?",{"text":81,"@type":77},"The paper shows a geometric mismatch: PPO-Clip measures policy discrepancy with an implicit Euclidean metric, which is inconsistent with the Riemannian geometry induced by KL divergence on the policy manifold.",{"name":83,"@type":74,"acceptedAnswer":84},"What is RIPO and how does it fix the issue?",{"text":85,"@type":77},"Riemannian Isometric Policy Optimization (RIPO) enforces isometric policy updates on the Riemannian manifold by adapting the clipping boundary to local manifold geometry, balancing exploration and exploitation and improving stability via a favorable bias–variance trade-off.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]