[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82868-en":3,"doc-seo-82868-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82868,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","Constraint-Aware Reinforcement Learning for Planning with LLMs (CARL)","Large Language Models (LLMs) often generate plans that violate task constraints, reducing reliability in real-world use. The problem stems from insufficient mechanisms to inject constraint information into the generation process. This work introduces Constraint-Aware Reinforcement Learning (CARL), an RL framework that strengthens intrinsic constraint focus by adding a constraint-aware reward from distributional changes between constrained and unconstrained outputs. CARL uses KL divergence of log-probabilities as a learning signal, is compatible with multiple RL algorithms, avoids external solvers or top models, and enables scalable end-to-end constraint-aware planning. Experiments on BlocksWorld, TravelPlanner, and TEval show significant gains over RFT and state-of-the-art reasoning models, with clearer constraint attribution.","CARL: Constraint-Aware Reinforcement Learning for Planning with  \nLLMs  \nQiuyi Qi♠♢ * , Jinjian Zhang♢ * , Mutian Bao♠♢ * , Tian Liang♠♢ , Guocong Li♠♢ ,  \nDongnan Liu♢ , Wei Zhou♢ , Jie Liu♣ , Ming Kong♠†,  \nLinjian Mo♢†, Feng Zhang♠ , Qiang Zhu♠†♠ Zhejiang University, ♢ Ant Group, ♣ City University of Hong Kong  \n{qiqiuyi,[zhuq}@zju.edu.cn](zhuq}@zju.edu.cn)  \narXiv :2607 .04854v 1 [ cs .AI] 6 Jul 2026  \nAbstract  \nDespite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model’s intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs’ intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model’s output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and TEval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.  \n1 Introduction  \nLarge Language Models (LLMs) have demonstrated remarkable capabilities in reasoning, tool utilization, and world knowledge modeling, positioning them as powerful candidates for complex planning tasks—a cornerstone of cognitive AI systems (Huang et al., 2022b ; Ahn et al., 2022) . Plan-  \n* Q. Qi, J. Zhang and M. Bao contributed equally to this work.  \n†Q. Zhu, M. Kong and L. Mo are corresponding authors.  \nQ. Zhu is with the College of Artificial Intelligence, Shanghai Institute for Advanced Study, Zhejiang University. M. Kong is with the School of Earth Sciences, Zhejiang University. L. Mo is with the Ant Group.  \nAttribution Scores Constraints  \nLLM Responses  \nFigure 1: A typical case of planning tasks. The upper box demonstrates that a query can be decomposed into agoal and constraints, while the lower box shows that our CARL exhibits a higher focus on constraints, ultimately outperforming RFT in planning.  \nning entails generating a sequence of executable actions to achieve a goal while strictly adhering to a set of constraints (Newell et al., 1958 ; Kartam and Wilkins, 1990) . For instance, a travel itinerary must not only satisfy the destination and timeline but also comply with specific constraints such as budget limits, transportation preferences, or dietary requirements.  \nDespite these capabilities, LLMs consistently struggle to generate constraint-compliant plans in practice (Wei et al., 2025 ; Huang et al., 2024) . On  \nthe challenging real-world benchmark TravelPlanner (Xie et al., 2024a), DeepSeek-R1 (Guo et al., 2025), a model renowned for its general reasoning performance, achieves a pass rate of only 12.2%, falling significantly short of human-level performance. This disparity is not due to weak reasoning ability but rather reflects a fundamental limitation:  \nLLMs lack the capacity to systematically incorporate constraints into their generation process. Empirical studies (Xie et al., 2024b) corroborate this, revealing that LLMs frequently neglect constraints during planning and exhibit low attribution scores for constraint-related tokens in the input.  \nExisting approaches primarily circumvent this issue by offloading constraint reasoning to external scaffolds. Common paradigms include planthen-execute, which decomposes complex queries int","cbCaip2jFiYQ6oI9","https://ap.wps.com/l/cbCaip2jFiYQ6oI9","pdf",4981787,1,25,"English","en",105,"# Introduction\n## Problem: constraint-violating planning by LLMs\n## Existing approaches and limitations\n## Proposed method: CARL\n## Training signal: constraint-aware reward","[{\"question\":\"Why do LLMs struggle with constraint-compliant planning?\",\"answer\":\"LLMs often fail to systematically incorporate constraint information during generation, leading to frequent constraint neglect and low attribution to constraint-related tokens.\"},{\"question\":\"How does CARL improve LLM planning with constraints?\",\"answer\":\"CARL introduces a constraint-aware reinforcement learning framework that uses a reward derived from distributional shifts between outputs under constrained versus unconstrained inputs.\"},{\"question\":\"What learning signal does CARL use for its constraint-aware reward?\",\"answer\":\"CARL computes KL divergence between the log-probabilities of the model in constrained and unconstrained settings, using it as a continuous reward signal to guide optimization.\"}]",1784183569,63,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"constraint-aware-reinforcement-learning-for-planning-with-llms-carl","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/constraint-aware-reinforcement-learning-for-planning-with-llms-carl/82868/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do LLMs struggle with constraint-compliant planning?","Question",{"text":75,"@type":76},"LLMs often fail to systematically incorporate constraint information during generation, leading to frequent constraint neglect and low attribution to constraint-related tokens.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does CARL improve LLM planning with constraints?",{"text":80,"@type":76},"CARL introduces a constraint-aware reinforcement learning framework that uses a reward derived from distributional shifts between outputs under constrained versus unconstrained inputs.",{"name":82,"@type":73,"acceptedAnswer":83},"What learning signal does CARL use for its constraint-aware reward?",{"text":84,"@type":76},"CARL computes KL divergence between the log-probabilities of the model in constrained and unconstrained settings, using it as a continuous reward signal to guide optimization.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]