[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-327715-105":59,"doc-detail-327715-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","when-aggregate-alignment-misleads-auditing-policy-repair-without-per-state-expert-actions","When Aggregate Alignment Misleads - Auditing Policy Repair Without Per-State Expert Actions","","Agentic AI systems increasingly repair decision policies, yet evaluating such edits is hard when per-state expert action labels are missing. This study examines hotel-pricing policy repair where an editor receives only region-level diagnostic feedback summarizing how its price distribution differs from a benchmark across time, inventory, and market regions. Without benchmark code, rewards, or outcomes, it proposes constrained table edits. On 5,000 held-out episodes, an LLM editor attains RevPAR 108.47 (95% CI 107.61–109.34), closely matching 108.75 (107.81–109.68) while improving episode composition distance; stronger alignment is partially decoupled from revenue lift.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/when-aggregate-alignment-misleads-auditing-policy-repair-without-per-state-expert-actions/327715/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/when-aggregate-alignment-misleads-auditing-policy-repair-without-per-state-expert-actions/327715.png","ImageObject",300,407,{"name":92,"@type":93},"Oliver","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-30","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":24},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What makes auditing agentic policy repair difficult in this work?","Question",{"text":112,"@type":113},"The evaluator lacks per-state expert action labels and also does not receive benchmark implementation, reward numbers, or held-out outcomes. The editor only gets region-level diagnostic feedback comparing its behavior to a benchmark.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What environment and policy-editing constraints are used for the experiments?",{"text":117,"@type":113},"Experiments use a hotel-pricing simulator with a target-action table indexed by time, inventory, and market. The agent can only apply constrained rectangular integer-shift edits to that table.",{"name":119,"@type":110,"acceptedAnswer":120},"What is the main finding about using aggregate alignment for evaluation?",{"text":121,"@type":113},"Region-level diagnostics can enable an LLM editor to achieve strong closed-loop performance, but evaluation must rely on whether diagnostic feedback becomes reliable for closed-loop outcomes. A single behavioral distance or aggregate match can be misleading.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},327715,1790793819,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":24,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145,"preview_page_count":4},8796095461610,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","arXiv :2607 .03386v 1 [ cs .AI] 3 Jul 2026  \nWhen Aggregate Alignment Misleads: Auditing Policy  \nRepair  \nWithout Per-State Expert Actions  \nPeiying Zhu Sidi Chang  \nBlossom AI Blossom AI Labs  \nSan Francisco, USA Tokyo, Japan  \nAbstract  \nAgentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61–109.34), close to the benchmark policy’s 108.75 (107.81–109.68), with paired gap (LLM minus benchmark)  \n-0.276 and 95% CI [-0.692, 0.146] . A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor’s distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from  \n1.153 to 0.609 . This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: nonsemantic proposers with up to 2,500 evaluations fall 8 .77–14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR  \n94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91 . These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.  \nKeywords: agent evaluation, policy repair, diagnostic feedback, behavioral auditing, revenue management  \n1 Introduction  \nAgentic AI systems are increasingly asked to modify decision systems rather than merely answer questions. They write code, tune operational rules, update workflows, and propose policy edits. This creates a new evaluation problem. The agent’s output is not a label or a prediction; it is an intervention on a decision policy. If the intervention looks plausible under a superficial metric, the system may still fail in the operational states that matter.  \nRecent work has emphasized language models as agents that reason, act, and interact with environments [3, 10] . Other benchmarks evaluate agents through artifact edits validated by external execution, such as code patches for software tasks [2] . Our setting follows the same broad direction but studies a different object: an agent edits a decision policy, and the evaluator must judge whether the policy repair is trustworthy without per-state expert action labels.  \nThis paper studies agentic policy repair under missing per-state expert action labels. The agent is given a policy to edit and diagnostic feedback about how the policy differs from a benchmark. It is not given the benchmark action for each state, the benchmark implementation, reward numbers, or held-out outcomes. It must propose  \nconstrained edits to the policy. The evaluator then asks whether the edited policy is better.  \nThis setting differs from standard supervised imitation and from reward optimization. In imitation learning, the learner observes expert state-action pairs. In reward optimization, the learner receives a scalar objective. In diagnostic-feedback repair, the agent receives aggregate behavioral summaries. These summaries may say that a region of the state space overuses or underuses certain actions, but they do not say what action to take in a particular state.  \nWe use hotel pricing as a controlled testbed. The policy is a target-actio","cbCaitFVFdJAwmU8","https://ap.wps.com/l/cbCaitFVFdJAwmU8","pdf",353828,11,"English","# Abstract\n# 1 Introduction\n## Agentic policy modification as a new evaluation problem\n## Diagnostic-feedback repair vs imitation and reward optimization\n## Hotel pricing as the controlled testbed\n## Benchmark discipline vs optimality\n## Audit fidelity risk: aggregate alignment","[{\"question\":\"What makes auditing agentic policy repair difficult in this work?\",\"answer\":\"The evaluator lacks per-state expert action labels and also does not receive benchmark implementation, reward numbers, or held-out outcomes. The editor only gets region-level diagnostic feedback comparing its behavior to a benchmark.\"},{\"question\":\"What environment and policy-editing constraints are used for the experiments?\",\"answer\":\"Experiments use a hotel-pricing simulator with a target-action table indexed by time, inventory, and market. The agent can only apply constrained rectangular integer-shift edits to that table.\"},{\"question\":\"What is the main finding about using aggregate alignment for evaluation?\",\"answer\":\"Region-level diagnostics can enable an LLM editor to achieve strong closed-loop performance, but evaluation must rely on whether diagnostic feedback becomes reliable for closed-loop outcomes. A single behavioral distance or aggregate match can be misleading.\"}]","When Aggregate Alignment Misleads - Auditing Policy Repair Without Per-State Expert Actions | PDF",1789977896,28]