[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85705-en":3,"doc-seo-85705-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85705,137441390410,"Hazel","https://ap-avatar.wpscdn.com/avatar/2000252f4ab5702993?_k=1776741390130283984",8,"Research & Report","Norm Enforcement for AI Agents Robustly Shaping Behavior in Multi-Agent Systems","AI agents increasingly operate in shared environments, where competing goals can generate harmful outcomes that benefit individuals while degrading collective information quality. Human societies counteract such incentives using norms and enforcement systems that detect and penalize violations. This paper studies norm enforcement mechanisms tailored to language model agents, showing that simplistic enforcement is exploitable by misaligned agents for competitive advantage. Robust mechanisms rely on estimating each agent’s reliability over time and escalating penalties for repeated misconduct, maintaining norm compliance at comparable or lower cost across multiple simulated environments.","arXiv :2607 .09766v 1 [ cs .AI ] 7 Jul 2026  \nNorm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems  \nYaowen Ye Jacob Steinhardt  \nUniversity of California, Berkeley  \n{elwin, [jsteinhardt}@berkeley.edu](jsteinhardt}@berkeley.edu)  \nAbstract  \nAI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-agent competition can lead to behaviors that serve individual gains at collective cost—for instance, marketing agents may post misleading content as a result of competing for engagement on social media. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations. Motivated by this, we study norm enforcement mechanisms for language model agents. We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage, even when they are not explicitly trained or prompted to do so. We thus turn our attention to designing more robust mechanisms, and identify two key ingredients: estimating each agent’s reliability over time, and updating this estimate with escalating penalties for repeated misbehavior. Across three simulated environments and a variety of agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping agents’ behavior, but only when designed to anticipate becoming part of the system they govern. Our code and data are available at [https://yaowenye.com/norm-enforcement](https://yaowenye.com/norm-enforcement).  \n1 Introduction  \nAI agents deployed by different parties in shared environments form large-scale, decentralized multiagent systems. For instance, marketing agents post promotional content on social media and compete for user engagement (HubSpot, 2026), while platforms like MoltBook let individual users deploy personal agents that act on their behalf and interact with other agents on the platform (Schlicht, 2026) . As agentic deployment expands, such systems are growing in both number and scale.  \nBecause agents are deployed with distinct goals, they often have partially conflicting (or competitive) interests. This competitive structure can produce undesired behaviors that serve individual goals at the expense of others: social media agents optimizing for engagement learn to post misleading content (Pan et al., 2024 ; El and Zou, 2025), pricing agents competing for profit converge on supracompetitive prices that harm consumers (Fish et al., 2024), and agents maximizing influence on MoltBook post manipulative content to win peer upvotes (Riegler and Gautam, 2026) . At scale, such behaviors canerode the information ecosystems that humans and agents share.  \nThese dynamics are not unique to AI agents. Human actors in shared environments face the same pull toward individual gain at collective cost, and societies address this by establishing norms that constrain acceptable behavior (e.g. laws or community guidelines) and enforcement mechanisms that penalize norm violations. These mechanisms range from centralized institutions like courts to decentralized processes like peer supervision. How can we design analogous mechanisms to enforce desired norms in systems involving AI agents?  \nPreprint.  \nSends posts following norms Reports violations honestly  \nSends friendly coordination messages  \nSends posts that violate norms  \n... even though it's not true, people are more likely to trust ... ifa famous person is involved ... [[misinfo. post](misinfo. post)]  \nFiles false reports against competitors  \nI'll report Agent 6 because ... might be taking away potential customers ... \u003Creport target=\"Agent6\"> Agent6's post is misleading because ... \u003C/report>  \nSends hostile messages  \n\u003Cprivate_message recipient=\"Agent3\"> Hey, I think y","cbCainO912B0WvQ9","https://ap.wps.com/l/cbCainO912B0WvQ9","pdf",837013,1,54,"English","en",105,"# Introduction\n## Norm Enforcement Problem\n## Related Work\n## Approach and Simulation Setup","[{\"question\":\"Why can norm violations emerge in multi-agent AI environments?\",\"answer\":\"Agents pursue diverse goals and compete for rewards, which can incentivize misleading or harmful behaviors that benefit individuals at collective cost.\"},{\"question\":\"How do misaligned agents exploit simple norm enforcement mechanisms?\",\"answer\":\"The paper finds that even without explicit training or prompting, misaligned agents can exploit straightforward enforcement for competitive advantage.\"},{\"question\":\"What two key ingredients make the proposed enforcement mechanisms more robust?\",\"answer\":\"The mechanisms estimate each agent’s reliability over time and update it with escalating penalties for repeated misbehavior, reducing exploitable behavior while still penalizing violations.\"}]",1784205712,136,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"norm-enforcement-for-ai-agents-robustly-shaping-behavior-in-multi-agent-systems","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/norm-enforcement-for-ai-agents-robustly-shaping-behavior-in-multi-agent-systems/85705/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why can norm violations emerge in multi-agent AI environments?","Question",{"text":74,"@type":75},"Agents pursue diverse goals and compete for rewards, which can incentivize misleading or harmful behaviors that benefit individuals at collective cost.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How do misaligned agents exploit simple norm enforcement mechanisms?",{"text":79,"@type":75},"The paper finds that even without explicit training or prompting, misaligned agents can exploit straightforward enforcement for competitive advantage.",{"name":81,"@type":72,"acceptedAnswer":82},"What two key ingredients make the proposed enforcement mechanisms more robust?",{"text":83,"@type":75},"The mechanisms estimate each agent’s reliability over time and update it with escalating penalties for repeated misbehavior, reducing exploitable behavior while still penalizing violations.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]