[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118935-en":3,"doc-seo-118935-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118935,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","Machine Learning and the Multiagent Alignment Problem - dissertation","Machine Learning and the Multiagent Alignment Problem studies how AI alignment can fail in multiagent settings when policies interact with distribution shifts and strategic responses. It argues that standard interventions designed under static assumptions fall short once feedback changes participation, labels, and covariate distributions over time. The work develops dynamic views of alignment, provides theoretical bounds on short-term violation, and proposes online formulations using sequential optimization, feedback control, and reinforcement learning with constrained objectives.","UC Santa Cruz  \nUC Santa Cruz Electronic Theses and Dissertations  \nTitle  \nMachine Learning and the Multiagent Alignment Problem  \nPermalink  \n[https://escholarship.org/uc/item/4t2588sp](https://escholarship.org/uc/item/4t2588sp)  \nAuthor  \nRaab, Reilly  \nPublication Date  \n2024  \nCopyright Information  \nThis work is made available under the terms of a Creative Commons AttributionShareAlike License, available at [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/)  \n[Peer reviewed|Thesis/dissertation](Peer reviewed|Thesis/dissertation)  \n[eScholarship.org](eScholarship.org) Powered by the California Digital Library  \nUniversity of California  \nUNIVERSITY OF CALIFORNIA  \nSANTA CRUZ  \nMACHINE LEARNING AND THE MULTIAGENT ALIGNMENT PROBLEM  \nA dissertation submitted in partial satisfaction of the requirements for the degree of  \nDOCTOR OF PHILOSOPHY  \nin  \nCOMPUTER SCIENCE AND ENGINEERING  \nby  \nReilly Raab  \nMarch 2024  \nThe dissertation of Reilly Raab is approved:  \n\n| Assistant Professor Yang Liu, Chair |\n| --- |\n| Professor Luca de Alfaro |\n| Professor Lise Getoor |\n| Professor Mingyan Liu |\n\nProfessor Emeritus Daniel Friedman  \nPeter Biehl  \nVice Provost and Dean of Graduate Studies  \nCopyright © 2024 Reilly Raab  \nContents  \nAbstract vi  \n1 Introduction 1  \n1.1 The Rapid Development of AI ...................... 1  \n1.2 The Alignment Problem .......................... 2  \n1.3 Standard Interventions Fall Short .................... 3  \n1.4 Contributions of this Dissertation .................... 4  \n2 Preliminaries 5  \n2.1 Empirical, Black-Box Function Optimization .............. 5  \n2.1.1 An Empirical Approximation .................. 6  \n2.1.2 The Smooth Black-Box ...................... 7  \n2.1.3 Iterative Refinement ....................... 8  \n2.1.4 Example: Binary Classification ................. 9  \n2.2 Present Normative Interventions ..................... 11  \n2.2.1 Group Fairness .......................... 12  \n2.2.2 Ethics Inferred from Examples .................. 15  \n2.2.3 The Tacit Assumption ...................... 16  \n2.3 Past Performance is no Guarantee of Future Results .......... 17  \n2.3.1 Feedback and Strategic Response ................ 18  \n2.4 Related Work ................................ 19  \n2.4.1 Fairness Subject to Distribution Shift .............. 20  \n2.4.2 Modelled Dynamics of Fairness Interventions ......... 21  \n2.4.3 Safe Reinforcement and Online Learning ............ 22  \n3 Alignment is not Static 23  \n3.1 Mechanisms for Distribution Shift .................... 24  \n3.1.1 Label Shift ............................. 25  \n3.1.2 Covariate Shift .......................... 26  \n3.1.3 Participation Rates ........................ 26  \n3.2 Models of Dynamics ............................ 27  \n3.2.1 Distributions as Function of Policy ............... 28  \n3.2.2 State Dependence ......................... 29  \n3.2.3 Markov Transitions ........................ 31  \n3.3 Baseline ML Polices ............................ 32  \n3.3.1 Repeated Risk Minimization ................... 33  \n3.3.2 Repeated Gradient Descent ................... 34  \n3.3.3 Distributional Robustness .................... 34  \n3.4 The Failure of Myopia ........................... 36  \n3.4.1 A Recommendation System Example .............. 36  \n3.4.2 A Geometric Picture of Misalignment .............. 38  \n3.5 The Potential Harm of Present-Normative Interventions ....... 40  \n3.5.1 A Setting for Unintended Selection ............... 40  \n3.5.2 Simulation Results ........................ 46  \n3.6 Adversarial Bounds on Short-Term Alignment Violations ....... 48  \n3.6.1 A Lipschitz Bound ........................ 51  \n3.6.2 Demographic Parity subject to Label Shift ........... 53  \n3.6.3 Numerical Validation ....................... 55  \n4 Alignment with Dynamics 57  \n4.1 Sequential Policies as Optimization Programs ............. 58  \n4.1.1 Constrained Projected Gradient ................. 60  \n4.1.2 Application to","cbCaitgXGTEtpv92","https://ap.wps.com/l/cbCaitgXGTEtpv92","pdf",7211910,1,126,"English","en",105,"# Introduction\n## The Rapid Development of AI\n## The Alignment Problem\n## Standard Interventions Fall Short\n## Contributions of this Dissertation\n# Preliminaries\n## Empirical, Black-Box Function Optimization\n## Present Normative Interventions\n## Past Performance is no Guarantee of Future Results\n## Related Work\n# Alignment is not Static\n## Mechanisms for Distribution Shift\n## Models of Dynamics\n## Baseline ML Policies\n## The Failure of Myopia\n## The Potential Harm of Present-Normative Interventions\n## Adversarial Bounds on Short-Term Alignment Violations\n# Alignment with Dynamics\n## Sequential Policies as Optimization Programs\n## Alignment via Feedback Control\n## New Possibilities for Algorithmic Fairness\n# Bringing Alignment Online\n## Alignment as an RL Problem\n## Scheduled Lagrangian Regularization\n## Bounding Regret\n## Experiments\n# Conclusion","[{\"question\":\"Why do standard alignment interventions fail in multiagent and dynamic settings?\",\"answer\":\"The dissertation explains that interventions often rely on static assumptions, but multiagent interaction can cause feedback to change participation and distributions over time. This undermines the intended effect of present-normative constraints.\"},{\"question\":\"What kinds of distribution shift are analyzed in the work?\",\"answer\":\"It considers label shift, covariate shift, and changes in participation rates. These shifts arise from policy-driven dynamics and strategic responses.\"},{\"question\":\"How does the dissertation propose bringing alignment online?\",\"answer\":\"It formulates alignment as a reinforcement learning problem and introduces methods such as constrained RL, scheduled Lagrangian regularization, and regret bounds. The goal is to achieve alignment while accounting for sequential decision-making and evolving distributions.\"}]","Machine Learning and the Multiagent Alignment Problem - dissertation | PDF",1785721069,318,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"machine-learning-and-the-multiagent-alignment-problem-dissertation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/machine-learning-and-the-multiagent-alignment-problem-dissertation/118935/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do standard alignment interventions fail in multiagent and dynamic settings?","Question",{"text":75,"@type":76},"The dissertation explains that interventions often rely on static assumptions, but multiagent interaction can cause feedback to change participation and distributions over time. This undermines the intended effect of present-normative constraints.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What kinds of distribution shift are analyzed in the work?",{"text":80,"@type":76},"It considers label shift, covariate shift, and changes in participation rates. These shifts arise from policy-driven dynamics and strategic responses.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the dissertation propose bringing alignment online?",{"text":84,"@type":76},"It formulates alignment as a reinforcement learning problem and introduces methods such as constrained RL, scheduled Lagrangian regularization, and regret bounds. The goal is to achieve alignment while accounting for sequential decision-making and evolving distributions.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]