[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85275-en":3,"doc-seo-85275-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85275,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies","This paper develops a model-free reinforcement learning framework for continuous-time extended mean field control, where both dynamics and rewards depend on the joint distribution of states and controls. Deterministic feedback policies are used so the state–action distribution is obtained directly via a push-forward of the state law, avoiding optimization over stochastic kernels. A model-free sensitivity result for parameterized McKean–Vlasov dynamics yields deterministic policy gradients expressed through advantage-rate functions on Wasserstein space, refined with local value/advantage-rate representations that add measure-derivative terms. A martingale-based learning principle motivates a deep deterministic policy-gradient algorithm with particles, measure-dependent networks, and temporal-difference learning.","arXiv :2607 . 11005v1 [math .OC] 13 Jul 2026  \nActor-Critic Learning for Extended Mean Field Control with  \nDeterministic Policies  \nZiheng Cheng* Xin Guo† Huyn Pham ‡ Yufei Zhang§  \nAbstract  \nThis paper develops a model-free reinforcement learning framework for continuous-time extended mean field control problems, where both the dynamics and reward may depend on the joint distribution of states and controls. We adopt deterministic feedback policies, under which the state–action distribution is induced directly as a push-forward of the state law. This avoids optimization over stochastic kernels and bypasses key limitations of existing approaches in extended mean field settings.  \nWe first establish a model-free sensitivity formula for parameterized McKean–Vlasov dynamics and use it to derive a deterministic policy gradient formula expressed through an advantage-rate function on the Wasserstein space. We then refine this formula by introducing local value and advantage-rate representations that depend on the state, action, and joint state–action distribution, yielding a policy gradient that includes both action derivatives and measure-derivative terms with respect to the control distribution. These characterizations lead to a martingale-based learning principle and motivate a continuous-time deep deterministic policy-gradient algorithm combining particle approximations, measure-dependent neural networks, temporal-difference learning, and exploration in either action or parameter space.  \nNumerical experiments on stochastic Cucker–Smale consensus control and optimal liquidation with trade crowding demonstrate the efficiency, stability, and robustness of the proposed method, including problems with explicit dependence on the control distribution.  \n1 Introduction  \n(Extended) MFC. (Extended) mean field control (MFC) provides a tractable framework for large population stochastic games, by optimizing the collective behavior of homogeneous and interacting agents through a central planner [1] . In the mean-field limit, the controlled state of a (representative) agent is governed by a McKean-Vlasov system  \ndX = b (s, X, αs, P (X ,αs ))ds + σ(s, X, αs, P (X ,αs ))dWs, (1.1)  \nwhere both the drift and diffusion coefficients depend on the distribution of the state Xα and the control α . Although this limit removes the explicit dependence on the number of agents, the associated extended MFC is intrinsically an infinite-dimensional control problem: the value function and the optimal feedback generally depend not only on time and the individual state, but also on the state distribution. The most fundamental theoretical ingredients in the development  \n* University of California, Berkeley. Email: ziheng [cheng@berkeley.edu](cheng@berkeley.edu)[ ](cheng@berkeley.edu)†University of California, Berkeley. Email: [xinguo@berkeley.edu](xinguo@berkeley.edu)[ ](xinguo@berkeley.edu)‡Ecole Polytechnique, CMAP, Email: [huyen.pham@polytechnique.edu](huyen.pham@polytechnique.edu)  \n§ Imperial College London. Email: [yufei.zhang@imperial.ac.uk](yufei.zhang@imperial.ac.uk)  \nof MFC theory are the dynamic programming principle for the lifted value function defined on the space of probability measures, and the invariance property of this value function along the deterministic state flow (see e.g., [28, 8]) .  \nContinuous-time RL with deterministic policy. In extended mean field control, both the dynamics and reward may depend on the joint distribution of states and controls. To address this challenge, we adopt deterministic feedback policies. A deterministic policy φ(t, x, µ) selects an action directly based on the current time, state and state law, so that the state law together with the feedback map completely determines the associated state–action distribution through the push-forward measure Γφt = 􀀀id, φ(t, ·, µ t)􀀁\\#µt, with µt being the current state distribution. For continuous-time reinforcement learning (RL), recent analysis using deterministic po","cbCaioDBPNTOb6af","https://ap.wps.com/l/cbCaioDBPNTOb6af","pdf",370238,1,27,"English","en",105,"# Abstract\n# Introduction\n## (Extended) MFC\n## Continuous-time RL with deterministic policy\n## Learning for discrete-time MFC\n## Our work for continuous-time extended MFC","[{\"question\":\"What does the proposed algorithm incorporate and how is it evaluated?\",\"answer\":\"It motivates a continuous-time deep deterministic policy-gradient algorithm using particle approximations, measure-dependent neural networks, temporal-difference learning, and exploration in either action or parameter space, with numerical experiments on consensus control and optimal liquidation showing efficiency and robustness.\"}]",1784202208,68,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"actor-critic-learning-for-extended-mean-field-control-with-deterministic-policies","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/actor-critic-learning-for-extended-mean-field-control-with-deterministic-policies/85275/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"What does the proposed algorithm incorporate and how is it evaluated?","Question",{"text":75,"@type":76},"It motivates a continuous-time deep deterministic policy-gradient algorithm using particle approximations, measure-dependent neural networks, temporal-difference learning, and exploration in either action or parameter space, with numerical experiments on consensus control and optimal liquidation showing efficiency and robustness.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,115,120,123,127],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":113,"slug":114},30,"research-report",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":98,"slug":130},19,"General","general"]