[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-142138-105":59,"doc-detail-142138-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","a-finite-time-analysis-of-two-time-scale-actor-critic-methods-conference-paper","A Finite-Time Analysis of Two Time-Scale Actor-Critic Methods - Conference Paper","","Actor-critic (AC) reinforcement learning methods are successful in practice, yet their non-asymptotic convergence behavior and finite-sample complexity remain largely unresolved, especially beyond i.i.d. settings. This paper provides a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. conditions. The study establishes guarantees for reaching first-order stationary points and deriving finite-time performance bounds. Comparisons are included against decoupled actor-critic variants that require more sampling and stronger data assumptions.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-finite-time-analysis-of-two-time-scale-actor-critic-methods-conference-paper/142138/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-finite-time-analysis-of-two-time-scale-actor-critic-methods-conference-paper/142138.png","ImageObject",300,407,{"name":92,"@type":93},"Paura","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-29","2026-08-25",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",11,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"Why does the paper focus on finite-time analysis for actor-critic methods?","Question",{"text":113,"@type":114},"Asymptotic convergence of two time-scale actor-critic is established, but the finite-sample complexity and non-asymptotic convergence questions are still open and important for choosing learning rates and sample budgets.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What assumption does the analysis target compared with some prior decoupled methods?",{"text":118,"@type":114},"The paper analyzes two time-scale actor-critic under a non-i.i.d. setting, avoiding i.i.d.-type data assumptions that many decoupled analyses rely on.",{"name":120,"@type":111,"acceptedAnswer":121},"How do two time-scale actor-critic algorithms update actor and critic differently?",{"text":122,"@type":114},"Both actor and critic are updated simultaneously, but the actor changes more slowly via a smaller step size, while the critic uses a larger step size to track its faster dynamics.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},142138,1787675717,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":41,"preview_page_count":4},13056712833777,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","A Finite-Time Analysis of Two Time-Scale Actor-Critic Methods  \nYue Wu  \nDepartment of Computer Science University of California, Los Angeles Los Angeles, CA 90095 [ywu@cs.ucla.edu](ywu@cs.ucla.edu)  \nPan Xu  \nDepartment of Computer Science University of California, Los Angeles Los Angeles, CA 90095 [panxu@cs.ucla.edu](panxu@cs.ucla.edu)  \nWeitong Zhang  \nDepartment of Computer Science University of California, Los Angeles Los Angeles, CA 90095 [weightzero@cs.ucla.edu](weightzero@cs.ucla.edu)  \nQuanquan Gu  \nDepartment of Computer Science University of California, Los Angeles Los Angeles, CA 90095 [qgu@cs.ucla.edu](qgu@cs.ucla.edu)  \nAbstract  \nActor-critic (AC) methods have exhibited great empirical success compared with other reinforcement learning algorithms, where the actor uses the policy gradient to improve the learning policy and the critic uses temporal difference learning to estimate the policy gradient. Under the two time-scale learning rate schedule, the asymptotic convergence of AC has been well studied in the literature. However, thenon-asymptotic convergence and finite sample complexity of actor-critic methods are largely open. In this work, we provide a non-asymptotic analysis for two timescale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point (i.e., krJ(􀀒)k22 􀀔 􀀏) of thToanoeanbm-cepaorpeklernxmleboandgun, futhfonisr tiotoJe fiime(􀀒r-sts)c,wwaloecor(-i:g)ithpeole-tidscmoeplenalxyiys.  \n1 Introduction  \nActor-Critic (AC) methods [2, 16] aim at combining the advantages of actor-only methods and critic-only methods, and have achieved great empirical success in reinforcement learning [31, 1] . Specifically, actor-only methods, such as policy gradient [28] and trust region policy optimization [24], utilize a parameterized policy function class and improve the policy by optimizing the parameters of some performance function using gradient ascent, whose exact form is characterized by the Policy Gradient Theorem [28] . Actor-only methods can be naturally applied to continuous setting but suffer from high variance when estimating the policy gradient. On the other hand, critic-only methods, such as temporal difference learning [26] and Q-learning [32], focus on learning a value function (expected cumulative rewards), and determine the policy based on the value function, which is recursively approximated based on the Bellman equation. Although the critic-only methods can efficiently learn a satisfying policy under tabular setting [14], they can diverge with function approximation under continuous setting [33] . Therefore, it is natural to combine actor and critic based methods to achieve the best of both worlds. The principal idea behind actor-critic methods is simple: the critic tries to learn the value function, given the policy from the actor, while the actor can estimate the policy gradient based on the approximate value function provided by the critic.  \n34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.  \nIf the actor is fixed, the policy remains unchanged throughout the updates of the critic. Thus one can use policy evaluation algorithm such as temporal difference (TD) learning [27] to estimate the value function (critic) . After many steps of the critic update, one can expect a good estimation of the value function, which in turn enables an accurate estimation of the policy gradient for the actor. A more favorable implementation is the so-called two time-scale actor-critic algorithm, where the actor and the critic are updated simultaneously at each iteration except that the actor changes more slowly (with a small step size) than the critic (with a large step size) . In this way, one can hope the critic will be well approximated even after one step of update. From the theoretical perspective, the asymptotic analysis of two time-scale actor-critic methods has been established in [6, 16] . In spec","cbCaislYrPGjTmCH","https://ap.wps.com/l/cbCaislYrPGjTmCH","pdf",282717,12,"English","# Introduction\n## Background on actor-critic and two time-scales\n## Motivation for finite-time (non-asymptotic) analysis\n## Related work: decoupled actor-critic methods","[{\"question\":\"Why does the paper focus on finite-time analysis for actor-critic methods?\",\"answer\":\"Asymptotic convergence of two time-scale actor-critic is established, but the finite-sample complexity and non-asymptotic convergence questions are still open and important for choosing learning rates and sample budgets.\"},{\"question\":\"What assumption does the analysis target compared with some prior decoupled methods?\",\"answer\":\"The paper analyzes two time-scale actor-critic under a non-i.i.d. setting, avoiding i.i.d.-type data assumptions that many decoupled analyses rely on.\"},{\"question\":\"How do two time-scale actor-critic algorithms update actor and critic differently?\",\"answer\":\"Both actor and critic are updated simultaneously, but the actor changes more slowly via a smaller step size, while the critic uses a larger step size to track its faster dynamics.\"}]","A Finite-Time Analysis of Two Time-Scale Actor-Critic Methods - Conference Paper | PDF"]