[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-142137-105":59,"doc-detail-142137-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","fast-two-time-scale-stochastic-gradient-method-with-applications-in-reinforcement-learning","Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning","","Two-time-scale optimization provides a unified framework for policy evaluation and policy optimization in reinforcement learning, where an upper-level objective relies on gradients computed through the solution of a lower-level root-finding problem for a strongly monotone operator. This work introduces a new algorithm that attains substantially faster convergence by incorporating averaging steps to refine operator estimates at both levels prior to updating variables, removing direct variable coupling. Finite-time convergence rates are derived across strong convexity, convexity, Polyak–Łojasiewicz, and general non-convex settings, improving upon the best-known complexity. Applied to RL, the method yields novel online sample-based procedures that match or surpass state of the art, supported by numerical simulations.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/fast-two-time-scale-stochastic-gradient-method-with-applications-in-reinforcement-learning/142137/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/fast-two-time-scale-stochastic-gradient-method-with-applications-in-reinforcement-learning/142137.png","ImageObject",300,407,{"name":92,"@type":93},"Arica Lee","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-29","2026-08-25",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What problem does the two-time-scale optimization framework model in reinforcement learning?","Question",{"text":113,"@type":114},"It models RL policy evaluation and policy optimization where an upper-level gradient evaluation depends on solving a lower-level equation finding the root of a strongly monotone operator.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does the proposed fast method improve convergence compared with standard two-time-scale stochastic approximation?",{"text":118,"@type":114},"It adds averaging steps to denoise and improve estimates of the stochastic operators in both lower and upper levels before updating decision variables, eliminating direct coupling and accelerating performance.",{"name":120,"@type":111,"acceptedAnswer":121},"What kinds of objective-function structures are covered by the finite-time convergence analysis?",{"text":122,"@type":114},"The analysis establishes finite-time convergence rates under strong convexity, convexity, the Polyak–Łojasiewicz condition, and general non-convexity.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},142137,1787675701,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145,"preview_page_count":4},8796096645457,"https://ap-avatar.wpscdn.com/avatar/800003749518d68ffe3?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345340919836971","Fast Two-Time-Scale Stochastic Gradient Method with Applications in  \nReinforcement Learning  \nSihan Zeng SIHAN . ZENG @JPMCHASE . COM  \nJPMorgan AIResearch, United States  \nThinh T. Doan THINHDOAN @VT. EDU  \nVirginia Tech, United States  \nEditors: Shipra Agrawal and Aaron Roth  \nAbstract  \nTwo-time-scale optimization is a framework introduced in Zeng et al. (2024) that abstracts a range of policy evaluation and policy optimization problems in reinforcement learning (RL) . Akinto bi-level optimization under a particular type of stochastic oracle, the two-time-scale optimization framework has an upper level objective whose gradient evaluation depends on the solution of a lower level problem, which is to find the root of a strongly monotone operator. In this work, we propose a new method for solving two-time-scale optimization that achieves significantly faster convergence than the prior arts. The key idea of our approach is to leverage an averaging step to improve the estimates of the operators in both lower and upper levels before using them to update the decision variables. These additional averaging steps eliminate the direct coupling between the main variables, enabling the accelerated performance of our algorithm. We characterize the finite-time convergence rates of the proposed algorithm under various conditions of the underlying objective function, including strong convexity, convexity, Polyak-Lojasiewicz condition, and general non-convexity. These rates significantly improve over the best-known complexity of the standard two-time-scale stochastic approximation algorithm. When applied to RL, we show how the proposed algorithm specializes to novel online sample-based methods that surpass or match the performance of the existing state of the art. Finally, we support our theoretical results with numerical simulations in RL. Keywords: Bi-level optimization, accelerated stochastic optimization, reinforcement learning  \n1. Introduction  \nInspired by the two-time-scale optimization framework for reinforcement learning (RL) introduced in Zeng et al. (2024), we study the following optimization problem  \nθ‹ “ argmin θPRd hpθq, (1)  \nwhere h : Rd Ñ R is a continuously differentiable (possibly nonconvex) function. The gradient of h can only be accessed through a special stochastic oracle F : Rd ˆ Rr ˆ X Ñ R. The three arguments to F are the decision variable θ P Rd, an auxiliary variable ω P Rr, and a random variable X over the sample space X. Given any θ , Fpθ,ω, Xq returns an unbiased estimate of the gradient ∇hpθq only under a single “correct” setting of ω when the random variable X follows some distribution ξ  \n∇θhpθq “ EX„ξrFpθ,ω‹ pθq, Xqs, (2)  \nwhere ω‹ pθq P Rr is defined through another stochastic sampling operator G : Rd ˆ Rr ˆ X Ñ R, as the solution to the equation  \nEX„ξrGpθ,ω‹ pθq, Xs “ 0. (3)  \n© 2024 S. Zeng & T.T. Doan.  \nZENG DOAN  \nIn this work, the operator G is assumed to be strongly monotone with respect to ω, implying that ω‹ pθq exists and is unique for any θ . Given any ω ‰ ω‹ pθq, the stochastic gradient Fpθ,ω, Xq maybe arbitrarily biased. By the first-order optimality condition, it is obvious that solving (1) is equivalent to finding a pair of solutions pθ‹,ω‹q that solve the following two coupled nonlinear equations  \n\\# EXEX„„ξξ rFprGpθθ‹‹,, ωω‹‹,, XqsXqs ““ 00, (4)  \nAs detailed in Section 4, a wide range of RL methods, such as gradient-based temporal difference learning (Sutton et al., 2008, 2009) and actor-critic algorithms for policy optimization (Yang et al., 2019 ; Wu et al., 2020 ; Chen and Zhao, 2024), aim to solve problems that are special cases of (4) . Thus, by developing and analyzing algorithms for the unified framework (4), we can apply them to various RL problems and immediately deduce performance guarantees, without having to tailor analysis for each specific problem. Hong et al. (2023); Zeng et al. (2024) pioneer the effort to construct such a unified framework and study solving (4) with ","cbCaigfJYj6EmHY0","https://ap.wps.com/l/cbCaigfJYj6EmHY0","pdf",599837,47,"English","# Introduction\n## Main Contributions","[{\"question\":\"What problem does the two-time-scale optimization framework model in reinforcement learning?\",\"answer\":\"It models RL policy evaluation and policy optimization where an upper-level gradient evaluation depends on solving a lower-level equation finding the root of a strongly monotone operator.\"},{\"question\":\"How does the proposed fast method improve convergence compared with standard two-time-scale stochastic approximation?\",\"answer\":\"It adds averaging steps to denoise and improve estimates of the stochastic operators in both lower and upper levels before updating decision variables, eliminating direct coupling and accelerating performance.\"},{\"question\":\"What kinds of objective-function structures are covered by the finite-time convergence analysis?\",\"answer\":\"The analysis establishes finite-time convergence rates under strong convexity, convexity, the Polyak–Łojasiewicz condition, and general non-convexity.\"}]","Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning | PDF",118]