[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-142140-105":59,"doc-detail-142140-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","two-time-scale-off-policy-td-learning-non-asymptotic-analysis-over-markovian-samples","Two Time-scale Off-Policy TD Learning - Non-asymptotic Analysis over Markovian Samples","","Gradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios, where TD updates rely on data from a different behavior policy. Two time-scale TD with gradient correction (TDC) has shown strong performance, but prior non-asymptotic results assumed i.i.d. samples only. This work establishes the first non-asymptotic convergence analysis for two time-scale TDC under non-i.i.d. Markovian sample paths with linear function approximation.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/two-time-scale-off-policy-td-learning-non-asymptotic-analysis-over-markovian-samples/142140/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/two-time-scale-off-policy-td-learning-non-asymptotic-analysis-over-markovian-samples/142140.png","ImageObject",300,407,{"name":92,"@type":93},"Oliver Hayes","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-01","2026-08-25",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",12,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What is the main contribution regarding two time-scale TDC convergence?","Question",{"text":113,"@type":114},"The paper provides the first non-asymptotic convergence analysis for two time-scale TDC under non-i.i.d. Markovian samples and linear function approximation.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How do the convergence behaviors differ between diminishing and constant stepsizes?",{"text":118,"@type":114},"With diminishing stepsize, two time-scale TDC can converge at rate O(…); with constant stepsize, it can converge exponentially fast but incurs a non-vanishing error.",{"name":120,"@type":111,"acceptedAnswer":121},"What stepsize strategy is proposed to improve performance under constant stepsize?",{"text":122,"@type":114},"The authors propose a TDC variant using blockwise diminishing stepsize and show it asymptotically converges with arbitrarily small error at a blockwise linear convergence rate.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},142140,1787675735,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145,"preview_page_count":4},687207020761,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples  \nTengyu Xu  \nDepartment of Electrical and Computer Engineering  \nThe Ohio State University  \n[xu.3260@osu.edu](xu.3260@osu.edu)  \nShaofeng Zou  \nDepartment of Electrical Engineering  \nUniversity at Buffalo, The State University of New York  \n[szou3@buffalo.edu](szou3@buffalo.edu)  \nYingbin Liang  \nDepartment of Electrical and Computer Engineering  \nThe Ohio State University  \n[liang.889@osu.edu](liang.889@osu.edu)  \nAbstract  \nGradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios. Among them, the two time-scale TD with gradient correction (TDC) algorithm has been shown to have superior performance. In contrast to previous studies that characterized the non-asymptotic convergence rate of TDC only under identical and independently distributed (i.i.d.) data samples, we provide the first non-asymptotic convergence analysis for two time-scale TDC under anon-i.i.d. Markovian sample path and linear function approximation. We show that the two time-scale TDC can converge as fast as O( lot2t3) under diminishing stepsize, and can converge exponentially fast under constant stepsize, but at the cost of a non-vanishing error. We further propose a TDC algorithm with blockwisely diminishing stepsize, and show that it asymptotically converges with an arbitrarily small error at a blockwisely linear convergence rate. Our experiments demonstrate that such an algorithm converges as fast as TDC under constant stepsize, and still enjoys comparable accuracy as TDC under diminishing stepsize.  \n1 Introduction  \nIn practice, it is very common that we wish to learn the value function of a target policy based on data sampled by a different behavior policy, in order to make maximum use of the data available. For such off-policy scenarios, it has been shown that conventional temporal difference (TD) algorithms [24, 25] and Q-learning [33] may diverge to infinity when using linear function approximation [2] . To overcome the divergence issue in off-policy TD learning, [27, 26, 17] proposed a family of gradient-based TD (GTD) algorithms, which were shown to have guaranteed convergence in off-policy settings and are more flexible than on-policy learning in practice [18, 23] . Among those GTD algorithms, the TD with gradient correction (TDC) algorithm has been verified to have superior performance [17] [9] and is widely used in practice. To elaborate, TDC uses the mean squared projected Bellman error as the objective function, and iteratively updates the function approximation parameter with the assistance of an auxiliary parameter that is also iteratively updated. These two  \n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.  \nparameters are typically updated with stepsizes diminishing at different rates, resulting the two time-scale implementation of TDC, i.e., the function approximation parameter is updated at a slower time-scale and the auxiliary parameter is updated at a faster time-scale.  \nThe convergence of two time-scale TDC and general two time-scale stochastic approximation (SA) have been well studied. The asymptotic convergence has been shown in [4, 6] for two time-scale SA, and in [26] for two time-scale TDC, where both studies assume that the data are sampled inan identical and independently distributed (i.i.d.) manner. Under non-i.i.d. observed samples, the asymptotic convergence of the general two time-scale SA and TDC were established in [14, 36] . All the above studies did not characterize how fast the two time-scale algorithms converge, i.e, they did not establish the non-asymptotic convergence rate, which is specially important for a two time-scale algorithm. In order for two time-scale TDC to perform well, it is important to properly choose the relative scaling rate of the stepsizes for the two time-scale iterations. In practice, this can be done by fixing one stepsize a","cbCaigPuP5ZTahHv","https://ap.wps.com/l/cbCaigPuP5ZTahHv","pdf",1890487,11,"English","# Introduction\n## Background: Off-policy TD divergence and GTD/TDC\n## Two time-scale implementation\n## Prior asymptotic vs missing non-asymptotic rates\n## Challenges under non-i.i.d. Markovian samples and stepsize design","[{\"question\":\"What is the main contribution regarding two time-scale TDC convergence?\",\"answer\":\"The paper provides the first non-asymptotic convergence analysis for two time-scale TDC under non-i.i.d. Markovian samples and linear function approximation.\"},{\"question\":\"How do the convergence behaviors differ between diminishing and constant stepsizes?\",\"answer\":\"With diminishing stepsize, two time-scale TDC can converge at rate O(…); with constant stepsize, it can converge exponentially fast but incurs a non-vanishing error.\"},{\"question\":\"What stepsize strategy is proposed to improve performance under constant stepsize?\",\"answer\":\"The authors propose a TDC variant using blockwise diminishing stepsize and show it asymptotically converges with arbitrarily small error at a blockwise linear convergence rate.\"}]","Two Time-scale Off-Policy TD Learning - Non-asymptotic Analysis over Markovian Samples | PDF",28]