[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84227-en":3,"doc-seo-84227-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84227,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","The Optimal Sample Complexity of Learning Autoregressive Chain of Thought","Provides a theoretical PAC-learning analysis for exact-trace learning of autoregressive chain-of-thought models. In the realizable setting, the sample complexity for learning full traces is bounded by the standard multiclass rate for the local next-token class, governed by Daniely–Shalev-Shwartz dimension. Exact-trace loss is shown to incur no dependence on rollout length, despite the all-or-nothing correctness requirement. A new parity dimension refines DS dimension, stays rollout-stable, controls one-inclusion density, and explains why DS dimension alone can increase under rollout.","arXiv :2607 .07423v 1 [ cs .LG] 8 Jul 2026  \nThe Optimal Sample Complexity of Learning Autoregressive  \nChain-of-Thought  \nZhiyuan Li  \nToyota Technological Institute at Chicago  \n[zhiyuanli@ttic. edu](zhiyuanli@ttic. edu)  \nAbstract  \nWe prove that, in the realizable PAC setting, the sample complexity of exact-trace learning for full autoregressive Chain-of-Thought traces is upper bounded by the standard multiclass rate of the local next-token class, where this rate is governed by the Daniely–Shalev-Shwartz dimension. Under exact-trace loss, one wrong action makes the whole trace incorrect; nevertheless, for every stopping rule halt and every pointwise halt-halting local class H,  \nnεP,δAC (Rollhalt (H)) = O 􀀒 DSdim(H)ε+~~ ~~log(1/δ)􀀓 ,  \nwith no dependence on rollout length. The dependence on DSdim (H) is worst-case optimal, since one-step stopping recovers ordinary multiclass learning of H.  \nThe proof introduces parity dimension, a rollout-stable refinement of DS dimension based on even pseudo-cubes. It controls one-inclusion density via a low-coordinate spanning theorem on finite restrictions and, unlike DS dimension itself, does not increase under autoregressive rollout. We also show why this detour is necessary: DS dimension can increase under rollout.  \n1 Introduction  \nChain-of-Thought supervision exposes more than a final answer: it gives the intermediate tokens, actions, or reasoning steps along a supervised trace (Joshi et al. , 2025; Hanneke et al. , 2026b) . Inan autoregressive model, these coordinates are generated by a single shared local rule. The rule is queried at the current state, its action is appended to the transcript, and the process repeats until a stopping condition is met. Under exact-trace loss, the learner is correct only if the entire generated trace is correct. One wrong action makes the whole trace wrong.  \nAt first sight, this all-or-nothing loss over a variable-length sequence should increase sample complexity. The label space consists of complete action strings, and exact correctness requires every coordinate of the generated trace to be right. The coordinates, however, are not chosen independently: they are produced by repeatedly querying the same local next-action rule. The central question is whether this shared autoregressive structure completely removes the apparent statistical cost of predicting an entire trace.  \nProblem formulation. We write Rollhalt(H) for the complete-trace class obtained by rolling out a local next-action class H under a deterministic stopping rule halt. Formally, the state space has aligned form X = E × A∗ , the local class is H ⊆ AX , and halt decides when to stop from the initial state and the emitted suffix. The environment coordinate E stores exogenous context that can affect the future rollout but is not necessarily emitted as an action string. It may include the  \nprompt, task state, tool observations in interactive language-model agents such as Yao et al. (2023), or visual context represented by multimodal encoders such as Radford et al. (2021) . The rollout model is formalized in Definitions 2.1 to 2.3. Exact-trace learning is ordinary multiclass learning of Rollhalt(H) ⊆ (A∗ )X by Proposition 2 .5, whose labels are complete action strings.  \nExisting routes and the optimal-rate question. For ordinary multiclass PAC learning, the sharp realizable benchmark is the Daniely–Shalev-Shwartz dimension DSdim, recalled in Definition 2.9 . Since exact-trace learning is multiclass learning of Rollhalt(H), the general multiclass theory already characterizes the rollout problem in terms of DSdim(Rollhalt(H)) . Thus the question is not whether the rollout class is PAC learnable, but how its complexity compares with that of the local next-action class H.  \nIndeed, if the stopping rule stops after one action, then Rollhalt(H) is just H, with labels written as length-one strings. Therefore any theorem stated in terms of the local class must in general pay at least the or","cbCaioK1vEowxJes","https://ap.wps.com/l/cbCaioK1vEowxJes","pdf",474431,1,33,"English","en",105,"# Abstract\n# Introduction\n## Problem formulation\n## Existing routes and the optimal-rate question\n## Question\n## Main result\n## Theorem 1.1","[{\"question\":\"What learning problem does the paper study for autoregressive chain-of-thought?\",\"answer\":\"It studies realizable PAC learning under exact-trace loss, where the learner must predict an entire generated token/action trace correctly. Under exact-trace loss, any single wrong action makes the whole trace incorrect.\"},{\"question\":\"What is the main result about sample complexity?\",\"answer\":\"The paper proves that, in the realizable PAC setting, exact-trace learning of full autoregressive chain-of-thought traces has sample complexity upper bounded by the standard multiclass rate for the local next-token class. This bound is governed by Daniely–Shalev-Shwartz dimension.\"},{\"question\":\"Why is parity dimension introduced, and what does it fix?\",\"answer\":\"The proof introduces parity dimension as a rollout-stable refinement of DS dimension based on even pseudo-cubes. It controls one-inclusion density and, unlike DS dimension itself, does not increase under autoregressive rollout, avoiding a failure mode where DS dimension can grow during rollout.\"}]",1784194161,83,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"the-optimal-sample-complexity-of-learning-autoregressive-chain-of-thought","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/the-optimal-sample-complexity-of-learning-autoregressive-chain-of-thought/84227/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What learning problem does the paper study for autoregressive chain-of-thought?","Question",{"text":75,"@type":76},"It studies realizable PAC learning under exact-trace loss, where the learner must predict an entire generated token/action trace correctly. Under exact-trace loss, any single wrong action makes the whole trace incorrect.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the main result about sample complexity?",{"text":80,"@type":76},"The paper proves that, in the realizable PAC setting, exact-trace learning of full autoregressive chain-of-thought traces has sample complexity upper bounded by the standard multiclass rate for the local next-token class. This bound is governed by Daniely–Shalev-Shwartz dimension.",{"name":82,"@type":73,"acceptedAnswer":83},"Why is parity dimension introduced, and what does it fix?",{"text":84,"@type":76},"The proof introduces parity dimension as a rollout-stable refinement of DS dimension based on even pseudo-cubes. It controls one-inclusion density and, unlike DS dimension itself, does not increase under autoregressive rollout, avoiding a failure mode where DS dimension can grow during rollout.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]