[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86316-en":3,"doc-seo-86316-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86316,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","From Expressivity to Sample Complexity Narrow Teachers for Transformers via C-RASP","The document presents theoretical progress on understanding Transformers by connecting expressivity with learnability. It derives preliminary sample complexity bounds for learning C-RASP constructions using Transformers, inspired by loss-landscape analyses of “narrow teacher” models under a PAC learning framework. Building on a guess-and-check learning view, it shows how C-RASP Transformers can yield bounds, clarifying why constant-size constructions such as Dyck-1 and specific counting languages are learnable.","arXiv :2607 . 1 1760v 1 [ cs .LG] 13 Jul 2026  \nFrom Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP  \nMichael Rizvi-Martel 1 ∗ Satwik Bhattamishra2 Guillaume Rabusseau 1 Michael Hahn4  \n1Mila & Universit de Montral 2University of Oxford 4 Saarland University  \nAbstract  \nA theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs) . There is much work analyzing the expressivity of attention-based models. By proposing handcrafted weights or using computational complexity arguments, a large amount of past theoretical works have sought to characterize which tasks are and which are not in the hypothesis class of Transformer models. However, little work investigates the learnability of such solutions. In this work, we make progress towards this goal.  \nInspired by recent loss landscape analysis work, we propose preliminary sample complexity bounds for learning C-RASP constructions with Transformers.  \n1 Introduction  \nTheoretical understanding of Transformer models is crucial to better understanding the capacities and limitations of modern large language models (LLMs) . Current theoretical analyses of transformers focus primarily on expressivity, characterizing what these models can or cannot encode in their weights, and situating them in known complexity classes [2, 5, 7, 6] . Notably, RASP [8] and C-RASP [9, 10] established an equivalence between transformers and handcrafted programming languages with specific features. However, much of this work is largely detached from learnability, leaving open the question of how such expressive capabilities can actually be acquired in training. Recent work by [4] suggests that generalization in deep learning can be largely attributed to the volume of ”good” solutions in the loss landscape. This idea is formalized by [3], who show that, within a PAC learning framework, a randomly initialized network generalizes well if there exists a “narrow teacher” network consistent with the labels. They derive a sample complexity bound based on the probability of sampling such a narrow teacher. Building on this framework, we demonstrate how C-RASP Transformers, which define a large class of constructions, can yield sample complexity bounds. These results shed light on how constant-size constructions such as Dyck-1 and an bn are easily learned by Transformers.  \nOur main contributions are as follows: (i) We extend the framework of Buzaglo et al. to Transformers and provide a constraint-counting argument showing how wide student networks can implement narrow teachers. (ii) To the best of our knowledge, our work is among the first to bridge expressivity and learnability by transforming a class of expressivity results into sample complexity results.  \nBackground: A Brief Introduction to C-RASP RASP is a programming language introduced by [8] characterizing the expressivity of Transformers. C-RASP [9] is a variant allowing compilation in future-masked soft-attention Transformers with no restrictions on the input length. C-RASP operations are either boolean (e.g., comparison or logical AND) or pertain to counting (e.g., counting the number of tokens up to position i) . Algorithm 1 gives an example program for recognizing Dyck-1 [9, 1], the language of well-balanced parentheses with 1 parenthesis type.  \n∗ Corresponding author. Contact: [michael.rizvi-martel@mila.quebec](michael.rizvi-martel@mila.quebec)  \n[Preprint.](Preprint.)  \n2 Theoretical Contribution  \nProblem Setup We assume that N sequences of length at most T are sampled i.i.d. from some distribution D s.t. S := {xn }Nn=1 ∼ DN , where xn ∈ V ≤T and V = {1, ... , |V|} is a vocabulary of discrete tokens. We also assume the existence of a teacher model h∗ : V ≤T → {±1} . The true risk of a predictor h : V ≤T → {±1} is LD := Px∼D(h(x)  h∗ (x)) i.e. the probability of having an example for which h and the teacher disagree. The empirical risk of h is","cbCaie9vsItmD6Uf","https://ap.wps.com/l/cbCaie9vsItmD6Uf","pdf",256761,1,9,"English","en",105,"# Abstract\n# Introduction\n## Background: A Brief Introduction to C-RASP\n# Theoretical Contribution\n## Problem Setup\n## Learning Algorithm\n## Teacher-Equivalence and Main Result","[{\"question\":\"What is the central goal of the work on C-RASP and Transformers?\",\"answer\":\"To connect Transformer expressivity results to learnability by deriving sample complexity bounds for learning C-RASP constructions with Transformers.\"},{\"question\":\"What framework is used to motivate narrow teacher learnability?\",\"answer\":\"A PAC-learning perspective where generalization is explained by the presence of a consistent “narrow teacher” network, and sample complexity is tied to the probability of sampling such solutions.\"},{\"question\":\"How do the results explain learning of constant-size constructions like Dyck-1?\",\"answer\":\"They show that C-RASP Transformers can provide sample complexity guarantees, implying that constant-size constructions such as Dyck-1 and certain counting languages are easily learned.\"}]",1784210449,23,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"from-expressivity-to-sample-complexity-narrow-teachers-for-transformers-via-c-rasp","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/from-expressivity-to-sample-complexity-narrow-teachers-for-transformers-via-c-rasp/86316/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the central goal of the work on C-RASP and Transformers?","Question",{"text":75,"@type":76},"To connect Transformer expressivity results to learnability by deriving sample complexity bounds for learning C-RASP constructions with Transformers.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What framework is used to motivate narrow teacher learnability?",{"text":80,"@type":76},"A PAC-learning perspective where generalization is explained by the presence of a consistent “narrow teacher” network, and sample complexity is tied to the probability of sampling such solutions.",{"name":82,"@type":73,"acceptedAnswer":83},"How do the results explain learning of constant-size constructions like Dyck-1?",{"text":84,"@type":76},"They show that C-RASP Transformers can provide sample complexity guarantees, implying that constant-size constructions such as Dyck-1 and certain counting languages are easily learned.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]