[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85376-en":3,"doc-seo-85376-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85376,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks","A theoretical framework explains how Transformer language models develop inductive reasoning abilities. It studies a generalized class of inductive tasks unifying synthetic benchmarks such as in-context n-grams and multi-hop reasoning. The work proves that attention-model training dynamics can be restricted to an interpretable low-dimensional invariant manifold, where learning is described by a small set of coordinates. It characterizes how data statistics drive competition between in-context learning and in-weights learning, how random initialization selects winning circuits, and how the manifold’s coordinate system detects learned circuits.","Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks  \nTiberiu Musat∗ ETH Zurich  \nTiago Pimentel  \nETH Zurich  \nNicholas Zucchet  \nStanford  \nThomas Hofmann  \nETH Zurich  \narXiv :2607 . 1 1875v 1 [ cs .LG] 13 Jul 2026  \nAbstract  \nWe present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis moretractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the ‘winning’ circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.  \nInvariant Manifold of Inductive Reasoning  \nFigure 1: The Invariant Manifold of Inductive Reasoning (IMIR) is an interpretable low-dimensional subspace of the parameter space which training trajectories never leave. Each induction circuit resides and evolves within a subspace of the IMIR, depicted schematically as a colored plane.  \n1 Introduction  \nWhile great progress has been made toward understanding the circuits (i.e. subnetworks, or weight structures) present in large language models [1–4], we are still far from understanding the learning dynamics that underlie their formation. As a result, skill acquisition in language models remains mostly unpredictable [5], which complicates model development and poses AI safety issues. A better understanding of circuit emergence could reduce these risks, provide practical insights into how to interpret the internal mechanics of trained models, as well as help design better training pipelines [6] . In this work, we focus on the emergence of circuits responsible for inductive abilities: general-purpose  \n∗ Correspondence [at](at tiberiu@musat.ai)[ tiberiu@musat.ai](at tiberiu@musat.ai), [tiago.pimentel@inf.ethz.ch](tiago.pimentel@inf.ethz.ch)  \nPreprint.  \ncapabilities that allow a language model to recognize patterns, infer rules, and perform abstract reasoning on the fly based on the provided context, rather than relying on static, memorized patterns. Inductive abilities of transformers have been studied extensively by prior work, both empirically and theoretically. However, inductive abilities are typically studied in isolation, limited to specific tasks such as in-context n-grams [7–9] or k-hop induction [10–13] . Without a unifying theoretical framework, our understanding progresses slowly, and three important phenomena remain particularly puzzling. 1 Circuit competition is a fundamental aspect of language model training [14–16] . However, most previous theoretical works assume staged learning algorithms where only one component is trained at a time, while others are kept frozen [17, 18] . A few works study the dynamical interaction of different components during training, but are limited to small transformers with only one [19] or two [20] attention heads. 2 Data distributional properties strongly influence the emergence of in-context learning [21–23], but a mechanistic understanding is currently limited to phenomenological models [24] . 3 Randomly initialized networks cont","cbCairt5ZWMdDQ15","https://ap.wps.com/l/cbCairt5ZWMdDQ15","pdf",1769284,1,42,"English","en",105,"# Abstract\n# Invariant Manifold of Inductive Reasoning\n# 1 Introduction\n# Circuit Competition and Applications","[{\"question\":\"What is the Invariant Manifold of Inductive Reasoning (IMIR)?\",\"answer\":\"IMIR is an interpretable low-dimensional subspace of the model’s parameter space where training trajectories remain under gradient descent. Each inductive circuit evolves inside a subspace of IMIR rather than requiring full high-dimensional parameter movement.\"},{\"question\":\"How does the framework unify different inductive reasoning tasks?\",\"answer\":\"It analyzes a generalized class of inductive tasks that includes synthetic task families reported in prior literature, such as in-context n-grams and multi-hop reasoning. This unification enables consistent theoretical treatment across these task types.\"},{\"question\":\"What mechanisms determine circuit competition during training?\",\"answer\":\"The document explains that data statistics govern the competition between in-context learning and in-weights learning. It also states that random initialization can determine which circuit “wins,” with sharp phase transitions between initialization regions favoring different solutions.\"}]",1784202972,106,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"invariant-learning-dynamics-of-transformers-in-inductive-reasoning-tasks","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/invariant-learning-dynamics-of-transformers-in-inductive-reasoning-tasks/85376/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the Invariant Manifold of Inductive Reasoning (IMIR)?","Question",{"text":75,"@type":76},"IMIR is an interpretable low-dimensional subspace of the model’s parameter space where training trajectories remain under gradient descent. Each inductive circuit evolves inside a subspace of IMIR rather than requiring full high-dimensional parameter movement.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the framework unify different inductive reasoning tasks?",{"text":80,"@type":76},"It analyzes a generalized class of inductive tasks that includes synthetic task families reported in prior literature, such as in-context n-grams and multi-hop reasoning. This unification enables consistent theoretical treatment across these task types.",{"name":82,"@type":73,"acceptedAnswer":83},"What mechanisms determine circuit competition during training?",{"text":84,"@type":76},"The document explains that data statistics govern the competition between in-context learning and in-weights learning. It also states that random initialization can determine which circuit “wins,” with sharp phase transitions between initialization regions favoring different solutions.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]