[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84423-en":3,"doc-seo-84423-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84423,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers","Transformers can show two-stage training dynamics during real-world optimization: for example, training GPT-2 on the Counterfact dataset transitions from syntactically incorrect answers to syntactically correct but semantically incorrect ones, and finally to predominantly semantically correct outputs. Existing theory rarely explains this feature-level phenomenon. This paper provides a rigorous mathematical analysis by modeling disentangled syntax–semantics feature structure in a simplified transformer setting, and derives conditions connecting the two-stage process to attention-weight spectral properties, yielding a first provable feature-level two-stage optimization result.","arXiv :2502 .2068 1v 3 [ cs .CL] 11 Jul 2026  \nDisentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers  \nZixuan Gong, Shijia Li, Yong Liu, Jiaye Teng  \nAbstract  \nTransformers may exhibit two-stage training dynamics during the real-world training process. For instance, when training GPT-2 on the Counterfact dataset, the answers progress from syntactically incorrect to syntactically correct to semantically correct. However, existing theoretical analyses hardly account for this feature-level two-stage phenomenon, which could be conceptually attributed to disentangled two-type features like syntax and semantics. In this paper, we theoretically demonstrate how the twostage training dynamics potentially occur in transformers. Specifically, we analyze the feature learning dynamics induced by the aforementioned disentangled two-type feature structure, grounding our analysis in a simplified yet illustrative setting that comprises normalized ReLU self-attention and structured data. Such disentanglement of feature structure is general in practice, e.g., natural languages contain syntax and semantics, and proteins contain primary and secondary structures. To our best knowledge, this is the first rigorous result regarding a feature-level two-stage optimization process in transformers within this theoretical framework. A corollary further indicates that such a two-stage process is closely related to the spectral properties of attention weights.  \nIndex Terms  \nFeature-level two-stage learning, Optimization dynamics, Finite-time convergence, Feature learning, Feature disentanglement.  \nI. INTRODUCTION  \nTRansformers [1] have emerged as foundational architectures with broad applications across multiple research domains,  \nsuch as natural language processing [2], [3], computer vision [4], [5], etc. Recently, large language models (LLM) based on decoder-only transformer architectures further demonstrate impressive capabilities, excelling in various downstream tasks [6],[7] . However, it remains an essential issue to delve into why LLMs exhibit such remarkable performance. Fortunately, exploring the optimization dynamics in transformers presents a promising approach for investigating the possible factors that contribute to this behavior.  \nEmpirically, it is widely known that transformers exhibit staged learning behaviors. For instance, when fine-tuning GPT-2 on the Counterfact dataset in Figure 1 (details in Appendix D-B), we observe the following phenomenon: at initial (epoch 1), most predictions are both syntactically and semantically incorrect. At midpoint (epoch 5), we observe a significant decrease in training loss; all predictions meet syntactic requirements, but most remain semantically incorrect and inconsistent with the true answers. At convergence (epoch 100), all predictions are syntactically correct, with most being semantically correct and achieving a small training loss. Overall, the model’s answers progress from syntactically incorrect to syntactically correct to semantically correct, exhibiting two-stage training dynamics for syntactic and semantic information.  \nMotivated by this phenomenon, for various tasks like language tasks, protein structure prediction tasks, or classic supervised learning tasks, we can disentangle feature structure into two types: elementary knowledge (like syntactic information), and specialized knowledge (like semantic information) . Such disentanglement is empirically general in both NLP and biological research [8]–[12] . Additionally, the corresponding two-stage learning process has been revealed in vision field [13] . Based on the above discussion, it is natural to infer that knowledge may be acquired following an elementary-then-specialized principle. However, this leaves the following critical theoretical question:  \nThe key question:  \nHow does the disentangled two-type feature structure theoretically induce the  \nfeature-level two-stage training dy","cbCairn2A23359of","https://ap.wps.com/l/cbCairn2A23359of","pdf",2870784,1,47,"English","en",105,"# Introduction\n## Two-stage training behavior in practice\n## Feature disentanglement motivation and key theoretical question","[{\"question\":\"What two-stage training dynamics do the authors observe in transformers?\",\"answer\":\"During training, predictions first shift from syntactically and semantically incorrect to syntactically correct while still semantically inconsistent, and finally to mostly syntactically and semantically correct with low loss.\"},{\"question\":\"How does the paper explain the feature-level two-stage phenomenon theoretically?\",\"answer\":\"It models feature learning under a disentangled two-type feature structure, treating elementary (e.g., syntax) and specialized (e.g., semantics) features as distinct learning components within a simplified transformer framework.\"},{\"question\":\"What connection does the paper make between two-stage learning and attention weights?\",\"answer\":\"A corollary shows that the two-stage process is closely related to the spectral properties of the attention weights.\"}]",1784195542,118,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"disentangling-feature-structure-a-mathematically-provable-two-stage-training-dynamics-in-transformers","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/disentangling-feature-structure-a-mathematically-provable-two-stage-training-dynamics-in-transformers/84423/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What two-stage training dynamics do the authors observe in transformers?","Question",{"text":75,"@type":76},"During training, predictions first shift from syntactically and semantically incorrect to syntactically correct while still semantically inconsistent, and finally to mostly syntactically and semantically correct with low loss.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the paper explain the feature-level two-stage phenomenon theoretically?",{"text":80,"@type":76},"It models feature learning under a disentangled two-type feature structure, treating elementary (e.g., syntax) and specialized (e.g., semantics) features as distinct learning components within a simplified transformer framework.",{"name":82,"@type":73,"acceptedAnswer":83},"What connection does the paper make between two-stage learning and attention weights?",{"text":84,"@type":76},"A corollary shows that the two-stage process is closely related to the spectral properties of the attention weights.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]