[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85260-en":3,"doc-seo-85260-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85260,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Singular Perturbations and Hierarchical Learning in Two-Layer Neural Networks","We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are jointly optimized while a perturbative parameter controls the relative training speeds of the first and second layers. Building on prior conjectures, the work proves recovery of the constant and linear components of the hidden link function at sharp explicit time thresholds, then analyzes how the quadratic component emerges and reshapes the dynamics.","arXiv :2607 . 10869v 1 [ cs .LG] 12 Jul 2026  \nSINGULAR PERTURBATIONS AND HIERARCHICAL LEARNING IN TWO-LAYER NEURAL NETWORKS  \nCÉDRIC GERBELOT  \nUnité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon  \nJEAN-CHRISTOPHE MOURRAT  \nUnité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon and CNRS  \nAbstract . We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in [13], who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.  \n1. Introduction  \nIn the effort to understand the success of gradient-based optimization methods for deep learning [11], it is crucial to find models that are amenable to theoretical analysis while retaining practical relevance. Among such models, two-layer neural networks remain a significant technical challenge despite a flurry of recent contributions: in the absence of convexity, it is difficult to find analytically tractable approximations of the dynamical system induced by a gradient flow on the parameters. In that regard, several approaches have appeared to go beyond the kernel regime [37, 26] which, despite the appealing simplicity of the ensuing (linear) dynamics, falls short of explaining the ability of neural networks to perform feature learning [21, 55 , 28] . Notably, kernel-based analyses of neural network training dynamics do not show how these architectures adapt to latent low-dimensional structure in the data, thereby avoiding the curse of dimensionality [6, 57 , 29] . We summarize in what follows, without claim of exhaustivity, two of these approaches that motivate the present work.  \nOn the one hand, the series of papers [44, 19 , 48] proposes to study infinitely wide two-layer neural networks and shows that, upon an appropriate scaling of the network and parameter initialization, the evolution of the empirical measure of the weights may be described by a non-linear transport equation corresponding to a Wasserstein gradient flow on a convex functional. Although this approach allows one to provide significant generalization guarantees in certain settings [20], the arguments often rely on infinite training time (see also [54]), and quantitative convergence results still largely rely on Polyak-Lojasiewicz assumptions, see e.g. [18] .  \nOn the other hand, a vast body of work studies the high-dimensional dynamics of various instances of gradient methods on simple problems, with random design data models displaying latent low-dimensional structure. Examples include single [56, 9 , 15 , 13] and multi-index regression [33, 23 , 5 , 2 , 10 , 24 , 14],  \nE-mail addresses: [cedric.gerbelot-barrillon@ens-lyon.fr](cedric.gerbelot-barrillon@ens-lyon.fr) , [jean-christophe.mourrat@ens-lyon.fr](jean-christophe.mourrat@ens-lyon.fr).  \nDate: July 14, 2026 .  \nJCM acknowledges the support of the ERC MSCA grant SLOHD (101203974) .  \n2  \nas well as binary and","cbCaisNbMAJkOoVS","https://ap.wps.com/l/cbCaisNbMAJkOoVS","pdf",1139936,1,37,"English","en",105,"# Introduction\n## Motivation and background\n## Related approaches\n## Challenge of joint two-layer learning","[{\"question\":\"What is the main learning setup studied in the paper?\",\"answer\":\"The paper analyzes the population gradient flow of an infinitely wide two-layer neural network that learns a misspecified single-index model in high dimension, with both layers optimized jointly.\"},{\"question\":\"How does the paper model hierarchical learning between the two layers?\",\"answer\":\"A perturbative parameter tunes the relative training speed between the first and second layer, leading to explicit timescales where different components of the hidden function are recovered.\"},{\"question\":\"What key components of the hidden link function are recovered, and how is the quadratic component treated?\",\"answer\":\"The constant and linear components are recovered within predicted timescales at sharp thresholds, while the onset of learning the quadratic component is analyzed as it continues to influence later dynamics and alters the weight empirical measure behaviour.\"}]",1784202125,93,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"singular-perturbations-and-hierarchical-learning-in-two-layer-neural-networks","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/singular-perturbations-and-hierarchical-learning-in-two-layer-neural-networks/85260/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main learning setup studied in the paper?","Question",{"text":75,"@type":76},"The paper analyzes the population gradient flow of an infinitely wide two-layer neural network that learns a misspecified single-index model in high dimension, with both layers optimized jointly.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the paper model hierarchical learning between the two layers?",{"text":80,"@type":76},"A perturbative parameter tunes the relative training speed between the first and second layer, leading to explicit timescales where different components of the hidden function are recovered.",{"name":82,"@type":73,"acceptedAnswer":83},"What key components of the hidden link function are recovered, and how is the quadratic component treated?",{"text":84,"@type":76},"The constant and linear components are recovered within predicted timescales at sharp thresholds, while the onset of learning the quadratic component is analyzed as it continues to influence later dynamics and alters the weight empirical measure behaviour.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]