[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83374-en":3,"doc-seo-83374-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83374,1099514068365,"Aurelia","https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?_k=1776742907772140068",8,"Research & Report","Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima","Gradient descent theory highlights sharpness, defined as the largest eigenvalue of the objective Hessian, with classical stability requiring step size uniformly below twice the reciprocal of sharpness; deep neural network training often violates this. Prior work established a normal form for large-step GD near an isolated flat minimum in overparametrised least-squares with a single scalar output. This paper extends the framework to vector-valued overparametrised least-squares and to neighbourhoods of manifolds of flat minima, covering applications such as matrix factorisation.","arXiv :2607 .08380v 1 [ cs .LG] 9 Jul 2026  \nDynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima  \nLachlan Ewen MacDonald René Vidal  \nInnovation in Data Engineering and Science (IDEAS)  \nUniversity of Pennsylvania  \nPennsylvania, PA 19104  \n[lemacdonald@protonmail.com](lemacdonald@protonmail.com)  \nAbstract  \nAn important quantity in the theory of gradient descent (GD) is the sharpness, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work [39] bridges this gap in the setting of overparametrised leastsquares with a single scalar output, providing a normal form for large-step GD in a neighbourhood of an isolated flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with vector-valued outputs (including regression with arbitrarily many observations), and (2) to a neighbourhood of a manifold of flat minima (which we show is essential for applications such as matrix factorisation) .  \nWe generalise both the normal form and all three convergence theorems of [39] to this broader setting, overcoming several technical challenges, including the solution of a singular partial differential equation via a novel method that may be of independent interest. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.  \n1 Introduction  \nA natural way of finding a minimum of a smooth function ℓ : Rp → R is by iterating the gradient descent (GD) map  \nx →7 x − η∇ℓ(x) (1)  \nfor some choice of step size η > 0. Originally introduced by Cauchy almost two centuries ago [10], the method has more recently found success as the quintessential technique for training deep neural networks (DNNs) . Unfortunately, despite its age, fame and simplicity, GD remains poorly understood in its application to deep learning (DL) .  \nA critical quantity in the theory of GD is the sharpness, i.e. the largest eigenvalue λ1 of the Hessian of ℓ . Classical theories of the convergence of GD typically assume both convexity of the objective ℓ and the pointwise stability condition  \nη \u003C 2/λ1 (x), (2)  \nwithout which GD diverges in the case of quadratic objectives. Both of these conditions are typically violated in the training of DNNs in practice, but convergence to a global minimiser is frequently observed nonetheless. The use of large step size in particular is often observed to accelerate convergence [13] and results in an implicit bias toward “flat minima”(i.e. minima with small sharpness  \nPreprint.  \nλ 1 ), which have been associated with better statistical performance [31] . Although tools from optimisation theory have shown some success in removing the stability condition (2) in convex settings [23, 24, 25, 4, 5, 52, 51, 53], convergence proofs with large step size in the non-convex settings of DL have stubbornly resisted analysis by these means.  \nIt is gradually becoming clear that analysis of GD in the regimes appropriate to DL instead require the tools of dynamical systems theory [35, 15, 39] . Of special significance for the present work is [39], which vindicates the dynamical systems approach in giving quantitative convergence theorems for gradient descent with large step size in “codimension 1” least squares problems, corresponding toa single scalar output and (p − 1)-dimensional manifold M ⊂ Rp of minimisers. Since extension of the results of [39] is the central concern of the present paper, we briefly recall them here. The contributions of [39]: A normal form is a change of c","cbCaifFQqu2khsDp","https://ap.wps.com/l/cbCaifFQqu2khsDp","pdf",1172605,1,56,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"What does “sharpness” mean in the gradient descent analysis discussed in the paper?\",\"answer\":\"Sharpness is defined as the largest eigenvalue of the objective Hessian. It determines stability thresholds for the gradient descent step size.\"},{\"question\":\"Why do classical convergence conditions fail in deep neural network training?\",\"answer\":\"Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of sharpness and often assume convexity, but deep networks commonly violate these assumptions in practice.\"},{\"question\":\"What new problem setting does the paper extend beyond prior work?\",\"answer\":\"It generalises large-step GD near isolated flat minima to (1) overparametrised least-squares with vector-valued outputs and (2) neighbourhoods of a manifold of flat minima, which is important for applications such as matrix factorisation.\"}]",1784187073,141,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"dynamics-of-gradient-descent-with-large-step-size-near-a-manifold-of-flat-minima","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/dynamics-of-gradient-descent-with-large-step-size-near-a-manifold-of-flat-minima/83374/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What does “sharpness” mean in the gradient descent analysis discussed in the paper?","Question",{"text":75,"@type":76},"Sharpness is defined as the largest eigenvalue of the objective Hessian. It determines stability thresholds for the gradient descent step size.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why do classical convergence conditions fail in deep neural network training?",{"text":80,"@type":76},"Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of sharpness and often assume convexity, but deep networks commonly violate these assumptions in practice.",{"name":82,"@type":73,"acceptedAnswer":83},"What new problem setting does the paper extend beyond prior work?",{"text":84,"@type":76},"It generalises large-step GD near isolated flat minima to (1) overparametrised least-squares with vector-valued outputs and (2) neighbourhoods of a manifold of flat minima, which is important for applications such as matrix factorisation.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]