[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84424-en":3,"doc-seo-84424-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84424,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Training Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization","We analyze the landscape and training dynamics of diagonal linear networks in a linear regression task when parameters are perturbed by isotropic normal noise during training. This noise can be interpreted as a stochastic sharpness-aware minimization (SAM) method. The perturbations induce a weighted mixture of fractional norm penalties, forcing rapid layer balancing and reshaping the loss landscape to favor shrinkage-thresholding solutions. Balancing layers matches minimizing average sharpness and the Hessian trace across factorization choices, and noise level governs regularization effects on shrinkage, thresholding, and balancing speed.","arXiv :2503 . 1 189 1v2 [ cs .LG] 10 Jul 2026  \nTraining Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization  \nGabriel Clara [gclara@berkeley.edu](gclara@berkeley.edu)  \nSimons Institute for the Theory of Computing University of California, Berkeley  \nBerkeley, CA 94720, United States of America  \nSophie Langer [s.langer@rub.de](s.langer@rub.de)  \nFaculty of Mathematics Ruhr University Bochum 44780 Bochum, Germany  \nJohannes Schmidt-Hieber [a.j.schmidt-hieber@utwente.nl](a.j.schmidt-hieber@utwente.nl)  \nFaculty of Electrical Engineering, Mathematics, and Computer Science University of Twente  \n7522 NB, Enschede, The Netherlands  \nAbstract  \nWe analyze the landscape and training dynamics of diagonal linear networks in a linear regression task, with the network parameters being perturbed by isotropic normal noise during training. The addition of such noise may be interpreted as a stochastic form of sharpness-aware minimization (SAM) and we prove several results that relate its action on the underlying landscape and training dynamics to the sharpness of the loss. In particular, the noise induces a weighted mixture of fractional norm penalties on the network parameters, which forces the individual layers to balance at a fast rate and changes the underlying landscape to favor solutions that result from a shrinkage-thresholding operator applied to the true parameter. We show that balancing the layers equates to minimizing the average sharpness, as well as the trace of the Hessian matrix, among all possible factorizationsof the same linear predictor. Further, we characterize how the noise level of the normal perturbations acts as a regularization parameter, with exact descriptions of its effect on shrinkage, thresholding, and balancing speed.  \nKeywords: sharpness-aware minimization, diagonal linear networks, gradient descent, algorithmic regularization  \n1 Introduction  \nTraining deep neural networks via empirical risk minimization requires solving difficult nonconvex optimization problems (Li et al., 2018) . The computed networks must strike the right balance between minimizing the empirical loss and generalizing well to unseen data. Sharpness-aware minimization (SAM) refers to a family of algorithmic regularization techniques, designed to nudge optimization routines towards flat regions of the underlying loss landscape (Foret et al., 2021) . In the context of training neural networks, flat minima of the empirical loss are thought to generalize well to unseen data, due to robustness against reasonable perturbations of the loss landscape (Hochreiter and Schmidhuber, 1994) . Although empirical studies promise improved model generalization when training with SAM  \n©2026 Gabriel Clara, Sophie Langer, and Johannes Schmidt-Hieber.  \nLicense: CC-BY 4.0, see [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/) .  \nClara, Langer, and Schmidt-Hieber  \n(Dziugaite and Roy, 2017; Foret et al., 2021), its theoretical underpinnings are incomplete (Andriushchenko and Flammarion, 2022; Wen et al., 2023a; Dinh et al., 2017) .  \nSAM acts via minimization of a surrogate loss that explicitly penalizes the sharpness of points on the original loss surface. This penalty can be defined in various ways, leading to distinct variants of SAM. Depending on the data and the model, these variants may generate dissimilar outcomes in practice (Andriushchenko et al., 2023b), highlighting the need for thorough investigations of their action on the underlying loss landscape and the associated training dynamics.  \nFor a more formal discussion, fix a loss function L : Rd → R≥0, for example the empirical loss of a specific model. For a neural network model, the empirical loss surface features many local and global minima (Li et al., 2018) . In particular, while each θ ∗ ∈ arg min L(θ) will fit the observed data well, different choices of θ∗ may generalize better or worse to unseen data. Consequently, we wish to ma","cbCaiheVVpOpi9GK","https://ap.wps.com/l/cbCaiheVVpOpi9GK","pdf",2021859,1,62,"English","en",105,"# Introduction","[{\"question\":\"What problem do the authors study regarding diagonal linear networks?\",\"answer\":\"They study the landscape and training dynamics of diagonal linear networks for linear regression under isotropic normal parameter perturbations during training.\"},{\"question\":\"How does the added isotropic noise relate to sharpness-aware minimization (SAM)?\",\"answer\":\"The noise is interpreted as a stochastic form of SAM, linking its effect on training dynamics to the sharpness of the loss.\"},{\"question\":\"What do the results say about layer balancing and the role of noise level?\",\"answer\":\"The noise induces penalties that force layers to balance quickly; balancing corresponds to minimizing average sharpness and Hessian trace over factorization choices. The noise level acts as a regularization parameter controlling shrinkage, thresholding, and balancing speed.\"}]",1784195551,156,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"training-diagonal-linear-networks-with-stochastic-sharpness-aware-minimization","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/training-diagonal-linear-networks-with-stochastic-sharpness-aware-minimization/84424/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem do the authors study regarding diagonal linear networks?","Question",{"text":75,"@type":76},"They study the landscape and training dynamics of diagonal linear networks for linear regression under isotropic normal parameter perturbations during training.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the added isotropic noise relate to sharpness-aware minimization (SAM)?",{"text":80,"@type":76},"The noise is interpreted as a stochastic form of SAM, linking its effect on training dynamics to the sharpness of the loss.",{"name":82,"@type":73,"acceptedAnswer":83},"What do the results say about layer balancing and the role of noise level?",{"text":84,"@type":76},"The noise induces penalties that force layers to balance quickly; balancing corresponds to minimizing average sharpness and Hessian trace over factorization choices. The noise level acts as a regularization parameter controlling shrinkage, thresholding, and balancing speed.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]