[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123456-en":3,"doc-seo-123456-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123456,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Incremental Gauss-Newton Descent for Machine Learning - Paper","Stochastic Gradient Descent (SGD) is widely used in machine learning but can struggle with slow convergence in flat regions of the loss landscape. This work introduces Incremental Gauss-Newton Descent (IGND), a modification of SGD that exploits approximate second-order information using the Gauss-Newton approximation. IGND keeps essentially the same computational cost as SGD, converges faster on certain problem classes, can be accelerated, and is easier to tune due to a scalar scaling effect. Results show IGND can outperform SGD in simulations and performs at least as well in worst cases, applied to supervised and reinforcement learning.","Incremental Gauss-Newton Descent for Machine Learning  \nMikalai Korbit† Mario Zanon†  \narXiv :2408 .05560v1 [ cs .LG] 10 Aug 2024  \nDYSCO (Dynamical Systems, Control, and Optimization)†  \nIMT School for Advanced Studies Lucca, Italy  \nAugust 13, 2024  \nAbstract  \nStochastic Gradient Descent (SGD) is a popular technique used to solve problems arising in machine learning. While very effective, SGD also has some weaknesses and various modifications of the basic algorithm have been proposed in order to at least partially tackle them, mostly yielding accelerated versions of SGD. Filling a gap in the literature, we present a modification of the SGD algorithm exploiting approximate second-order information based on the Gauss-Newton approach. The new method, which we call Incremental Gauss-Newton Descent (IGND), has essentially the same computational burden as standard SGD, appears to converge faster on certain classes of problems, and can also be accelerated. The key intuition making it possible to implement IGND efficiently is that, in the incremental case, approximate second-order information can be condensed into a scalar value that acts as a scaling constant of the update. We derive IGND starting from the theory supporting Gauss-Newton methods in a general setting and then explain how IGND can also be interpreted as a well-scaled version of SGD, which makes tuning the algorithm simpler, and provides increased robustness. Finally, we show how IGND can be used in practice by solving supervised learning tasks as well as reinforcement learning problems. The simulations show that IGND can significantly outperform SGD while performing at least as well as SGD in the worst case.  \n1 Introduction  \nMany problems in Machine Learning (ML) are formulated as optimization problems that can be solved by gradient-based methods. Stochastic Gradient Descent (SGD) is a first-order stochastic optimization algorithm formally introduced by [35] which has become the backbone optimization algorithm for many machine learning tasks. While being effective and computationally simple, SGD can in some cases converge slowly in the flat regions of the loss function landscape [22] . Many acceleration methods have been proposed to overcome this issue, among others, momentum-based approaches [7, 23 , 29 , 38], adaptive learning rates [10, 14 , 49] or a combination of the two [9, 18 , 46] . Recent successes in training very large neural networks can be partially attributed to the advances in such optimization algorithms [42] .  \nAnother approach to improve first-order methods consists in (approximately) accounting for secondorder information [45], such that the descent direction is calculated by solving the linear system Hd =−g where H is the (approximate) Hessian matrix and g is the gradient of the loss function. The adaptation of these algorithms to ML problems is challenging due to the high dimensionality of the search space, which makes it hard to compute and store second-order sensitivities as well as to solve the (typically dense) linear system, which scales cubically with the dimension of the search space. This is particularly unfortunate since second-order derivatives carry information about the curvature of the loss  \nlandscape which can yield faster convergence, and these methods are scale-invariant, i.e., more robust and easier to tune.  \nMotivated by the observations above, the so-called Gauss-Newton (GN) Hessian approximation scheme [30] has been re-examined for application in the stochastic setting. The most promising approaches include the stochastic generalized Gauss-Newton method (SGN) [12] that couples a regularized Levenberg-Marquardt Hessian approximation with a Conjugate Gradient (CG) solver, the Nonlinear Least Squares Rank-L method (NLLSL) [6] and SGN2 [40] that estimate the Jacobian with a rank-L approximation and SARAH estimate respectively, as well as SMW-GN [34] that uses the ShermanMorrison-Woodbury formula to efficiently solve the linea","cbCaieVoZcarOJW6","https://ap.wps.com/l/cbCaieVoZcarOJW6","pdf",2178887,1,27,"English","en",105,"# Abstract\n# 1 Introduction\n# 2 Related Work","[{\"question\":\"What problem does IGND address in SGD-based machine learning optimization?\",\"answer\":\"IGND targets cases where SGD converges slowly, especially in flat regions of the loss landscape, by improving the descent direction using approximate second-order information.\"},{\"question\":\"How does IGND incorporate second-order information while keeping SGD-like cost?\",\"answer\":\"IGND uses a Gauss-Newton-based approximation in the incremental setting and condenses the approximate second-order information into a scalar that scales the update.\"},{\"question\":\"Where can IGND be applied and how does it compare to SGD?\",\"answer\":\"IGND can be used for supervised learning and reinforcement learning. Numerical experiments indicate it can outperform SGD in some cases while performing at least as well as SGD in the worst case.\"}]","Incremental Gauss-Newton Descent for Machine Learning - Paper | PDF",1785816619,68,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"incremental-gauss-newton-descent-for-machine-learning-paper","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/incremental-gauss-newton-descent-for-machine-learning-paper/123456/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does IGND address in SGD-based machine learning optimization?","Question",{"text":75,"@type":76},"IGND targets cases where SGD converges slowly, especially in flat regions of the loss landscape, by improving the descent direction using approximate second-order information.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does IGND incorporate second-order information while keeping SGD-like cost?",{"text":80,"@type":76},"IGND uses a Gauss-Newton-based approximation in the incremental setting and condenses the approximate second-order information into a scalar that scales the update.",{"name":82,"@type":73,"acceptedAnswer":83},"Where can IGND be applied and how does it compare to SGD?",{"text":84,"@type":76},"IGND can be used for supervised learning and reinforcement learning. Numerical experiments indicate it can outperform SGD in some cases while performing at least as well as SGD in the worst case.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]