[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121658-en":3,"doc-seo-121658-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121658,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","Improving Energy Conserving Descent for Machine Learning - Theory and Practice","Energy Conserving Descent (ECD) is developed as a theoretical optimization framework and ECDSep, a gradient-based algorithm, is proposed for both convex and non-convex problems. ECD recasts optimization as the physical evolution of a chaotic, energy-conserving dynamical system, enabling analytic control of result distributions, especially for low-loss regions in generic high-dimensional settings without symmetries. The study refines dynamics and chaos mechanisms, reduces hyper-parameter tuning, and shows competitive or improved performance against SGD, Adam, and AdamW across diverse tasks while outlining analysis limitations and directions for further gains.","Improving Energy Conserving Descent for Machine Learning: Theory and Practice  \narXiv :2306 .00352v 1 [ cs .LG] 1 Jun 2023  \nG. Bruno De Luca∗1 [gbdeluca@stanford.edu](gbdeluca@stanford.edu)  \nAlice Gatti∗ [agatti0@proton.me](agatti0@proton.me)  \nEva Silverstein∗1  \n[evas@stanford.edu](evas@stanford.edu)  \n1 Stanford Institute for Theoretical Physics, Stanford University, Stanford, CA 94306, USA  \nAbstract  \nWe develop the theory of Energy Conserving Descent (ECD) and introduce ECDSep, a gradient-based optimization algorithm able to tackle convex and non-convex optimization problems. The method is based on the novel ECD framework of optimization as physical evolution of a suitable chaotic energy-conserving dynamical system, enabling analytic control of the distribution of results – dominated at low loss – even for generic high-dimensional problems with no symmetries. Compared to previous realizations of this idea, we exploit the theoretical control to improve both the dynamics and chaos-inducing elements, enhancing performance while simplifying the hyper-parameter tuning of the optimization algorithm targeted to different classes of problems. We empirically compare with popular optimization methods such as SGD, Adam and AdamW on a wide range of machine learning problems, finding competitive or improved performance compared to the best among them on each task. We identify limitations in our analysis pointing to possibilities for additional improvements.  \n1 Introduction  \nThe novel Energy Conserving Descent (ECD) framework for optimization introduced in [6] was shown in small experiments to be competitive with standard algorithms such as stochastic gradient descent with momentum (SGDM) while offering a theoretically predictable distribution of optimization results. Our contribution is to leverage  \n∗ Equal contribution.  \nand improve this theoretical understanding to systematically enhance the performance and demonstrate an overall advantage compared to standard methods (Adam [18], AdamW [25] and SGDM [29]) on a diverse suite of small/medium-scale machine learning benchmarks. This succeeds via a new parameter enabling stronger concentration of the results at small loss, combined with the use of a separable Hamiltonian as in previously developed sampling versions of the algorithm [27 , 28], which allow a more effective integration. Overall in our experiments, ECD is competitive with the better of Adam and SGDM, without need for learning rate (lr) scheduling required by SGDM for competitive performance. Along the way we describe limitations and derive theoretical predictions for further improvement.  \n2 ECD Theory and formulaic improvements  \nECD is formulated, in analogy to classical mechanics in physics, as a discretization of chaotic energy-conserving Hamiltonian evolution on a 2n-dimensional phase space of positions Θ ([e.g. an](e.g. an) n-dimensional space of neural network weights and biases) and momenta Π . The distribution of results in Θ – either along a given trajectory, or among multiple trajectories – is given by  \nµ (Θ) = Z dn Πδ(H(Π , Θ) − E) = ΩnΠ|~~ ~~ΠH|~~ ~~n|~~ ~~−1 = Ωn|~~ ~~1Θ|~~ ~~Π/d|~~ ~~nt1 , (1)  \nwhere E is a constant, the conserved energy, and H (Π , Θ) is a time-independent Hamiltonian function which depends on the objective function F (Θ) and on the magnitude |Π| of the momentum. The continuum evolution equations are dΠ/dt = −∂H/∂Θ, dΘ/dt =∂H/∂Π . An example we will focus on is a Hamiltonian describing the kinetic energy of a particle with Θ-dependent mass ∼ 1/V[F(Θ)] [6 , 27 , 28]  \nH (Π , Θ) = V[F(Θ)]Π2 = 1VΘ˙2 (2)  \nV [F(Θ)] ∝ (F(Θ) − F0 )η , η ⩾ 1 (3)  \nalong with energy-conserving momentum rotations to enhance chaotic behavior [6](see §3.2) . The continuum-equivalent dynamics obtained from the logarithm of (2) separates the position and momentum dependencies allowing more robust numerical integration [27 , 28] . For (2), the measure (1) gives  \nµ (Θ) = E (n−2)/2 Γπ(nn//22)V [F(Θ)]−n/2 . (4)  \nWe","cbCailou8D6L8aEM","https://ap.wps.com/l/cbCailou8D6L8aEM","pdf",1413157,1,28,"English","en",105,"# Abstract\n# Introduction\n# ECD Theory and formulaic improvements","[{\"question\":\"What is Energy Conserving Descent (ECD) and how does it relate to optimization?\",\"answer\":\"ECD frames optimization as a physical evolution of a chaotic, energy-conserving dynamical system. The resulting framework controls the distribution of optimization outcomes in a theoretically predictable way.\"},{\"question\":\"What problems can ECDSep solve?\",\"answer\":\"ECDSep is designed to handle both convex and non-convex optimization problems, supported by the energy-conserving chaotic evolution viewpoint.\"},{\"question\":\"How does ECD compare with common optimizers like SGD and Adam?\",\"answer\":\"Experiments compare ECD against SGD, Adam, and AdamW across many machine learning tasks. Results are competitive or improved relative to the best baseline on each task, with notes on limitations and possible further improvements.\"}]","Improving Energy Conserving Descent for Machine Learning - Theory and Practice | PDF",1785805999,71,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"improving-energy-conserving-descent-for-machine-learning-theory-and-practice","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/improving-energy-conserving-descent-for-machine-learning-theory-and-practice/121658/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is Energy Conserving Descent (ECD) and how does it relate to optimization?","Question",{"text":75,"@type":76},"ECD frames optimization as a physical evolution of a chaotic, energy-conserving dynamical system. The resulting framework controls the distribution of optimization outcomes in a theoretically predictable way.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What problems can ECDSep solve?",{"text":80,"@type":76},"ECDSep is designed to handle both convex and non-convex optimization problems, supported by the energy-conserving chaotic evolution viewpoint.",{"name":82,"@type":73,"acceptedAnswer":83},"How does ECD compare with common optimizers like SGD and Adam?",{"text":84,"@type":76},"Experiments compare ECD against SGD, Adam, and AdamW across many machine learning tasks. Results are competitive or improved relative to the best baseline on each task, with notes on limitations and possible further improvements.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]