[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-290371-105":59,"doc-detail-290371-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","a-dynamical-systems-perspective-on-machine-learning-optimization","A Dynamical Systems Perspective on Machine Learning Optimization","","The thesis studies how to understand optimization algorithms at the heart of machine learning by recasting gradient-based training as discrete-time dynamical systems in parameter space. It contrasts classical optimization theory with dynamical systems analysis and shows that this viewpoint enables deeper study of training dynamics beyond standard metrics, including multi-agent settings, difficult-to-characterize objectives, and convergence analysis. The work develops three complementary frameworks for learning representations, differential Nash equilibria, and robust control under minibatch noise and nonlinear dynamics.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-dynamical-systems-perspective-on-machine-learning-optimization/290371/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-dynamical-systems-perspective-on-machine-learning-optimization/290371.png","ImageObject",300,407,{"name":92,"@type":93},"Eliana","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-04","2026-09-17",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",14,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"Why does the thesis frame machine learning optimization as a dynamical systems problem?","Question",{"text":113,"@type":114},"It treats gradient-based optimizers as discrete-time dynamical systems whose update rules evolve parameters over time, enabling dynamical tools to analyze training dynamics and convergence beyond conventional metrics.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What is studied about learning meaningful representations in the thesis?",{"text":118,"@type":114},"The thesis uses linear autoencoders as a tractable model to analyze symmetry breaking, comparing regularization-based approaches with a proposed rotation-augmented gradient that yields stronger symmetry breaking and faster convergence.",{"name":120,"@type":111,"acceptedAnswer":121},"How does the thesis address optimization in multi-agent settings?",{"text":122,"@type":114},"It proposes double Follow-the-Ridge for general-sum differentiable games and provides local convergence guarantees, showing convergence to and only to differential Nash equilibria.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},290371,1789641512,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145,"preview_page_count":4},4398048949847,"https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267","A DYNAMICAL SYSTEMS PERSPECTIVE ON MACHINE LEARNING OPTIMIZATION  \nby  \nXuchan Bao  \nA thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy  \nDepartment of Computer Science  \nUniversity of Toronto  \n© Copyright 2026 by Xuchan Bao  \nA Dynamical Systems Perspective on Machine Learning Optimization  \nXuchan Bao  \nDoctor of Philosophy  \nDepartment of Computer Science  \nUniversity of Toronto  \n2026  \nAbstract  \nThe rapid advancement of machine learning has been enabled in large part by effective optimization algorithms for training increasingly complex models. Given the central role of optimization in machine learning, developing deeper understanding of these algorithms is essential. Traditionally, studies of optimization in machine learning have drawn extensively from classical optimization theory, applying tools such as convex analysis, Lipschitz smoothness assumptions, and convergence rate bounds to understand neural network training behavior. Meanwhile, dynamical systems analysis—which shares fundamental connections with optimization—has developed in parallel, with widespread applications across physics, engineering, economics, and biology to analyze complex iterative processes and temporal evolution. Despite their close relationship, these fields evolved largely independently with different focuses and techniques.  \nViewing machine learning optimization from a dynamical systems perspective offers significant advantagesand new insights. By treating gradient-based optimizers as discrete-time dynamical systems with update rules acting on parameter spaces, we can apply powerful analytical tools from dynamical systems theory to study neural network training. This perspective enables training dynamics analysis beyond conventional optimization metrics, extends naturally to multi-agent settings with potentially conflicting objectives, allows analysis of optimization procedures where explicit objective functions are difficult to characterize, and provides systematic approaches for analyzing convergence properties under diverse problem conditions.  \nThis thesis views machine learning optimization from the dynamical systems perspective through three complementary angles. First, we study how neural networks learn meaningful representations using linear autoencoders as a tractable model. We analyze different approaches to symmetry breaking, showing that regularization-based methods provide weak symmetry breaking, whereas the proposed rotation-augmented gradient achieves stronger symmetry breaking and faster convergence to desired representations. Second, we examine multi-agent optimization, which is significantly more complex than single-objective optimization. We propose double Follow-the-Ridge for finding differential Nash equilibria in general-sum differentiable games and provide the first local convergence guarantees for general-sum games, ensuring local convergence to and only to differential Nash equilibria. Third, we extend a recent robust control framework for analyzing machine learning optimization to handle neural network training with minibatch noise and nonlinear dynamics.  \nOur framework reveals new insights into important differences between momentum methods that align with empirical observations.  \nTo my parents, Danhua and Xudong  \nAcknowledgements  \nI want to thank my parents Danhua and Xudong for their unconditional support and encouragement throughout my life. Being scientists yourselves, you planted in me the curiosity and respect for science at an early age. Growing up, you have always respected my interests and choices, offering advice when I needed it. To me, home is a safe space with like-minded friends. I could not have asked for better parents.  \nI want to thank my partner and best friend Cem. We shared everything, our PhD journeys being one of them. The journey must have been tough, but with you by my side, every obstacle felt smaller. It is incredibly rare to find someone w","cbCaimsi5C1dDeUS","https://ap.wps.com/l/cbCaimsi5C1dDeUS","pdf",6691313,144,"English","# Abstract\n## Motivation and Background\n## Dynamical Systems Viewpoint\n## Three Complementary Angles\n## Conclusion of Contributions","[{\"question\":\"Why does the thesis frame machine learning optimization as a dynamical systems problem?\",\"answer\":\"It treats gradient-based optimizers as discrete-time dynamical systems whose update rules evolve parameters over time, enabling dynamical tools to analyze training dynamics and convergence beyond conventional metrics.\"},{\"question\":\"What is studied about learning meaningful representations in the thesis?\",\"answer\":\"The thesis uses linear autoencoders as a tractable model to analyze symmetry breaking, comparing regularization-based approaches with a proposed rotation-augmented gradient that yields stronger symmetry breaking and faster convergence.\"},{\"question\":\"How does the thesis address optimization in multi-agent settings?\",\"answer\":\"It proposes double Follow-the-Ridge for general-sum differentiable games and provides local convergence guarantees, showing convergence to and only to differential Nash equilibria.\"}]","A Dynamical Systems Perspective on Machine Learning Optimization | PDF",363]