[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123030-en":3,"doc-seo-123030-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123030,8796095461564,"Liam","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Stochastic Optimization for Machine Learning: Investigations on Bilevel Optimization and Large Learning Rates - Dissertation","Stochastic optimization underpins modern machine learning by enabling efficient loss minimization on large-scale datasets. This dissertation studies theoretical properties and empirical behavior of bilevel optimization methods, together with the emergence of large learning rates in learning dynamics. A new Moving-Average Stochastic Bilevel Algorithm (MA-SOBA) is proposed for stochastic bilevel optimization under standard smoothness assumptions. The work further develops a multi-agent decentralized setting via a Moving-Average Decentralized Stochastic Bilevel Optimization (MA-DSBO) algorithm, improving per-iteration complexity by reducing quadratic dependence on dimension to linear dependence. Finally, training dynamics of gradient descent are analyzed in quadratic regression models using large learning rates, motivated by the Edge of Stability phenomenon.","UC Davis  \nUC Davis Electronic Theses and Dissertations  \nTitle  \nStochastic Optimization for Machine Learning: Investigations on Bilevel Optimization and Large Learning Rates  \nPermalink  \n[https://escholarship.org/uc/item/4h6002gb](https://escholarship.org/uc/item/4h6002gb)  \nAuthor  \nChen, Xuxing  \nPublication Date  \n2024  \nPeer reviewed|Thesis/dissertation  \n[eScholarship.org](eScholarship.org) Powered by the California Digital Library  \nUniversity of California  \nStochastic Optimization for Machine Learning: Investigations on Bilevel Optimization and Large  \nLearning Rates  \nBy  \nXUXING CHEN  \nDISSERTATION  \nSubmitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY  \nin  \nApplied Mathematics  \nin the  \nOFFICE OF GRADUATE STUDIES  \nof the  \nUNIVERSITY OF CALIFORNIA  \nDAVIS  \nApproved:  \n\n| Krishnakumar Balasubramanian, Chair |\n| --- |\n| Jesús De Loera |\n| Mina Karzand |\n\nShiqian Ma Committee in Charge 2024  \n© Xuxing Chen, 2024 . All rights reserved.  \nTo my family  \nii  \nContents  \nAbstract v  \nAcknowledgments vi  \nChapter 1 . Introduction 1  \n1.1. Preliminaries 1  \n1.2. Outline of the Dissertation 7  \nChapter 2 . Stochastic Bilevel Optimization 9  \n2.1. Introduction 9  \n2.2. Proposed Framework: the MA-SOBA Algorithm 14  \n2.3. Theoretical Analysis 16  \n2.4. Min-Max Bilevel Optimization 19  \n2.5. Experiments 22  \n2.6. Conclusion 32  \nChapter 3 . Decentralized Stochastic Bilevel Optimization 33  \n3.1. Introduction 33  \n3.2. Preliminaries 37  \n3.3. DSBO Algorithm with Improved Per-Iteration Complexity 39  \n3.4. Numerical experiments 46  \n3.5. Conclusion 48  \nChapter 4 . Training Dynamics of Gradient Descent for Quadratic Regression 49  \n4.1. Introduction 49  \n4.2. Analyzing a discrete dynamical system with cubic map 54  \n4.3. Applications to quadratic regression models 60  \n4.4. Experimental investigations 64  \n4.5. Conclusion 66  \nAppendix A. Additional Experiments, Proofs, and Discussions 69  \nA.1 . Proofs of Theorems in Chapter 2 69  \nA.2 . Discussions on the Prior Works Related to Chapter 2 97  \nA.3 . Additional Experiments on Heterogeneous Data of Chapter 3 99  \nA.4 . Proofs of Theorems in Chapter 3 100  \nA.5 . Discussions on the Prior Works Related to Chapter 3 127  \nA.6 . Experimental Investigations of Chapter 4 129  \nA.7 . Proofs of Theorems in Chapter 4 135  \nA.8 . Auxiliary Results in Chapter 4 152  \nBibliography 154  \nAbstract  \nStochastic optimization is fundamental to modern machine learning and deep learning problems. It provides various algorithmic frameworks, such as stochastic gradient descent (SGD), adaptive gradient algorithm (ADAGRAD) and adaptive moment estimation (ADAM), to efficiently minimize loss functions constructed from large-scale datasets. In this dissertation, we explore the theoretical properties and empirical performance of bilevel optimization algorithms and the phenomenon of large learning rates in machine learning. First, we introduce a novel algorithm, the Moving-Average Stochastic Bilevel Algorithm (MA-SOBA), designed for solving stochastic bilevel optimization under standard smoothness assumptions. Next, we extend the scope of bilevel optimization algorithms from single-agent training to a multi-agent context, i.e., distributed training, by proposing the MovingAverage Decentralized Stochastic Bilevel Optimization (MA-DSBO) algorithm. This approach improves the per-iteration complexity of previous methods, reducing the quadratic dependency on dimensionality to linear dependency. Lastly, inspired by the Edge of Stability (EoS) phenomenon observed in modern deep learning, we examine the training dynamics of gradient descent in a class of quadratic regression models with large learning rates—-a scenario that classical optimization theory struggles to explain.  \nAcknowledgments  \nFirst and foremost, I would like to express my deepest gratitude to my advisors, Prof. Krishnakumar Balasubramanian and Prof. Shiqian Ma. Their patience and professionalism provide","cbCainGOt26I5Iu1","https://ap.wps.com/l/cbCainGOt26I5Iu1","pdf",5170134,1,177,"English","en",105,"# Chapter 1. Introduction\n## Preliminaries\n## Outline of the Dissertation\n# Chapter 2. Stochastic Bilevel Optimization\n## Introduction\n## Proposed Framework: the MA-SOBA Algorithm\n## Theoretical Analysis\n## Min-Max Bilevel Optimization\n## Experiments\n## Conclusion\n# Chapter 3. Decentralized Stochastic Bilevel Optimization\n## Introduction\n## Preliminaries\n## DSBO Algorithm with Improved Per-Iteration Complexity\n## Numerical experiments\n## Conclusion\n# Chapter 4. Training Dynamics of Gradient Descent for Quadratic Regression\n## Introduction\n## Analyzing a discrete dynamical system with cubic map\n## Applications to quadratic regression models\n## Experimental investigations\n## Conclusion\n# Appendix A. Additional Experiments, Proofs, and Discussions\n## Proofs of Theorems in Chapter 2\n## Discussions on the Prior Works Related to Chapter 2\n## Additional Experiments on Heterogeneous Data of Chapter 3\n## Proofs of Theorems in Chapter 3\n## Discussions on the Prior Works Related to Chapter 3\n## Experimental Investigations of Chapter 4\n## Proofs of Theorems in Chapter 4\n## Auxiliary Results in Chapter 4","[{\"question\":\"What problem does the dissertation focus on in machine learning optimization?\",\"answer\":\"It focuses on stochastic optimization methods for bilevel problems and on understanding the behavior of learning when learning rates become large.\"},{\"question\":\"What is MA-SOBA and what is it designed for?\",\"answer\":\"MA-SOBA is a Moving-Average Stochastic Bilevel Algorithm proposed to solve stochastic bilevel optimization under standard smoothness assumptions.\"},{\"question\":\"How does the decentralized method MA-DSBO improve efficiency?\",\"answer\":\"MA-DSBO extends bilevel optimization to multi-agent distributed training and improves per-iteration complexity by reducing quadratic dependence on dimensionality to linear dependence.\"}]","Stochastic Optimization for Machine Learning: Investigations on Bilevel Optimization and Large Learning Rates - Dissertation | PDF",1785814263,446,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"stochastic-optimization-for-machine-learning-investigations-on-bilevel-optimization-and-large-learning-rates-dissertation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/stochastic-optimization-for-machine-learning-investigations-on-bilevel-optimization-and-large-learning-rates-dissertation/123030/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the dissertation focus on in machine learning optimization?","Question",{"text":75,"@type":76},"It focuses on stochastic optimization methods for bilevel problems and on understanding the behavior of learning when learning rates become large.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is MA-SOBA and what is it designed for?",{"text":80,"@type":76},"MA-SOBA is a Moving-Average Stochastic Bilevel Algorithm proposed to solve stochastic bilevel optimization under standard smoothness assumptions.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the decentralized method MA-DSBO improve efficiency?",{"text":84,"@type":76},"MA-DSBO extends bilevel optimization to multi-agent distributed training and improves per-iteration complexity by reducing quadratic dependence on dimensionality to linear dependence.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]