[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118011-en":3,"doc-seo-118011-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118011,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Adaptive Optimization Algorithms for Machine Learning - Dissertation","Machine learning relies on fast, reliable optimization to train increasingly large models and datasets. This dissertation studies adaptivity in learning optimizers and develops contributions across multiple directions of adaptive behavior. It addresses personalization via personalized loss, provable post-training adaptation through meta-learning, and real-time hyperparameter learning via hyperparameter variance reduction. It also delivers fast global convergence and scalable second-order approaches using low-dimensional updates, with improved algorithms and convergence guarantees accompanied by strengthened analyses of practical methods.","arXiv :2311 . 10203v1 [ cs .LG] 16 Nov 2023  \nAdaptive Optimization Algorithms for Machine Learning  \nDissertation by  \nSlavom´ır Hanzely  \nIn Partial Fulfillment of the Requirements  \nFor the Degree of  \nDoctor of Philosophy  \nKing Abdullah University of Science and Technology Thuwal, Kingdom of Saudi Arabia  \n©September, 2023  \nSlavom´ır Hanzely  \nAll rights reserved  \n [orcid.org/0009-0006-2640-0354](orcid.org/0009-0006-2640-0354)  \n[slavomir-hanzely.github.io](slavomir-hanzely.github.io)  \n2  \nEXAMINATION COMMITTEE PAGE  \nThe dissertation of Slavom´ır Hanzely is approved by the examination committee.  \nCommittee Chairperson: Peter Richt´arik  \nCommittee Members: Eric Moulines, Martin Jaggi, Ajay Jasra, Di Wang  \n3  \nABSTRACT  \nAdaptive Optimization Algorithms for Machine Learning  \nSlavom´ır Hanzely  \nMachine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This dissertation investigates adaptivity in machine learning optimizers. The ensuing chapters are dedicated to various facets of adaptivity, including:  \n1. personalization and user-specific models via personalized loss (Chapter 2 and  \n3),  \n2. provable post-training model adaptations via meta-learning (Chapter 3),  \n3. learning unknown hyperparameters in real time via hyperparameter  \nvariance reduction (Chapter 4),  \n4. fast O (k−2) global convergence of second-order methods via stepsized Newton method regardless of the initialization and choice basis (Chapter 5),  \n[5. fast](5. fast) and scalable second-order methods via low-dimensional updates (Chapter 6) .  \nThis thesis contributes novel insights, introduces new algorithms with improved convergence guarantees, and improves analyses of popular practical algorithms.  \n4  \nACKNOWLEDGEMENTS  \nI would like to express my deepest gratitude and appreciation to all those who have contributed to my research. First and foremost, I am immensely grateful to my supervisor, Peter Richt´arik, for their invaluable expertise, guidance, and continuous support throughout this research journey. Their deep understanding of the subject matter, insightful feedback, and commitment to academic excellence have been instrumental in shaping the direction and quality of my research. I would like to extend my sincere appreciation to the members of my dissertation committee. I am indebted to my colleagues and peers in the research group for their intellectual contributions, fruitful discussions, and collaborative spirit. Namely, I’d like to thank Abdurakhmon Sadiev, Adil Salim, Ahmed Khaled, Alexander Tyurin, Aritra Dutta, Artavazd Maranjyan, Avetik Karagulyan, Aymeric Dieleveut, Dmitry Kamzolov, Dmitry Kovalev, Dmitry Pasechnyuk, Dan Alistarh, Eduard Gorbunov, Egor Shulgin, El Houcine Bergou, Elnur Gasanov, Filip Hanzely, Grigory Malinovsky, Hanmin Li, Igor Sokolov, Ilyas Fatkhullin, Ivan Agarsky, Ivan Ilin, Kay Yi, Kaja Gruntkowska, Konstantin Burlachenko, Konstantin Mishchenko, Laurent Condat, Lukang Sun, Martin Tak´aˇc, Mher Safaryan, Michal Grudzien, Nicolas Loizou, Peter Richt´arik, Rafal Szlendak, Robert M. Gower, Rustem Islamov, Samuel Horv´ath, Sarit Khirirat, Si Yi Meng, Xun Qian, Yury Demidovich, Zhize Li. Their diverse perspectives and collective knowledge have broadened my horizons and inspired new ideas throughout the course of this research. I am deeply grateful to my friends and family for their unwavering support, understanding, and encouragement throughout this demanding journey. Their belief in my abilities and continuous motivation have been the driving force behind my perseverance.  \n5  \nContents  \nExamination committee page 2  \nAbstract 3  \nAcknowledgements 4  \n1 Introduction 12  \n1.1 Adaptivity ............................... 12  \n1.2 Adaptation to unknown hyperparameters .............. 13  \n1.2.1 Optimal minibatch size .................... 14  \n1.2.2 Approximating the optimal minibatch size ......... 14  \n1","cbCaio31kcmEXf9n","https://ap.wps.com/l/cbCaio31kcmEXf9n","pdf",4527443,1,193,"English","en",105,"# Introduction\n## Adaptivity\n## Adaptation to unknown hyperparameters\n## Adaptation to geometry\n## Adaptivity to high dimensions\n## Adaptivity to heterogeneity of the data distribution\n## Adaptation to users/clients\n## Overview of objective functions\n## Overview of assumptions\n## Organization of the thesis\n# Lower bounds and optimal algorithms for personalized federated learning","[{\"question\":\"What is the dissertation’s main topic?\",\"answer\":\"The dissertation investigates adaptivity in machine learning optimizers, focusing on how optimization algorithms can adapt to personalization, unknown hyperparameters, and problem geometry.\"},{\"question\":\"How does the work handle personalization in optimization?\",\"answer\":\"Personalization is addressed through personalized loss, with dedicated chapters covering personalized federated learning and user-specific modeling.\"},{\"question\":\"What methods are developed for fast convergence and scalability?\",\"answer\":\"The dissertation introduces fast global convergence results for second-order methods and scalable second-order approaches using low-dimensional updates and stepsized Newton-style techniques.\"}]","Adaptive Optimization Algorithms for Machine Learning - Dissertation | PDF",1785680744,486,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"adaptive-optimization-algorithms-for-machine-learning-dissertation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/adaptive-optimization-algorithms-for-machine-learning-dissertation/118011/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the dissertation’s main topic?","Question",{"text":75,"@type":76},"The dissertation investigates adaptivity in machine learning optimizers, focusing on how optimization algorithms can adapt to personalization, unknown hyperparameters, and problem geometry.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the work handle personalization in optimization?",{"text":80,"@type":76},"Personalization is addressed through personalized loss, with dedicated chapters covering personalized federated learning and user-specific modeling.",{"name":82,"@type":73,"acceptedAnswer":83},"What methods are developed for fast convergence and scalability?",{"text":84,"@type":76},"The dissertation introduces fast global convergence results for second-order methods and scalable second-order approaches using low-dimensional updates and stepsized Newton-style techniques.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]