[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118039-en":3,"doc-seo-118039-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118039,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","Advancing Automated Machine Learning: Neural Architectures and Optimization Algorithms","Automated Machine Learning (AutoML) aims to automate complex machine-learning workflows, but its effectiveness depends on carefully engineered methods. This dissertation investigates Neural Architecture Search (NAS) and optimization algorithms as two core pillars. It improves NAS stability and robustness via perturbation-based regularization, recasts NAS as distribution learning, and extends it to collaborative filtering for more efficient, accurate recommendations. It also analyzes optimization in transformer architectures, identifies an optimization gap, and introduces symbolic program discovery to automatically generate superior optimization methods, including the widely adopted Lion algorithm.","UCLA  \nUCLA Electronic Theses and Dissertations  \nTitle  \nAdvancing Automated Machine Learning: Neural Architectures and Optimization Algorithms  \nPermalink  \n[https://escholarship.org/uc/item/2f40c1w4](https://escholarship.org/uc/item/2f40c1w4)  \nAuthor  \nChen, Xiangning  \nPublication Date  \n2023  \nPeer reviewed|Thesis/dissertation  \n[eScholarship.org](eScholarship.org) Powered by the California Digital Library  \nUniversity of California  \nUNIVERSITY OF CALIFORNIA Los Angeles  \nAdvancing Automated Machine Learning: Neural Architectures and Optimization Algorithms  \nA dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science  \nby  \nXiangning Chen  \n2023  \n© Copyright by Xiangning Chen 2023  \nABSTRACT OF THE DISSERTATION  \nAdvancing Automated Machine Learning:  \nNeural Architectures and Optimization Algorithms  \nby  \nXiangning Chen  \nDoctor of Philosophy in Computer Science  \nUniversity of California, Los Angeles, 2023  \nProfessor Cho-Jui Hsieh, Chair  \nThe field of Automated Machine Learning (AutoML) has gained immense attention for its ability to automate complex machine learning tasks, yet it is still an evolving discipline requiring nuanced approaches to be fully realized. This thesis, \"Advancing Automated Machine Learning: Neural Network Architectures and Optimization Algorithms,\" provides a comprehensive investigation into two foundational pillars: Neural Architecture Search (NAS) and optimization algorithms.  \nIn the first half of the thesis, we confront the inherent challenges of stability and robustness in NAS, enhancing its reliability through a perturbation-based regularization scheme. This allows for more consistent and dependable architecture choices. Furthermore, we extend the traditional paradigms of NAS by framing it as a distribution learning problem, and additionally, by applying it to collaborative filtering. These extensions not only broaden the applicability of NAS but also lead to marked improvements in the efficiency and accuracy of recommendation systems.  \nThe latter part of the thesis focuses on the role of optimization in achieving high per-  \nformance, particularly in transformer architectures. We identify a critical optimization gap and propose strategies for its mitigation, emphasizing the necessity of a transition from purely architecture-based search to include optimization techniques. Then we delve into a groundbreaking approach to optimization algorithm design through symbolic program discovery. This framework automatically discover new optimization methods that outperform traditional algorithms, thereby introducing an unprecedented level of automation in the development of optimization techniques. Our developed Lion algorithm has been widely adopted by the community. This not only advances the state-of-the-art in optimization algorithms but also significantly augments the capabilities and reach of AutoML systems.  \nBy addressing these multifaceted challenges in both neural architecture and optimization algorithm design, this thesis presents a coherent, unified contribution to the advancement of Automated Machine Learning. It is hoped that these collective insights serve as a robust foundation for future research in the ever-evolving landscape of AutoML.  \nThe dissertation of Xiangning Chen is approved.  \nWei Wang  \nKai-Wei Chang  \nMani Srivastava  \nCho-Jui Hsieh, Committee Chair  \nUniversity of California, Los Angeles  \n2023  \nTo my parents  \nv  \nTABLE OF CONTENTS  \n1 Introduction ...................................... 1  \n2 Stabilize and Robustify Neural Architecture Search ............. 6  \n2.1 Problem Settings ................................. 6  \n2.2 Performance Collapse of DARTS ........................ 7  \n2.3 Stabilizing Neural Architecture Search via Perturbation-based Regularization 8  \n2.3.1 Origins of Instability in the DARTS ................... 9  \n2.3.2 Proposed method ............................. 10  \n2.3.3 Search ","cbCaii4KSEjbUxGK","https://ap.wps.com/l/cbCaii4KSEjbUxGK","pdf",8765257,1,186,"English","en",105,"# Introduction\n# Stabilize and Robustify Neural Architecture Search\n## Problem Settings\n## Performance Collapse of DARTS\n## Stabilizing Neural Architecture Search via Perturbation-based Regularization\n# Neural Architecture Search as Distribution Learning\n## The Proposed Approach-DrNAS\n## Discussions and Relationship to Prior Work\n# Neural Architecture Search in Collaborative Filtering\n## Existing Interaction Functions (IFCs)\n## Proposed Method\n## Empirical Study","[{\"question\":\"What are the two main research pillars of this dissertation?\",\"answer\":\"The dissertation focuses on Neural Architecture Search (NAS) and optimization algorithms, treating them as foundational pillars for advancing AutoML.\"},{\"question\":\"How does the thesis improve NAS stability and robustness?\",\"answer\":\"It proposes a perturbation-based regularization scheme to mitigate instability and enhance reliability of architecture choices.\"},{\"question\":\"What role does optimization play, especially for transformer architectures?\",\"answer\":\"The thesis identifies an optimization gap in transformer-related performance and proposes strategies to address it, emphasizing a shift from architecture-only search to optimization-aware methods.\"}]","Advancing Automated Machine Learning: Neural Architectures and Optimization Algorithms | PDF",1785680955,469,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"advancing-automated-machine-learning-neural-architectures-and-optimization-algorithms","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/advancing-automated-machine-learning-neural-architectures-and-optimization-algorithms/118039/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are the two main research pillars of this dissertation?","Question",{"text":75,"@type":76},"The dissertation focuses on Neural Architecture Search (NAS) and optimization algorithms, treating them as foundational pillars for advancing AutoML.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the thesis improve NAS stability and robustness?",{"text":80,"@type":76},"It proposes a perturbation-based regularization scheme to mitigate instability and enhance reliability of architecture choices.",{"name":82,"@type":73,"acceptedAnswer":83},"What role does optimization play, especially for transformer architectures?",{"text":84,"@type":76},"The thesis identifies an optimization gap in transformer-related performance and proposes strategies to address it, emphasizing a shift from architecture-only search to optimization-aware methods.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]