[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81495-en":3,"doc-seo-81495-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},81495,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Accelerated Fully First-Order Methods for Bilevel and Minimax Optimization","The paper develops accelerated fully first-order methods for bilevel optimization (BLO) under strong convexity in the lower-level problem. A perturbed, restarted accelerated fully first-order approach (PRAF2BA) is proposed using only fully first-order oracles, targeting efficient approximation and second-order stationarity. As a key application, the method yields perturbed restarted accelerated gradient descent ascent (PRAGDA) for nonconvex-strongly-convex minimax optimization. For BLO cases without strong convexity, regularity conditions are identified and an inexact gradient-free method (IGFM) is presented with provable polynomial-time tractability, plus hardness results for general convex lower-level functions.","arXiv :2405 .00914v3 [math .OC] 9 Jul 2024  \nAccelerated Fully First-Order Methods for Bilevel and  \nMinimax Optimization  \nChris Junchi Li⋄  \nDepartment of Electrical Engineering and Computer Sciences⋄  \nUniversity of California, Berkeley  \nJuly 10, 2024  \nAbstract  \nWe present in this paper novel accelerated fully first-order methods in Bilevel Optimization (BLO) . Firstly, for BLO under the assumption that the lower-level functions admit the typical strong convexity assumption, the (Perturbed) Restarted Accelerated Fully First-order methods for Bilevel Approximation ((P)RAF2BA) algorithm leveraging fully first-order oracles is proposed, whereas the algorithm for finding approximate first-order and second-order stationary points with state-of-the-art oracle query complexities in solving complex optimization tasks. Secondly, applying as a special case of BLO the nonconvex-strongly-convex (NCSC) minimax optimization, PRAF2BA rediscovers perturbed restarted accelerated gradient descent ascent (PRAGDA) that achieves the state-of-the-art complexity for finding approximate second-order stationary points. Additionally, we investigate the challenge of finding stationary points of the hyper-objective function in BLO when lower-level functions lack the typical strong convexity assumption, where we identify several regularity conditions of the lower-level problems that ensure tractability and present hardness results indicating the intractability of BLO for general convex lower-level functions. Under these regularity conditions we propose the Inexact Gradient-Free Method (IGFM), utilizing the Switching Gradient Method (SGM) as an efficient sub-routine to find an approximate stationary point of the hyper-objective in polynomial time. Empirical studies for real-world problems are provided to further validate the outperformance of our proposed algorithms.  \n1 Introduction  \nBilevel optimization (BLO) has received increasing attention owing to its remarkable capability in addressing crucial machine learning tasks by revealing the inner structure of many (otherwise oblique) machine learning optimization problems, such as meta-learning [FFS+18, BHTV19, JLLP20, RL17, HAMS21], hyperparameter optimization [FFS+18, Ped16, FH19, SCHB19, GFPS20, AM22a], continual learning [PLSS21], out-of-distribution learning [ZLP+22], adversarial training [GPAM+20, SND18, WCJ+21, LJJ20a, LJJ20b, WL20], composite optimization [GHZY21], reinforcement learning [KT99, HWWY23, KZH+21, SZB20], causal learning [JV22, LSR+22, ABGLP19], neural architecture search [LSY19, WGS+22, ZL17, ZSP+21], etc. Formally, BLO aims to optimize the upper-level (UL) function f (x, y) under the constraint that y is minimized with respect to the lower-level (LL) function g (x, y) on a closed convex set Y ⊆ Rdy . Mathematically, it can be formulated as  \nmin f (x, y) where Y ∗ (x) ≜ arg ming(x, y) is the LL solution mapping (1) x∈Rd x,y∈Y∗(x) y∈Y  \nLet LL value function be g ∗ (x) ≜ miny∈Y g (x, y) . Problem (1) can be transformed via hyperobjective approaches [Dem02, DZ20, LMY+20, LLZZ21]  \nxindx 􀀚 φ (x) ≜ y n(x) f (x, y)􀀛 (2)  \nwhere φ (x) is called the hyper-objective function of BLO problem (1) . It transforms the problem into the composition of a simple BLO [SS17] w.r.t. the LL variable y and an unconstrained singlelevel optimization w.r.t. the UL variable x. This reformulation naturally leads to two foundational questions. The first question involves  \nThe secP1ond: Fquind an optimalestion involves LL variable yb ∈ Y ∗ (xb) such that φ (xb) = f(xb, yb) for a given xb P2: Find a UL variable xb that is a stationary point of φ (x)  \nBLO with LLSC. When the LL function is strongly convex, both questions previously proposed are relatively easy to solve. The lower-level strong convexity (LLSC) ensures Y ∗ (x) to bea singleton, and therefore simplifies (2) into φ (x) = f (x, y∗ (x)), where the LL optimal solution y ∗ (x) = arg miny∈Y g (x, y) can be found via gradient descent on g. For simpl","cbCaihrKGAGHMmzu","https://ap.wps.com/l/cbCaihrKGAGHMmzu","pdf",1540752,2,1,48,"English","en",105,"# Abstract\n# Introduction\n## Bilevel optimization and hyper-objective reformulation\n## Minimax optimization as a special case\n## BLO without lower-level strong convexity","[{\"question\":\"What problem does the paper focus on?\",\"answer\":\"The paper focuses on accelerated fully first-order algorithms for bilevel optimization (BLO), including minimax optimization as a special case.\"},{\"question\":\"What is PRAF2BA and what assumptions does it use?\",\"answer\":\"PRAF2BA is a perturbed, restarted accelerated fully first-order method for bilevel approximation that leverages fully first-order oracles and assumes the lower-level functions satisfy typical strong convexity conditions.\"},{\"question\":\"How does the paper handle BLO when lower-level strong convexity is not available?\",\"answer\":\"It studies tractability via regularity conditions, proves hardness for general convex lower-level functions, and proposes an inexact gradient-free method (IGFM) using the Switching Gradient Method (SGM) to find an approximate stationary point in polynomial time.\"}]",1784173813,121,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"accelerated-fully-first-order-methods-for-bilevel-and-minimax-optimization","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/accelerated-fully-first-order-methods-for-bilevel-and-minimax-optimization/81495/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper focus on?","Question",{"text":75,"@type":76},"The paper focuses on accelerated fully first-order algorithms for bilevel optimization (BLO), including minimax optimization as a special case.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is PRAF2BA and what assumptions does it use?",{"text":80,"@type":76},"PRAF2BA is a perturbed, restarted accelerated fully first-order method for bilevel approximation that leverages fully first-order oracles and assumes the lower-level functions satisfy typical strong convexity conditions.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the paper handle BLO when lower-level strong convexity is not available?",{"text":84,"@type":76},"It studies tractability via regularity conditions, proves hardness for general convex lower-level functions, and proposes an inexact gradient-free method (IGFM) using the Switching Gradient Method (SGM) to find an approximate stationary point in polynomial time.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]