[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86367-en":3,"doc-seo-86367-105":29,"detail-sidebar-cat-0-en-105":95},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86367,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",8,"Research & Report","On the Condition Number Dependency in Bilevel Optimization","Bilevel optimization minimizes an upper-level objective whose decision is constrained by the solution of a lower-level problem. The study derives oracle complexity for locating an ϵ-stationary point using first-order methods under a nonconvex–strongly-convex (NC-SC) structure, where the lower-level problem is strongly convex. Prior near-optimal upper bounds in ϵ remain unclear in their condition-number dependence, motivating new results. The paper establishes a lower bound of Ω(κ_y^{5/2}ϵ^{-2}), sharpening condition-number gaps between bilevel and minimax formulations, and extends the analysis to high-order smooth functions, stochastic oracles, and convex hyper-objectives with improved bounds for specific regimes.","arXiv :2511 .22331v3 [math .OC] 12 Jul 2026  \nOn the Condition Number Dependency in Bilevel Optimization  \nLesi Chen Jingzhao Zhang  \nIIIS, Tsinghua University  \n[chenlc23@mails.tsinghua.edu.cn](chenlc23@mails.tsinghua.edu.cn) , [jingzhaoz@mail.tsinghua.edu.cn](jingzhaoz@mail.tsinghua.edu.cn)  \nJuly 14, 2026  \nAbstract  \nBilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an ϵ-stationary point with first-order methods when the upper-level problem is nonconvex, and the lower-level problem is strongly convex. Recent works (Ji et al., ICML 2021; Arbel and Mairal, ICLR 2022; Chen et al., JMLR 2025) achieve a ˜O(κ¯4yϵ −2) upper bound that is near-optimal in ϵ, which can be reduced to ˜O(κ¯7y/2ϵ −2) by a naive application of Nesterov acceleration in the inner loop, where κ¯y is the global condition number. However, the optimal dependency on the condition number is unknown. In this work, we establish anew Ω(κ5y/2ϵ −2) lower bound, where κy \u003C κ¯y is the lower-level condition number that is of the same order as κ¯y when the smoothness constants are O(1) . Our lower bound establishes the first provable gap in terms of condition number dependency between bilevel problems and minimax problems in this setup. Our lower bounds can be extended to various settings, including high-order smooth functions, stochastic oracles, and convex hyper-objectives: (1) For second-order and arbitrarily smooth problems, we show lower bounds of Ω(κ31y/14ϵ −12/7) and Ω(κ21y/10ϵ −8/5) , respectively. (2) For convex-strongly-convex problems, we improve the previously best lower bound (Ji and Liang, JMLR 2022) from Ω(κy / √ϵ) to Ω(κ3y/2/ √ϵ) . (3) For smooth stochastic problems, we also show a lower bound of Ω(κ4yϵ −4) .  \n1 Introduction  \nWe study the first-order oracle complexity for solving the bilevel optimization  \nmin F (x) = f(x, y ∗ (x)), y ∗ (x) = arg min g (x, y) . (1)  \nx∈Rd x y∈Rdy  \nThe goal is to minimize the hyper-objective F (x), which is implicitly defined by two explicit objectives f (x, y) and g(x, y) and the arg min operator in g(x, y) with respect to y. This formulation originates from the two-player general-sum Stackelberg game [58], and reflects the sequential nature of the decision process of two players x and y. Bilevel optimization of this form got extensive attention in the machine learning community due to its wide applications, such as meta-learning [55], hyper-parameter tuning [8, 19 , 47], generative adversarial networks [22], and reinforcement learning [25, 34, 67] .  \nTo ensure that Problem (1) is well-defined, we follow [21 , 25 , 29] to study the standard nonconvexstrongly-convex (NC-SC) setting, where the lower-level function g (x, y) is strongly convex in y while the upper-level function f (x, y) can be nonconvex. In NC-SC bilevel problems, a first-order method applied to the hyper-objective F (x) requires second-order information of g because  \n∇F(x) = ∇xf (x, y ∗ (x)) − ∇2xyg (x, y ∗ (x))[∇2yyg (x, y ∗ (x))]−1∇y f (x, y ∗ (x)) . (2)  \nIn large-scale problems, directly inverting the Hessian matrix ∇2yyg (x, y) is very expensive. There are mainly two lines of work to address this issue. The first line studies algorithms based on Hessian-vector product (HVP) oracles. Among them, the best-known theoretical guarantees [4, 29] show that they can find an ϵ-stationary point of F (x) with O (κ¯4yϵ −2) and O (κ¯9yϵ −4) first-order oracle complexity for smooth deterministic and stochastic problems, respectively, where κ¯y is the global condition number for the problem  \nTable 1: We present the oracle complexities of different deterministic methods for finding an ϵ-stationary point for smooth NC-SC bilevel problems, where κy and κ¯y refers to lower-level and global condition number respectively, as defined in Definition 2.2 .  \n\n| Oracle | Method | Complexity |\n| --- | --- | --- |\n| HVP | BA [21] |","cbCaim77atS76Odp","https://ap.wps.com/l/cbCaim77atS76Odp","pdf",1890032,1,36,"English","en",105,"# Introduction\n## Bilevel optimization setup\n## NC-SC setting and gradient structure\n## Oracle complexity and prior work\n## Condition-number dependency results","[{\"question\":\"What problem setting is analyzed in the document?\",\"answer\":\"The document studies bilevel optimization in a nonconvex–strongly-convex (NC-SC) setting, where the upper-level objective can be nonconvex while the lower-level function is strongly convex in the inner variable.\"},{\"question\":\"What complexity question does the document focus on?\",\"answer\":\"It investigates first-order oracle complexity for finding an ϵ-stationary point of the resulting hyper-objective, especially how this complexity depends on the condition numbers of the bilevel structure.\"},{\"question\":\"What is the main lower bound result regarding condition number dependence?\",\"answer\":\"The paper establishes a lower bound of Ω(κ_y^{5/2}ϵ^{-2}), showing a provable gap in condition-number dependency between bilevel problems and related minimax problems in the same setup.\"},{\"question\":\"How are the lower bounds extended to other scenarios?\",\"answer\":\"The document extends lower bounds to high-order smooth functions, stochastic oracles, and convex hyper-objectives, including improved bounds for convex–strongly-convex problems and additional regimes for smooth stochastic problems.\"}]",1784211034,91,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":90,"head_meta":92,"extra_data":94,"updated_unix":27},"on-the-condition-number-dependency-in-bilevel-optimization","",{"@graph":35,"@context":89},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/on-the-condition-number-dependency-in-bilevel-optimization/86367/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81,85],{"name":72,"@type":73,"acceptedAnswer":74},"What problem setting is analyzed in the document?","Question",{"text":75,"@type":76},"The document studies bilevel optimization in a nonconvex–strongly-convex (NC-SC) setting, where the upper-level objective can be nonconvex while the lower-level function is strongly convex in the inner variable.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What complexity question does the document focus on?",{"text":80,"@type":76},"It investigates first-order oracle complexity for finding an ϵ-stationary point of the resulting hyper-objective, especially how this complexity depends on the condition numbers of the bilevel structure.",{"name":82,"@type":73,"acceptedAnswer":83},"What is the main lower bound result regarding condition number dependence?",{"text":84,"@type":76},"The paper establishes a lower bound of Ω(κ_y^{5/2}ϵ^{-2}), showing a provable gap in condition-number dependency between bilevel problems and related minimax problems in the same setup.",{"name":86,"@type":73,"acceptedAnswer":87},"How are the lower bounds extended to other scenarios?",{"text":88,"@type":76},"The document extends lower bounds to high-order smooth functions, stochastic oracles, and convex hyper-objectives, including improved bounds for convex–strongly-convex problems and additional regimes for smooth stochastic problems.","https://schema.org",{"og:url":51,"og:type":91,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":93,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":96},[97,101,105,109,114,119,124,127,132,135,139],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},"Exam",70,"exam",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},5,"Comic",60,"comic",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},6,"Technology",50,"technology",{"id":120,"doc_module":4,"doc_module_name":45,"category_name":121,"show_sort_weight":122,"slug":123},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":125,"slug":126},30,"research-report",{"id":128,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":130,"slug":131},9,"Religion & Spirituality",20,"religion-spirituality",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":130,"slug":134},"World Cup","world-cup",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":136,"slug":138},10,"Lifestyle","lifestyle",{"id":140,"doc_module":4,"doc_module_name":45,"category_name":141,"show_sort_weight":110,"slug":142},19,"General","general"]