[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117270-en":3,"doc-seo-117270-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},117270,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","Nonsmooth Implicit Differentiation - Deterministic and Stochastic Convergence Rates","The work studies efficient computation of the derivative of a fixed point arising from a parametric nondifferentiable contraction mapping. It investigates iterative differentiation (ITD) and approximate implicit differentiation (AID), highlighting the key obstacle that the chain rule fails in the nonsmooth setting. Deterministic results provide linear convergence rates for nonsmooth ITD and new, near-optimal linear rates for AID and improved ITD. A stochastic method, NSID, is introduced for composite maps where the inner function is available only through stochastic unbiased estimators, and convergence rates matching the smooth best known results are established, supported by illustrative experiments.","Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates  \nRiccardo Grazzi 1 Massimiliano Pontil 1 2 Saverio Salzo 3 1  \nAbstract  \nWe study the problem of efficiently computing the derivative of the fixed-point of a parametric nondifferentiable contraction map. This problem has wide applications in machine learning, including hyperparameter optimization, meta-learning and data poisoning attacks. We analyze two popular approaches: iterative differentiation (ITD) and approximate implicit differentiation (AID) . A key challenge behind the nonsmooth setting is that the chain rule does not hold anymore. We build upon the work by Bolte et al. (2022), who prove linear convergence of nonsmooth ITD under a piecewise Lipschitz smooth assumption. In the deterministic case, we provide a linear rate for AID and an improved linear rate for ITD which closely match the ones for the smooth setting. We further introduce NSID, a new stochastic method to compute the implicit derivative when the contraction map is defined as the composition of an outer map and an inner map which is accessible only through a stochastic unbiased estimator. We establish rates for the convergence of NSID, encompassing the best available rates in the smooth setting. We also present illustrative experiments confirming our analysis.  \n1. Introduction  \nIn this paper, we study the problem of efficiently approximating a generalized derivative (or Jacobian) of the solution map of the parametric fixed point equation  \nwpλq “ Φpwpλq,λq pλ P Rm q, (1)  \nwhen Φ is not differentiable, but only piecewise differentiable. We address both the case that Φ can be explicitly  \n1 CSML, Istituto Italiano di Tecnologia, Genoa, Italy 2Department of Computer Science, University College, London, UK 3Dipartimento di Ingegneria Informatica, Automatica e Gestionale, Universit La Sapienza, Rome, Italy. Correspondence to: Riccardo Grazzi \u003Criccardo.grazzi@iit.it> .  \nProceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024 . Copyright 2024 by the author(s) .  \nevaluated, and the case that Φ has the composite form  \nΦpw,λq “ GpTpw,λq,λq  \n(2)  \nTpw,λq “ ErˆTξ pwpλq,λqs,  \nwhere the external map G can be evaluated, but the inner map T is accessible only via a stochastic estimator ˆTξ , with ξ a random variable.  \nA main motivation for computing the implicit derivative of (1) is provided by bilevel optimization, which aims to minimize an upper level objective function of wpλq. Important examples are given by hyperparameter optimization and meta-learning (Franceschi et al., 2018 ; Lee et al., 2019), where (1) expresses the optimality conditions of a lowerlevel minimization problem. Further examples include learning a surro˜gate model for data poisoning attacks (Xiao et al.,  \n2015 ; Munoz-Gonzlez et al., 2017), deep equilibrium models (Bai et al., 2019) or OptNet (Amos & Kolter, 2017) . All these problems may present nonsmooth mappings Φ . For instance, consider hyperparameter optimization or data poisoning attacks for SVMs, or meta-learning for image classification, where Φ is evaluated through the forward pass of a neural net with RELU activations (Bertinetto et al., 2019 ; Lee et al., 2019 ; Rajeswaran et al., 2019) . In addition, when such settings are applied to large datasets, evaluating the map Φ would be too costly, but we can usually apply stochastic methods through the composite stochastic structure in (2), where only T involves a computation on the full training set (e.g., a gradient descent step) .  \nNowadays, automatic differentiation techniques (Griewank & Walther, 2008) popular for deep learning, can also be used to efficiently, i.e. with a cost of the same order of that of approximating wpλq, approximate Jacobian-vector (or vector-Jacobian) products of wpλq by relying only on an implementation of an iterative solver for problem (1) . There are two main approaches to achieve this: ITerative Differentiation (ITD) (e","cbCaigYbkvJv5jB4","https://ap.wps.com/l/cbCaigYbkvJv5jB4","pdf",543061,1,25,"English","en",105,"# Introduction\n## Problem setting: parametric fixed point and implicit derivative\n## Applications in machine learning and optimization\n## Differentiation approaches: ITD and AID\n## Challenges in nonsmooth and stochastic settings","[{\"question\":\"What problem does the paper address?\",\"answer\":\"It addresses efficient approximation of the generalized derivative (Jacobian) of the solution map for a parametric fixed point equation when the mapping is only piecewise differentiable.\"},{\"question\":\"Why does nonsmooth implicit differentiation break common methods?\",\"answer\":\"Because in the nonsmooth setting the chain rule does not hold, so convergence analysis for ITD and AID cannot rely on smooth-case differentiation properties.\"},{\"question\":\"What are ITD, AID, and the new NSID method?\",\"answer\":\"ITD differentiates through solver iterations, AID solves an implicit linear system for Jacobian-vector products, and NSID computes implicit derivatives in a stochastic composite setting where the inner map is accessible only via an unbiased estimator.\"}]",1785674905,63,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"nonsmooth-implicit-differentiation-deterministic-and-stochastic-convergence-rates","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/nonsmooth-implicit-differentiation-deterministic-and-stochastic-convergence-rates/117270/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does the paper address?","Question",{"text":74,"@type":75},"It addresses efficient approximation of the generalized derivative (Jacobian) of the solution map for a parametric fixed point equation when the mapping is only piecewise differentiable.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why does nonsmooth implicit differentiation break common methods?",{"text":79,"@type":75},"Because in the nonsmooth setting the chain rule does not hold, so convergence analysis for ITD and AID cannot rely on smooth-case differentiation properties.",{"name":81,"@type":72,"acceptedAnswer":82},"What are ITD, AID, and the new NSID method?",{"text":83,"@type":75},"ITD differentiates through solver iterations, AID solves an implicit linear system for Jacobian-vector products, and NSID computes implicit derivatives in a stochastic composite setting where the inner map is accessible only via an unbiased estimator.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]