[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119720-en":3,"doc-seo-119720-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119720,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","On Tilted Losses in Machine Learning - Theory and Applications","Exponential tilting is widely used in statistics, probability, information theory, and optimization, yet it has rarely been applied in machine learning. This work introduces tilted empirical risk minimization (TERM), a simple extension of ERM that uses exponential tilting to flexibly control the influence of individual losses. TERM can adjust outlier sensitivity for fairness or robustness, offers variance-reduction benefits for generalization, approximates tail-probability behavior, and connects to objectives such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization.","arXiv :2 109 .06 14 1v 3 [ cs .LG] 1 Jun 2023  \nOn Tilted Losses in Machine Learning: Theory and Applications  \nTian Li˚ [tianli@cmu.edu](tianli@cmu.edu)  \nComputer Science Department Carnegie Mellon University Pittsburgh, PA 15213, USA  \nAhmad Beirami˚: [beirami@google.com](beirami@google.com)  \nGoogle Research  \nNew York, NY 10011, USA  \nMaziar Sanjabi [maziars@fb.com](maziars@fb.com)  \nMeta AI  \nMenlo Park, CA 94025, USA  \nVirginia Smith [smithv@cmu.edu](smithv@cmu.edu)  \nMachine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA  \nEditor: Zaid Harchaoui  \nAbstract  \nExponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization.  \nWe study a simple extension to ERM—tilted empirical risk minimization (TERM)—which uses exponential tilting to flexibly tune the impact of individual losses. The resulting framework has several useful properties: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to the tail probability of losses. Our work makes connections between TERM and related objectives, such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO) . We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and show that the framework can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance.  \nDespite the straightforward modification TERM makes to traditional ERM objectives, we find that the framework can consistently outperform ERM and deliver competitive performance with state-of-the-art, problem-specific approaches.  \nKeywords: Exponential tilting, empirical risk minimization, Value-at-Risk, superquantile optimization, fairness, robustness.  \n˚ Equal contribution.  \n:Work done at Meta AI.  \n©2023 Tian Li˚ , Ahmad Beirami˚ , Maziar Sanjabi, Virginia Smith.  \nLicense: CC-BY 4.0, see [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/. Attribution)[. Attribution](https://creativecommons.org/licenses/by/4.0/. Attribution) requirements are provided at  \n[http://jmlr.org/papers/v24/21-1095.html](http://jmlr.org/papers/v24/21-1095.html).  \nLi˚ , Beirami˚ , Sanjabi, Smith  \n1 Introduction  \nMany statistical estimation procedures rely on the concept of empirical risk minimization (ERM), in which the parameter of interest, θPΘĎRd, is estimated by minimizing an average loss over the data tx1 ,...,xNu:  \nRpθq :“ 1N  \nÿ fpxi;θq . iPrNs  \n(1)  \nAlthough ERM is widely used in machine learning, it is known to perform poorly in situations where average performance is not an appropriate surrogate for the problem of interest. Significant research has thus been devoted to developing alternatives to traditional ERM for diverse applications, such as learning in the presence of noisy/corrupted data (Khetanet al., 2018; Jiang et al., 2018), performing classification with imbalanced data (Lin et al. , 2017; Malisiewicz et al., 2011), ensuring that subgroups within a population are treated fairly (Hashimoto et al., 2018; Samadi et al., 2018), or developing solutions with favorable out-of-sample performance (Duchi and Namkoong, 2019) .  \nIn this paper, we suggest that deficiencies in ERM can be flexibly addressed via a unified framework, tilted empirical risk minimization (TERM) . TERM encompasses a famil","cbCaisMp7PuNxfsZ","https://ap.wps.com/l/cbCaisMp7PuNxfsZ","pdf",3263541,1,79,"English","en",105,"# Introduction\n## Perspectives on Exponential Tilting","[{\"question\":\"What problem does tilted empirical risk minimization (TERM) address compared with ERM?\",\"answer\":\"ERM averages losses, which can be a poor surrogate when the optimization target depends on more than mean performance. TERM introduces a tunable exponential-tilting mechanism to control how individual losses affect training.\"},{\"question\":\"How does TERM influence outliers and support fairness or robustness?\",\"answer\":\"By tuning the exponential tilting applied to losses, TERM can either decrease or increase the influence of outliers. This enables robustness improvements and can be used to enforce fairness between subgroups.\"},{\"question\":\"With what known risk objectives and frameworks does TERM relate?\",\"answer\":\"TERM can be connected to Value-at-Risk and Conditional Value-at-Risk, and it also relates to distributionally robust optimization (DRO).\"}]","On Tilted Losses in Machine Learning - Theory and Applications | PDF",1785725951,199,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"on-tilted-losses-in-machine-learning-theory-and-applications","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/on-tilted-losses-in-machine-learning-theory-and-applications/119720/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does tilted empirical risk minimization (TERM) address compared with ERM?","Question",{"text":75,"@type":76},"ERM averages losses, which can be a poor surrogate when the optimization target depends on more than mean performance. TERM introduces a tunable exponential-tilting mechanism to control how individual losses affect training.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does TERM influence outliers and support fairness or robustness?",{"text":80,"@type":76},"By tuning the exponential tilting applied to losses, TERM can either decrease or increase the influence of outliers. This enables robustness improvements and can be used to enforce fairness between subgroups.",{"name":82,"@type":73,"acceptedAnswer":83},"With what known risk objectives and frameworks does TERM relate?",{"text":84,"@type":76},"TERM can be connected to Value-at-Risk and Conditional Value-at-Risk, and it also relates to distributionally robust optimization (DRO).","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]