[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327322-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327322-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","distributionally-robust-listwise-preference-optimization","Distributionally Robust Listwise Preference Optimization","","Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. This work studies listwise preference optimization under ranking-label uncertainty, where the observed order over a candidate list is ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. A pointwise total-variation robust Plackett–Luce objective yields an exact decomposition and tractable worst-case ranking. It provides convex offline guarantees and online optimization improvements with more reliable candidate expansion.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/distributionally-robust-listwise-preference-optimization/327322/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/distributionally-robust-listwise-preference-optimization/327322.png","ImageObject",300,407,{"name":42,"@type":43},"Emma Wilson","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-08","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",13,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address in listwise preference optimization?","Question",{"text":63,"@type":64},"It addresses robustness when the observed ranking label over a candidate list is uncertain due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model-induced noise.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the proposed method model ranking-label uncertainty?",{"text":68,"@type":64},"It allows the ranking-label distribution to vary within a pointwise total-variation ambiguity set around the empirical ranking, conditional on a realized candidate list.",{"name":70,"@type":61,"acceptedAnswer":71},"What is the key technical contribution regarding the worst-case ranking?",{"text":72,"@type":64},"The robust loss decomposes into the nominal Plackett–Luce loss plus a worst-case correction, and the worst-case ranking is obtained by sorting current implicit scores, reducing computation from K! enumeration to O(K log K).","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327322,1790293969,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},3848291630094,"https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","Distributionally Robust Listwise Preference  \nOptimization  \narXiv :2607 .0 17 15v 1 [ cs .AI] 2 Jul 2026  \nXudong Wu  \nThe University of Hong Kong Hong Kong SAR  \nJian Qian  \nThe University of Hong Kong Hong Kong SAR  \nPangpang Liu  \nYale University New Haven, CT, USA  \nVaneet Aggarwal  \nPurdue University West Lafayette, IN, USA  \nJiayu Chen  \nThe University of Hong Kong Hong Kong SAR  \nAbstract  \nExisting robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preferencepair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett–Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from K! enumeration to O (K log K) . This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global ϵ-suboptimality with O (ϵ−2) sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak ngdexmrtyntleaanshnte(ϵhls)hdMoe pimrreopprs-eeeeoolopbubussettariodeitnryln.ileyrimpInreesoflrfgnine Lormmean, makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.  \n1 Introduction  \nLearning from human preferences has become a central mechanism for aligning large language models, with reinforcement-learning-from-human-feedback (RLHF) and direct preference optimization (DPO) serving as standard recipes for instruction-tuned models [Christiano et al., 2017, Ouyang et al., 2022, Rafailov et al., 2023] . Most existing analyses and algorithms formulate preference feedback as pairwise supervision under a Bradley–Terry (BT) model [Bradley and Terry, 1952, Rafailov et al., 2023], where each training example identifies a chosen response and a rejected response. However, many modern preference datasets are naturally listwise: for a single prompt, multiple candidate responses are available, and the supervision may contain a full or partial ranking. This has motivated Plackett–Luce (PL) listwise preference objectives [Plackett, 1975, Xia et al., 2008, Liu et al., 2025, Song et al., 2024], which exploit the relative ordering of multiple candidates rather than reducing the feedback to isolated pairwise comparisons.  \nPreprint.  \nWhile these works establish listwise preference optimization, they do not address robustness to uncertainty in the observed ranking label itself. To the best of our knowledge, this is the first work to study robust listwise preference optimization for LLM alignment under conditional ranking-label ambiguity.  \nRobust preference optimization has developed largely along a different axis. Recent robust DPOstyle methods introduce distributional uncertainty over the empirical preference dataset, the prompt distribution, or the preference-pair distribution [Wu et al., 2025, Mandal et al., 2025, Xu et al., 2026] . These approaches address an important question: how should alignment behave when the distribution from which prompts or preference pairs are sampled is perturbed? They do not directly address a different and common source of uncertainty in listwise supervision: even after conditioning on the same prompt and the same candidate list, the observed ranking label itself may be unreliable.  \nThis paper studies this conditional ranking-label uncertainty. Giv","cbCaio5K5gBZK8i5","https://ap.wps.com/l/cbCaio5K5gBZK8i5","pdf",651637,48,"English","# Abstract\n# 1 Introduction","[{\"question\":\"What problem does the paper address in listwise preference optimization?\",\"answer\":\"It addresses robustness when the observed ranking label over a candidate list is uncertain due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model-induced noise.\"},{\"question\":\"How does the proposed method model ranking-label uncertainty?\",\"answer\":\"It allows the ranking-label distribution to vary within a pointwise total-variation ambiguity set around the empirical ranking, conditional on a realized candidate list.\"},{\"question\":\"What is the key technical contribution regarding the worst-case ranking?\",\"answer\":\"The robust loss decomposes into the nominal Plackett–Luce loss plus a worst-case correction, and the worst-case ranking is obtained by sorting current implicit scores, reducing computation from K! enumeration to O(K log K).\"}]","Distributionally Robust Listwise Preference Optimization | PDF",1789976978,121]