[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85721-en":3,"doc-seo-85721-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85721,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels","Direct Preference Optimization (DPO) is a prominent approach for aligning large language models with human preferences by avoiding explicit reward modeling and reinforcement learning optimization. Its effectiveness depends on preference data quality, and noisy preference labels can degrade alignment. This work introduces a bilevel optimization framework with theoretical guarantees to recover the DPO optimum from noisy data under assumptions. It also derives a prior weighting form under asymmetric label-flipping noise and proposes PACMR-DPO using task-agnostic meta-knowledge without metadata. Central-difference approximation plus LoRA fine-tuning enables scalable higher-order training, yielding improved results on TL;DR summarization and Anthropic HH dialogue across multiple noise rates.","Metadata-Free Meta-Reweighted Direct Preference Optimization  \nunder Noisy Preference Labels  \nHua Qu  \nXi’an Jiaotong University [qh@mail.xjtu.edu.cn](qh@mail.xjtu.edu.cn)  \nYifan Li Xi’an Jiaotong University [4123158005@stu.xjtu.edu.cn](4123158005@stu.xjtu.edu.cn)  \nXiaodong Yuan Xi’an Jiaotong University [xiaodongyuan@163.com](xiaodongyuan@163.com)  \narXiv :2607 .09796v 1 [ cs .LG] 9 Jul 2026  \nAbstract  \nDirect Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning optimization. However, its performance depends heavily on the quality of preference data, and noisy preference data in real-world settings can weaken alignment performance. To address this issue, we propose a bilevel optimization framework and prove, under certain assumptions, that this framework can recover the DPO optimum under clean data. We further derive a prior form for the learnable weighting function under asymmetric label-flipping noise. Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method that enables meta-learning even when metadata is completely unavailable. To reduce the high cost of higher-order gradients in LLM meta-learning, we combine central-difference approximation with LoRA fine-tuning and develop a scalable training scheme. Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.  \nKeywords Direct Preference Optimization 􀀁 preference noise 􀀁 meta-learning 􀀁 prompt augmentation consistency 􀀁 finite difference 􀀁 LoRA fine-tuning  \n1 Introduction  \nAligning large language models (LLMs) with human preferences is important [1, 2 , 3] . Reinforcement learning from human feedback (RLHF) has been shown to be effective for aligning LLMs with human preferences [4] . A typical RLHF pipeline first obtains an SFT policy through supervised fine-tuning, then learns a reward model (RM) from human preference data, and finally optimizes the policy with reinforcement learning algorithms such as PPO while constraining its distance from a reference model to avoid excessive policy drift.  \nAlthough RLHF plays an important role in LLM alignment, its training pipeline is relatively complex. It requires training an additional reward model and policy model, and the reinforcement learning stage involves frequent sampling, which leads to high computational and memory costs [5, 6] . Direct Preference Optimization (DPO) provides a simpler alternative for preference alignment: it directly optimizes the LLM from human preference data without explicitly learning a reward model and avoids the complexity of reinforcement learning [7] .  \nDespite greatly simplifying the traditional RLHF pipeline, DPO still depends heavily on the quality of preference data. Prior studies have shown that preference-data noise can degrade both the stability and final performance of alignment training [8, 9] . Therefore, effective DPO learning under noisy preference data has become an important problem in preference alignment.  \nExisting work has mainly addressed this problem by improving the noise robustness of DPO through loss correction, label smoothing, distributionally robust optimization, or noisy-sample filtering [10, 11 , 12 , 13 , 14 , 15] .  \nHowever, these methods often rely on predefined noise forms, fixed reweighting rules, or heuristic reliability estimates, which makes it difficult to simultaneously maintain interpretability and training effectiveness under complex noise conditions.  \nRecently, several studies have introduced meta-learning into preference optimization to adaptively characterize noisy preference samples and learn sample-level weights [16, 17] . These methods show that meta-learning can provide a more flexible mechanis","cbCaidF2plYlCtPO","https://ap.wps.com/l/cbCaidF2plYlCtPO","pdf",6631946,1,41,"English","en",105,"# Introduction\n## RLHF and DPO overview\n## Impact of noisy preference labels\n## Related work and limitations\n## Proposed PACMR-DPO and contributions","[{\"question\":\"为什么 DPO 对偏好数据质量特别敏感？\",\"answer\":\"DPO 的学习目标直接依赖偏好数据，噪声偏好标签会削弱对齐训练的稳定性与最终效果，因此效果高度受数据质量影响。\"},{\"question\":\"文中提出的 PACMR-DPO 如何在缺少干净元数据时进行元学习加权？\",\"answer\":\"PACMR-DPO 不依赖干净的元偏好标签，而是将与任务无关的提示增强一致性作为元知识注入外层优化，从而在完全无元数据的情况下仍能进行元学习与重加权。\"},{\"question\":\"如何降低 LLM 元学习中高阶梯度带来的训练成本？\",\"answer\":\"文中结合中心差分近似与在 LoRA 参数空间中的扰动，形成可扩展的训练方案，从而减少双层优化所需的内存与计算开销。\"}]",1784205805,103,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-preference-labels","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-preference-labels/85721/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么 DPO 对偏好数据质量特别敏感？","Question",{"text":75,"@type":76},"DPO 的学习目标直接依赖偏好数据，噪声偏好标签会削弱对齐训练的稳定性与最终效果，因此效果高度受数据质量影响。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"文中提出的 PACMR-DPO 如何在缺少干净元数据时进行元学习加权？",{"text":80,"@type":76},"PACMR-DPO 不依赖干净的元偏好标签，而是将与任务无关的提示增强一致性作为元知识注入外层优化，从而在完全无元数据的情况下仍能进行元学习与重加权。",{"name":82,"@type":73,"acceptedAnswer":83},"如何降低 LLM 元学习中高阶梯度带来的训练成本？",{"text":84,"@type":76},"文中结合中心差分近似与在 LoRA 参数空间中的扰动，形成可扩展的训练方案，从而减少双层优化所需的内存与计算开销。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]