[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84443-en":3,"doc-seo-84443-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84443,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization","Diffusion large language models (dLLMs) offer higher inference throughput than autoregressive models, but achieving comparable reasoning performance requires reinforcement learning tailored to dLLMs’ characteristics. The work introduces Distribution Matching Policy Optimization (DMPO), an RL fine-tuning method that matches the dLLM policy distribution to an optimal reward-tilted distribution via cross-entropy optimization. It addresses small-batch training using a weight baseline subtraction technique and improves accuracy on multiple reasoning benchmarks without supervised finetuning.","Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy  \nOptimization  \nYuchen Zhu * 1 Wei Guo * 1 Jaemoo Choi 1 Petr Molodyk 1 Bo Yuan 1 Molei Tao † 1 Yongxin Chen † 1  \narXiv :2510 .08233v 3 [ cs .LG] 12 Jul 2026  \nAbstract  \nDiffusion large language models (dLLMs) are promising alternatives to autoregressive large language models (AR-LLMs), as they potentially allow higher inference throughput. Reinforcement learning (RL) is crucial to enabling dLLMs to achieve performance comparable to that of AR-LLMs on important tasks, such as reasoning.  \nHowever, RL algorithms well-suited to dLLMs’unique characteristics have yet to be developed.  \nThis paper proposes Distribution Matching Policy Optimization (DMPO), a principled and theoretically grounded RL fine-tuning method specifically designed to enhance the reasoning capabilities of dLLMs by matching the dLLM policy distribution to the optimal, reward-tilted one through cross-entropy optimization. We identify a key implementation challenge with small training batch sizes and propose several effective solutions based on a novel weight baseline subtraction technique.  \nDMPO exhibits superior performance on multiple reasoning benchmarks without supervised finetuning, achieving up to a 39 .63 percentage-point improvement in accuracy over prior non-DMPORL baselines and 67.97 percentage points over the base model, underscoring the effectiveness of the distribution-matching framework. Our code is available at [https://github.com/yuche](https://github.com/yuche)n-zhu-zyc/DMPO.  \n1. Introduction  \nAutoregressive large language models (AR-LLMs) have demonstrated remarkable capabilities in addressing sophisticated reasoning tasks, such as solving challenging math  \n*Equal contribution †Equal advising 1 Georgia Institute of Technology. Correspondence to: Yuchen Zhu \u003C[yzhu738@gatech.edu](yzhu738@gatech.edu) >, Yongxin Chen \u003C[yongchen@gatech.edu](yongchen@gatech.edu) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \nquestions and completing coding tasks (Jaech et al., 2024 ; Anthropic, 2025 ; Guo et al., 2025a ; Novikov et al., 2025 ; Kimi Team et al., 2025) . While these models form their amazing capabilities from pretraining on massive text corpora, the main powerhouse behind the success is scaling the post-training phase with reinforcement learning (RL) techniques, such as Proximal Policy Optimization (PPO, Schulman et al. (2017)) and Group Relative Policy Optimization (GRPO, Shao et al. (2024)), which enhance model abilities through exploration of reward functions and go beyond static datasets. While possessing extraordinary competence, AR-LLMs are known to be expensive for inference due to their sequential, fixed left-to-right generation order, which currently prohibits large-scale deployment.  \nTo address such issues, diffusion large language models (dLLMs) have been investigated as an alternative to the AR models. Unlike AR-LLMs, dLLMs iteratively refine a sequence from masked inputs, enabling any-order generation, and have shown promising performance on text modeling tasks. dLLMs such as LLaDA (Nie et al., 2025b) and Dream (Ye et al., 2025), have demonstrated competitive performances on many tasks compared to similar-size AR baselines. Recently, commercial models such as Mercury (Inception Labs et al., 2025) and Gemini Diffusion (DeepMind) have demonstrated the ability to achieve significantly higher inference throughput without sacrificing generation quality, suggesting dLLMs as a promising direction for lan  \nguage modeling. However, one question that remains largely unanswered is how to transfer the success of RL on LLMs to dLLMs, thereby further scaling up their capability.  \nDesigning RL algorithms for dLLMs faces two major challenges. Due to the bidirectional nature of dLLMs, estimating the log probability of generated sequences is more expensive than for AR models,","cbCaikTA89FAtsTZ","https://ap.wps.com/l/cbCaikTA89FAtsTZ","pdf",1326659,1,26,"English","en",105,"# Introduction\n## Diffusion LLMs and RL Motivation\n## Challenges in Adapting RL to dLLMs\n## Proposed Method: DMPO","[{\"question\":\"What problem does DMPO address in training diffusion LLMs for reasoning?\",\"answer\":\"DMPO targets the lack of RL algorithms tailored to dLLMs, aiming to improve reasoning performance by transferring RL success from AR-LLMs to diffusion models.\"},{\"question\":\"How does DMPO differ from reward-maximization RL approaches?\",\"answer\":\"DMPO matches the entire reward-tilted policy distribution rather than focusing only on maximizing a reward mode, using cross-entropy optimization within a stochastic optimal control-based framework.\"},{\"question\":\"What challenge arises from small training batch sizes, and how is it handled?\",\"answer\":\"The paper identifies implementation difficulties when training batches are small, and proposes solutions including a weight baseline subtraction technique to stabilize training.\"}]",1784195665,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"enhancing-reasoning-for-diffusion-llms-via-distribution-matching-policy-optimization","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/enhancing-reasoning-for-diffusion-llms-via-distribution-matching-policy-optimization/84443/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does DMPO address in training diffusion LLMs for reasoning?","Question",{"text":75,"@type":76},"DMPO targets the lack of RL algorithms tailored to dLLMs, aiming to improve reasoning performance by transferring RL success from AR-LLMs to diffusion models.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does DMPO differ from reward-maximization RL approaches?",{"text":80,"@type":76},"DMPO matches the entire reward-tilted policy distribution rather than focusing only on maximizing a reward mode, using cross-entropy optimization within a stochastic optimal control-based framework.",{"name":82,"@type":73,"acceptedAnswer":83},"What challenge arises from small training batch sizes, and how is it handled?",{"text":84,"@type":76},"The paper identifies implementation difficulties when training batches are small, and proposes solutions including a weight baseline subtraction technique to stabilize training.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]