[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82942-en":3,"doc-seo-82942-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82942,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","What Does a Discrete Diffusion Model Learn","Discrete diffusion models raise a foundational question: what is learned by negative-ELBO minimization—denoisers, score ratios, bridge plug-in predictors, or leave-one-out–style predictors? A rigorous CTMC ELBO derivation shows the oracle distance identity: the negative ELBO equals the data entropy plus the path KL to the oracle reverse process, not a bound. The unique optimizer is the conditional expectation of the true reverse jump rate, and token-factorizing noise yields exact coordinates with closed-form conversions, recovering multiple literature losses as special cases.","arXiv :2607 .0538 1v 1 [ cs .LG] 6 Jul 2026  \nWhat Does a Discrete Diffusion Model Learn?  \nRodrigo Casado Noguerales∗1, Bernhard Sch¨olkopf1,2,3 , Thomas Hofmann 1 , and Aran  \nRaoufi 1  \n1 ETH Zurich  \n2 Max Planck Institute for Intelligent Systems, T¨ubingen  \n3 ELLIS Institute T¨ubingen  \nJuly 7, 2026  \nAbstract  \nWhat does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process, boundary terms included, we prove the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one, not merely a bound. Its unique optimizer is therefore the conditional expectation of the true reverse jump rate given the current noisy state, and its irreducible cost is the rate at which the forward process Zt destroys information about the clean data Z0 ,− ~~d~~dtI (Z0 ;Zt ), so every noising process shares the same best achievable negative ELBO: the data entropy. For sequences with token-factorizing noise, the oracle projection yields three exact coordinates for the optimizer: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them. This framework identifies which law each loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases; explains why denoiserand cavity coincide for masked diffusion but not for uniform diffusion; proves that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite; and calibrates ELBO implementations exactly at initialization. Every identity is verified numerically, without approximation, on an exactly solvable model.  \n∗ Correspondence: [rcasado@ethz.ch](rcasado@ethz.ch)  \nContents  \n1 Introduction 3  \n2 A general theory of discrete diffusion 4  \n2.1 The general theory ..................................... 4  \n2.2 Discrete diffusion for sequential data ........................... 11  \n3 Background and related work 17  \n4 A continuous-time discrete diffusion framework 19  \n4.1 Variational formulation of diffusion models ....................... 19  \n4.2 CTMC formulation ..................................... 21  \n4.3 The CTMC ELBO ..................................... 24  \n4.4 Importance sampling, time clocks, and the empirical ELBO .............. 29  \n5 What negative ELBO minimization really optimizes 31  \n5.1 ELBO as distance to the oracle reverse process ..................... 31  \n5.2 The ELBO as a projection problem and the Pythagorean decomposition ....... 33  \n5.3 The oracle ELBO is the rate of information loss ..................... 35  \n5.4 The universal NELBO floor ................................ 36  \n5.5 An alternative proof of the Oracle Distance theorem .................. 37  \n5.6 A Bregman-divergence and generator-matching view .................. 39  \n6 Sequence modeling 41  \n6.1 Token-factorizable noising processes ........................... 41  \n6.2 Three exact token-wise factorizations of the reverse rate ................ 42  \n6.3 Parameterizations in the literature ............................ 45  \n7 Masked, uniform, and GIDD noising processes 49  \n7.1 Common CTMC form ................................... 49  \n7.2 Masked diffusion ...................................... 50  \n7.3 Uniform diffusion ...................................... 52  \n7.4 Generalized interpolating discrete diffusion ....................... 56  \n8 Numerical verification 60  \n9 Conclusion 60  \n1 Introduction  \nDiscrete diffusion models offer a natural route to generative modeling on language and other categorical data: corrupt a sequence directly on its alphabet, then learn to run the corruption backward [AJH+21 , ","cbCaiaXSmEMXa3mt","https://ap.wps.com/l/cbCaiaXSmEMXa3mt","pdf",1170853,1,66,"English","en",105,"# Introduction\n# A general theory of discrete diffusion\n## Discrete diffusion for sequential data\n# Background and related work\n# A continuous-time discrete diffusion framework\n## The CTMC ELBO\n# What negative ELBO minimization really optimizes\n## ELBO as distance to the oracle reverse process\n# Sequence modeling\n## Token-factorizable noising processes\n# Masked, uniform, and GIDD noising processes\n## Masked diffusion","[{\"question\":\"What does negative-ELBO minimization learn in discrete diffusion models?\",\"answer\":\"It learns the reverse jump process induced by the model, with the negative ELBO exactly matching the data entropy plus a path KL to the oracle reverse process. The unique optimizer corresponds to the conditional expectation of the true reverse jump rate given the current noisy state.\"},{\"question\":\"Why do the same neural-network outputs not always correspond to the same learned diffusion process?\",\"answer\":\"A network output is not a reverse process until the conversion to jump rates is specified. Interpreting the same output in the wrong coordinate changes the continuous-time Markov chain being trained and sampled from.\"},{\"question\":\"How do masked diffusion and uniform diffusion differ regarding denoiser versus cavity predictions?\",\"answer\":\"In masked diffusion, denoiser and cavity coincide because at masked positions the local observation carries no information about the clean token. For uniform and hybrid diffusion, this cancellation does not occur, so the coordinate conversions are intrinsic to the model.\"}]",1784184193,166,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"what-does-a-discrete-diffusion-model-learn","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/what-does-a-discrete-diffusion-model-learn/82942/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What does negative-ELBO minimization learn in discrete diffusion models?","Question",{"text":74,"@type":75},"It learns the reverse jump process induced by the model, with the negative ELBO exactly matching the data entropy plus a path KL to the oracle reverse process. The unique optimizer corresponds to the conditional expectation of the true reverse jump rate given the current noisy state.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why do the same neural-network outputs not always correspond to the same learned diffusion process?",{"text":79,"@type":75},"A network output is not a reverse process until the conversion to jump rates is specified. Interpreting the same output in the wrong coordinate changes the continuous-time Markov chain being trained and sampled from.",{"name":81,"@type":72,"acceptedAnswer":82},"How do masked diffusion and uniform diffusion differ regarding denoiser versus cavity predictions?",{"text":83,"@type":75},"In masked diffusion, denoiser and cavity coincide because at masked positions the local observation carries no information about the clean token. For uniform and hybrid diffusion, this cancellation does not occur, so the coordinate conversions are intrinsic to the model.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]