[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85531-en":3,"doc-seo-85531-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85531,8796095360427,"Lucas Martin","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","MetaState Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models","Discrete diffusion language models generate text by iteratively denoising a masked sequence, yet standard designs condition each step only on the current hard-masked tokens and discard intermediate continuous representations after sampling and remasking. This creates the Information Island problem, where rich step-wise information cannot propagate across the denoising trajectory, undermining multi-step reasoning. MetaState introduces persistent fixed-size working memory via three recurrent modules around a frozen dLLM, trained with K-step unrolling, improving reasoning and code generation by 4.5 percentage points on average.","arXiv :2603 .0 133 1v 3 [ cs .CL] 10 Jul 2026  \nMetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models  \nKejing Xia1, Mingzhe Li2, Lixuan Wei3, Zhenbang Du1, Xiangchi Yuan1, Dachuan Shi1, Qirui Jin1, Wenke Lee1  \n1Georgia Institute of Technology 2University of Massachusetts Amherst  \n3Harvard University  \nAbstract  \nDiscrete diffusion language models (dLLMs) generate text by iteratively denoising a masked sequence. However, standard dLLMs condition each denoising step solely on the current hard-masked sequence, while intermediate continuous representations are discarded after sampling and remasking. We term this bottleneck the Information Island issue: continuous information remains isolated within individual denoising steps and fails to propagate across the trajectory. This bottleneck is especially harmful for reasoning, which requires intermediate reasoning state to be preserved and updated across many denoising steps. To address this limitation, we introduce MetaState, a lightweight recurrent augmentation that equips a frozen dLLM backbone with persistent, fixed-size working memory. MetaState comprises three modules with a shared time conditioner: a cross-attention Mixer that reads backbone activations into memory slots, a GRU-style Updater that integrates information across steps, and a cross-attention Injector that writes the updated memory back into the backbone. We train these modules with a dedicated K-step unrolling pipeline to learn multi-step dynamics. MetaState adds only ∼0.6% trainable parameters while keeping the backbone frozen, and consistently improves reasoning performance over frozen baselines on mathematical reasoning and code generation benchmarks, with an average gain of 4.5 percentage points across all evaluations.  \nOur code is available at [https://github.com/Les1a/MetaState](https://github.com/Les1a/MetaState).  \n1 Introduction  \nAutoregressive (AR) language models factorize the joint distribution over sequences into a product of conditional probabilities, producing one token per forward pass (Radford et al., 2018; 2019; Brown et al., 2020) . Although this paradigm underlies many recent foundation models, the left-to-right causal structure prevents parallel decoding and limits the use of bidirectional context. Discrete diffusion language models (dLLMs) have recently emerged as a non-autoregressive alternative (Li et al., 2025; Sahoo et al., 2024; Gonget al., 2024) . Starting from a fully corrupted sequence, dLLMs iteratively denoise to recover clean text and update arbitrary positions using bidirectional attention (Austin et al., 2021a) . When scaled to billions of parameters, dLLMs achieve quality comparable to that of autoregressive models while retaining the advan-  \nBase  Base + MetaState  Instruct  Instruct + MetaState  \nFigure 2: Performance comparison between MetaState and frozen baselines on reasoning benchmarks for LLaDA-8B and Dream-7B in both Instruct and Base versions.  \ntages of decoding parallelism and generation flexibility, as demonstrated by the LLaDA series (Nie et al., 2025; Zhu et al., 2025a; Bie et al., 2025) and Dream (Ye et al., 2025) .  \nCurrent standard dLLMs nevertheless share a limitation that we term the Information Island issue. In the diffusion formulation, the inter-step process is Markovian in the discrete sequence state: each transition conditions on the current masked tokens xt, while the continuous hidden representation ht computed at that step is not carried forward explicitly,  \nT  \np θ (x0:T ) = p(xT)  p θ (xt−1 | xt), xt−1 = S (ht) . (1)  \nAt each denoising step t, the model computes a high-dimensional hidden representation ht that encodes substantially richer information than the discrete tokens passed to the nextstep. Beyond token-level predictive semantics, ht also captures long-range dependencies and global sequence structure information. However, the sampling-and-remasking interface S maps this rich continuous state to ","cbCaimziWHjheenH","https://ap.wps.com/l/cbCaimziWHjheenH","pdf",1572976,1,27,"English","en",105,"# Abstract\n# Introduction\n## Information Island issue\n## MetaState approach","[{\"question\":\"What is the Information Island issue in discrete diffusion language models?\",\"answer\":\"Continuous hidden representations computed at each denoising step are discarded after sampling and remasking, so rich information cannot propagate across steps. The next step therefore relies on a lossy discrete token sequence, repeatedly losing intermediate signals needed for reasoning.\"},{\"question\":\"How does MetaState improve reasoning in discrete diffusion models?\",\"answer\":\"MetaState adds persistent working memory to a frozen dLLM using a recurrent loop with three modules: a cross-attention Mixer to read activations into memory slots, a GRU-style Updater to integrate information across steps, and a cross-attention Injector to write updated memory back into the backbone.\"},{\"question\":\"What training strategy and performance gains does MetaState report?\",\"answer\":\"MetaState trains its modules with a dedicated K-step unrolling pipeline to learn multi-step dynamics while keeping the backbone frozen. It adds about 0.6% trainable parameters and achieves an average improvement of 4.5 percentage points on mathematical reasoning and code generation benchmarks.\"}]",1784204212,68,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"metastate-persistent-working-memory-enhances-reasoning-in-discrete-diffusion-language-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/metastate-persistent-working-memory-enhances-reasoning-in-discrete-diffusion-language-models/85531/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the Information Island issue in discrete diffusion language models?","Question",{"text":75,"@type":76},"Continuous hidden representations computed at each denoising step are discarded after sampling and remasking, so rich information cannot propagate across steps. The next step therefore relies on a lossy discrete token sequence, repeatedly losing intermediate signals needed for reasoning.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does MetaState improve reasoning in discrete diffusion models?",{"text":80,"@type":76},"MetaState adds persistent working memory to a frozen dLLM using a recurrent loop with three modules: a cross-attention Mixer to read activations into memory slots, a GRU-style Updater to integrate information across steps, and a cross-attention Injector to write updated memory back into the backbone.",{"name":82,"@type":73,"acceptedAnswer":83},"What training strategy and performance gains does MetaState report?",{"text":84,"@type":76},"MetaState trains its modules with a dedicated K-step unrolling pipeline to learn multi-step dynamics while keeping the backbone frozen. It adds about 0.6% trainable parameters and achieves an average improvement of 4.5 percentage points on mathematical reasoning and code generation benchmarks.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]