[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85057-en":3,"doc-seo-85057-105":30,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85057,1099514067415,"Rowan","https://ap-avatar.wpscdn.com/avatar/100002539d78ffe74a7?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779092875211072502",8,"Research & Report","Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models","Scaling Large Language Models often relies on enlarging Transformer backbones, which triggers costly additional pretraining and higher training/serving cost. This work tests whether a fixed backbone can keep improving by allocating more computation per token via continued pretraining. It introduces Hidden Decoding, a sequence-length expansion that splits each token into multiple streams with independent embeddings, retaining intermediate key–value caches as context. To make it scalable, Stream-Factorized Attention reduces attention cost from quadratic to near-linear in the expansion factor. Experiments show consistent gains on multiple benchmarks, including 100B+ scale models (WeLM-HD4-80B and WeLM-HD4-617B).","arXiv :2607 .08 186v 1 [ cs .CL] 9 Jul 2026  \nHidden Decoding at Scale: Latent Computation Scaling for  \nLarge Language Models  \nWeChat AI Team  \nAbstract  \nScaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Depth-recurrent (looped) Transformers pursue this goal but are hard to scale, because looped computation does not fit naturally with the pipeline parallelism used to train the largest models. We add computation along the sequencelength dimension, where the extra computation is simply a longer input and stays compatible with standard large-model training. We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining (CPT) . It expands each token into 􀀼 streams with independent embedding tables and keeps the intermediate streams’ key–value cache as context, so each token performs more internal computation without adding or widening Transformer layers. To keep this affordable at scale, we introduce Stream-Factorized Attention, in which most layers attend only within each stream and only a few layers mix across streams, reducing the attention cost from quadratic to roughly linear in 􀀼 . Experiments support two scaling results. At frontier scale, we train WeLM-HD4-80B and WeLM-HD4-617B at 􀀼 =4 and improve their matched non-HD baselines, making Hidden Decoding the first demonstrated sequence-length scaling method at the 100B+ MoE scale. Across expansion factors, the gains grow as 􀀼 increases, showing that sequence-length expansion is a practical fixed-backbone scaling path for frontier-scale LLMs.  \nAvg . accuracy (8B)  \n83  \n82  \n81  \n80  \n79  \n78  \n77  \n76  \n(a) Hidden Decoding scales with n  \nFrontierMath  \nPHYBench  \nMathArena Apex  \nGPQA Diamond  \nHLE  \nMMMLU  \n(b) Early SFT-only gains on WeLM-HD4 models  \n\n| \u003Cbr> +0.5\u003Cbr>\u003Cbr> | +1.0 |  | +2.0 | +3.2\u003Cbr>+3.7 | +4.0 |\n| --- | --- | --- | --- | --- | --- |\n|  | +1.2 |  | +2.1 |  |  |\n|  | +1.0 | + | 1.8 |  |  |\n|  | +1.2 +1.1 |  |  |  WeLM\u003Cbr> WeLM | -HD4-80B\u003Cbr>-HD4-617B |\n|  |  |  |  |  |  |\n\nn = 1 n = 2 n = 4 n = 8  \nExpansion factor  \n0 1 2 3 4 Improvement over Base Δ (pts)  \nFigure 1 | Hidden Decoding scales and improves frontier models. (a) On the dense Qwen3- 8B-Base, average accuracy grows steadily with the expansion factor 􀀼 . (b) Improvement Δ (in points) of WeLM-HD4-80B and WeLM-HD4-617B over their matched non-HD counterparts, WeLM-80B and WeLM-617B, after the same early SFT-only post-training.  \n1. Introduction  \nScaling has been a primary source of progress in Large Language Models (LLMs) . Increasing model size, training data, and training compute has consistently led to stronger models [16, 21] . However, for already strong foundation models, further scaling the Transformer backbone is costly: it often requires another round of large-scale pretraining and increases both training and serving costs. This motivates the setting of this paper: improving an existing backbone by giving each token more internal computation while keeping the Transformer backbone fixed.  \nRecent reasoning models and test-time scaling studies show that, even with fixed parameters, spending more computation per problem can improve accuracy [9, 27, 29, 39] . This motivates agoal that recent latent-reasoning work has started to pursue: moving part of the extra thinking computation from visible tokens into the model’s internal computation.  \nA prominent direction toward this goal is recurrent-depth or looped Transformers, which add computation by reusing the same Transformer blocks several times on each token [10, 31, 38, 54] . This is a direct way to increase per-token computation with a fixed set of Transformer weights. However, it is hard to scale. Training the largest MoE models ","cbCaivp64le0EhqV","https://ap.wps.com/l/cbCaivp64le0EhqV","pdf",2940016,4,1,30,"English","en",105,"# Abstract\n# 1. Introduction\n## Motivation for fixed-backbone scaling\n## Limits of looped Transformers with pipeline parallelism\n## Sequence-dimension computation as the scaling path\n## Hidden Decoding approach in CPT","[{\"question\":\"What do experiments show about the scaling behavior of Hidden Decoding?\",\"answer\":\"Across expansion factors, gains increase as the expansion factor grows, indicating sequence-length expansion is a practical fixed-backbone scaling route. At frontier scale, models such as WeLM-HD4-80B and WeLM-HD4-617B improve over matched non-HD baselines and demonstrate sequence-length scaling at the 100B+ MoE scale.\"}]",1784200696,76,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":28},"hidden-decoding-at-scale-latent-computation-scaling-for-large-language-models","",{"@graph":36,"@context":77},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/hidden-decoding-at-scale-latent-computation-scaling-for-large-language-models/85057/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"What do experiments show about the scaling behavior of Hidden Decoding?","Question",{"text":75,"@type":76},"Across expansion factors, gains increase as the expansion factor grows, indicating sequence-length expansion is a practical fixed-backbone scaling route. At frontier scale, models such as WeLM-HD4-80B and WeLM-HD4-617B improve over matched non-HD baselines and demonstrate sequence-length scaling at the 100B+ MoE scale.","Answer","https://schema.org",{"og:url":52,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,114,119,122,126],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":46,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":46,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":22,"slug":113},"research-report",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":98,"slug":129},19,"General","general"]