[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84148-en":3,"doc-seo-84148-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84148,2336464648746,"Skyler","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Trees from Marginals Autoregressive Drafting with Factorized Priors","Speculative decoding increases the interactivity of autoregressive language models by shifting computation into extra tokens generated during a single forward pass, but factorized draft models suffer from degraded acceptance due to independence assumptions at later draft positions. The work analyzes this limitation and proposes Weaver, a lightweight autoregressive adapter that builds proposal trees from top predictions of a factorized drafter. Weaver restores conditional dependencies while avoiding full vocabulary projection. It enables rollback-free tree verification for Gated Delta Net layers via optimized CUDA kernels in SGLang, achieving 4.37× speedup over autoregressive decoding and improving over a DFlash baseline by 24.7%.","Trees from Marginals: Autoregressive drafting with factorized priors  \nYuma Oda, Ryan Mathieu, Roman Knyazhitskiyand Artur Chakhvadze1  \nAbstract  \nSpeculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply at later draft positions. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top 􀀁 prediction of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding the cost of a full vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback free tree verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37 × speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7% .  \n1Correspondence [to ](to artur@getmirai.co)[artur@](to artur@getmirai.co)[g](to artur@getmirai.co)[etmirai.co](to artur@getmirai.co)  \nmtbench gsm8k  \nmath500 aime25  \nhumaneval mbpp  \nlcbsharechat  \noverall  \nInteractivity (tok/s/sequence)  \nAR baseline  DFlash  DDTree  DFlash TfM (ours)  \nFigure 1: Comparison of decoding speed for Qwen3.6 27B in SGLang in bfloat16 precision on different datasets with default sampling parameters (temperature 1.0, reasoning on) . Our method outperforms an optimized DFlash baseline by 25% on average.  \n1 Introduction  \n1.1 Motivation  \nThe dominant neural architecture powering large language models is an autoregressive transformer[1]. It allows for parallel likelihood evaluation and training, but generation remains sequential, since the generated sequence has to be sampled autoregressively one token per step.  \nFor a given hardware setup, the time spent on each generation step is a function of communication and computation cost. Communication involves moving the weights, activations and KV cache between DRAM and SRAM1, while computation consists mostly of matrix multiplications and dot product attention operations. A reasonably good inference implementation can overlap computation and communication to a high degree, so the overall time spent on a single decoding step can be well approximated as  \nlatency = max ( aritcompuhmetictation~~ ~~costperformance , communicmemory~~ ~~baatniodnwicostdth)  \n1 For the sake of simplicity we only consider single GPU deployments in this exposition.  \nWhen generating a single sequence, this latency is dominated by the memory traffic due to streaming weights from DRAM, which makes generation memorybound. This is undesirable because of compute underutilization and poor energy efficiency per token, as DRAM accesses consume orders of magnitude more energy than arithmetic operations[2] .  \nIn order to increase the ratio of computations to memory traffic, server LLM inference implementations make use of batching – decoding multiple sequencesin parallel. Batching increases the cost of computation while keeping the weight traffic constant. Unfortunately, it is not a free lunch, as larger batch sizes result in increased activation and KV cache traffic, and the latter quickly becomes the bottleneck for long sequences.  \nAdditionally, increased batch size reduces interactivity1, which is critical for user facing applications and asynchronous reinforcement learning algorithms, where the policy gradient bias depends on the speed of rollout completion[3],[4] .  \nSpeculative decoding[5] is an orthogonal approach for increasing the arithmetic intensity of generation while simultaneously increasing interactivity and maintaining constant KV cache traffic. The idea is to use a fast auxiliary draft model to generate a sequence of proposal tok","cbCaimJoAlMA9yqZ","https://ap.wps.com/l/cbCaimJoAlMA9yqZ","pdf",933183,1,35,"English","en",105,"# Abstract\n# 1 Introduction\n## 1.1 Motivation\n## 1.2 Our contributions","[{\"question\":\"Why do acceptance rates of factorized draft models degrade at later draft positions?\",\"answer\":\"Because factorized draft models use an independence assumption, their marginal token predictions diverge from the true autoregressive conditionals as draft length increases, sharply reducing acceptance rates later in the draft.\"},{\"question\":\"What does Weaver do to improve proposal quality without full vocabulary projection?\",\"answer\":\"Weaver constructs proposal trees from the top predictions of a factorized drafter and restores conditional dependencies between proposed tokens, while avoiding the cost of a full vocabulary projection.\"},{\"question\":\"How is fast verification supported for models with Gated Delta Net layers?\",\"answer\":\"The approach derives a rollback-free tree verification algorithm and implements optimized CUDA kernels in SGLang to enable fast verification during speculative decoding.\"}]",1784193438,88,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"trees-from-marginals-autoregressive-drafting-with-factorized-priors","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/trees-from-marginals-autoregressive-drafting-with-factorized-priors/84148/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do acceptance rates of factorized draft models degrade at later draft positions?","Question",{"text":75,"@type":76},"Because factorized draft models use an independence assumption, their marginal token predictions diverge from the true autoregressive conditionals as draft length increases, sharply reducing acceptance rates later in the draft.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What does Weaver do to improve proposal quality without full vocabulary projection?",{"text":80,"@type":76},"Weaver constructs proposal trees from the top predictions of a factorized drafter and restores conditional dependencies between proposed tokens, while avoiding the cost of a full vocabulary projection.",{"name":82,"@type":73,"acceptedAnswer":83},"How is fast verification supported for models with Gated Delta Net layers?",{"text":84,"@type":76},"The approach derives a rollback-free tree verification algorithm and implements optimized CUDA kernels in SGLang to enable fast verification during speculative decoding.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]