[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85110-en":3,"doc-seo-85110-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85110,687197207057,"Sage","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","A Practical Investigation of Training-free Relaxed Speculative Decoding","Speculative decoding accelerates sampling from an autoregressive LLM by drafting tokens with a faster auxiliary model and verifying them in parallel with the LLM. While standard speculative decoding is lossless through exact rejection and resampling that preserve the LLM’s sampling distribution, newer research relaxes this constraint for further speed-ups, controllable capability-speed trade-offs, or even capability gains. The work investigates training-free relaxed techniques, unifies methods, benchmarks on modern setups, and distills practitioner-oriented findings and takeaways.","arXiv :2607 .08690v 1 [ cs .LG] 9 Jul 2026  \nA Practical Investigation of Training-free Relaxed Speculative Decoding  \nGuoxuan Xia, Luka Ribar, [g.xia21@imperial.ac.uk](g.xia21@imperial.ac.uk)  \nPaul Balanca [paulb@graphcore.ai](paulb@graphcore.ai)  \nWork done at Graphcore  \nAbstract  \nSpeculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM’s sampling distribution.  \nRecent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners.  \nImportant takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters. Code is available at: [https://github.com/graphcore-research/relaxed-spec-dec-repro](https://github.com/graphcore-research/relaxed-spec-dec-repro).  \n1 Introduction  \nLarge language models (LLMs) dominate modern applications of machine learning (Sajadieh et al., 2026) . Language modelling as a task is itself in turn dominated by autoregressive (AR) models (Vaswani et al., 2017 ; Brown et al., 2020), i.e. models that predict one token after another in series given previous tokens. Such AR models represent the frontier of capability, enabling transformative outcomes in programming, scientific research, and many other applications (Sajadieh et al., 2026) . A core issue, however, is the inherent slowness of generation that arises from the requirement to sequentially produce tokens, especially in an era of abundant parallel compute resources (You et al., 2026) . Applications often require lower latencies in order to satisfy user requirements (Anthropic, 2025 ; OpenAI, 2026b)– intuitively, snappier LLM responses enable greater human productivity.  \nOne solution to this problem is speculative decoding (spec-dec) (Leviathan et al., 2023), which is motivated by the idea that for text generation, weaker but faster models may still be able to “speculate” decent short-range spans of text. Given an AR LLM, spec-dec employs a low-latency auxiliary model to “draft” tokens quickly, which are then “verified” in parallel using the LLM. This better utilises parallel compute and reduces overall latency. A key feature of this algorithm is that “bad” draft tokens are stochastically rejected and resampled during verification in such a way that strictly preserves the LLM’s output distribution, giving theoretically lossless acceleration. Speculative decoding is seeing increasing adoption across a wide range of models, such as DeepSeek and Qwen (DeepSeek-AI, 2024 ; Qwen Team, 2026), and inference frameworks such as vLLM and SGLang (Kwon et al., 2023 ; Zheng et al., 2024 ; Schmitt-Ulms et al., 2025) .  \nRecently, a new body of research has argued that the strict distribution preservation of speculative decoding is disadvantageous; generally, it proposes to relax this requirement in favour of sampling from an alternative distribution to the original LLM. There are a number of motivations for doing this, for example, to enable the option of trading off task capability for speed-up (Holsman et al., 2025), increasing speed-up by accepting semantically valid tokens that strict speculative decoding would reject (Bachmann et al., 2025), or even improving task capability (Yuan et al., 2024) . In this work, we perform a practical investigation of such relaxed speculative decoding approaches (in particular, those that are training","cbCaif7W2qxABDab","https://ap.wps.com/l/cbCaif7W2qxABDab","pdf",1386673,4,1,29,"English","en",105,"# Introduction\n## Speculative decoding and lossless guarantees\n## Relaxed speculative decoding motivations\n## Contributions overview\n# Methodology and algorithms\n## Strict speculative decoding primer\n## Relaxed speculative decoding framework","[{\"question\":\"What problem does speculative decoding address in autoregressive LLMs?\",\"answer\":\"It addresses the slowness of token-by-token generation by using an auxiliary model to draft tokens and the LLM to verify them in parallel, improving latency.\"},{\"question\":\"How does standard speculative decoding remain lossless?\",\"answer\":\"It stochastically rejects and resamples “bad” draft tokens during verification so the overall sampling distribution matches the original LLM’s distribution exactly.\"},{\"question\":\"What is the focus of training-free relaxed speculative decoding techniques?\",\"answer\":\"They relax the strict distribution-preservation requirement, sampling from alternative target distributions to enable speed-ups, capability–speed trade-offs, or potential capability gains without additional training.\"}]",1784201163,73,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"a-practical-investigation-of-training-free-relaxed-speculative-decoding","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/a-practical-investigation-of-training-free-relaxed-speculative-decoding/85110/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does speculative decoding address in autoregressive LLMs?","Question",{"text":75,"@type":76},"It addresses the slowness of token-by-token generation by using an auxiliary model to draft tokens and the LLM to verify them in parallel, improving latency.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does standard speculative decoding remain lossless?",{"text":80,"@type":76},"It stochastically rejects and resamples “bad” draft tokens during verification so the overall sampling distribution matches the original LLM’s distribution exactly.",{"name":82,"@type":73,"acceptedAnswer":83},"What is the focus of training-free relaxed speculative decoding techniques?",{"text":84,"@type":76},"They relax the strict distribution-preservation requirement, sampling from alternative target distributions to enable speed-ups, capability–speed trade-offs, or potential capability gains without additional training.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]