LiLiCorr - Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding

Sep 15, 2026

Speculative decoding speeds up autoregressive language-model inference by letting a small draft propose future tokens that the target model verifies in parallel, improving throughput when drafts are both long and cheap. Diffusion-style block drafters like DFlash predict token slots independently but are trained only on per-position marginals, producing blocks that are individually plausible yet jointly incoherent. LiLiCorr correlates per-slot marginal distributions with a lightweight likelihood-based scoring scheme, enabling coherent accepted draft blocks with minimal added latency.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

AccessFree to download

Introduction

Overview of LiLiCorr

Marginals issue in diffusion-style block drafters

Coherence recovery approaches

arXiv : 2608 .20530v 1 [cs .CL] 20 Aug 2026 LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding Matan Rusanovsky* Yoav Miron Roy Uziel Omer Belhasin Ran Zilberstein Maor Ashkenazi Michael Elad NVIDIA Abstract Speculative decoding accelerates language-model inference by drafting future tokens that the target model verifies in parallel. A diffusion-style block head such as DFlash isan attractive drafter, predicting an entire block of future tokens in a single forward pass. However, it is trained on per-position marginals rather than the joint block distribution, so the tokens it emits are individually plausible yet jointly incoherent. We introduce LiLiCorr, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions such a drafter already produces. It keeps the top-K tokens at each position as candidates and processes them jointly, producing for each an in and an out vector. These vectors link candidates at consecutive positions: a pair matches when the earlier candidate’s out vector has high cosine similarity with the later candidate’s in vector. These matches capture the block’s joint structure without ever materializing the full joint distribution, which is exponential in the block length. Training makes the correct pairings score highest while pushing competing ones down, so coherent blocks outscore incoherent ones. One lightweight network pass produces all the vectors, and the pairwise scores are then computed in parallel as batched matrix operations, leaving only a cheap greedy walk sequential. We further co-train the DFlash drafter with LiLiCorr, soit learns to propose candidates that correlate into longer accepted sequences. Over the vanilla DFlash drafter, LiLiCorr raises acceptance length on every benchmark, by 9 to 19%, while its single-pass scoring head accounts for only about 2.8% of the per-block latency. Against DFlash and two concurrently developed methods that also restore co herence at draft time, LiLiCorr delivers the highest throughput in 70 of 72 settings: nine benchmarks at two target sizes under greedy and temperature-one decoding, anda throughput sweep over six concurrencies, two input lengths and three entropy tiers, with all systems equally optimized on a common serving stack. Extending LiLiCorr to inputs an order of magnitude longer than it was trained on preserves that lead. 1 Introduction Speculative decoding accelerates autoregressive language-model inference by letting a small draft model propose several future tokens that the target model verifies in parallel, keeping the longest correct prefix (8, 3) . Because decoding is memory-bandwidth bound, verifying a multitude of tokens together costs little more than generating just one token. The resulting speedup depends on two quantities: the acceptance length, the expected number of draft tokens accepted per step, and the cost of the drafter itself. A good drafter must raise the former without inflating the latter. Much effort therefore goes into raising acceptance length, with designs ranging from feature-autoregressive heads such as EAGLE (9, 10, 11) to multi-head predictors such as Medusa (2) . A particularly promising approach predicts an entire block of future tokens in a single forward pass via a parallel diffusion-style drafter, such as DFlash (4), offering a long speculative window at very low cost. * Correspondence to Matan Rusanovsky, [mrusanovsky@nvidia.com](mrusanovsky@nvidia.com). × “The dog meows sweetly” in out ✓“The cat meows softly” (a) the marginal drafter is jointly wrong (b) one pass over all candidates (c) cheap greedy decoding one network pass, all slots at once argmax walk, no network weights Figure 1: Overview of LiLiCorr. (a) DFlash produces a distribution logit over the vocabulary at each slot in one forward pass, and taking the most probable token of each slot independently yields a sequence that is locally probable but

FAQs

How does speculative decoding accelerate language-model inference?

A draft model proposes several future tokens, and the target model verifies them in parallel, committing the longest correct prefix. Since verification is memory-bandwidth bound, parallel checks add little cost compared to generating a single token.

What problem limits diffusion-style block drafters like DFlash?

DFlash is trained with supervision only on per-position marginals. This makes each slot token prediction plausible, while the assembled block can be jointly incoherent because the true joint block distribution is largely unconstrained.

What does LiLiCorr introduce to improve draft coherence?

LiLiCorr keeps top-K tokens per slot and computes joint coherence signals by correlating per-position candidates across consecutive slots. It scores in one lightweight network pass, then performs a cheap greedy walk to commit the block.

Sep 15, 2026

Uploaded by Evangeline

LiLiCorr - Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding_Page 1

Open in WPS to view all 24 pages