Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters - Paper Abstract and Introduction

Q: What problem does the paper address in speculative decoding training?

It addresses a train-inference mismatch: block-style draft models train with full-block supervision even though speculative decoding accepts only the longest left-to-right accepted prefix and discards tokens after the first rejection.

Q: How does Accept-Until-Fail (AUF) modify training?

AUF keeps cross-entropy support only through the accepted prefix and the first failing token, masking out later positions. This approximates prefix-sensitive supervision in a mask-only drafter setting.

Q: What impact does AUF have on performance in the reported experiments?

With fixed drafter backbones and serving settings on Qwen3-8B, AUF increases the DFlash drafter’s average emitted length across six benchmarks and shows consistent gains, with transfer to another head as well.

Sep 21, 2026

Uploaded by Aurora

Speculative decoding accelerates autoregressive generation by drafting a block of tokens that the target model verifies left-to-right, committing only the longest accepted prefix. Block (DLM-style) drafters are trained with full-block cross-entropy supervising every position, even though inference discards tokens after the first rejection. This work introduces Accept-Until-Fail (AUF), using teacher-forced motivation to concentrate cross-entropy support only through the first predicted failure in mask-only block drafting. AUF is a single CE-support change without auxiliary objectives, verifier rollouts, or pipeline modifications. On Qwen3-8B serving settings, AUF raises average emitted length on multiple benchmarks and transfers to another head while several ablation results clarify the role of prefix conditioning and position-decay weighting.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

1 Introduction

Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters Tianjian Yang 1 , Meng Li2,3∗ 1 School of Electronics Engineering and Computer Science, Peking University, Beijing, China 2Institute for Artificial Intelligence, Peking University, Beijing, China 3 School of Integrated Circuits, Peking University, Beijing, China [2300012738@stu.pku.edu.cn](2300012738@stu.pku.edu.cn), [meng.li@pku.edu.cn](meng.li@pku.edu.cn) arXiv :2607 .0 1893v 1 [ cs .AI] 2 Jul 2026 Abstract Speculative decoding accelerates autoregressive generation by drafting a block of tokens that the target model verifies left-to-right, committing only the longest accepted prefix. Block (DLM-style) drafters predict the whole block in parallel, which is fast but trained with a full-block cross-entropy that supervises every position against the gold continuation—eventhough inference discards every token after the first rejection. Recent acceptance-aware objectives patch this by reweighting the full-block loss; we instead use teacher-forced learning asa motivation for how supervision should concentrate on the accepted prefix. A mask-only block drafter has no input-side channel for gold-prefix conditioning, so AUF approximates that prefix-sensitive supervision on the loss side by keeping the cross-entropy support only through the drafter’s first predicted failure. AUF is a single, detached change to the CE support—no auxiliary objective, no verifier rollouts, and no change to the inference pipeline or the exactness contract. Within fixed drafter backbones and serving settings on Qwen3-8B, AUF raises the DFlash drafter’s average emitted length τ, averagedover six benchmarks, from 2 .40 to 2.61, with a gain on every benchmark, and transfers to Domino’s two-branch head (2 .56 to 2.68) . Two findings sharpen the picture: the decay-only baseline reaches higher token accuracy on the shared block mask yet decodes worse, and on DFlash, once AUF truncates the support, the standard exponential position-decay weighting becomes empirically inert. 1 Introduction Speculative decoding (SD) accelerates autoregressive language-model inference by letting a lightweight draft model propose several future tokens and letting the target model verify them in one parallel pass (Leviathan, Kalman, and Matias 2023; Chen et al. 2023) . Because verification accepts only the longest target-consistent prefix, the practical speedup of SD is governed less by isolated token accuracy than by the draft model’s ability to produce a consecutive prefix that the target will accept. A drafter that predicts later tokens correctly after an early mistake still wastes those predictions: the verifier rejects the suffix once the prefix is broken. This observation changes how draft-model training should be viewed. Standard alignment objectives train the drafter to ∗Corresponding author. match target tokens over a full block, often with a handdesigned position prior—a fixed weight that decays with block position, encoding the inductive bias that earlier tokens matter more (DFlash, for instance, uses an exponential decay (Chen, Liang, and Liu 2026)). These objectives recognize that early positions matter more, but they still supervise tokens whose contribution to throughput is conditional on all previous draft tokens being accepted. Draft-model training is therefore not only a token-alignment problem; it must also reflect the left-to-right prefix contract used by the verifier. Our starting point is the analogy to autoregressive draftmodel training. With teacher-forced maximum likelihood, an autoregressive drafter receives the gold prefix on its input side, so the loss at position i is conditioned on the event that positions < i are already correct (Li et al. 2025) . Gold supervision is the strong signal, and the design question is which positions are entitled to receive it. Amasked or diffusion-style block drafter such as DFlash cannot obtain this c

FAQs

What problem does the paper address in speculative decoding training?

It addresses a train-inference mismatch: block-style draft models train with full-block supervision even though speculative decoding accepts only the longest left-to-right accepted prefix and discards tokens after the first rejection.

How does Accept-Until-Fail (AUF) modify training?

AUF keeps cross-entropy support only through the accepted prefix and the first failing token, masking out later positions. This approximates prefix-sensitive supervision in a mask-only drafter setting.

What impact does AUF have on performance in the reported experiments?

With fixed drafter backbones and serving settings on Qwen3-8B, AUF increases the DFlash drafter’s average emitted length across six benchmarks and shows consistent gains, with transfer to another head as well.

Sep 21, 2026

Uploaded by Aurora