Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitors

Jul 16, 2026

Upload by Aurelia

Low-bit quantization makes small reasoning models cheap to deploy but degrades their chain-of-thought, motivating decoder-side monitoring to intervene when generation goes wrong. A martingale based on token log-probabilities is analyzed and shown to be an incorrect observable: its increments form a mean-zero martingale under the model’s own sampling law, remaining blind to confident repetition and failing to trigger useful sequential alarms. A training-free controller uses a degeneration-aware repetition score and a calibrated e-process CUSUM-style detector, improving selective detection on GSM8K while highlighting non-termination as a dominant failure mode.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Abstract

Introduction

arXiv :2607 . 1 13 17v 1 [ cs .AI] 13 Jul 2026 Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitors El Hassane Ettifouri [eettifouri@novelis.io](eettifouri@novelis.io) Novelis Research, Paris, France ORCID: [https: // orcid. org/ 0000-0001-5299-9053](https: // orcid. org/ 0000-0001-5299-9053) Ayoub Belfatmi [abelfatmi@novelis.io](abelfatmi@novelis.io) Novelis Research, Paris, France ORCID: [https: // orcid. org/ 0009-0005-4010-794X](https: // orcid. org/ 0009-0005-4010-794X) Mahaman Sanoussi Yahaya Alassan [syahaya@novelis.io](syahaya@novelis.io) Novelis Research, Paris, France ORCID: [https: // orcid. org/ 0009-0006-0825-4701](https: // orcid. org/ 0009-0006-0825-4701) Walid Dahhane [wdahhane@novelis.io](wdahhane@novelis.io) Novelis Research, Paris, France ORCID: [https: // orcid. org/ 0000-0001-5387-3380](https: // orcid. org/ 0000-0001-5387-3380) Abstract Low-bit quantization makes small reasoning models cheap to deploy but degrades their chain-of-thought, and a recurring folk remedy is to monitor the decoder and intervene when generation “goes wrong”. A natural candidate for such a monitor is a martingale built from token log-probabilities, controlled by a concentration inequality. We show, both analytically and empirically, that this observable is the wrong basis for a decoder monitor: the increment log p(wt ) + Ht is a mean-zero martingale by construction under the model’s own sampling law, so it is blind to exactly the failure it is meant to catch—confident repetition—and, when naively checked throughout generation, it does not provide a useful sequential alarm. We replace it with a training-free decoding controller built on two ingredients that are individually standard but, to our knowledge, not previously combined for this purpose: (i) a degeneration-aware alarm score that fuses token uncertainty with an explicit verbatimrepetition signal, and (ii) a calibrated e-process-based sequential detector. The raw product process is Ville-valid under a conditional-mean null; the deployed CUSUM-floored statistic is treated more cautiously as an empirical change detector because the alarm score is history-dependent and autocorrelated. On GSM8K with DeepSeek-R1-Distill-Qwen-1.5B, the controller reduces the observed verbatim-degeneration signals in this pilot and, once calibrated, becomes a selective detector of failing traces (ϕ ≈ 0 .3, precision ≈ 0.6 versus a 0.38 base rate) rather than firing on 93% of generations as the uncalibrated version does. Wereport a positive but statistically inconclusive accuracy trend (INT4: 63% → 69%, paired McNemar p = 0 . 18 at n=100), an honest token-budget cost of +28%, and the observation—counter to the motivating intuition—that on GSM8K the dominant failure of these models is non-termination, not looping. We release the decoder, the calibration and re-scoring tools, and all traces. This is a preliminary study whose main contribution is methodological: a clear account of why a tempting token-level observable is inadequate, and a calibrated, honestly-evaluated replacement. 1 Introduction Distilled reasoning models such as DeepSeek-R1-Distill (DeepSeek-AI, 2025) put competitive mathematical reasoning within reach of commodity hardware, and 4-bit post-training quantization pushes them further, onto a single consumer GPU or even a CPU. The price is well documented: low-bit quantization degrades mathematical reasoning disproportionately, inflating method and execution errors and lengthening chains of thought, with errors that tend to appear early and cascade (Li et al., 2025; Liu et al., 2025) . This points to a different intervention point. Rather than waiting for a finished answer and re-ranking it, one can act while the reasoning trace is still being generated. Because a handful of early tokens is often enough to send a quantized model down a bad path, a decoder-side monitor ma

FAQs

Why is token log-probability an inadequate observable for decoding monitors?

Centering token log-probability under the model’s own sampling law yields a mean-zero martingale whose signal reflects self-consistency of sampling, not trajectory correctness. As a result, confident repetition produces near-zero increments, leaving the monitor silent when it should act.

What replacement mechanism does the paper propose for monitoring decoding failures?

It proposes a training-free decoding controller that combines a degeneration-aware alarm score fusing token uncertainty with an explicit verbatim-repetition signal, with a calibrated e-process-based sequential detector. The deployed CUSUM-floored statistic is treated as an empirical change detector due to history dependence and autocorrelation.

What experimental findings are reported about decoder failures under quantization?

On GSM8K with DeepSeek-R1-Distill-Qwen-1.5B, the controller reduces verbatim-degeneration signals and, after calibration, becomes a selective detector of failing traces. The study also reports that the dominant failure mode is non-termination rather than looping on this benchmark.

Jul 16, 2026

Upload by Aurelia