REDDIT：用回放式分布编辑纠正ASR模型生成的时间戳漂移而不遗忘

Jul 16, 2026

Upload by Aria

Modern autoregressive ASR systems generate timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners. However, generated timestamps drift across long non-speech spans: the transcript can stay linguistically plausible while the time axis shifts away from the audio. Benchmarks with gap and long-gap settings across 15 systems reveal this non-speech-induced drift. Naive timestamp-corrected fine-tuning can improve targeted alignment yet cause forgetting. REDDIT is a lightweight two-stage post-training method that corrects timestamps while matching the frozen base distribution on non-timestamp tokens, avoiding catastrophic forgetting. Experiments on Whispertiny show large mIoU gains and reduced out-of-domain alignment errors with minimal parameter updates.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Non-speech-induced timestamp drift

Limitations of post-hoc alignment and naive fine-tuning

REDDIT approach and contributions

REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing Cheng-Kang Chou∗1, Ming-To CHUANG∗1, Ke-Han Lu 1 , Chan-Jan Hsu2 , Hung-yi Lee3 1National Taiwan University 2Carnegie Mellon University 3NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE) arXiv :2607 .05364v 1 [ cs .CL] 6 Jul 2026 Abstract—Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT (REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model’s own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whispertiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning). Index Terms—automatic speech recognition, model-generated timestamps, non-speech gap, replay-based learning, post-training, catastrophic forgetting I. INTRODUCTION Automatic speech recognition (ASR) and audio-language models increasingly need to answer not only what was said, but also when it was said. Autoregressive speech systems expose this interface by generating time as part of the decoded output, either as timestamp tokens in ASR models such as Whisper or as word-level and text-form temporal predictions in speech-aware language models [1], [3]–[5] . This self-generated timestamp interface avoids the need for a separate VAD, frame-level aligner, or forced-alignment module at inference time. This convenience introduces a failure mode that transcript-centric metrics can miss. Speech evaluation already extends beyond lexical content, e.g., tonality [46]; here, the missing dimension is temporal placement. Because the time axis is generated by the same decoder that emits text, a model can produce plausible content while assigning it to the wrong region of the audio. We focus on non-speech-induced timestamp drift: when speech follows a long silent or non-speech span, self-generated timestamps may place the following speech too early, too late, or under a globally displaced time axis, rendering subtitle timing unusable despite having an acceptable word error rate or task-level transcript quality. A likely reason this failure is under-tested is the mismatch between common training and long-gap deployment conditions. Timestamped ASR examples often begin near speech onset, and VAD-based trimming can further increase speech density [11] . During training, this practice may reduce the model’s exposure to long non-speech prefixes and strengthen a prior for early timestamp tokens near 0s. Post-hoc alignment pipelines can refine boundaries after decoding, but they operate at a different level than correcting the model’s native * Equal contribution. predictions. VAD segmentation, forced alignment, and attentionbased alignment add inference-time m

FAQs

什么是ASR中“非语音诱导的时间戳漂移”？

当长时间静音或非语音段之后出现语音时，模型自生成的时间戳可能把后续语音放到过早/过晚或整体时间轴偏移的位置，导致字幕时间不可靠，即使词错率或转写内容质量仍可能看起来合理。

为什么仅做时间戳校正的朴素微调会带来“遗忘”问题？

朴素的时间戳校正微调可能在目标时间戳数据上改善对齐，但会在非目标ASR行为上退化，说明难点不只是时间戳监督，而是需要对时间编辑具有抗遗忘能力。

REDDIT如何在纠正时间戳的同时避免灾难性遗忘？

REDDIT采用两阶段后训练：先在模型回放的解码器上下文下编辑时间戳目标，并将非时间戳token的分布锚定到冻结基模型；再从阶段一的检查点进行短的编辑前缀精修，从而实现时间轴纠正而不过度破坏原有行为。

Jul 16, 2026

Upload by Aria