Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

Jul 16, 2026

Upload by Ezra

Speech large language models encode speech into much longer sequences than text, creating a key bottleneck during autoregressive decoding. Existing adapter-level downsampling improves efficiency but may discard fine-grained details irrecoverably. SpeechKV instead applies learned pooling directly to the KV cache of speech tokens inside the Speech LLM, enabling internal fusion of speech and text while accelerating decoding. Trained on 71K hours of speech data, it compresses speech to near text-level granularity, achieving baseline-matching or slightly better accuracy with 6.6% gains on out-of-domain entity recognition and 2.3% on OpenASR, plus at least 1.49× decoding speedups scaling with audio length.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Problem of speech sequence length

Trade-off in early compression

SpeechKV approach

Related Work

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs Ke-Han Lu 12∗, Keqi Deng 1†, Ruchao Fan 1 , Rui Zhao 1 , Jinyu Li 1 1Microsoft, USA 2National Taiwan University, Taiwan [d12942024@ntu.edu.tw](d12942024@ntu.edu.tw), [keqideng@microsoft.com](keqideng@microsoft.com) arXiv :2607 .06827v1 [ ee ss .AS] 7 Jul 2026 Abstract—Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length. Index Terms—Speech LLM, KV cache compression, automatic speech recognition I. INTRODUCTION Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks [1]–[4] . Building on this success, researchers have extended these models to speech modalities, giving rise to a growing family of Speech LLMs [5]–[20] . A typical Speech LLM consists of three components: a pre-trained speech encoder, a modality adapter, and a large language model backbone. The speech encoder maps input speech into a sequence of acoustic frames. The modality adapter projects these frames into the LLM’s input space, and the LLM then processes speech embeddings alongside text embeddings to enable multimodal understanding. A key challenge in Speech LLMs is the length of speech sequences. For example, in English ASR, a single text token typically corresponds to approximately 300 ms of speech, yet common speech encoders emit one frame every 80 ms, yielding roughly 4 × more sequence length for the same semantic information. This poses a substantial efficiency bottleneck during autoregressive generation, where the model must attend to all preceding positions at each decoding step, resulting in increased latency and memory consumption that scale with sequence length. A straightforward approach is to downsample speech se quences at the adapter level into more compact representations ∗ Work done while Ke-Han Lu was an intern at Microsoft. †Corresponding author. [7], [15]–[17], [19]–[21] . However, such early compression may risk losing information and fails to exploit the representational capacity of the LLM itself. This creates a trade-off between efficiency and recognition performance. In this work, we aim at resolving this trade-off by deferring speech compression into the LLM itself. Recent LLM research has shown that Transformer layers often produce redundant key-value representations [22],[23] . In our preliminary experiments, we also observe that Speech LLMs naturally merge nearby speech information in their early layers and operate on near-duplicate representations in the deeper layers (see Section V-B) . Based on this observation, we propose SpeechKV, which applies a learned pooling operation to merge the speech key-value sequences at an intermediate LLM layer as illustrated in Figure 1 . This allows the LLM layer to learn which speech features to preserve and how to fuse speech with text through the attention mechanism. We evaluate SpeechKV on a Speech LLM with Qwen3-1.7B backbone trained on 71K hours of ASR data. Interestingly, when compressing the KV se

FAQs

What efficiency bottleneck do Speech LLMs face during decoding?

Speech encoders output frames much more frequently than text tokens map to semantics, producing far longer sequences. This forces autoregressive generation to attend over more positions each step, increasing latency and memory usage.

Why can adapter-level speech downsampling hurt performance?

Early compression can remove fine-grained information that the model cannot recover later. It creates a trade-off between efficiency and recognition quality.

How does SpeechKV perform KV compression and what benefit does it bring?

SpeechKV uses learned pooling to compress the speech KV cache at an intermediate layer inside the Speech LLM. The model still learns how to preserve speech features and fuse speech with text via attention, while achieving decoding speedups of at least 1.49× and maintaining or slightly improving accuracy.

Jul 16, 2026

Upload by Ezra