PM-KVQ Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

Q: How does PM-KVQ improve calibration without collecting long-context data?

It introduces a calibration strategy with positional interpolation, leveraging short calibration data and approximating the long-context Key-cache distribution by interpolating positional effects from RoPE.

Jul 16, 2026

Upload by Logic

Reasoning-capable large language models (LLMs) using long chain-of-thought (CoT) techniques face major memory overhead from the key-value (KV) cache required to store historical context. Posttraining KV cache quantization can reduce memory usage, yet existing short-context methods degrade performance on long-CoT models due to cumulative quantization error and calibration mismatch from RoPE. PM-KVQ introduces progressive mixed-precision quantization with block-wise memory allocation and a positional interpolation calibration strategy, yielding up to 8% better reasoning and 2.73–5.18× throughput under the same memory budget.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Abstract

Introduction

arXiv :2505 . 186 10v2 [ cs .CL] 11 Jul 2026 PM-KVQ: PROGRESSIVE MIXED-PRECISION KV CACHE QUANTIZATION FOR LONG-COT LLMS Tengxuan Liu∗1 ,2 , Shiyao Li† ∗1 ,2 , Jiayi Yang∗3, Tianchen Zhao 1 , Feng Zhou4 , Xiaohui Song4 , Guohao Dai5 ,2 , Shengen Yan2 , Huazhong Yang 1 , Yu Wang‡ 1 1Tsinghua University, 2Infinigence-AI, 3 Columbia University, 4 OPPO AI Center, Beijing, China, 5 Shanghai Jiaotong University ABSTRACT Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques. However, this long-CoT reasoning process imposes substantial memory overhead due to the large Key-Value (KV) Cache memory overhead. Posttraining KV Cache quantization has emerged as a promising compression technique and has been extensively studied in short-context scenarios. However, directly applying existing methods to long-CoT LLMs causes significant performance degradation due to the following two reasons: (1) Large cumulative er ror: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cumulative quantization error. (2) Short-context calibration: Due to Rotary Positional Embedding (RoPE), the use of short-context data during calibration fails to account for the distribution of less frequent channels in the Key Cache, resulting in performance loss. We propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs to address the above issues in two folds: (1) To reduce cumulative error, we design a progressive quantization strategy to gradually lower the bit-width of the KV Cache in each block. Then, we propose block-wise memory allocation to assign a higher bit-width to more sensitive transformer blocks. (2) To increase the calibration length without additional overhead, we propose a new calibration strategy with positional interpolation that leverages short calibration data with positional interpolation to approximate the data distribution of long-context data. Extensive experiments on 7B–70B longCoT LLMs show that PM-KVQ improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73–5.18 × throughput over the original 16-bit LLMs. Our code is available at [https://github.com/thu-nics/PM-KVQ](https://github.com/thu-nics/PM-KVQ). 1 INTRODUCTION Recently, many pioneers have developed remarkable reasoning Large Language Models (LLMs) with long Chain-of-Thoughts (CoT) techniques, such as OpenAI-o1 (OpenAI, 2024), DeepSeekR1 (Guo et al., 2025), QwQ (Team, 2025), and so on. To achieve better algorithmic performance, these long-CoT reasoning LLMs are trained to generate up to 128K tokens with multiple complex rationales from different perspectives (Guo et al., 2025) . However, this long-CoT process demands significant memory overhead (∼ 10GB–100GB) to store the Key-Value (KV) Cache as the history information, which limits the practical application scenarios for such long-CoT LLMs. To mitigate the substantial memory overhead of long-CoT LLMs, various KV Cache compression methods have been proposed (Liu et al., 2024c; Yang et al., 2024; Su et al., 2025; Xiao et al., ∗Equal contribution. †Program leader. ‡Corresponding author: Yu Wang ([yu-wang@tsinghua.edu.cn](yu-wang@tsinghua.edu.cn)) . 2023; Fu et al., 2024) . Among them, Post-training KV Cache Quantization is a promising compression technique that has already been well explored in short-context scenarios (e.g., <8K tokens) . QServe (Lin* et al., 2024) and MiKV (Yang et al., 2024) observe that the Key Cache has more outliers than the Value Cache, leading to higher quantization error. More importantly, the outliers in the Key Cache persist in certain channels. To this end, they propose a channel-wise equalization method to migrate the outliers from the Key tensor to the Query tensor, thereby significantly reducing the

FAQs

How does PM-KVQ improve calibration without collecting long-context data?

It introduces a calibration strategy with positional interpolation, leveraging short calibration data and approximating the long-context Key-cache distribution by interpolating positional effects from RoPE.

Jul 16, 2026

Upload by Logic