Differentiable Proxy Learning for Adaptive Quantization Control in H.264 Video Coding

Jul 16, 2026

Upload by Levi

H.264 remains a widely deployed video coding format, yet optimizing codec parameters such as the quantization parameter (QP) is difficult because standard codecs are non-differentiable. This paper introduces a differentiable proxy learning method for H.264 intra coding, enabling adaptive quantization control. The proxy is built on a variable-rate learned compression model and becomes differentiable w.r.t. codec QP via soft indexing. The learned proxy approximates the rate-distortion behavior of H.264 under global-QP and spatial-QP, improving perceptual and machine-vision trade-offs, including BD-rate reductions up to 17.12%.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Abstract

Introduction

Proposed Method

Differentiable proxy learning for H.264 intra codec

Proxy-based adaptive quantization framework

Differentiable Proxy Learning for Adaptive Quantization Control in H.264 Video Coding Qihan Xu and Ivan V. Baji School of Engineering Science, Simon Fraser University Burnaby, BC, Canada arXiv :2607 . 10478v1 [ ee ss .IV] 11 Jul 2026 Abstract—H.264 has been the most widely used video coding format for the past two decades due to its relative simplicity, efficiency, and wide availability of software and hardware implementations. However, optimizing codec parameters such asthe quantization parameter (QP) for specific objectives (e.g., perceptual quality or machine vision tasks) is challenging due to the non-differentiable nature of standard video codecs. While differentiable proxies have recently been used to enable gradientbased optimization around standard codecs, their fidelity to the target codec is rarely explicitly characterized. In this paper, we propose a differentiable proxy learning method for H.264 intra codec to enable adaptive quantization control. Built upon a variable-rate learned compression model, the proposed proxy is made differentiable with respect to codec QP through a softindexing mechanism. It is then trained to approximate the ratedistortion behavior of H.264 under two quantization settings: global-QP, which uses one QP per image, and spatial-QP, which assigns QPs at the macroblock level. Using the frozen trained proxy, we develop a proxy-based adaptive quantization (AQ) framework for both perceptual optimization and machine vision tasks. Experimental results demonstrate that the proposed proxies closely approximate the rate-distortion behavior of H.264 intra codec. The resulting proxy-based AQ framework consistently improves rate-task trade-offs over fixed-QP H.264 baselines, achieving BD-rate reduction of up to 17. 12% for semantic segmentation and 15.30% for MS-SSIM. Index Terms—H.264, video coding, differentiable proxy, quantization I. INTRODUCTION Image and video coding has become increasingly important due to the rapid growth of visual data. H.264 [1] has remained widely used for video coding over the past two decades due to its relative simplicity, high coding efficiency, and broad availability in software and hardware implementations. For standard codecs like H.264, adaptive quantization (AQ) is a practical way to improve rate-distortion and rate-task trade-offs without changing the codec itself. However, directly optimizing quantization parameter (QP) assignments is difficult: standard codecs are non-differentiable, and the relationship between QP, bitrate, and reconstruction is highly coupled, which prevents gradient-based end-to-end optimization. One way to address this issue is to introduce differentiable proxies for the target codec during training [2] . Existing works explore several strategies toward this goal. Some methods retain the overall codec structure while replacing non-differentiable operations such as quantization or entropy This work was supported by the SFU–Huawei Canada Joint Lab and the NSERC grant RGPIN-2021-02485 . coding-related components with differentiable analytical approximations, enabling gradient propagation through the codec pipeline [3], [4] . Other approaches use neural simulation modules to approximate the output behavior of a standard codec during training. Instead of implementing a full learned codec, these methods train networks to predict the decoded reconstruction and/orbitrate of the target codec, thereby enabling gradient-based optimization while avoiding repeated codec invocation [5]–[7] . More recently, learned image and video compression models [8], [9] have been adopted as differentiable codec proxies. These methods train neural compression models to mimic thereconstruction and bitrate behavior of standard codecs, and use the resulting differentiable proxies to optimize preprocessing, postprocessing, or downstream task performance [10]–[12] . Despite this progress, most prior work uses the differentiable proxy primarily as

FAQs

Why is adaptive quantization (AQ) hard to optimize for H.264 with gradient-based methods?

Standard codecs like H.264 are non-differentiable, and the relationship among QP, bitrate, and reconstruction is highly coupled, which blocks end-to-end gradient optimization.

What differentiable proxy is proposed for H.264 intra coding?

The method learns a differentiable proxy based on a variable-rate learned compression model and makes it differentiable with respect to the codec QP using a soft-indexing mechanism.

How does the paper evaluate the proxy and what benefits does it bring?

The proxy is trained and assessed by approximating H.264 intra rate-distortion behavior under global-QP and spatial-QP, then used in a frozen form to train AQ networks for perceptual and machine vision tasks, improving rate-task trade-offs and achieving BD-rate reductions.

Jul 16, 2026

Upload by Levi