A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems

Jul 16, 2026

Upload by Ezra

Rate–distortion (RD) theory underpins efficient information representation, yet computing RD quantities in high-dimensional settings is difficult. Classical iterative solvers such as the Blahut–Arimoto algorithm become impractical due to the curse of dimensionality and hard mutual-information terms. This survey reviews neural learn-to-compute approaches that parameterize distributions and objective functionals with flexible networks. It covers three method families—variational inference, neural mutual-information estimation, and dual-form optimization—linking theory, algorithms, and consistency, while highlighting remaining large-scale challenges.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Rate–distortion framework

Related RD formulations

Classical computation challenges

Learn-to-compute paradigm with neural methods

A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems Shitong Wu, Sicheng Xu, Lingyi Chen, Qiang Sun, Huihui Wu, Hao Wu, and Wenyi Zhang, Senior Member, IEEE arXiv :2607 .054 17v 1 [ cs .IT] 26 Jun 2026 Abstract—Rate–distortion (RD) theory and its related formulations play a central role in understanding efficient information representation, but computing these quantities remains challenging in high-dimensional settings. Classical iterative methods such as the Blahut–Arimoto algorithm become impractical in highdimensional domains due to the curse of dimensionality and the intractability of mutual-information terms. Recent advances in neural modeling and differentiable optimization offer a promising alternative through a learn-to-compute paradigm, in which probability distributions and objective functionals are represented by flexible neural parameterizations. This survey presents an overview of neural approaches for evaluating the RD-type objectives. We present three representative families of methods: variational inference, neural mutual-information estimation, and dual-form optimization. By reviewing their theoretical principles, algorithmic techniques, and consistency properties, we elucidate how these methods collectively transform classical RD-type problems into scalable differentiable objectives suitable for deep learning, though challenges remain in large-scale applications. Together, these perspectives offer promising avenues for scaling information-theoretic computation to complex, high-dimensional machine learning systems. Index Terms—Rate-distortion theory, Learn-to-Compute paradigm, Neural estimation, Information bottleneck. I. INTRODUCTION Machine learning systems are increasingly designed to process high-dimensional sensory data such as images, videos, and natural language. As the scale and complexity of these modalities continue to grow, the trade-off between preserving essential information and constraining the resources required for storage, transmission, and computation becomes increasingly pronounced. Rate–distortion (RD) theory [1] provides one of the most principled information-theoretic frameworks for characterizing this trade-off, formally specifying the minimum rate required to achieve a given reconstruction fidelity. Several related formulations can be understood within a unified RD framework by adopting different distortion measures. For Shitong Wu, Sicheng Xu, Lingyi Chen, and Hao Wu are with the Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China. Qiang Sun and Wenyi Zhang are with the Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, Anhui 230027, China. Huihui Wu is with the Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo, Zhejiang 315200, P.R. China. Corresponding author: Wenyi Zhang ([email: wenyizha@ustc.edu.cn](email: wenyizha@ustc.edu.cn)). This work was partially supported by National Natural Science Foundation of China(Grants 12271289 and 62231022) and by Ningbo Yongjiang Talent Program. instance, the information bottleneck (IB) principle [2] focuses on extracting task-relevant information, while the indirect rate–distortion (iRD) [3] formulation addresses scenarios in which the source signal is observable only through a noisy or incomplete measurement channel. These RD-derived formulations have become increasingly valuable in machine learning systems. On the one hand, they provide theoretical foundations for learned image and video compression schemes [4]; on the other hand, they offer conceptual tools for analyzing representation learning and generalization in deep neural networks [5] . Collectively, these developments underscore the renewed and growing interest in RD-type problems, which have become central to understanding information processing in high-dimensional regimes of machine learni

FAQs

Why are classical RD computation methods difficult in high-dimensional problems?

Classical iterative methods like Blahut–Arimoto scale poorly because discretization grows exponentially with dimension, and mutual-information terms are intractable in large spaces.

What is the learn-to-compute (LtC) paradigm for RD-type objectives?

LtC represents probability distributions and objective functionals with parameterized neural models, and replaces original quantities with tractable differentiable estimators for end-to-end optimization via stochastic gradients.

Which three representative neural method families are reviewed for RD-type objectives?

The survey reviews variational inference, neural mutual-information estimation, and dual-form optimization, emphasizing their theoretical basis, algorithmic techniques, and consistency properties.

Jul 16, 2026

Upload by Ezra