Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning

Jul 16, 2026

Actor-critic reinforcement learning relies on learned critics, yet critic quality is often judged only indirectly via return, temporal-difference error, or value loss. The work introduces critic complexity as an additional diagnostic and intervention axis. It measures complexity with spectral effective-rank entropy, summarizing singular-value distributions of critic weight matrices, and tracks it alongside return and Monte Carlo value-estimation bias in TD3 and PPO. Results show measurable, training-correlated complexity with heterogeneous relationships across algorithms, tasks, and hyperparameters, and allow direct complexity control via a spectral-entropy penalty.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Related Work

arXiv :2607 .00452v 1 [ cs .LG] 1 Jul 2026 Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning Konstantin Garbers Peking University [konstantin. garbers25@stu. pku. edu. cn](konstantin. garbers25@stu. pku. edu. cn) July 2, 2026 Abstract Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or value loss. Critic complexity is introduced asan additional diagnostic and intervention dimension for actor-critic reinforcement learning. The analysis uses spectral effective-rank entropy, a rank-like summary of the singular-value distributions of critic weight matrices, to assess critic model complexity. Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias. The results show that critic complexity is measurable throughout training and is systematically associated with training behavior, while also making clear that the relationship is heterogeneous across algorithms, tasks, and hyperparameters. A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss. This intervention reliably changes the targeted spectral quantity, demonstrating that critic complexity can be controlled rather than only observed. Return effects are treated as task-dependent evidence rather thanas a general performance claim, because overall complexity-control results vary. 1 Introduction Actor-critic reinforcement learning depends on learned critics to estimate values and guide policy improvement. This makes the critic a central source of both progress and failure. If the critic overestimates certain states or actions, propagates bootstrapping errors, or learns an unnecessarily irregular value function, the actor may optimize against a distorted objective. Much of actor-critic research is therefore concerned with improving critics. Double Q-learning reduces maximization bias by separating action selection from value evaluation [1, 2], while TD3 extends this idea to continuous control through clipped double critics, delayed policy updates, and target-policy smoothing [4] . PPO provides a useful on-policy contrast, where the critic is trained under a different update regime [3] . These methods improve critic reliability indirectly through targets, architectures, or optimization procedures. A complementary route is the critic training process itself: whether critic complexity can be measured, related to performance, and controlled during training. The guiding hypothesis is Occamstyle: among critics that capture the relevant value structure, a simpler critic may be more reliable, matching broader neural-network generalization arguments that use norm-based and spectral quantities as complexity proxies [6, 7] . This does not imply that lower complexity is always better. A critic that is too simple may underfit, while a critic that is too complex may overestimate, become unstable, or represent sharp value artifacts. To make this hypothesis testable, critic complexity is measured using spectral effective-rank entropy. This metric summarizes how diffusely a critic layer uses its singular directions: high entropy means many directions contribute, while low entropy means the spectrum is concentrated in fewer dominant directions [9] . This quantity is then tracked during TD3 and PPO training, compared with return and Monte Carlo estimates of value-estimation bias, and directly controlled in separate experiments by adding a spectral-entropy penalty to the critic loss. The main conclusion is deliberately narrow. Critic spectral complexity is measurable and controllable, and it is related to actor-critic performance in structured but task-dependent ways. Spectral-entropy regularization reliably reduces critic rank-like complexity and improves TD3/HalfCheetah-v4 performance in the tested setting, but the ret

FAQs

How is critic complexity defined and measured in this work?

Critic complexity is measured using spectral effective-rank entropy, a rank-like summary of singular-value distributions of critic weight matrices. Higher entropy indicates broader usage of singular directions, while lower entropy indicates concentration in fewer dominant directions.

What evidence links critic complexity to learning behavior and performance?

Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias. The study finds a systematic association with training behavior and value-estimation bias, but the relationship is heterogeneous across algorithms, tasks, and hyperparameters.

Can critic complexity be controlled rather than only observed?

Yes. The paper evaluates a direct intervention by adding a spectral-entropy penalty to the critic loss, which reliably changes the targeted spectral quantity. This demonstrates that critic complexity can be controlled during critic training.

Jul 16, 2026

Upload by Lucas Martin