Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation

Jul 16, 2026

Upload by Rowan

Data-driven off-policy policy iteration for continuous-time linear quadratic regulation is reformulated to remove explicit critic identification. The method anchors the Riccati equation at a known stabilizing gain and rewrites optimality as a policy-space residual, then uses an endpoint null-space projection to eliminate the value-matrix term from the integral data equation. Under a projected rank condition, the data equation matches the policy-space residual and each least-squares update coincides with the Kleinman iteration, preserving stability and convergence while reducing regression dimensions and rank assumptions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract and Index Terms

Introduction

Problem Background and Motivation

Data-Driven Policy Iteration and Critic–Actor Regression

Key Question: Is Critic Identification Necessary?

Related Policy-Space and Model-Based Results

Gap in Existing Off-Policy Integral Regression

Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation Jiacheng Wu, Yang Zhu∗ , Hongye Su, Senior Member, IEEE arXiv :2607 .08204v1 [ ee ss . SY] 9 Jul 2026 Abstract—For continuous-time linear quadratic regulation with unknown system matrices, data-driven off-policy policy iteration typically estimates the value matrix and the improved feedback gain through a joint critic–actor regression. We show that the critic is not needed in the policy-improvement step. The key is to anchor the Riccati equation at a known stabilizing gain and express optimality as a policy-space residual. An endpoint null-space projection then removes the value-matrix term from the integral data equation. This yields a critic-free, actor-only least-squares update computed directly from input-state data. Under a veriﬁable projected rank condition, the resulting data equation is equivalent to the policy-space residual equation, and each update coincides with the Kleinman iteration. Thus, the stabilizing and convergence properties of Kleinman iteration are retained without a critic regression. We further show that the conventional off-policy full-rank condition decomposes into an endpoint critic rank condition and a projected actor rank condition. The proposed method removes the rank requirement needed for critic identiﬁcation while retaining the one needed for policy improvement. The repeated least-squares dimension is reduced from n (n + 1)/2 + mn to mn. Finally, comparative simulations validate the effectiveness of the proposed algorithm. Index Terms—Policy iteration, linear quadratic regulation, data-driven control, critic-free reinforcement learning, endpoint null-space projection. I. INTRODUCTION CONTINUOUS-time linear quadratic regulation is a fun damental problem in optimal control [1] . When the system matrices are known, the optimal feedback gain can be obtained by solving the algebraic Riccati equation (ARE) . In many practical applications, however, accurate knowledge of the system matrices is unavailable, which has motivated extensive research on data-driven optimal control [2], reinforcement learning (RL) [3], and adaptive dynamic programming [4] . For this problem, data-driven policy iteration (PI) provides a natural way to implement Riccati-based optimal control from data [5]–[7] . Starting from an admissible feedback gain, PI alternates between policy evaluation and policy improvement, and its model-based form is closely related to the classical Kleinman iteration. In the data-driven setting, Vrabie et al. [8] introduced an on-policy PI scheme based on integral RL, where the feedback gain is updated from closed-loop inputstate data rather than by directly solving the ARE. This approach reduces the reliance on a complete system model, Jiacheng Wu, Yang Zhu, and Hongye Su are with State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China (e-mail: ji[achengwu@zju.edu.cn](achengwu@zju.edu.cn); [zhuyang88@zju.edu.cn](zhuyang88@zju.edu.cn); [hysu@iipc.zju.edu.cn](hysu@iipc.zju.edu.cn);). * Corresponding author: Yang Zhu. but its policy-improvement step still requires partial knowledge of the system dynamics. To further reduce model dependence, Jiang et al. [9] developed an off-policy PI framework in which integral regression equations are constructed from trajectories generated by a behavior input, so that the unknown system matrices need not be explicitly identiﬁed. This framework has motivated a broad range of extensions and applications, including inverse optimal control [10], permanent magnet synchronous motor [11], singularly perturbed systems [12], and networked [13] or multi-agent systems [14], [15] . Nevertheless, the resulting regression treats the value matrix of the current policy and the improved feedback gain as simultaneous unknowns. As a result, each policy update still requires a joint critic–acto

FAQs

What theoretical results ensure the method’s stability and equivalence to Kleinman iteration?

With a verifiable projected rank condition, the resulting data equation is equivalent to the policy-space residual equation, so each update coincides with the Kleinman iteration, retaining its stabilizing and convergence properties.

Jul 16, 2026

Upload by Rowan