Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function

Jul 16, 2026

Hamilton–Jacobi (HJ) reachability offers rigorous safety and reachability guarantees for continuous-time dynamical systems, yet numerical computation suffers from the curse of dimensionality. Deep reinforcement learning (DRL) scales through sample-based learning, but standard RL uses additive cumulative rewards, while reachability is inherently non-additive. The work develops a semantics-preserving discounted reach-based value function and derives a non-additive Bellman operator whose unique fixed point matches the HJ formulation. Discounting renders the operator contractive, enabling existence, uniqueness, and convergence of value iteration, and provides a principled RL interpretation for solving the same fixed-point equation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

Abstract

I. Introduction

A. Background & Motivation

B. Related Work & Contributions

Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton–Jacobi Reachability and Reinforcement Learning Isabelle El-Hajj, Prashant Solanki, Jasper van Beers, Coen de Visser, and Erik-Jan van Kampen arXiv :2607 .07893v1 [ ee ss . SY] 8 Jul 2026 Abstract—Hamilton-Jacobi (HJ) reachability provides rigorous safety and reachability guarantees for continuous-time dynamical systems, but its numerical solution suffers from the curse of dimensionality. Deep reinforcement learning (DRL), by contrast, offers scalable sample-based methods. However, RL is typically built around additive cumulative rewards; whereas, reachability objectives are inherently non-additive. This mismatch makes a direct bridge between HJ reachability and RL nontrivial. Recent discounted formulations have either introduced contraction by altering the original reachability semantics, or preserved exact semantics on the HJ side without a corresponding Bellman fixedpoint characterization. In this paper, we close this gap by building on a semantics-preserving discounted reach-based value function and deriving a non-additive Bellman operator whose unique fixed point exactly matches the value function in the HJ formulation. We prove that discounting makes this operator contractive, yielding existence, uniqueness, and convergence of value iteration. Furthermore, we establish the equivalence between the HJ and Bellman characterizations, and show that RL can be interpreted as a sample-based approximation scheme for the same fixedpoint equation. This yields a principled and semantically exact connection between HJ reachability and RL, enabling learningbased methods to approximate reachability value functions while preserving their safety-critical meaning. As a result, the proposed framework opens the door to scalable, data-driven computation of reachable sets and safety certificates in high-dimensional systems. Numerical experiments demonstrate close agreement with Hamilton–Jacobi solutions, confirm preservation of reachability semantics via alignment of zero level sets, and support the interpretation of reinforcement learning as a sample-based solver of the proposed Bellman operator. Index Terms—Hamilton-Jacobi reachability, Non-additive Bellman Operator, Reinforcement learning, Reach cost, Safetycritical control. I. INTRODUCTION A. Background & Motivation Hamilton–Jacobi (HJ) reachability provides a principled framework for analyzing safety and reachability properties of continuous-time dynamical systems [18] . By characterizing reachable or safe sets as sublevel or superlevel sets of a value function satisfying a Hamilton–Jacobi variational inequality (HJVI), this framework offers strong semantic guarantees and has become a cornerstone of safety-critical control [3], [7] . However, its practical utility remains limited by the curse of dimensionality: numerical solutions of the associated partial All authors are with the section of Control & Simulation at the Faculty of Aerospace Engineering at Delft University of Technology. differential equations scale exponentially with the state dimension [11], [12] . A large body of work has therefore sought to improve the scalability of HJ reachability, including decomposition methods, level-set approximations, and learning-based surrogates [8], [14], [16], [17], [30] . Learning methods that are constrained by partial differential equation (PDEs), such as DeepReach [4], approximate the time-dependent HJ value function using a neural implicit representation trained to satisfy the PDE together with terminal and boundary conditions. While such approaches enforce local PDE consistency, they do not by themselves provide a global Bellman fixed-point characterization of the solution. In parallel, deep reinforcement learning (DRL) has emerged as a powerful paradigm for sequential decision-making and control in high-dimensional systems. Across games, robotics, fluid dynami

FAQs

What core gap does the paper address between HJ reachability and reinforcement learning?

It addresses the mismatch between HJ reachability objectives, which are inherently non-additive, and standard RL frameworks built on additive cumulative rewards, requiring an operator-theoretic bridge that preserves reachability semantics.

How does the paper construct a Bellman operator compatible with discounted reachability?

It builds on a semantics-preserving discounted reach-based value function and derives a non-additive Bellman operator whose unique fixed point exactly matches the HJ value function.

What does discounting enable for the proposed operator?

Discounting makes the operator contractive, leading to existence, uniqueness, and convergence of value iteration, and supporting a stable fixed-point solution perspective.

Jul 16, 2026

Upload by Clementine