VLM-AR3L：用于强化学习绝对与相对奖励的视觉-语言模型

Jul 16, 2026

Upload by Himbo

Designing effective reward functions remains a major challenge in reinforcement learning, especially in open-ended and visually complex settings where goals are abstract and hard to quantify. VLM-AR3L introduces a vision-language framework that interprets visual observations in the context of natural-language goals and learns both absolute and relative rewards from VLM-generated preference labels. The absolute model evaluates individual states, while the relative model infers progress or regression by comparing consecutive observations, combining stability with comparative robustness.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Problem: Reward function design in RL

Prior approaches and limitations

Proposed framework: VLM-AR3L

Contributions: dual-reward design and relative reward benefits

VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning Kuan-Chen Chen 1 , Winston Chen 1 , Wei-Fang Sun2 and Min-Chun Hu 1 1Department of Computer Science, National Tsing Hua University 2NVIDIA AI Technology Center (NVAITC) arXiv :2607 .00483v2 [ cs .RO] 2 Jul 2026 Abstract Designing effective reward functions remains a major challenge in reinforcement learning (RL), particularly in open-ended environments where task goals are abstract and difficult to quantify. In this work, we present VLM-AR3L, a framework that leverages Vision-Language Models (VLMs) to provide both absolute and relative rewards for RL. VLM-AR3L interprets an agent’s visual observations in the context of a natural language task goal, and learns both absolute and relative rewards from VLM-generated preference labels. The absolute reward model predicts scalar evaluations for individual states, while the relative reward model compares consecutive observations to infer progress or regression toward the task goal. Their integration combines the stability of state-based evaluation with the robustness of comparative supervision. We evaluate VLM-AR3L across benchmarks spanning classic control, manipulation, and open-world embodied tasks, with a particular focus on Minecraft given its visual complexity and long-horizon decision-making requirements. Experimental results show that VLM-AR3L consistently outperforms prior VLM-based reward learning methods. Videos and code are available on the project website: [https://vlm-ar3l.github.io/](https://vlm-ar3l.github.io/) . 1 Introduction Designing effective reward functions remains a major challenge in reinforcement learning (RL) [Laud, 2004; Leike et al., 2018; OpenAI, 2019; Gupta et al., 2022] . This challenge is especially pronounced in open-ended or visually complex environments, where task goals are often abstract and difficult to specify precisely. For example, in open-world environments such as Minecraft, agents are often assigned high-level goals (e.g.,“build a shelter”) that require long-horizon planning and semantic understanding, while intermediate states provide little explicit reward signal. As a result, hand-crafting reward functions in such settings is typically impractical, due to ambiguous objectives, sparse feedback, and the absence of privileged state information. To address this bottleneck, recent approaches have explored leveraging large pre-trained models as sources of reward, enabling agents to learn from high-level task descriptions or multimodal alignment signals. Large language models (LLMs) have been shown to provide structured code or textual feedback for RL agents in text-based or programmatic settings. However, these methods often rely on privileged state access or handcrafted programmatic interfaces, limiting their applicability in real-world visual domains. When working with visual observations, contrastive vision-language models (cVLMs) such as CLIP are widely used to compute similarity scores between observations and goals for reward shaping. While simple and scalable, these methods typically require task-specific fine-tuning or retraining to be effective across different domains. To move beyond direct similarity-based alignment, preference-based reward modeling has emerged as a promising alternative. These methods learn reward functions by comparing observation pairs and inferring preferences that are typically obtained from human feedback or large pretrained models. This formulation enables finer-grained feedback and greater robustness in visually diverse or ambiguous environments. For instance, RL-VLM-F [Wang et al., 2024] leverages generative vision-language models (VLMs) to generate pairwise preferences, which are then used to train a reward model that assigns scalar values to individual states. We call this approach absolute reward. However, we observe that absolute reward signals can be inconsistent across training steps

FAQs

VLM-AR3L解决强化学习中什么核心难题？

它解决奖励函数难以设计的问题，尤其是在开放式、视觉复杂环境中，任务目标抽象且难以量化时更为突出。

VLM-AR3L如何得到绝对奖励与相对奖励？

VLM-AR3L在自然语言任务目标语境下解析智能体的视觉观测，并利用VLM生成的偏好标签学习奖励。绝对奖励评估单个状态的标量值，相对奖励通过比较连续观测来推断朝目标的进展或退化。

为什么相对奖励能提升训练稳定性与学习鲁棒性？

文中指出，绝对奖励在训练过程中可能因不断暴露新状态而出现不一致，导致长时域任务难优化；相对奖励通过比较监督提供更好的时间一致性，并能缓解全局状态顺序不明确（如循环结构）带来的问题。

Jul 16, 2026

Upload by Himbo