A Descriptive and Normative Theory of Human Beliefs in RLHF

Jul 16, 2026

Upload by Himbo

Human preferences in RLHF are commonly treated as reflections of a human reward function or corresponding optimal state-action values, but human belief about an agent’s capabilities can also shape which options are judged preferable. The study asks whether labelers’ capability beliefs influence the preferences they provide, and what ideal belief set should be held. A preference model incorporating such beliefs is introduced, including a normative bound on policy error from belief mismatch. A human study and synthetic experiments show capability beliefs significantly affect preferences and that assuming agent optimality can be suboptimal, improving RLHF practice.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Introduction

Problem Motivation: Rational Preferences in RLHF

Descriptive vs. Normative Questions

Proposed Belief-Aware Preference Modeling

arXiv :2506 .0 1692v2 [ cs .AI] 10 Jul 2026 A Descriptive and Normative Theory of Human Beliefs in RLHF Sylee Dandekar College of Information and Computer Sciences University of Massachusetts Amherst Shripad V. Deshmukh College of Information and Computer Sciences University of Massachusetts Amherst Frank Chiu College of Information and Computer Sciences University of Massachusetts Amherst W. Bradley Knox Department of Computer Science The University of Texas at Austin Scott Niekum College of Information and Computer Sciences University of Massachusetts Amherst [sdandekar@umass. edu](sdandekar@umass. edu) [svdeshmukh@umass. edu](svdeshmukh@umass. edu) [fchiu@umass. edu](fchiu@umass. edu) [bradknox@cs.utexas. edu](bradknox@cs.utexas. edu) [sniekum@umass. edu](sniekum@umass. edu) Reviewed on OpenReview: [https: // openreview. net/ forum? id= YdW0KZwPeT](https: // openreview. net/ forum? id= YdW0KZwPeT) Abstract Human preferences in RLHF are typically modeled as a function of the human’s reward function or corresponding optimal state-action values. In this work, we propose that human beliefs about the capabilities of the agent being trained also play a key role in preference generation. We examine two questions related to this hypothesis, one descriptive and one normative, respectively: Do human labelers’ beliefs about agent capabilities affect the preferences that they provide? And what is the ideal set of beliefs about an agent—and resulting preferences—for humans to have? We propose a new preference model that incorporates human beliefs and provide a normative theory that bounds the error on the final learned policy based on the mismatch between the human’s beliefs and an idealized set of beliefs. We then confirm via a human study that beliefs about agent capabilities do, in fact, significantly affect preferences and can be influenced through simple interventions. Additionally, we empirically show through synthetic experiments that it is often suboptimal for human preference labelers to assume agent optimality. Collectively, these results theoretically and empirically demonstrate how reducing the mismatch between human beliefs and agent capabilities can lead to more performant RLHF and point toward new best practices for RLHF practitioners. 1 Introduction Reinforcement learning from human feedback (RLHF) is one of the main tools used to align powerful AI systems (Kaufmann et al., 2023) . Alignment is important for numerous reasons, such as minimizing unintentional harm, ensuring safety and control, and increasing public trust and legal compliance (Ji et al. , 2023; Amodei et al., 2016) . In order to use RLHF for alignment more effectively, researchers require highquality data from humans that express rational preferences. This begs the question: How should rationality of preferences be defined within RLHF? Standard approaches to RLHF assume that humans provide preferences based only on what actually happened in the trajectories, i.e., the return-based interpretation (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022; Brown et al., 2019) . This partial return-based approach does not consider events that could have occurred while taking a risky action or what might happen after the end of the trajectory segment. Knox et al. (2024) show that the regret-based model addresses this by looking at the quality of actions taken, rather than just the summed reward of the observed outcomes. Figure 1: Illustration of scenario in which preference for a more optimal partial path can lead to a worse post-RLHF policy. (Path 1): The car drives along the edge of a cliff but straight to the destination. This path takes less time but requires greater capabilities (both during and after) to avoid catastrophic outcomes.(Path 2): The car takes a longer path far away from the cliff, reducing risk. The dotted lines indicate different possible paths that could occur when executing a post-R

FAQs

What new factor does the document propose affects preference generation in RLHF?

It proposes that human beliefs about the capabilities of the agent being trained play a key role alongside reward-function or value-based interpretations.

What descriptive question and normative question does the work examine?

Descriptively, it asks whether labelers’ beliefs about agent capabilities affect the preferences they provide. Normatively, it asks what the ideal set of beliefs—and resulting preferences—should be for humans to hold.

How are the paper’s theoretical claims validated?

The document reports confirmation via a human study showing capability beliefs significantly affect preferences and can be changed via simple interventions, and it further supports findings with synthetic experiments indicating that assuming agent optimality can be suboptimal.

Jul 16, 2026

Upload by Himbo