Preference Conditioned Multi-Objective Reinforcement Learning Decomposed Diversity-Driven Policy Optimization

Jul 16, 2026

Upload by Anda

Multi-objective reinforcement learning (MORL) trains agents to balance conflicting objectives, typically using preference-conditioned policies to let users select trade-offs. Prior preference-conditioned approaches are fragile and often fail to recover dense Pareto fronts due to two structural issues: destructive advantage cancellation from early scalarization and representational mode collapse across the preference space. D3PO (Decomposed, Diversity-Driven Policy Optimization) reorganizes optimization by decomposing per-objective learning signals and applying preference weighting only after trust-region stabilization, coupled with a scaled diversity regularizer. On standard high-dimensional and many-objective benchmarks, D3PO finds broader, higher-quality Pareto fronts, outperforming prior methods in hypervolume and expected utility using a single deployable policy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Abstract

Introduction

Motivation for MORL

Challenges in preference-conditioned methods

D3PO overview

arXiv :2602 .07764v2 [ cs .LG] 10 Jul 2026 Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization Tanmay Ambadkar Sourav Panda, Shreyash Kale, Jonathan Dodge, Abhinav Verma Pennsylvania State University University Park, PA 16802 {ambadkar, sbp5911, shreyash, dodge, [verma}@psu.edu](verma}@psu.edu) Abstract Multi-objective reinforcement learning (MORL) seeks to train agents capable of balancing conflicting objectives. While single preference-conditioned policies offer a highly scalable solution, existing approaches remain brittle in practice, frequently failing to recover dense Pareto fronts. We demonstrate that this failure stems from two structural pathologies: destructive advantage cancellation caused by premature Early Scalarization (ES), and representational mode collapse across the preference space. To overcome these bottlenecks, we introduce D3PO, a PPObased framework that fundamentally reorganizes multi-objective optimization. By preserving per-objective learning signals through a decomposed pipeline and integrating preferences only after trust-region stabilization (Late-Stage Weighting), D3PO improves credit assignment under conflicting objectives. Concurrently, a scaled diversity regularizer encourages behavioral divergence proportional topreference distance. D3PO operates entirely within the efficient linear scalarization regime shared by standard deep MORL baselines. By reducing information loss caused due to linear scalarization rather than relying on expensive non-linear utility functions, it suggests that optimization bottlenecks play a significant role. Across available standard benchmarks, including high-dimensional and many-objective environments, D3PO consistently discovers broader, higher-quality Pareto fronts than prior methods, exceeding state-of-the-art hypervolume and expected utility using a single deployable policy. 1 Introduction Reinforcement learning (RL) successfully trains agents to make sequential decisions by maximizing a single scalar reward [1] . However, real-world applications (such as autonomous driving or logistics) require agents to simultaneously optimize multiple, often conflicting objectives. Optimizing a single reward function collapses this richness, frequently leading to suboptimal behaviors and motivating the field of Multi-Objective Reinforcement Learning (MORL) . MORL decomposes objectives into a vector of reward signals. When objectives conflict, a single global optimum is unattainable. Instead, the goal is to learn a set of Pareto-optimal policies [2], allowing users to select desired trade-offs via preference weight vectors [3, 4] . This non-uniqueness of optimal solutions introduces severe algorithmic challenges. The agent must respond to a potentially infinite set of preference queries, and conflicting objective gradients can point in opposing directions, thereby destabilizing policy updates and impairing sample efficiency [5] . In fact, our empirical ∗[https://ambadkar.com/d3po](https://ambadkar.com/d3po) Preprint. Minecart Lunar Lander Building9d Figure 1: Overview of the D3 PO framework. (1) Multi-Head Critic: The critic estimates independent per-objective values V (i)(s,ω) to compute unweighted advantages A (i) . (2) PPO Surrogate Losses: The clipping mechanism is applied to each advantage stream independently Eq. 1, stabilizing the learning signal before scalarization. (3) Late-Stage Weighting: Preference weights ω are applied only to the stabilized surrogate losses Eq. 3, reducing cancellation of gradients prior to optimization.(4) Diversity Regularizer: A diversity loss Eq. 2 is added to force behavioral separation between different preference queries, incurs a penalty under representational collapse, discouraging such solutions during optimization. diagnostics reveal that per-objective gradients actively oppose each other (exhibiting negative cosine similarity) in 36 to 53 pe

FAQs

为什么单一偏好条件化的MORL方法在实践中难以恢复密集帕累托前沿？

文中指出失败来源于两个结构性问题：早期标量化导致优势抵消（destructive advantage cancellation），以及跨偏好空间的表征模式坍塌（representational mode collapse）。

D3PO通过哪些机制改善多目标信用分配与梯度冲突？

D3PO采用分解式流程保留每个目标的学习信号，并在信任域稳定后（Late-Stage Weighting）再应用偏好权重，从而减少优化前梯度抵消，并提升在冲突目标下的信用分配能力。

D3PO如何在不同偏好查询之间鼓励行为多样性？

D3PO引入了经过缩放的多样性正则项，使行为随偏好距离产生分离；同时，该正则项会对表征坍塌引入惩罚，促使训练避免退化到“安全平均行为”。

Jul 16, 2026

Upload by Anda