STAGE - Controlled Objective Admission for Multi-Preference LLM Alignment

Sep 15, 2026

Multi-preference alignment is commonly treated as scalarization, which combines reward dimensions but leaves unclear when each preference should enter policy optimization. STAGE introduces a stability-guided active-set controller for controlled objective admission. Starting with a small active set, it retains admitted objectives and expands using reward-deviation gates with a patience budget. A probing phase orders dimensions hard-to-easy and adaptive weighting emphasizes underperforming active objectives, improving held-out benchmark averages.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Curriculum Progression

Related Work

arXiv :2608 . 16553v1 [ cs .CL] 17 Aug 2026 August 18, 2026 STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment Yongqi Tong Zhenyu Zhang Ruirui Wang Kewei Fu Shaoqing Lin Sijie Dong Jiang-Ming Yang Xin Zhang Jianshe Li Ant International Correspondence: [tongyongqi.yq@ant-intl.com](tongyongqi.yq@ant-intl.com) Abstract Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose STAGE, a stability-guided active-set controller for controlled objective admission. STAGE starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that STAGE obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF. Keywords: RLHF, multi-preference alignment, curriculum learning, objective admission, reinforcement learning 1 Introduction . . . Curriculum Progression Stage 1 (Pref {1}) Stage 2 (Prefs {1 and 2}) Final Stage 􀀢 (All Prefs) Figure 1: Overview of STAGE. A probing phase orders preference dimensions by early gain and volatility. Training begins with a small active set P 􀀺, optimizes it with adaptive preference weighting (Eq. 9), and expands when both the Instantaneous Stability Gate (ISG) and Windowed Stability Gate (WSG) are satisfied or the patience budget 􀀩􀀼􀀰􀁇 is reached. For deployed LLMs, alignment is not a single behavior to improve. A policy update can make answers more helpful while weakening safety, factuality, refusal behavior, candor, reasoning quality, or creativity. RLHF therefore faces a multi-objective training problem, even when feedback is implemented as an aggregate preference reward [Christiano et al., 2017, Stiennon et al., 2020, Ouyang et al., 2022] . Existing multi-objective alignment methods mainly ask how to represent, combine, or schedule these criteria: reward composition and Reward Soups build scalar or interpolated objectives [Coste et al., 2024, Ramé et al., 2023]; Pareto-or gradient-based methods adjust update directions or weights [Yang et al., 2024, He and Maghsudi, 2025]; and data-centric methods such as RCS and Curri-DPO schedule the preference pairs [Williams, 2024, Xu et al., 2025, Pattnaik et al., 2024] . Within this broader setting, we study a narrower operational choice that arises when multi-objective RLHF is organized as a curriculum: should training expose the policy to all objectives from the start, or expand the objective set over stages? If training is staged, when is the current set of criteria stable enough to absorb another preference dimension? In a 15-dimensional inventory, all-at-once optimization may let easy objectives dominate or amplify conflicts, while rigid sequencing may introduce shifts that damage earlier behavior. We propose STAGE, a stability-guided active-set controller for multi-preference RLHF. Rather than introducing a new reward model or DPO loss, STAGE schedules which objectives are active, retains introduced objectives, and expands when recent reward-deviation signals are low enough under its heuristic. It uses a short probing phase for hardto-easy ordering, adaptive weighting for underperforming active dimensions, and pointwise/windowed gates before expansion. The core contribution is explicit objective admission. STAGE combines cumulative retention

FAQs

What problem does STAGE address in multi-preference RLHF training?

STAGE targets the temporal decision of when each preference dimension should become active during policy optimization, beyond simple scalarization.

How does STAGE expand the active preference set?

It starts with a small active set, retains admitted objectives, and expands when stability gates indicate low recent reward deviation or when a patience budget is exhausted.

How does STAGE determine the order of preference dimensions?

It uses a probing phase to estimate a hard-to-easy order based on early gain and volatility, then applies adaptive weighting to underperforming active dimensions.

Sep 15, 2026

Uploaded by Emma Wilson