SMOPD - Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation - read online free

Sep 15, 2026

Uploaded by Anda

Specialize-and-Merge Online Policy Distillation (SMOPD) improves multi-reward reinforcement learning when reward dimensions differ in signal density. Existing Group reward-Decoupled Normalization Policy Optimization (GDPO) normalizes each reward dimension before aggregation, but still fails to balance dense rewards with fine-grained scores against sparse rewards with binary feedback. SMOPD uses two stages: reward-priority training of multiple reward-specialized teachers, then online policy distillation to merge them into a single student policy with balanced optimization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Motivation: reward-density imbalance

Proposed method: Specialize-and-Merge Online Policy Distillation (SMOPD)

Stage 1: Specialize

Stage 2: Merge

Experimental validation and results

arXiv : 2608 .03092v2 [cs .LG] 2 1 Aug 2026 SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation Wen Wang‡1,2, Jiahua Bao‡1, Tu Yongsiqi‡1, Yihao Liu‡1, Haotian Zhou‡1, Haoxuan Ma‡1, Mengyu Zhou†1, Wenkui Fan‡1, Junwei He2 , Xiaoxi Jiang1 and Guanjun Jiang1 1 Qwen Large Model Application Team, Alibaba, 2University of Chinese Academy of Sciences ‡Work done during an internship at Alibaba.†Corresponding author. We aim to improve model performance in multi-reward reinforcement learning training process. Existing Group reward-Decoupled Normalization Policy Optimization (GDPO) has mitigated the issue of reward signals masking one another during direct scalarization by normalizing each reward dimension separately before aggregation. However, our experiments show that GDPO still struggles to balance reward signals with different granularities. Specifically, in some particular training tasks, the model may receive a dense reward that assigns fine-grained scores ranging from 0.1 to 1.0, together with a sparse reward that provides only binary feedback of either 0 or 1 . In such cases, we find that the sparse reward may provide an insufficient optimization signal, preventing its corresponding capability from being effectively reinforced. Therefore, how can we strengthen the optimization signal from the sparse reward without sacrificing the capability already learned from the fine-grained reward? To overcome this limitation, we propose Specialize-and-Merge Online Policy Distillation (SMOPD), a two-stage training method for multi-reward optimization. Stage1-Specialize: SMOPD first employs reward-priority configurations to train multiple reward-specialized teachers, allowing each reward to be learned under conditions where its signal can effectively drive optimization. Stage2-Merge: SMOPD then utilizes online policy distillation to combine the reward-specialized capabilities of these teachers into a single student policy, while maintaining balanced task-level optimization. To validate our method, we conduct experiments on two multi-reward settings: complementary rewards (tool-calling accuracy and format) and conflicting rewards (helpful and harmless rewards) . Based on above settings, SMOPD outperforms GDPO across 1.5B, 3B and 7B backbones. 1. Introduction Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models with human preferences Ouyang et al. (2022); Christiano et al. (2017) . In multi-reward alignment, prior work commonly adopts GRPO Shao et al. (2024) to optimize a single policy with several reward signals. However, because GRPO first sums different rewards and then computes group-relative advantages, reward dimensions with different scales or reward combinations can mask one another during scalarization, causing the final advantage to lose reward-specific information. GDPO Liu et al. (2026b) mitigates this aggregation-level issue by normalizing each reward dimension separately before aggregation, thereby preserving the contribution of each reward in the training signal. Building on this reward-level decomposition, GD2 PO Liu et al. (2026a) further addresses conflicts among reward dimensions by filtering rollouts with severe reward-wise disagreement and reweighting queries according to reward consensus. Despite these improvements, GDPO and GD2 PO still focus on how observed reward signals are normalized and combined within a single policy. This leaves open a different problem: Reward dimensions can differ not only in scale, but also in how often they provide informative learning signals. Group-based advantage estimation relies on reward differences among rollouts sampled for the same prompt. Dense reward distributions can rank sampled responses in most rollout groups, continuously providing reliable optimization signals. Sparse reward distributions, by contrast, may assign the same value to most responses in a group

FAQs

What problem does SMOPD address in multi-reward reinforcement learning?

SMOPD targets reward dimensions that provide different learning signal densities, where sparse binary rewards yield insufficient within-group variation and get overwhelmed by dense rewards.

How does GDPO fall short compared with SMOPD?

GDPO normalizes each reward dimension separately before aggregation, but it still cannot ensure balanced optimization when sparse rewards lack informative response-level differences within rollout groups.

What are SMOPD’s two training stages?

Stage 1 trains multiple reward-specialized teachers using reward-priority configurations, and Stage 2 merges them into a single student policy via online policy distillation.

Sep 15, 2026

Uploaded by Anda