Seeing What Matters - Visual Preference Policy Optimization for Visual Generation

Aug 20, 2026

Reinforcement learning is used for post-training visual generative models, with Group Relative Policy Optimization (GRPO) aligning generators to human preferences. Existing GRPO pipelines rely on a single scalar reward per sample, overlooking spatial and temporal structure and thereby failing to correct localized artifacts or capture fine-grained perceptual cues. Visual Preference Policy Optimization (ViPO) upgrades GRPO by converting scalar feedback into structured, pixel-level advantages. Using a Perceptual Structuring Module, ViPO builds spatially and temporally aware advantage maps via pretrained vision backbones. Experiments on image and video benchmarks show consistent improvements over vanilla GRPO in human-preference alignment and out-of-domain generalization, with architecture-agnostic, lightweight integration into standard GRPO training.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Introduction

Motivation: limits of scalar GRPO for visuals

Proposed approach: ViPO with perceptual structuring

Expected outcome: fine-grained spatial credit assignment

Seeing What Matters: Visual Preference Policy Optimization for Visual Generation Ziqi Ni 1 *, Yuanzhi Liang2∗, Rui Li2 ,3 , Yi Zhou 1†, Haibin Huang2 , Chi Zhang2 , Xuelong Li2† 1 Southeast University, 2Institute of Artificial Intelligence (TeleAI), China Telecom 3University of Science and Technology of China [zqni@seu.edu.cn](zqni@seu.edu.cn) , [liangyzh18@outlook.com](liangyzh18@outlook.com) , [yizhou.szcn@gmail.com](yizhou.szcn@gmail.com) , xuelong [li@ieee.org](li@ieee.org) arXiv :2511 . 18719v4 [ cs .CV] 15 May 2026 Abstract Reinforcement learning (RL) has become a powerful tool for post-training visual generative models, with Group Relative Policy Optimization (GRPO) increasingly used to align generators with human preferences. However, existing GRPO pipelines rely on a single scalar reward per sample, treating each image or video as a holistic entity and ignoring the rich spatial and temporal structure of visual content. This coarse supervision hinders the correction of localized artifacts and the modeling of fine-grained perceptual cues. We introduce Visual Preference Policy Optimization (ViPO), a GRPO variant that lifts scalar feedback into structured, pixel-level advantages. ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps, redistributing optimization pressure toward perceptually important regions while preserving the stability of standard GRPO. Across both image and video benchmarks, ViPO consistently outperforms vanilla GRPO, improving in-domain alignment with human-preference rewards and enhancing generalization on out-of-domain evaluations. The method is architecture-agnostic, lightweight, and fully compatible with existing GRPO training pipelines, providing a more expressive and informative learning signal for visual generation. 1. Introduction Reinforcement learning (RL) has recently emerged as an effective framework for aligning visual generative models [1, 13, 22–24, 26, 29, 40] with human preferences [4, 5], enabling scalable supervision beyond paired data. Among RL-based approaches, Group Relative Policy Optimization (GRPO) [6] has attracted attention for its group-wise *Equal contribution. Work done when Ziqi interned at Institute of Artificial Intelligence (TeleAI), China Telecom. †Corresponding authors. Figure 1 . Brief illustration of our work. Existing GRPO for visual generation assigns a single scalar advantage to the entire content, producing coarse feedback that often leads to sub-optimal results. In contrast, our ViPO converts this coarse signal into preference-aware feedback, enabling fine-grained alignment. This allows, for instance, differentiated optimization of the dancing doll and its background, yielding outputs that are more coherent, harmonious, and perceptually pleasing. comparison-based advantage formulation, which improves optimization stability and sample quality. Recent studies [38, 41] have successfully extended GRPO to diffusion and flow-based generators, confirming its potential for reinforcement-driven alignment in visual generation. However, GRPO was originally designed for token-level or sequence-level outputs, such as in language or reasoning tasks. When directly applied to visual data, this formulation assumes that each visual instance, whether a static image ora video, can be represented by a single scalar advantage, ignoring the rich spatial and temporal structure inherent in visual generation. Such simplification makes GRPO less sensitive to regional or semantic variations within visual content, limiting its ability to assign differentiated credit across spatial locations. Consequently, although the framework remains effective in principle, it provides insufficiently structured feedback for complex visual synthesis tasks. Specifically, this coarse feedback directly affects the visual quality and perceptual alignment of generated results. In convention

FAQs

Why does existing GRPO struggle with visual generation?

GRPO typically assigns a single scalar advantage to an entire image or video, ignoring spatial and temporal structure. This produces coarse feedback and weak credit assignment across regions and semantics, limiting perceptual fidelity.

What is the core idea behind ViPO?

ViPO reformulates GRPO’s advantage representation by lifting scalar rewards into structured, pixel-level advantages. It redistributes supervision according to perceptual relevance across regions rather than treating all pixels equally.

How does ViPO generate pixel-level advantage maps?

ViPO uses a Perceptual Structuring Module built on pretrained vision backbones to extract spatial and semantic relevance cues. These cues guide advantage assignment without requiring dense annotations.

Aug 20, 2026

Upload by Caleb Sterling