Policy-based Tuning of Autoregressive Image Models with Instance-and Distribution-Level Rewards

Jul 16, 2026

Upload by Finn

Autoregressive image generation models trained with maximum-likelihood estimation optimize token probability but do not directly target perceptual preference, semantic alignment, or distributional coverage. A reinforcement learning approach is proposed by formulating token-based AR synthesis as a Markov Decision Process and optimizing with Group Relative Policy Optimization (GRPO). The framework introduces a distribution-level Leave-One-Out FID (LOO-FID) reward using an exponential moving average of feature moments to encourage diversity and prevent mode collapse, combined with instance-level CLIP and HPSv2 rewards plus adaptive entropy regularization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages48

Reading Time2 h 1 min

Introduction

Motivation and Contributions

arXiv :2603 .23086v2 [ cs .LG] 9 Jul 2026 Policy-based Tuning of Autoregressive Image Models with Instance-and Distribution-Level Rewards Orhun Buğra Baran 1, Melih Kandemir2, and Ramazan Gokberk Cinbis 1 ,3 1 Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye {bugra,[gcinbis}@ceng.metu.edu.tr](gcinbis}@ceng.metu.edu.tr) 2 Department of Mathematics and Computer Science (IMADA), University of Southern Denmark, Odense, Denmark [kandemir@imada.sdu.dk](kandemir@imada.sdu.dk) 3 ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye Abstract. Autoregressive (AR) models are highly effective for image generation, yet their standard maximum-likelihood estimation training lacks direct optimization for sample quality and diversity. While reinforcement learning (RL) has been used to align diffusion models, these methods typically suffer from output diversity collapse. Similarly, concurrent RL methods for AR models rely strictly on instance-level rewards, often trading off distributional coverage for quality. To address these limitations, we propose a lightweight RL framework that casts token-based AR synthesis as a Markov Decision Process, optimized via Group Relative Policy Optimization (GRPO) . Our core contribution is the introduction of a novel distribution-level Leave-One-Out FID (LOO-FID) reward; by leveraging an exponential moving average of feature moments, it explicitly encourages sample diversity and prevents mode collapse during policy updates. We integrate this with composite instance-level rewards (CLIP and HPSv2) for strict semantic and perceptual fidelity, and stabilize the multi-objective learning with an adaptive entropy regularization term. Extensive experiments on LlamaGen and VQGAN architectures demonstrate clear improvements across standard quality and diversity metrics within only a few hundred tuning iterations. The results also show that the model can be updated to produce competitive sampleseven without Classifier-Free Guidance, and bypass its 2x inference cost. 1 Introduction Autoregressive (AR) models have recently re-emerged as a strong and scalable framework for high-fidelity image generation. By modeling pixel or token sequences with Transformer decoders, they unify visual synthesis with the language modeling paradigm underlying GPT-style systems [30] . Advances such 3 Accepted at the European Conference on Computer Vision (ECCV), 2026 . 2 Baran et al. as VQVAE and VQGAN tokenizers [10, 25], hierarchical scaling [41, 48], and continuous-token variants [12, 21] have enabled AR models like LlamaGen [39] to match or even surpass diffusion models [17] in both quality and throughput. Despite this progress, most AR generators remain trained solely via maximum likelihood estimation (MLE), which does not directly optimize semantic alignment or perceptual preference and thus limits post-training controllability. Reinforcement learning (RL) has proven central to shaping the behavior of large language models. Recent progress in large language models (LLMs) has been driven by post-training pipelines that combine supervised instruction tuning, preference learning, and policy optimization using methods such as PPO [36], DPO [31], or GRPO [37] . These approaches produce models that better follow instructions and align with human preferences [5, 27, 53] . A parallel trend has emerged in diffusionbased image generation, where models originally trained with re- Fig. 1: Maximum-likelihood training (Left): maximize true token xt probability from its groundtruth context. Proposed method (Right): maximize image-level correctness and distribution-level alignment. construction losses are post-trained using external reward models such as CLIP [29], HPSv2 [44], or ImageReward [46] . Existing approaches include offline supervised finetuning [13], differentiating through the denoising process [28, 45], and preference or policy optimizat

FAQs

Why do maximum-likelihood trained autoregressive image models struggle with quality and diversity goals?

They maximize token likelihood from ground-truth context but do not directly optimize sample quality, semantic alignment, or distributional coverage, which limits post-training controllability.

How does the proposed method optimize token-based AR synthesis?

It casts token generation as a Markov Decision Process and applies Group Relative Policy Optimization (GRPO) to fine-tune the autoregressive image generator.

What is the role of the Leave-One-Out FID (LOO-FID) reward?

LOO-FID is a distribution-level reward computed via an exponential moving average of feature moments, explicitly encouraging diversity and reducing mode collapse during policy updates.

Jul 16, 2026

Upload by Finn