Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

Jul 16, 2026

Upload by Himbo

Visual AutoRegressive modeling (VAR) enables a coarse-to-fine multi-scale autoregressive generation paradigm, delivering strong image generation quality. Yet VAR still faces multi-scale representation learning issues: lower scales emphasize global semantics, higher scales capture fine details, shared-scale architectures create optimization conflicts, and semantic mistakes at early scales propagate through the causal pipeline. A scale-aware token-routed Mixture of Experts (MoE) decouples learning across scales, while external self-supervised features and a residual feature aggregation design strengthen early semantic modeling. Extensive experiments show improved efficiency and generation quality, including better FID on ImageNet 256×256 with fewer training epochs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

Introduction

Motivation: Multi-scale representation learning issues in VAR

Proposed approach: MEPA with token-routed MoE

Semantic guidance at early scales

Experimental results and benchmark performance

arXiv :2607 .0037 1v 1 [ cs .CV] 1 Jul 2026 Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts Nuoyan Zhou 1 ,2†, Zhijun Tu2, Lei Yu2 , Kun Cheng 1, Jie Hu2 ∗ , Nannan Wang 1 ∗, and Xinghao Chen2 1 State Key Laboratory of Integrated Services Networks, Xidian University, Xi’an, China [nuoyanzhou@stu.xidian.edu.cn](nuoyanzhou@stu.xidian.edu.cn), [kuncheng.xidian@gmail.com](kuncheng.xidian@gmail.com), [nnwang@xidian.edu.cn](nnwang@xidian.edu.cn) 2 Huawei Technologies Co., Ltd. {zhijun.tu, yulei96, hujie23, [xinghao.chen}@huawei.com](xinghao.chen}@huawei.com) Abstract. Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256×256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs. 1 Introduction Autoregressive (AR) models, empowered by their remarkable scaling laws [19,30], have achieved tremendous success in natural language processing (NLP) [1, 2, 48, 50] . This success has inspired their adaptation to the computer vision [11, 51, 53, 56, 65], where researchers aim not only to replicate the breakthroughs witnessed in NLP, but also to explore AR modeling as a unified paradigm for both image generation and understanding tasks [47, 53] . A typical AR paradigm for image generation involves two stages: it first discretizes images into tokens via a tokenizer [6, 11, 26, 49, 52, 55, 61] and then utilizesan AR network to generate samples in a sequential manner. Built on this foundational paradigm, recent works [23, 35, 40, 42, 46, 49, 54, 60] have made great progress in image generation, demonstrating comparable performance to diffusion models. † This work was done during an internship at Huawei Technologies Co., Ltd. ∗ Corresponding authors 2 N. Zhou et al. (a) One architecture, multiple roles. (b) Semantic errors propagate to large scale. Fig. 1: Empirical analysis of representation space and expert load. (a) VAR learns different features across scales with a shared architecture, resulting in conflicts of optimization objectives.(b) Semantic errors at small and intermediate scales can propagate across scales and induce unpleasing results. Correct semantics at early stage are critical for predictions at later stage. Specifically, Visual AutoRegressive modeling (VAR) [49] stands out among AR methods for its high-quality and fast image generation. It has pioneered a coarse-to-fine autoregressive modeling via next scale prediction, which decomposes the image data into multi-scale residual representations and builds causality across scales. This design preserves the 2D structure of images and supports efficient parallel decoding, le

FAQs

What key problems does VAR still have in multi-scale representation learning?

Lower scales mainly learn global semantics while higher scales focus on fine details, and using a shared architecture across scales creates optimization conflicts. In addition, semantic errors at early scales can propagate through the causal autoregressive process and degrade final outputs.

How does the proposed MEPA method address these multi-scale issues?

MEPA introduces a token-routed Mixture of Experts (MoE) that adaptively decouples model capacity across scales, reducing adverse effects from conflicting optimization and enabling more specialized feature learning.

How is semantic modeling improved at early scales in MEPA?

MEPA incorporates external self-supervised features to enrich semantics at early stages and uses a residual feature aggregation scheme tailored to the VAR paradigm, which helps prevent semantic error propagation.

Jul 16, 2026

Upload by Himbo