VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Jul 16, 2026

Large-scale video diffusion models often generate visually impressive frames but suffer from geometric drift and unstable camera motion, especially in highly dynamic real-world scenes. VGGRPO (Visual Geometry GRPO) presents a latent geometry-guided post-training framework that preserves pretrained generalization while improving consistency. It introduces a Latent Geometry Model that links video diffusion latents to 4D-capable geometry reconstruction, enabling dynamic-scene extension. Group Relative Policy Optimization uses latent-space rewards for smooth camera trajectories and cross-view geometric reprojection coherence, eliminating costly VAE decoding.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

Introduction

Problem: geometric drift and unstable camera trajectories

Existing approaches and limitations

Proposed solution: VGGRPO framework

Latent geometry model and latent-space GRPO rewards

arXiv :2603 .26599v2 [ cs .CV] 11 Jul 2026 VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward Zhaochong An1 2 * , Orest Kupyn1,3 , Théo Uscidda1,4 , Andrea Colaco1 , Karan Ahuja1 , Serge Belongie2 , Mar Gonzalez-Franco1 and Marta Tintore Gazulla1 1 Google, 2University of Copenhagen, 3University of Oxford, 4 CREST-ENSAE, Institut Polytechnique de Paris [https://zhaochongan.github.io/projects/VGGRPO](https://zhaochongan.github.io/projects/VGGRPO) Large-scale video diffusion models achieve impressive visual quality, yet often fail to preserve geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve thepretrained capacity while improving geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware video post-training. VGGRPO introducesa Latent Geometry Model (LGM) that stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view geometric coherence. Experiments on both static and dynamic benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent video generation. 1. Introduction Recent video diffusion models (An et al., 2026a; Liu et al., 2026b; Qiu et al., 2025; Wiedemer et al., 2025; Zhou et al., 2026) have achieved impressive visual fidelity and broad generalization by training on large volumes of diverse, high-quality data. However, they often lack 3D and motion consistency (Bhowmik et al., 2026; Gao et al., 2026b; Park et al., 2025; Wang et al., 2026b,c; Xue et al., 2025a), exhibiting geometric drift, unstable camera trajectory, and inconsistent scene structure. These issues are critical for downstream applications (Agarwal et al., 2026; An et al., 2025, 2026b; Gao et al., 2026a; Intelligence et al., 2026; Jiang et al., 2026; Le et al., 2025; Liu et al., 2026a; Renet al., 2026; Yuan et al., 2026) such as embodied AI and physics-aware simulation, where stable camera motion and coherent 3D geometry are required. To mitigate these issues, existing efforts largely follow two paradigms. The first paradigm injects geometric structure into the generator via additional conditioning modules (Cao et al., 2025; Ren et al., 2025; Yu et al., 2025) or extra loss components (Danier et al., 2025; Wu et al., 2026) . For example, point cloud-conditioned diffusion models (Cao et al., 2025; Ren et al., 2025) impose pixel-wise constraints from 3D inputs to improve static-scene consistency, while other approaches (Bai et al., 2025; Dai et al., 2026; Huang et al., 2025a; Zhang et al., 2025b) augment video diffusion with auxiliary geometry prediction to improve scene generation. While effective, these modifications often Prompt: Push-in on rugged vehicle crossing rocky desert under clear sky. | Baseline Keyframes | | | --- | --- | | | | | | | | VGGRPO Keyframes | | | --- | --- | | | | | | | Prompt: Tracking shot follows snowboard

FAQs

What main issue does VGGRPO address in current video diffusion models?

Current models often fail to preserve geometric consistency, leading to geometric drift, unstable camera trajectories, and inconsistent scene structure, especially in dynamic scenes.

How does VGGRPO improve geometric consistency without modifying the original architecture heavily?

VGGRPO performs latent-space geometry-guided video post-training, using a Latent Geometry Model to connect diffusion latents with 4D-capable geometry reconstruction rather than relying on architectural changes.

What rewards does VGGRPO use during latent-space Group Relative Policy Optimization?

It uses a camera motion smoothness reward to penalize jittery trajectories and a geometry reprojection consistency reward to enforce cross-view geometric coherence.

Jul 16, 2026

Upload by Olivia Brown