Beyond Time Shifts Adapting Omni LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

Jul 16, 2026

Upload by Skyler

Audio-visual generative models are evolving toward world simulators, where cross-modal synchronization becomes a key proxy for evaluating consistency of generated world dynamics and causal relations. Existing metrics largely assume structural correctness and reduce synchronization to temporal alignment, failing under structural hallucinations and asymmetric audio-video relations that currently require expert human annotation. The work resolves the relative-vs-absolute evaluation paradox by converting ranked human judgments into a continuous, globally consistent reference-free metric using SynthSync data, an Omni-LLM-based latent projection, and RGRPO.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Abstract

Introduction

Contributions of this paper

arXiv :2607 .09091v1 [ cs .CV] 10 Jul 2026 Beyond Time Shifts: Adapting Omni-LLM asa Reference-Free Evaluator for Generative Audio-Visual Models Yijie Qian 1 ,3⋆‡, Juncheng Wang2⋆, Chao Xu3 ,4 , Huihan Wang2 , Yuxiang Feng 1 , Yang Liu3 ,4 , Baigui Sun3 ,4 , Yong Liu 1†, and Shujun Wang2† 1 Zhejiang University, Hangzhou, China 2 The Hong Kong Polytechnic University, Hong Kong, China 3 IROOTECH TECHNOLOGY, China 4 Wolf 1069 b Lab, Sany Group, China ⋆ Equal contribution. † Corresponding authors. {[yijieqian@zju.edu.cn](yijieqian@zju.edu.cn), [wjc2830@gmail.com}](wjc2830@gmail.com})⋆ , {[yongliu@iipc.zju.edu.cn](yongliu@iipc.zju.edu.cn), [shu-jun.wang@polyu.edu.hk}](shu-jun.wang@polyu.edu.hk})† Abstract. As audio-visual generative models evolve into world simulators, cross-modal synchronization stands as a critical proxy for assessing the consistency of world dynamics and causality in generated content. However, existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment. Consequently, they fail on generative outputs, especially when exhibiting structural hallucinations and asymmetric cross-modal relations, which currently mandate expert human annotation to assess synchronization. This dependency introduces a critical paradox: human evaluators rely on relative, reference-dependent comparisons, whereas automated metrics require reference-free, absolute scalars. We resolve this paradox by proposing a framework that distills relative human perception into a continuous, globally consistent metric. First, we introduce SynthSync, a dataset of generative failures ranked via pairwise human annotations. Second, we adapt the Omni-LLM equipped with a continuous latent projection to translate relative human rankings into continuous absolute values. Third, we propose Real-Valued Group Relative Policy Optimization (RGRPO) to internalize the global causal structure of synchronization via listwise score distributions. Empirically, our metric achieves state-ofthe-art human preference alignment. We leverage this estimator to establish a standardized benchmark, advancing AV-Gen assessment from low-level signal correlation to visually grounded causality. Project page: [https://chenhaoqcdyq.github.io/BeyondTimeShifts](https://chenhaoqcdyq.github.io/BeyondTimeShifts) Keywords: Joint Audio-Video Generation · Audio-Visual Synchronization · Omni-LLM 1 Introduction Driven by systems like Sora-2 [3,47], Veo-3 [18,46], and Seedance-2 [4,53], audiovisual generative models [32, 35, 39, 56, 61] are evolving from visual synthesizers ‡ This work was conducted in collaboration with IROOTECH TECHNOLOGY. 2 Y. Qian et al. Two Cases generated by Veo-3.1 Contributions of this paper Fig. 1: Upper row: Two Veo-3.1 [18]-generated audio-video samples with evident mismatches between visual events and audio, showcasing structural hallucinations where the synthesized impact sounds (e.g., the golf strike) exhibit semantic and physical textures inconsistent with the visual cues, and a breakdown in asymmetric relationships where the visual (e.g., dog barking) lacks corresponding audio signal. Lower row: Our synchronization evaluation pipeline. We first introduce SynthSync (left), a dataset of authentic generative synchronization failures with pairwise preference annotations. We then fine-tune an omni-LLM as a reference-free continuous synchronization scorer using a new training paradigm (middle), followed by reinforcement post-training to better capture the global causal structure of synchronization judgments (right) . into nascent “world simulators” [3] . While these models achieve single-modality fidelity, the synchronization of cross-modal events remains fragile (shown in upper row of Fig. 1) . Such failures matter not only for perceptual realism, but also as a key observable proxy for whether a model has learned how actions and sounds relate in the real world. However, measuring

FAQs

What problem does the paper address in evaluating generative audio-visual synchronization?

Current evaluation metrics assume structural correctness and treat synchronization mainly as temporal alignment. This breaks down for modern generative outputs that show structural hallucinations, asymmetric cross-modal relations, and temporally diffuse events, making reliable evaluation hard without expert judgment.

How does the proposed method avoid reference-based evaluation?

The approach distills relative human preference comparisons into continuous absolute scores. It uses SynthSync pairwise human annotations, then adapts Omni-LLM with a continuous latent projection to produce a globally consistent reference-free synchronization metric.

What is RGRPO and what role does it play?

Real-Valued Group Relative Policy Optimization internalizes the global causal structure of synchronization judgments. It does so via listwise score distributions during reinforcement post-training to better reflect how humans assess causal alignment.

Jul 16, 2026

Upload by Skyler