COMET - Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models

Sep 15, 2026

Uploaded by Aurelia

Video multimodal large language models show strong performance, but fine-grained motion and temporal understanding remains fragile. The bottleneck goes beyond sparse frame sampling: many pipelines lack an end-to-end temporal modeling mechanism that explicitly represents frame-to-frame change, enables appearance–motion interaction, and captures temporal direction sensitivity. COMET introduces a temporally grounded framework with an explicit temporal motion branch and direction-aware optimization, improving action-centric and temporal reasoning benchmarks while keeping static perception on par.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Introduction

Problem: fragile temporal reasoning

Approach: temporally grounded COMET

Method Overview

Temporal motion branch with Taylor frame differences

Appearance–motion fusion via temporal attention bias

Optimization with temporal contrastive GRPO

Experiments and Results

COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models arXiv : 2608 .2 1030v 1 [cs .CV] 2 1 Aug 2026 Chenghua Zhu∗ Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School Peking University Shenzhen, China Kehan Jiang Peking University Beijing, China Zhaolu Kang∗ School of Software and Microelectronics Peking University Beijing, China Lei Wei Peking University Beijing, China Qifan Shi∗ School of Future Technology South China University of Technology Guangzhou, China Lianyu Hu Nanyang Technological University Singapore, Singapore Siyan Wu Aberdeen Institute of Data Science and Artificial Intelligence South China Normal University Foshan, China Guangyuan Dong National University of Singapore Singapore, Singapore Mingbo Yang Sun Yat-Sen University Guangzhou, China Rui Lu Pingan Technology Shenzhen, China Guibo Luo✉ Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School Peking University Shenzhen, China Abstract Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile. The core bottleneck is not only sparse frame sampling, but also the lack of a complete temporal modeling pipeline for explicitly representing frame-to-frame change, enabling appearance-motion interaction, and optimizing temporal direction sensitivity. We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation, appearance-motion fusion, and direction-aware optimization. Architecturally, COMET introduces a temporal motion branch built on Taylor frame differences and injects its motion evidence into the appearance stream via temporal attention bias-enhanced crossattention. For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal and strengthens the model’suse of directional motion patterns encoded by the temporal motion branch. The method achieves consistent overall improvements with a pronounced motion-temporal bias: on Qwen3-VL-8B, actioncentric tasks (STAR, SSv2) improve by 4.9% on average, temporal reasoning tasks (NExT-QA, CLEVRER, LLaVA-178K) by 2.1% over BL-GRPO, while static perception tasks (PerceptionTest) remain on par. The same gain pattern also transfers to InternVL2.5-8B, indicating that COMET generalizes across model families. ∗ These authors contributed equally to this work. MM’26, Rio de Janeiro, Brazil 2026. CCS Concepts • Computing methodologies → Computer vision; Natural language processing; Reinforcement learning. Keywords multimodal large language models, video reasoning, temporal understanding, video question answering 1 Introduction Video multimodal large language models (video MLLMs), represented by systems such as Qwen3-VL [1], InternVL2.5 [3], and VideoLLaMA 2 [4], have recently shown strong progress on video question answering, dense video understanding, and long-form reasoning. Yet their temporal reasoning remains fragile. Many systems answer correctly when static appearance cues or language priors are sufficient, but they still struggle when the answer depends on how a scene evolves over time, such as motion direction, action order, or changes in state. In such cases, temporally grounded evidence is often weaker than the final answer confidence. We argue that a major reason is that current video MLLMs still underuse video as a temporal signal. In many pipelines, videos are reduced to sparsely sampled static frames [17] that are then encoded independently, often as sparse frame tokens [2], before being compressed, pooled, or aligned with the language model. Explicit inter-frame interaction is absent, so temporal evidence becomes sparse and discontinuous. As a result, video data util

FAQs

What problem does COMET address in video multimodal large language models?

COMET targets fragile fine-grained motion-temporal understanding, caused by missing explicit frame-to-frame temporal representation and insufficient optimization for direction- or order-sensitive behavior.

How does COMET represent temporal motion explicitly?

COMET adds a temporal motion branch using Taylor-style frame differences, then injects motion evidence into the appearance stream through temporal attention bias-enhanced cross-attention.

What optimization strategy does COMET use to improve temporal reasoning?

COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage, turning temporal order into a direct learning signal and strengthening reliance on direction-sensitive motion patterns.

Sep 15, 2026

Uploaded by Aurelia