UniSkip Mamba Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization

Jul 16, 2026

Upload by Maeve

Audio-Visual Temporal Forgery Localization (AV-TFL) is increasingly urgent as AI-generated multimedia enables evidence fabrication and opinion manipulation. Existing approaches that use transformer temporal modeling or channel-wise multimodal fusion treat all frequency components equally, overfitting to high-frequency noise and losing robustness under real-world degradation. Frequency-domain ablation shows forgery-discriminative patterns concentrate in low/mid bands (0–0.15), while high-frequency parts mainly add noise; removing them improves detection. UniSkip-Mamba integrates unified multimodal sequence fusion and skip-scanning state space blocks with group-scan-merge regularization, achieving SOTA results on LAV-DF and AV-Deepfake1M with faster inference.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Frequency-band contribution analysis

Motivation and challenges in existing methods

Proposed framework and contributions

UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization Cangjin Qiu∗ , Quan Zhang∗ , Dan Jiang, and Ke Zhang† arXiv :2607 .04498v 1 [ cs .CV] 5 Jul 2026 Abstract—With the proliferation of AI-generated content, sophisticated multimedia manipulation has raised critical concerns about malicious applications such as opinion manipulation and evidence fabrication, making Audio-Visual Temporal Forgery Localization (AV-TFL) an urgent research frontier. Existing TFL methods have progressed along two main paradigms: Transformer-based temporal modeling and channel-wise multimodal fusion. While these approaches capture temporal dependencies and cross-modal correlations, they process all frequency components indiscriminately, leading to overfitting on highfrequency noise and limited robustness under real-world data degradation. Through systematic frequency domain analysis, we find that forgery-discriminative patterns concentrate in the low/mid-frequency range (normalized frequency 0–0.15), while high-frequency components primarily introduce noise—removing them even improves detection performance by +1.4% . Based on this phenomenon, we propose UniSkip-Mamba, a frequencyaware State Space Model framework that incorporates Unified Multimodal Sequence Fusion to preserve cross-modal phase relationships, and Skip-Scanning Mamba Blocks that implement frequency-aware regularization through a novel Group-ScanMerge mechanism, naturally biasing learning toward discriminative low/mid-frequency patterns (0–0.15) while maintaining representational completeness. We achieve state-of-the-art (SOTA) performance: 63.4% AP@0.95 on LAV-DF (+9.8% improvement) and 63.58% mAP on AV-Deepfake1M (+14.32% improvement), with 6× faster inference. Our frequency-domain analysis provides theoretical justification from a signal processing perspective for why skip-scanning inherently improves both accuracy and robustness. Index Terms—Temporal forgery localization, Mamba, Unified sequence. I. INTRODUCTION The proliferation of Artificial Intelligence Generated Content (AIGC) [1]–[3] has enabled sophisticated multimedia manipulation, raising critical concerns about malicious applications such as opinion manipulation and evidence fabrication. While the multimedia forensics community has made significant progress in Deepfake detection [4]–[8], existing methods predominantly rely on binary classification, failing to identify where manipulation occurs temporally. This limitation severely constrains their utility in practical scenarios such as judicial forensics and content moderation, where precise temporal boundaries of forged segments are indispensable. Consequently, Audio-Visual Temporal Forgery Localization (AV-TFL) has emerged as a critical research frontier. Before addressing the technical challenges of AV-TFL, we pose a fundamental question: In which frequency bands ∗ Cangjin Qiu and Quan Zhang contributed equally to this work. †Corresponding author: Ke Zhang. Cangjin Qiu and Ke Zhang are with Soochow University, Suzhou, China. Quan Zhang and Dan Jiang are with Tsinghua University, Beijing, China. Fig. 1. Motivating Observation: Frequency Band Contribution Analysis. We systematically remove each frequency band from input features and measure performance impact (mAP@0.95) . Key findings: (1) Low/midfrequency bands (0–0.15) are critical for detection, with removal causing up to 92.8% performance drop; (2) High-frequency removal (> 0. 15) has minimal impact (−0 .7% to +1 .4%), with stride-4 models actually improving—confirming that high frequencies introduce noise rather than discriminative information. This motivates our Skip-Scanning design as a frequency-aware regularization strategy. Comprehensive experimental setup and detailed analysis in Section IV-G. does discriminative forgery information reside? Through spectral analysis and systematic ablation experiments, we reveal that forgery-discriminative patterns

FAQs

What problem does AV-TFL address, and why is it important?

AV-TFL localizes the temporal segments where audio-visual content is forged. It is crucial for practical use in judicial forensics and content moderation where precise temporal boundaries are required.

Which frequency bands contain the most forgery-discriminative information?

Forgery-discriminative patterns concentrate in the low/mid-frequency range with normalized frequency 0–0.15. Removing these bands causes large performance drops, while removing high frequencies (>0.15) has minimal impact and can even improve results.

How does UniSkip-Mamba improve accuracy and robustness for AV-TFL?

UniSkip-Mamba uses unified multimodal sequence fusion to preserve cross-modal phase relationships and skip-scanning Mamba blocks with a group-scan-merge mechanism to apply frequency-aware regularization. This biases learning toward discriminative 0–0.15 patterns while filtering high-frequency noise.

Jul 16, 2026

Upload by Maeve