C3 ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection

Jul 16, 2026

Upload by Aurora

Active Speaker Detection identifies whether a visible person is speaking at each video moment. Audio–visual fusion approaches work well on clean data but lose accuracy under real-world corruptions such as background noise, occlusion, and simultaneous degradation of both modalities. The weakness stems from missing explicit consistency constraints that guide robust, semantically aligned cross-modal representations. C3 ASD introduces three complementary constraints: embedding-level inter-modality consistency, sequence-level intra-modality consistency via track-aware contrastive learning, and prediction-level consistency via knowledge distillation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Introduction

Problem: Robustness Under Multi-Modal Corruption

Key Idea: Multi-Level Consistency Constraints

Method Overview

Inter-Modality Consistency

Intra-Modality Consistency

Prediction-Level Consistency

arXiv :2607 .030 18v2 [ cs .CV] 7 Jul 2026 C 3 ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection Jin Hong 1 *, Jisoo Park 1 *, and Junseok Kwon 1 Chung-Ang University, Seoul, Republic of Korea {jindl465,susiehome,[jskwon}@cau.ac.kr](jskwon}@cau.ac.kr) * Equal contribution. Fig. 1. Multi-Level Consistency for Robust ASD. In real-world active speaker detection, audio and visual streams are often corrupted by noise and occlusion. Existing models learn modality-clustered representations fragile under such degradations. In contrast, our multi-level consistency framework produces class-aligned, modality-invariant embeddings robust across diverse corruption scenarios. Abstract. Active Speaker Detection determines whether a visible person in a video is speaking at each moment. While recent audio–visual fusion methods perform well on clean data, they degrade under realworld corruptions such as background noise, occlusion, or simultaneous modality degradation. We attribute this limitation to the absence of explicit consistency constraints that promote robust, semantically aligned representations across modalities. Without such guidance, models tend to learn fragile modality-specific shortcuts that fail under corrupted conditions. We propose C3 ASD, a multi-level consistency-driven framework with three complementary constraints: embedding-level inter-modality consistency aligns audio-visual representations during speech; sequencelevel intra-modality consistency separates speaking and non-speaking clusters via track-aware contrastive learning; and prediction-level consistency stabilizes fusion through knowledge distillation. Extensive experiments demonstrate significant improvements under diverse audio, visual and joint corruptions, while maintaining competitive performance on clean data. Keywords: Active Speaker Detection · Audio-Visual Consistency · Corruption Robustness 2 J. Hong et al. 1 Introduction Active Speaker Detection (ASD) determines whether a visible person in a video is speaking at each moment, serving as a fundamental task in video understanding with applications in video conferencing [11], human–robot interaction [20, 35, 37, 41], and multimedia retrieval [7] . Owing to its inherently multimodal nature, ASD relies on both auditory speech signals and visual facial movements, motivating the development of audio–visual fusion methods that exploit complementary information from the two modalities. Recent advances in audio–visual ASD have achieved strong performance using sophisticated fusion architectures, such as temporal convolutional networks, transformers, and attention mechanisms [25, 32, 38, 44, 48] . These methods typically extract modalityspecific features and aggregate them through various fusion strategies. Despite their success on clean data, they often treat audio and visual streams as separate sources to be combined, without explicitly modeling how these modalities should relate to and constrain each other. This oversight becomes particularly problematic in real-world scenarios where one or both modalities may be corrupted by noise, occlusion, or other distortions. In real-world settings, modality corruption is inevitable. Audio may be contaminated by background noise, music, or overlapping speech [36, 40], while visual inputs can degrade due to motion blur, occlusion, low resolution, or rapid head movement. More challenging are scenarios where both modalities are simultaneously unreliable, such as unconstrained videos in the wild [31] . Under such conditions, existing ASD models often experience significant performance degradation [18,42], indicating that current approaches fail to robustly and complementarily leverage audio and visual information. We attribute this limitation to how current models learn audio-visual representations. As shown in Fig. 1, existing models learn modality-clustered representations that are fragile under real-world

FAQs

What problem does C3 ASD address in active speaker detection?

C3 ASD targets the performance drop of audio–visual active speaker detection under real-world corruption like noise, occlusion, and simultaneous modality degradation.

Why do existing models degrade when modalities are corrupted?

The degradation is attributed to the lack of explicit consistency constraints, which allows models to learn fragile modality-specific shortcuts instead of robust, semantically aligned representations.

What are the three consistency constraints proposed by C3 ASD?

C3 ASD uses (1) embedding-level inter-modality consistency during actual speech, (2) sequence-level intra-modality consistency via track-aware contrastive learning, and (3) prediction-level consistency through knowledge distillation to stabilize fusion.

Jul 16, 2026

Upload by Aurora