MRUF: 多粒度路由与不确定性引导融合的鲁棒多模态情感分析

Jul 16, 2026

Multimodal sentiment analysis integrates language, vision, and acoustic cues, yet utterance-level modality quality can fluctuate due to occlusion, noise, motion blur, or transcript errors, causing conventional fusion to over-rely on unreliable signals. MRUF introduces reliability-aware fusion with multi-granularity routing and uncertainty-aware calibration. It summarizes sentiment-relevant representations, estimates utterance-level modality importance via leave-one-out error increases, predicts modality-wise uncertainty, and refines modality gates using inverse-variance reweighting, stabilized by modality-invariant contrastive alignment. Experiments on CMU-MOSI and CMU-MOSEI show consistent gains, with higher predicted uncertainty receiving lower fusion weights.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

Related Work

Multimodal Sentiment Analysis

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis† Haoran Ma 1 ,2 ,3 ,4 ,5 , Yinfeng Yu 1 ,2 ,3 ,4 ,5 ,B , and Liejun Wang 1 ,2 ,3 ,4 ,5 arXiv :2607 . 10599v 1 [ cs .AI] 12 Jul 2026 Abstract—Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentimentrelevant representations, performs subspace-and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMUMOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights. Index Terms— multimodal sentiment analysis, reliabilityaware fusion, uncertainty-aware fusion, modality routing, contrastive alignment. I. INTRODUCTION Multimodal sentiment analysis predicts sentiment by jointly modeling language, visual, and acoustic cues [1], [2],[3] . It is important for human-centered intelligent systems, such as human–machine interaction, driver monitoring, intelligent tutoring, and affective decision support. In practice, multimodal signals are often asynchronous, heterogeneous, and uneven in quality: visual streams may suffer from occlusion, motion blur, or head-pose changes; acoustic signals may be affected by background noise or channel variation; and transcripts may be incomplete or inaccurate. Such quality fluctuations can make a model over-trust unreliable evidence and produce unstable predictions. Existing studies mainly improve multimodal sentiment analysis through cross-modal interaction and robust representation learning. Fusion methods capture modality correlations via tensor fusion, low-rank fusion, attention, graph interaction, progressive alignment, or interaction enhancement [4], †This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 62463029 and 62472368 . 1 School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. 2Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. 3Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. 4Pengcheng Laboratory Xinjiang Network Node. 5Embodied Intelligence Joint Laboratory. BYinfeng Yu is the corresponding author (Email: yuyin[feng@xju.edu.cn](feng@xju.edu.cn)). [5], [6], [7], [8], [9], while robust methods handle missing or degraded modalities through calibration, contrastive learning, reconstruction, dynamic weighting, expert modeling, or modality-invariant distillation [10], [11], [12], [13], [14] . Similar reliability concerns also appear in audio-visual navigation, vision-and-language navigation, audio-visual source separation, speech processing, and visual detail enhancement [15], [16], [17], [18], [19], [20] . However, modality importance is often learned implicitly, and uncertainty is rarely used to directly calibrate modality weights during final fusion. These issues remain even with strong representation backbones. Disentangled or decoupled models improve representation quality by separating modality-invariant and modalityspecific factors or transferring knowledge across modalities [21], [22] . Nevertheless, a degraded modality may still receive excessive influence if final fusion is not gui

FAQs

What problem does MRUF address in multimodal sentiment analysis?

MRUF targets the reliability mismatch caused by utterance-level modality quality variation, where conventional fusion can over-trust occluded, noisy, blurred, or inaccurately transcribed modalities, leading to unstable predictions.

How does MRUF estimate the importance of each modality for an utterance?

MRUF supervises modality routing using leave-one-out error increases to infer utterance-level modality importance.

How does MRUF use uncertainty to improve fusion?

MRUF predicts modality-wise uncertainty and calibrates modality gates with inverse-variance reweighting, so modalities with higher predicted uncertainty receive lower fusion weights.

Jul 16, 2026

Upload by Evangeline