SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition Evidence and Limits

Q: What problem does the paper address in SHAP-weighted cross-modal expert fusion?

It addresses how to reduce SHAP attributions when experts have unequal feature dimensionalities, since mean-abs reduction can suppress high-dimensional cross-modal experts.

Q: How does sum-abs SHAP weighting affect performance on MELD and CMU-MOSEI?

On MELD 7-class emotion recognition, sum-abs XGAF reaches early-fusion-level weighted-F1 and significantly outperforms late fusion. On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF achieves higher weighted-F1 than early fusion with a small but statistically significant improvement.

Q: What do the ablation and diagnostic analyses suggest about where the gains come from?

Most gains come from adding cross-modal experts, especially the trimodal expert, rather than from rich per-sample routing. Mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate strongly on the trimodal expert.

Jul 16, 2026

Upload by Ophelia

Multimodal emotion and sentiment recognition often uses early fusion or late fusion, each with distinct trade-offs in accuracy, modularity, and cross-modal interaction. The study revisits XAI-guided adaptive fusion (XGAF), a tree-based mixture of unimodal and cross-modal experts whose sample-level expert weights are derived from TreeSHAP attribution magnitudes. When experts have unequal feature dimensionalities, SHAP attribution reduction must be sum-abs rather than mean-abs. On MELD, sum-abs XGAF matches early fusion under multiple face-sequence aggregators and significantly outperforms probability-average late fusion. On CMU-MOSEI, sum-abs XGAF improves weighted-F1 over early and late fusion with statistically supported gains. Ablations show most benefits come from adding cross-modal experts, not complex per-sample routing.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

arXiv :2607 .08573v 1 [ cs .AI] 9 Jul 2026 SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits Adis Alihodzic 1* and Selma Skopljakovic Hubljar 1 1* Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, Sarajevo, Bosnia and Herzegovina. *Corresponding author(s). E-mail(s): [adis.alihodzic@pmf.unsa.ba](adis.alihodzic@pmf.unsa.ba) ; Contributing authors: [selma.skoplajkovic@gmail.com](selma.skoplajkovic@gmail.com) ; Abstract Multimodal emotion and sentiment recognition commonly relies on either early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion is often accurate but monolithic, whereas late fusion is modular but can lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (XGAF), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. The study focuses on a methodological issue that becomes important when experts have unequal feature dimensionalities: reducing feature attributions by mean absolute SHAP values can suppress high-dimensional cross-modal experts, while reducing them by summed absolute SHAP values preserves total attribution mass. On MELD 7-class emotion recognition, the proposed sum-abs reduction closes the gap between the cross-modal expert mixture and early fusion across three face-sequence aggregators, with the Transformer variant reaching 0.5983 weighted-F1 compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs XGAF and early fusion on MELD (p = 1 .000), while XGAF remains significantly better than late fusion (p < 0.0001) . On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF reaches 0.6519 weighted-F1 compared with 0.6485 for early fusion and 0.5696 for late fusion, with a small but statistically significant improvement over early fusion (p = 0 .0452) . An expert-pool ablation indicates that most of the gain comes from adding crossmodal experts, especially the trimodal expert, rather than from rich per-sample routing. A focused three-way ablation further confirms that median-abs SHAP behaves similarly to mean-abs SHAP (0.5682 versus 0.5669 weighted-F1 on MELD), while sum-abs SHAP reaches early-fusion-level performance (0.5957 versus 0.5955) . Diagnostic analysis shows that mean-abs and median-abs weights are nearly uniform, whereas sum-abs weights become strongly concentrated on the trimodal expert. The main contribution is therefore not a new state-of-the-art recognition model, but a transparent empirical study of how SHAP attribution reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion. Keywords: multimodal fusion, emotion recognition, sentiment analysis, explainable artificial intelligence, SHAP, XGBoost, MELD, CMU-MOSEI, mixture of experts 1 Introduction Multimodal affective computing attempts to infer human emotion, sentiment, or related affective states from several complementary channels, most commonly text, speech, and visual cues. A sentence may look positive in text but sound sarcastic in speech, or a neutral phrase may become emotionally informative when facial expression is considered. This simple observation explains why fusion is not a minor implementation detail but one of the central design choices in multimodal emotion recognition. Benchmark datasets such as MELD [1] and CMU-MOSEI [2] have made it possible to compare modelling strategies under controlled experimental protocols, but the question of how modalities should be fused remains central. Recent surveys emphasise that modern MER systems must be evaluated not only by accuracy but also by fusion design, robustness to missing or noisy modalities, generalisation across users and datasets, and int

FAQs

What problem does the paper address in SHAP-weighted cross-modal expert fusion?

It addresses how to reduce SHAP attributions when experts have unequal feature dimensionalities, since mean-abs reduction can suppress high-dimensional cross-modal experts.

How does sum-abs SHAP weighting affect performance on MELD and CMU-MOSEI?

On MELD 7-class emotion recognition, sum-abs XGAF reaches early-fusion-level weighted-F1 and significantly outperforms late fusion. On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF achieves higher weighted-F1 than early fusion with a small but statistically significant improvement.

What do the ablation and diagnostic analyses suggest about where the gains come from?

Most gains come from adding cross-modal experts, especially the trimodal expert, rather than from rich per-sample routing. Mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate strongly on the trimodal expert.

Jul 16, 2026

Upload by Ophelia