Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust

Aug 4, 2026

Upload by Asher

Multimodal fusion using a multimodal transformer can perform early and late fusion effectively, yet self-attention alone was designed for unimodal token sequences and may harm modality alignment. Empirical findings show that relying only on self-attention makes the model vulnerable to missing modalities and input noise due to over-dependence on a single modality. Wasserstein Modality Alignment (WMA) introduces an implicit Wasserstein-distance based alignment without adding trainable parameters. Experiments across four datasets with 2- and 3-modality settings and early/late fusion demonstrate consistent gains in both performance and robustness over baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Related Work

Method: Wasserstein Modality Alignment (WMA)

Experiments

Results and Analysis

Conclusion

Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust Zhuo Zhi Department of Electronic and Electrical Engineering University College London ∗ Yuxuan Sun Department of Electronic and Electrical Engineering University College London Qiangqiang Wu Department of Computer Science City University of Hong Kong Ziquan Liu School of Electronic Engineering and Computer Science Queen Mary University of London ∗ Miguel Rodrigues Department of Electronic and Electrical Engineering University College London [zhuo.zhi.21@ucl. ac.uk](zhuo.zhi.21@ucl. ac.uk) [yuxuan. sun.22@ucl. ac.uk](yuxuan. sun.22@ucl. ac.uk) [qiangqwu2@cityu. edu.hk](qiangqwu2@cityu. edu.hk) [ziquan.liu@qmul. ac.uk](ziquan.liu@qmul. ac.uk) [m. rodrigues@ucl. ac.uk](m. rodrigues@ucl. ac.uk) Reviewed on OpenReview: [https: // openreview. net/ forum? id= 2IkaUZdB62](https: // openreview. net/ forum? id= 2IkaUZdB62) Abstract Multimodal fusion with a multimodal transformer is an effective method for both early and late fusion paradigms. However, in a multimodal transformer, the modality fusion is performed solely through the self-attention mechanism, which is originally designed for unimodal token sequences. To improve the self-attention mechanism for handling multimodal input, a parametric adapter model, like the Q-former in BLIP-2, is often used to align tokens from different modalities. Our empirical study unveils that only using the self-attention layer to perform the modality fusion makes the model less robust to missing modalities and input noise, as the model will overly rely on one certain modality. To improve the robustness of the transformer, our paper proposes an implicit approach based on Wasserstein distance that aligns tokens from different modalities without using any additional trainable parameters. Our empirical study shows that the implicit modality alignment improves the effectiveness of the multimodal Transformer in discriminative tasks, as well as its robustness to input noise and missing modalities. We conduct experiments on four downstream task datasets, including 2-modalities and 3-modalities tasks. We also consider different fusion paradigms, i.e., early and late fusion. The experimental results show that our proposed method has a significant improvement in both performance and robustness over all baselines across all datasets and fusion paradigms. 1 Introduction Multimodal machine learning (MML) mimics human perception by integrating multiple modalities such as text, audio, images, video, and sensor data to form a comprehensive understanding of the world. Many ∗ Corresponding author multimodal models have been applied to various tasks like multimodal medical diagnostics Hayat et al.(2022a), sentiment analysis Zadeh et al. (2018) and malicious speech detection Kiela et al. (2020) . Aligning heterogeneous data in multimodal learning is crucial since such data often exhibit distinct distributions, representations, and noise levels. Proper alignment enhances the uniform representation of these diverse data types, leading to improved performance and robustness in multimodal tasks Ghahremani Boozandani & Wachinger (2024); Liang et al. (2024); Kim et al. (2020) . To achieve better modality alignment, various strategies are applied in large-scale multimodal models, such as the Q-Former in BLIP-2 Li et al. (2023), contrastive learning in CLIP Radford et al. (2021) and Imagebind Girdhar et al. (2023) . Multimodal fusion based on a one-tower transformer, named as multimodal transformer (MT), by its flexibility and simplicity, are widely used for a variety of multimodal learning tasks Lee et al. (2023); Nagraniet al. (2021); Zhi et al. (2024); Ma et al. (2021) . Although the multimodal transformer can handle multimodal tokens as the input due to the flexibility of self-attention layers, it lacks a mechanism for modality alignment during the fine-tuning process. In other words, it is not opti

FAQs

Why does self-attention alone lead to weaker robustness in multimodal transformers?

Self-attention is originally designed for unimodal token sequences, so modality fusion becomes implicit through attention weights. This can cause the model to over-rely on a particular modality, making it less robust to missing modalities and input noise.

What is Wasserstein Modality Alignment (WMA) in this work?

WMA is an implicit approach that aligns tokens from different modalities using a Wasserstein-distance-based regularization. It does so without introducing additional trainable parameters.

How is WMA evaluated across tasks and fusion paradigms?

The study runs experiments on four downstream datasets, covering both 2-modality and 3-modality settings. It also compares early and late fusion paradigms, reporting improvements in performance and robustness across baselines.

Aug 4, 2026

Upload by Asher