Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

Jul 16, 2026

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) uses complementary audio and visual cues to improve robustness in adverse acoustics. Existing methods often project independently pretrained audio/visual encoders into an LLM space and fuse them as soft prompts, but they rarely correct cross-modal representational discrepancy. This work introduces an optimal transport (OT) semantic alignment framework that couples modality features to LLM linguistic embeddings, yielding probabilistic soft pseudo-labels for contrastive learning. Experiments on LRS3-TED show consistent gains and state-of-the-art results in clean and noisy conditions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Introduction

Background: AVSR and LLM-AVSR

Challenges: visual ambiguity and representational mismatch

Motivation: semantic alignment beyond temporal synchronization

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition* Xugang Lu 1 , Peng Shen 1 , Yu Tsao2 , Hisashi Kawai 1 1. National Institute of Information and Communications Technology, Kyoto, Japan 2. Research Center for Information Technology Innovation, Academia Sinica, Taiwan arXiv :2607 .0900 1v 1 [ cs . SD] 10 Jul 2026 Abstract—Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of crossmodal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modalityspecific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AVHuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve stateof-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs). Index Terms—Audio-visual ASR, optimal transport, feature fusion. I. INTRODUCTION Audio-visual speech recognition (AVSR) has attracted increasing attention due to its robustness in adverse acoustic environments. By leveraging complementary visual cues, such as modality-invariant features [1], modality temporal dynamics [2], and lip movements [3], AVSR systems can substantially alleviate performance degradation caused by background noise [3]–[6] . Recent advances in AVSR have shown that the effective integration of audio and visual information within Conformer-based encoder-decoder architectures, often combined with hybrid CTC/Attention objectives [7], can significantly improve recognition performance [3], [4] . With the rapid development of large language models (LLMs), recent studies have extended AVSR to LLM-based speech recognition frameworks (LLM-AVSR) [5], [8]–[10] . These systems typically employ pretrained modality-specific encoders to extract acoustic and visual representations, which are subsequently projected into the embedding space of an LLM and used as soft prompts for autoregressive transcription generation. Benefiting from the strong contextual modeling and linguistic reasoning capabilities of LLMs, such approaches have demonstrated improved robustness and generalization. However, visual speech recognition (VSR) inherently suffers from ambiguity because multiple phonemes may correspond to identical or highly similar lip shapes [6], [11] . This ambiguity becomes even more severe in continuous speech due to co-articulation effects, where the same viseme may correspond to different phonetic realizations depending on the surrounding context [6], [11] . Although acoustic information can partially compensate for this limitation, speech

FAQs

What limitation do most existing LLM-based audio-visual speech recognition methods have?

They typically fuse projected audio and visual representations without explicitly addressing representational discrepancy between the modalities, which can weaken cross-modal integration and semantic grounding.

How does the proposed optimal transport (OT) approach improve semantic alignment?

OT estimates probabilistic coupling matrices that relate acoustic/visual features to reference linguistic embeddings in the LLM space, producing soft pseudo-labels that supervise contrastive learning for cross-modal consistency.

What model components and evaluation results are reported?

The framework uses a Whisper-based acoustic encoder, an AVHuBERT-based visual encoder, and a LLaMA3.2-3B decoder, and it achieves consistent improvements and state-of-the-art performance on the LRS3-TED benchmark under both clean and noisy conditions across a range of SNRs.

Jul 16, 2026

Upload by River Wang