Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

Jul 16, 2026

Upload by Asher

Recognizing sentiment polarity, positive or negative, from speech is difficult because it depends on both vocal characteristics and the meaning of spoken words. A multimodal approach is introduced by combining audio with text information: transcripts are generated automatically using ASR, then translated into multiple languages using machine translation. Audio and multilingual text are fused through cascaded cross-modal transformer blocks. Knowledge distillation transfers the multimodal teacher into an audio-only student, improving performance without added inference cost. Experiments and ablations confirm that both automatic transcripts and translations help.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Problem setting and motivation

Proposed multimodal pipeline

Knowledge distillation and evaluation context

arXiv :2607 .066 1 1v 1 [ cs .CL] 7 Jul 2026 Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts Andrei-George Durduna , Victor Constantinescub , Radu Tudor Ionescua,b,∗ a Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania b Department of Data Science, PPC Romania, 30 Mircea Voda, Bucharest 030667, Romania Abstract Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account. To this end, we propose amultimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple languages via machine translation tools. Audio and multilingual text features are combined via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one. We further distill knowledge from the multimodal model, called teacher, into a unimodal (audio only) model, called student. We conduct experiments on a large-scale dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful. Moreover, we show that the audio-only model can be enhanced via distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at [https://github.com/andreidurdun/cross-modal-audio-sentiment](https://github.com/andreidurdun/cross-modal-audio-sentiment). Keywords: multimodal learning; audio sentiment analysis; audio polarity classification; cross-modal transformer 1. Introduction Sentiment polarity classification from speech is an actively studied task [2, 11, 28, 31, 44], having a broad range of real-world applications, such as customer service call analysis [5], virtual assistant adaptation [1], mental health monitoring [42], vehicle driver monitoring [52] and gaming experience adaptation [25], among others. As for most speech processing tasks nowadays, deep learning models [14, 18, 37, 41], especially audio foundation models [3, 9, 37, 38], represent the mainstream solution for polarity classification, due to their typically high accuracy levels. However, the task remains challenging, as it involves the concurrent analysis of several aspects, including tone (which transmits emotion), pitch (which conveys excitement or calmness), word usage (which reflects the polarity of the transmitted message), irony / sarcasm (which may often indicate the opposite opinion than the transmitted message) . Disentangling these aspects is a key pathway towards improving performance, yet this area remains largely underexplored in current literature. To this end, we propose a novel knowledge distillation (KD) pipeline [15], where the teacher model benefits from disentangled audio and text information. While the audio modality is readily available, the text transcript is not directly ∗ Corresponding author. E-mail address: [radu.ionescu@fmi.unibuc.ro](radu.ionescu@fmi.unibuc.ro) Preprint submitted to Elsevier July 9, 2026 | | Audio modality Student | | | --- | --- | --- | | | Tea(her | | | --- | --- | --- | | Automati( Spee(h Re(ognition | | | | | | | RoBERTa RoBERTuito Cross-Modal Transformer Classi'(ation Head Knowledge distillation Cross-Modal Transformer Cross-Modal Transformer | [CLS] | | --- | | | | | | [CLS]

FAQs

How are text inputs obtained for the multimodal sentiment model?

Text transcripts are generated automatically from speech using an ASR system. The transcripts are then translated into multiple languages with machine translation to create additional text modalities.

How are audio and multilingual text modalities combined in the proposed system?

The approach uses cascaded cross-modal transformer blocks that integrate modalities one by one until a joint representation is formed.

What does the knowledge distillation step achieve for the final audio-only model?

A multimodal teacher model transfers knowledge to an audio-only student model. This improves audio-only sentiment polarity performance without increasing computational overhead during inference.

Jul 16, 2026

Upload by Asher