SpeechSense - A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis

Sep 15, 2026

Uploaded by Maeve

Recent advances in AI have transformed speech processing, but effective speech understanding depends on discerning not only what is said, but also how it is said. Speech Sentiment Analysis decodes paralinguistic cues for real-world uses such as recruitment and customer service, yet existing work is limited by ASR-text cascades that discard prosody and tone, and by benchmarks that emphasize coarse emotions over nuanced interpersonal stances. SpeechSense introduces an 8-class prosody-driven taxonomy, a high-fidelity synthesized dataset with human validation, and experiments showing acoustic-aware models outperform text-only baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Abstract

Introduction

SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis Shicheng Ma Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, Hong Kong [shichengma@cuhk.edu.hk](shichengma@cuhk.edu.hk) Wenqian Cui Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, Hong Kong [wenqian.cui@link.cuhk.edu.hk](wenqian.cui@link.cuhk.edu.hk) Irwin King Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, Hong Kong [king@cse.cuhk.edu.hk](king@cse.cuhk.edu.hk) arXiv : 2608 . 1793 1v 1 [cs .CL] 18 Aug 2026 Abstract Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not just what is said, but how it is said. Speech Sentiment Analysis plays a critical role in decoding these paralinguistic cues for diverse real-world applications such as recruitment and customer service. However, existing Speech Sentiment Analysis research faces two primary limitations. First, dominant approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis. This process inevitably discards essential acoustic features like prosody and tone, failing to capture attitudinal meanings in acoustically ambiguous utterances. Second, current benchmarks suffer from a mismatch in label granularity, prioritizing basic emotions (e.g., happy, sad) over the nuanced interpersonal stances (e.g., confident, impatient) necessary for social sensitivity. To address these limitations, we propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis. Specifically, we define a specialized 8-class taxonomy of interpersonal stances detectable primarily through prosodic cues beyond lexical content alone. We then construct acurated dataset based on this taxonomy, built from high-fidelity speech synthesis and rigorous human validation. Comprehensive experiments across multi-modal LLMs, text-only LLMs, and speech encoders demonstrate that models with acoustic access consistently outperform text-only baselines. These results empirically validate the primacy of acoustic cues in detecting subtle speaker attitudes, highlighting the necessity of SpeechSense.1 CCS Concepts • Information systems → Sentiment analysis; • Computing methodologies → Language resources; Neural networks; • Applied computing → Sound and music computing. Keywords Speech Sentiment Analysis, Paralinguistics, Multi-modal Large Language Models, Synthetic Data 1 Dataset and supplementary materials: [https://github.com/Sher13cked/SpeechSense](https://github.com/Sher13cked/SpeechSense) This work is licensed under a Creative Commons Attribution 4 .0 International License. MM’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2213-4/2026/11 [https://doi.org/10.1145/3767308.3838667](https://doi.org/10.1145/3767308.3838667) ACM Reference Format: Shicheng Ma, Wenqian Cui, and Irwin King. 2026. SpeechSense: A ParalinguisticFocused Dataset for Fine-Grained Speech Sentiment Analysis. In Proceedings of the 34th ACM International Conference on Multimedia (MM’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 7 pages. [https://doi.org/10.1145/3767308.3838667](https://doi.org/10.1145/3767308.3838667) 1 Introduction Recent advances in Artificial Intelligence (AI) have dramatically improved machines’ ability to process and understand human speech, enabling applications ranging from automated transcription services [30] to end-to-end voice assistants [8, 9, 15, 39] . For effective speech understanding, AI models require not only recognizing what is said (linguistic information) but also discerning how it is said (paralinguistic information) [6, 8, 14], as paralinguistic information carries additional meanings beyond the spoken content. A notable task that involves paralingui

FAQs

What limitations affect current Speech Sentiment Analysis research?

Most approaches rely on text-centric pipelines that use ASR and then text analysis, discarding prosody and tone. Benchmarks also use coarse label granularity, focusing on basic emotions rather than nuanced interpersonal stances.

What is SpeechSense and how is it designed?

SpeechSense is a novel dataset for fine-grained speech sentiment analysis that prioritizes prosodic features. It defines a specialized 8-class taxonomy of interpersonal stances that can be detected primarily from prosodic cues beyond lexical content.

How is the dataset constructed and validated?

The dataset is constructed using high-fidelity speech synthesis and then rigorously validated by humans to ensure the labels align with the proposed interpersonal stance taxonomy.

Sep 15, 2026

Uploaded by Maeve