A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

Jul 16, 2026

A comparative study examines how geometric properties of speech-language-model (SLM) and conditional-flow-matching (CFM) modules affect steerable mixed-emotion speech synthesis. Emotion representations are characterized using linear probing and local intrinsic dimensionality (LID), followed by single-site and joint activation steering evaluations on multiple datasets. Results show SLM provides a clean, low-dimensional emotion subspace with strong speaker–emotion disentanglement, whereas CFM generalizes poorly across speakers due to entanglement. Joint steering raises intensity but degrades proportional control and speech quality in-distribution, guiding multi-site steering design.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Abstract

Introduction

Method

Geometry Analysis

A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models Siyi Wang 1 James Bailey 2 Ting Dang 1 arXiv :2607 .00946v 1 [ cs . SD] 1 Jul 2026 Abstract While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flowmatching (CFM) modules as activation steering sites for mixed-emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotionspecific subspace with strong speaker–emotion disentanglement, while CFM exhibitspoor crossspeaker generalization due to speaker–emotion entanglement.Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation. 1. Introduction Generating emotionally controllable speech is essential for applications such as conversational agents, audiobook narration, and assistive communication. Human emotional expression is nuanced, often involving mixed affective cues where multiple emotions coexist within a single utterance (Zhou et al., 2022 ; Cowen & Keltner, 2017), a complexity that current systems generally fail to control effectively. Existing emotion control methods operate through the model’s external interface: label-based approaches can 1The University of Melbourne, Australia 2Monash University, Australia. Correspondence to: Siyi Wang <[siyi.wang.4@student.unimelb.edu.au](siyi.wang.4@student.unimelb.edu.au) >. Published at ICML 2026 Workshop on the Machine Learning for Audio, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . enable explicit emotion conditioning but require costly annotated data and retraining (Cho et al., 2025 ; Gao et al., 2025), while prompt-based methods can describe target emotions but lack precise quantitative control over emotion proportions (Guo et al., 2023 ; Yang et al., 2025) . Activation steering bypasses these limitations by directly injecting learned direction vectors into intermediate activations at inference time, without retraining (Zou et al., 2023 ; Turner et al., 2023) . This paradigm has shown success in LLMs and text-to-image diffusion (Rodriguez et al., 2025 ; Rimsky et al., 2024) . State-of-the-art TTS systems increasingly adopt hybrid architectures combining a speech language model (SLM) with a conditional flow-matching (CFM) decoder (Du et al., 2024 ; Anastassiou et al., 2024 ; Zhou et al., 2026), where the SLM governs high-level prosodic structure and the CFM renders fine-grained acoustics, each a potential site for steering emotional expression. Wang et al. (2026) demonstrate composable mixed-emotion steering via the SLM, while Xie et al. (2025) achieve continuous single-emotion intensity control via CFM. However, no prior work has systematically compared the representation geometry at these two steering sites, examined how geometric properties relate to steering effectiveness, or investigated whether jointly steering both modules yields complementary or interfering effects. We present the first comparative study of SLM and CFMas activation steering sites for mixed-emotion synthesis. Through linear probing and local intrinsic dimensionality (LID) analysis of both modules’ representation geometry, combined with single-site and joint steering experiments on four datasets, our study reveals three key findings:(i) the SLM encodes emotions in geometrically distinct, lowdimensional subspaces with strong cross-speaker generaliza

FAQs

What is the main goal of the study on composable emotion steering?

The study aims to compare SLM and CFM as activation steering sites for mixed-emotion speech synthesis, focusing on how representation geometry relates to steering effectiveness.

How are emotion representations analyzed in the paper?

Emotion representations are characterized using linear probing and local intrinsic dimensionality (LID) to study how emotions are organized in the modules’ representation spaces.

What are the key findings when steering SLM versus CFM, and when steering jointly?

SLM steering offers stronger proportional control with better speaker–emotion disentanglement, while CFM steering yields stronger overall intensity but weaker speaker fidelity and poorer cross-speaker generalization; joint steering increases intensity but degrades proportional control and speech quality on in-distribution data.

Jul 16, 2026

Upload by Olivia Brown