Enhancing Flow Matching with a Unified Guidance Framework for Efficient and Robust Speech Synthesis

Jul 16, 2026

Upload by Logic

Flow Matching (FM) for speech generation is limited by high inference latency and timbre leakage. A unified guidance framework is presented to improve both efficiency and robustness via two complementary mechanisms. Data-guidance uses heterogeneous augmentation to encourage disentanglement between linguistic content and acoustic residue. Enhanced model-guidance combines trajectory rectification with an intrinsic guidance objective, distilling conditional knowledge into network weights and removing Classifier-Free Guidance (CFG) overhead. Experiments show nearly threefold faster inference and better speaker similarity than strong baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Abstract

Introduction

Timbre leakage in zero-shot generation

Inference latency in ODE-based generation

Unified guidance framework overview

Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis Zuda Yu∗, Qianhui Xu∗, Ting Chen∗∗, Junhui Zhang∗∗, Tao Fu∗∗, Hongjiang Yu∗∗, Qiangqing Wang∗∗, Yang Song∗∗ Zuoyebang, China {yuzuda, xuqianhui02, chenting08, zhangjunhui04, futao01, yuhongjiang, wangqiangqiang, [songyang](songyang}@zuoyebang.com)[}](songyang}@zuoyebang.com)[@zuoyebang.com](songyang}@zuoyebang.com) arXiv :2607 .00363v 1 [ cs . SD] 1 Jul 2026 Abstract Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage. To address these bottlenecks, we propose a unified guidance framework that enhances generation efficiency and robustness through two complementary strategies. On the data front, we introduce Data-guidance via heterogeneous augmentation, encouraging the model to disentangle linguistic content from acoustic residue. In parallel, we propose an enhanced Model-guidance mechanism that synergizes trajectory rectification with a novel intrinsic guidance objective. This approach distills conditional knowledge into network weights and straightens inference trajectory path, thereby eliminating Classifier-Free Guidance (CFG) overhead. Experiments demonstrate that our framework accelerates inference by nearly three times while effectively improving speaker similarity compared to state-of-the-art baselines. Audio samples are available here.1. Index Terms: speech synthesis, voice conversion, flow matching 1. Introduction Flow Matching (FM) [1] has rapidly advanced speech synthesis by modeling the continuous transformation from a simple prior to complex data distributions. This paradigm has demonstrated remarkable potential across various speech generation tasks. For instance, text-based approaches such as F5-TTS [2] and Matcha-TTS [3] achieve fully non-autoregressive speech generation by directly mapping text to acoustic features. Concurrently, discrete-token-based frameworks including the CosyVoice family [4–7], MaskGCT [8], and FireRedTTS [9] apply FM to convert semantic representations into mel-spectrograms. Furthermore, recent audio-language models like Kimi-Audio [10], GLM-4-Voice [11], and Step-Audio [12] increasingly rely on FM as a high-fidelity detokenizer for waveform reconstruction. Beyond speech synthesis, models such as Seed-VC [13] and StableVC [14] illustrate the capacity of FM to effectively disentangle speech attributes for voice conversion. Despite these impressive milestones, the widespread deployment of FM models in real-time scenarios remains fundamentally constrained by two distinct yet critical bottlenecks. The first bottleneck, timbre leakage, compromises the robustness of zero-shot generation. Token-based models rely on discrete semantic representations as conditioning inputs, yet *These authors contributed equally. **indicates the corresponding author. 1 [https://yuzuda283.github.io/unified-guidanc](https://yuzuda283.github.io/unified-guidanc)e-flow-matching/Interspeech2026_demo_samples/ extracting purely disentangled content is notoriously difficult. Traditional information bottlenecks such as Vector Quantization (VQ) [15] attempt to strip away speaker identity but often degrade linguistic intelligibility and pronunciation quality. Consequently, data-driven approaches have gained prominence. SeedTTS [16] mitigates leakage by retraining models on synthetic cross-speaker pairs generated via self-distillation, while SeedVC [13] successfully employs an external generative model to perturb the source audio during training, forcing the network to align with the target prompt. These strategies inspire us to extend single-stage perturbation into a more comprehensive dataguidance constraint. The second bottleneck, inference latency, stems from the intrinsic nature of ODE-based generation imposing two distinct computational costs. First, iterative ODE solvers require a high Number of F

FAQs

What problem does the unified guidance framework address in flow matching speech synthesis?

It targets two bottlenecks: high inference latency and timbre leakage, which reduce robustness in zero-shot generation.

How does Data-guidance reduce timbre leakage?

It uses dual-stage heterogeneous augmentation with model-driven cross-synthesis and signal-driven acoustic deformations to weaken the acoustic reliability of semantic tokens during training.

How does Enhanced Model-guidance improve efficiency and avoid CFG overhead?

It synergizes trajectory rectification with an intrinsic guidance objective, distilling conditional knowledge into network weights to straighten the inference trajectory and eliminate Classifier-Free Guidance’s extra conditional/unconditional passes.

Jul 16, 2026

Upload by Logic