Flow Matching-Based Speech Source Separation with Best-of-N Biometric Sampling

Jul 16, 2026

Upload by Oliver

Single-channel speech separation remains difficult for real-world deployment due to source permutation ambiguity, stochastic sampling variability in generative models, and the operational burden of long-recording inference with chunk-wise processing. A conditional flow-matching approach is introduced to yield an ordered two-source output conditioned on the mixture. A frozen speaker encoder provides source ordering in training and enables biometric best-of-N candidate selection plus chunk-level channel alignment. Libri2Mix evaluation uses SI-SDR, PESQ, and ESTOI, with downstream ASR (cpWER) and speaker verification (EER).

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

Background

Speech source separation

Speaker recognition

Best-of-N sampling

Flow Matching-Based Speech Source Separation with Best-of-N Biometric Sampling Anastasia Zorkina 1 Alexandr Anikin 1 Nikita Khmelev 1 2 Anastasiya Korenevskaya 1 Sergey Novoselov 1 2 Vladimir Volokhov 1 2 Maxim Korenevsky 2 Yuriy Matveev 1 arXiv :2607 .06088v 1 [ cs . SD] 7 Jul 2026 Abstract Single-channel speech separation remains challenging for real-world deployment due to source permutation ambiguity, sampling variability of generative models, and the difficulty of processing long recordings with chunk-wise inference. We address these issues with a conditional flowmatching-based method that produces an ordered two-source output conditioned on the mixture. A frozen speaker encoder defines the source order during training and is reused at inference for biometric best-of-N candidate selection and chunklevel channel alignment. We evaluate separation quality on Libri2Mix benchmark using SI-SDR, PESQ, and ESTOI, and measure downstream impact using cpWER for automatic speech recognition and EER for speaker verification. The results show that the proposed Transformer U-Net variant is competitive with strong baselines in objective separation metrics and achieves the lowest downstream automatic speech recognition and speaker verification error rates in all evaluated settings. 1. Introduction Speech source separation, also known as the “cocktail party problem”, is a fundamental task in audio processing and speech technologies (Li et al., 2025 ; Araki et al., 2025 ; Wang & Luo, 2025) . It aims to extract individual speech signals from a mixture, benefiting systems such as automatic speech recognition (ASR), speaker recognition (SR), intelligent voice assistants, and others. Despite rapid progress driven by deep learning, speech separation remains challenging due to acoustic complexity, the ill-posed nature of singlechannel inversion, varying overlap patterns, permutation ambiguity, and high speaker variability. 1ITMO University, Speech Processing Group, Russia 2 Speech Technology Center Ltd., R&D department, Russia. Correspondence to: Anastasia Zorkina <[zorkina@speechpro.com](zorkina@speechpro.com) >. Modern deterministic speech source separation systems based on Transformer and Conformer architectures achieve SI-SDRi above 24 dB on the WSJ0-2mix benchmark (Zhao et al., 2024 ; Shin et al., 2024) . Generative counterparts, including hybrid schemes (Lutati et al., 2024 ; Wang et al., 2024) and fully generative diffusion or flow matching models (Scheibler et al., 2023 ; Dong et al., 2025 ; Scheibler et al., 2025), offer new modeling paradigms that improve perceptual naturalness at the cost of sampling variance and higher computation. Deploying these systems in practice faces additional hurdles: non-causality, processing of long recordings, downstream integration, and computational efficiency. In this work, we propose a practical speech separation system based on conditional flow matching, built on generative speech enhancement models from the NVIDIA NeMo Toolkit (Juki et al., 2024 ; Ku et al., 2025) . We formulate two-speaker separation as a conditional generation of a structured waveform that contains both separated sources. Besides, we introduce a best-of-N biometric sampling procedure that selects the most speaker-disentangled generation among multiple stochastic candidates. The main contributions of this work are as follows: 1) A flow-matching-based speech separation method adapted from generative speech enhancement. 2) A best-of-N biometric criterion for inference-time candidate selection. 3) Long-form processing via chunk-wise generation combined with speaker-recognition-based channel tracking. 4) Evaluation of downstream ASR and SR performance after use of our speech source separation system. 2. Background Speech source separation. Formally, the task is to recover K signals s 1 , . . . , sK from a mixture m = Pk sk. Permutation-invariant training (PIT) (Yu et al., 2017 ; Kolbæk et al., 2017) resolves

FAQs

What problem does single-channel speech source separation face in real-world use?

It is limited by source permutation ambiguity, sampling variability from generative models, and the difficulty of handling long recordings via chunk-wise inference.

How does the proposed method determine an ordered two-source output?

It uses a conditional flow-matching model where a frozen speaker encoder defines the source order during training and is reused at inference for candidate selection and channel alignment.

How is the approach evaluated and what downstream effects are measured?

Separation quality is measured on Libri2Mix with SI-SDR, PESQ, and ESTOI, while downstream impact is assessed using cpWER for ASR and EER for speaker verification.

Jul 16, 2026

Upload by Oliver