Unified Architecture and Unsupervised Speech Disentanglement for Speaker Embedding-Free Enrollment in Personalized Speech Enhancement

Jul 16, 2026

Upload by Logic

Conventional speech enhancement (SE) improves speech perception by suppressing noise and reverberation without enrollment speech, while personalized speech enhancement (PSE) extracts a target speaker using enrollment speech as reference to address the cocktail party problem. Although SE and PSE can share similar architectures, PSE robustness degrades under variations in enrollment emotion, duration, and semantics. The work proposes USEF-PNet with a unified SE/PSE architecture and DSEF-PNet using unsupervised speech disentanglement and consistency constraints to isolate speaker-identity cues, improving real-world robustness. Experiments on Libri2Mix and VoiceBankDEMAND show consistent gains, including slight advantages with random enrollment duration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Abstract

Introduction

Unified Architecture and Unsupervised Speech Disentanglement for Speaker Embedding-Free Enrollment in Personalized Speech Enhancement Ziling Huang, Student Member, IEEE, Haixin Guan, Member, IEEE, Yanhua Long, Member, IEEE arXiv :2505 . 12288v2 [ ee ss .AS] 12 Jul 2026 Abstract—Conventional speech enhancement (SE) aims to improve speech perception and intelligibility by suppressing noise and reverberation without requiring enrollment speech, whereas personalized speech enhancement (PSE) addresses the cocktail party problem by extracting a target speaker’s speech using enrollment speech as a reference. While these two tasks tackle different yet complementary challenges in speech signal processing, they often share similar model architectures, with PSE incorporating an additional branch to process enrollment speech. This suggests the possibility of developing a unified model capable of efficiently handling both SE and PSE tasks, thereby simplifying deployment while maintaining high performance. However, PSE performance is highly sensitive to variations in enrollment speech, such as differences in emotional tone, duration, and semantic content, which limiting its robustness in real-world applications. To address these challenges, we propose two novel models, USEFPNet and DSEF-PNet, both extending our previous SEF-PNet framework. USEF-PNet introduces a unified architecture for processing enrollment speech, integrating SE and PSE into a single framework to enhance performance and streamline deployment. Meanwhile, DSEF-PNet incorporates an unsupervised speech disentanglement approach by pairing a mixture speech with two different enrollment utterances and enforcing consistency in the extracted target speech. This strategy effectively isolates highquality speaker identity information from enrollment speech, reducing interference from factors such as emotion and content, thereby improving PSE robustness. Additionally, we explore along-short enrollment pairing (LSEP) strategy to examine the impact of enrollment speech duration during both training and evaluation. Extensive experiments on the Libri2Mix and VoiceBankDEMAND datasets demonstrate that our proposed USEF-PNet, DSEF-PNet all achieve substantial performance improvements, with random enrollment duration performing slightly better. Our source code, model checkpoints, and datasets will be publicly available at [https://github.com/isHuangZiling/UDSEF-PNet](https://github.com/isHuangZiling/UDSEF-PNet). Index Terms—Personalized speech enhancement, Unified architecture, Unsupervised speech disentanglement, Long-short enrollment pairing, Speaker embedding-free network I. INTRODUCTION I N real-world applications multi-speaker conference such as smart home devices and systems, managing overlapping speech, environmental noise, and room reverberation poses significant challenges. A well-known problem in multi-speaker scenarios is the “cocktail party problem” [1], [2], wherein Yanhua Long is the Corresponding author. Ziling Huang, Yanhua Long are with Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University, Shanghai, 200234, China. Yanhua Long is also with the SHNU-Unisound Natural Human-Computer Interaction Lab, Shanghai Normal University. (e-mail: [hzlkycg111@163.com](hzlkycg111@163.com); [yanhua@shnu.edu.cn](yanhua@shnu.edu.cn)). Haixin Guan are with the Unisound AI Technology Co., Ltd., Beijing, China (e-mail: [guanhaixin@unisound.com](guanhaixin@unisound.com)). humans can effortlessly focus on a specific speaker and shift attention, yet replicating this capability in machines remains a significant challenge. Two effective approaches to address these challenges are conventional speech enhancement (SE) and personalized speech enhancement (PSE) . SE focuses on removing noise and reverberation to improve speech perception and intelligibility, with substantial progress driven by deep learning techniques [3]–[9], while PSE b

FAQs

What problem does the paper address in personalized speech enhancement (PSE)?

PSE relies on enrollment speech, but its performance is sensitive to changes in enrollment emotional tone, duration, and semantic content, which reduces robustness in real-world use.

How does USEF-PNet unify SE and PSE tasks?

USEF-PNet introduces a unified architecture that processes enrollment speech while integrating SE and PSE into a single framework, aiming to improve performance and simplify deployment.

What is the role of unsupervised speech disentanglement in DSEF-PNet?

DSEF-PNet pairs a mixture speech with two different enrollment utterances and enforces consistency in the extracted target speech, isolating high-quality speaker identity information and reducing interference from emotion and content.

Jul 16, 2026

Upload by Logic