Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation

Jul 16, 2026

Upload by Aurelia

State-of-the-art speech enhancement models rely on large labeled datasets, while singing voice separation typically faces limited training data. This work reframes singing voice separation as domain adaptation from speech enhancement to singing voice separation. Two adaptation strategies are evaluated: full fine-tuning and parameter-efficient fine-tuning with LoRA on both discriminative and generative models. Adaptation improves separation by 0.29–1.8 dB SDR; full fine-tuning achieves best SVS but harms enhancement via catastrophic forgetting. LoRA reaches competitive SVS while preserving speech enhancement capability with only 6–12% extra parameters and the generative model generalizes better to an unseen test set.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction

Method

Tasks related to separation of vocal signals

TEACHING SPEECH ENHANCEMENT MODELS TO SING: DOMAIN ADAPTATION FROM SPEECH ENHANCEMENT TO SINGING VOICE SEPARATION Paul A. Bereuter1, Mark D. Plumbley2, Alois Sontacchi 1 1Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria 2Department of Informatics, King’s College London, London, United Kingdom arXiv :2607 . 1 1630v 1 [ cs . SD] 13 Jul 2026 ABSTRACT State-of-the-art speech enhancement models benefit from large-scale labeled datasets, whereas singing voice separation models suffer from limited available training data. To address this limitation, we formulate singing voice separation as domain adaptation from speech enhancement to singing voice separation. We investigate two fine-tuning strategies: full fine-tuning and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA) on a discriminative and a generative model. Models with either adaptation strategy outperform the same architectures trained from scratch by 0.29-1.8 dBin Signal-to-Distortion-Ratio. Full fine-tuning yields the highest singing voice separation performance, but catastrophic forgetting degrades speech enhancement performance. LoRA fine-tuning achieves competitive singing voice separation performance while preserving the original speech enhancement capability with only 6-12% additional parameters compared to the base speech enhancement model. Furthermore, the generative model shows improved generalization to an unseen test set. The results demonstrate that adapting pretrained speech enhancement models is an effective strategy for training singing voice separation models in data-scarce scenarios. Index Terms— singing voice separation, speech enhancement, domain adaptation, low rank adaption 1. INTRODUCTION Classical speech enhancement (SE) has primarily addressed acoustic degradations such as environmental noise and reverberation, as tackled in organized challenges such as the Deep Noise Suppression (DNS) Challenge [1–3] . The recent transition to more universal enhancement scenarios, such as the Universality, Robustness, and Generalizability for Speech Enhancement (URGENT) Challenge [4], reflects a broader shift towards models capable of handling diverse interference conditions beyond traditional denoising tasks. Enabled by increasingly large-scale training datasets covering a wide range of degradations, these developments have led to the emergence of generative and hybrid discriminative–generative enhancement architectures [5–8] . Broadening the term speech enhancement, these models The computing infrastructure used in this work was funded by the digital research infrastructure project “Interactive Audiovisual Digital Twins of Performance Venues”, supported by the Austrian Federal Ministry of Education, Science and Research (BMFWF) . This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) [grant numbers EP/T019751/1, EP/Y028805/1, UKRI397] . For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) license to any Author Accepted Manuscript version arising. not only enable separation of speech from diverse interference conditions but also restoration of degraded speech structures. While subsets of large-scale SE datasets include singing voice as part of the target vocal signals, the associated interference conditions typically remain dominated by acoustic distortions such as environmental noise or reverberation, rather than structured musical accompaniment. In contrast, singing voice separation (SVS) models are commonly trained independently using music datasets such as MUSDB- 18-HQ [9] and MoisesDB [10], which provide isolated vocal and accompaniment stems for supervised training under realistic musical interference conditions. However, if combined, these datasets remain limited to approximately 35 hours of labeled data, whereas the recent URGENT challenge corpus [4] amounts to roughly 700 hours of audio. Di

FAQs

What problem does the paper address in training singing voice separation models?

It addresses the gap that singing voice separation usually has limited labeled training data compared with speech enhancement, which benefits from large-scale labeled datasets.

How do the authors perform domain adaptation from speech enhancement to singing voice separation?

They reformulate SVS as domain adaptation and evaluate two strategies: full fine-tuning of pretrained SE models and parameter-efficient fine-tuning using LoRA.

What is the main trade-off between full fine-tuning and LoRA fine-tuning?

Full fine-tuning yields the highest SVS performance but can degrade speech enhancement due to catastrophic forgetting, while LoRA preserves the original speech enhancement capability with only 6–12% additional parameters while still improving SVS.

Jul 16, 2026

Upload by Aurelia