H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR

Jul 16, 2026

Multi-talker automatic speech recognition (MTASR) struggles with accurate transcription of overlapping speech, especially in complex high-overlap conditions. Existing mixture-of-experts (MoE) solutions often use frame-independent routing that produces temporal myopia, and they rely only on downstream ASR objectives, yielding implicit and weakly grounded representations. H-SAGE introduces a speaker-aware global encoder with an overlap-aware auxiliary loss and a holistic gating mechanism that jointly evaluates global context and local cues, improving expert collaboration on LibriSpeechMix.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

MTASR problem and challenges

SIMO vs. SISO paradigms

Prior explicit speaker modeling and MoE routing

Proposed H-SAGE framework and contributions

H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR Yujie Guo 1 , Jiaming Zhou 1 , Yuhang Jia 1 , Yang Chen 1 , and Yong Qin 1 ,∗ 1TMCC, College of Computer Science, Nankai University, Tianjin, China {guoyujie02,zhoujiaming,2120240729,[2120230617](2120230617}@mail.nankai.edu.cn)[}](2120230617}@mail.nankai.edu.cn)[@mail.nankai.edu.cn](2120230617}@mail.nankai.edu.cn), [qinyong@nankai.edu.cn](qinyong@nankai.edu.cn) arXiv :2607 .0 1566v 1 [ cs . SD] 2 Jul 2026 Abstract—Multi-talker Automatic Speech Recognition (MTASR) faces significant challenges in accurately transcribing overlapping speech, particularly under complex high-overlap conditions. While recent Mixture-of-Experts (MoE) approaches have shown promise, they typically rely on frame-independent routing that leads to temporal myopia, and depend solely on the downstream ASR objective, which results in implicit andungrounded representation learning. To address these limitations, we propose Holistic Speaker-Aware Guided Experts (H-SAGE) for MoE-based MTASR. Specifically, we introduce a SpeakerAware Global Encoder to capture long-term dependencies, supervised by an auxiliary Overlap-Aware Loss that explicitly guides the model to discern acoustic states. Furthermore, we design a Holistic Gating Mechanism to arbitrate expertselection by jointly evaluating global context and local details. Experiments on LibriSpeechMix demonstrate that H-SAGE achieves consistent improvements over strong baselines, particularly in complex scenarios, validating that explicit acoustic guidance effectively enhances expert collaboration. Our code can be found at [https://github.com/NKU-HLT/H-SAGE](https://github.com/NKU-HLT/H-SAGE). Index Terms—multi-talker automatic speech recognition, cocktail party problem, Mixture-of-Experts, speaker-aware I. INTRODUCTION In recent years, Automatic Speech Recognition (ASR) has witnessed remarkable progress due to the advent of deep learning. State-of-the-art architectures have achieved humanlevel performance in clean, single-speaker scenarios. However, these systems often degrade significantly in realistic conversational environments, which are characterized by spontaneous turn-taking and frequent speech overlaps. This limitation has catalyzed the emergence of Multi-Talker ASR (MTASR) asa critical research frontier. Unlike traditional ASR, which assumes a monotonic mapping between a single acoustic stream and text, MTASR aims to simultaneously disentangle and transcribe overlapping utterances from multiple speakers, addressing the classic “Cocktail Party Problem”. Current end-to-end MTASR approaches fall into two paradigms: Single-Input Multiple-Output (SIMO) and SingleInput Single-Output (SISO) [1] . SIMO-based methods [2]–[4] utilize Permutation Invariant Training (PIT) [5] to assign speech to specific branches. While effective, they are constrained by a fixed number of output branches, limiting their flexibility when the speaker count is unknown or variable. In contrast, SISO approaches employ Serialized Output Training (SOT) [6] to generate a serialized transcription con- ∗ Corresponding author. This work was supported by the National Natural Science Foundation of China (62271270) . taining all speakers. This paradigm inherently handles variable speaker counts, offering superior flexibility. However, it relies heavily on the attention mechanism to implicitly disentangle overlapping speech. Without explicit guidance, the model often struggles to distinguish speakers in high-overlap segments, resulting in performance degradation. To mitigate this issue, recent research has pivoted towards augmenting SISO frameworks with explicit speaker modeling, utilizing auxiliary modules or objectives to capture discriminative speaker characteristics. For instance, CSE-SOT [7] employs multi-branch structures to capture distinct speaker representation. Similarly, SACTC [8] and SD-CTC [9] leverage improved connectionist temporal

FAQs

What core problem does H-SAGE target in multi-talker ASR?

H-SAGE targets the difficulty of transcribing overlapping speech accurately under complex high-overlap conditions.

Why do many existing MoE-based MTASR methods underperform?

They often use frame-independent routing that causes temporal myopia, and they depend mainly on the downstream ASR objective, leading to implicit, insufficiently grounded representation learning.

How does H-SAGE improve expert routing and representation learning?

It adds a Speaker-Aware Global Encoder supervised by an auxiliary Overlap-Aware Loss, and uses a Holistic Gating Mechanism that assigns experts by combining global acoustic context with local frame-level cues.

Jul 16, 2026

Upload by Olivia Brown