Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech

Jul 16, 2026

A system for Task 1 of the MLC-SLM 2026 Challenge addresses multilingual two-speaker conversational speech by tightly coupling speaker diarization with an adapted Qwen3-ASR-1.7B recognizer. The diarization front end performs VAD, subsegment generation, CAMPPlus speaker embeddings, two-speaker spectral clustering, and RTTM-based segmentation, then groups segments by language or region for decoding. ASR adaptation uses supervised full fine-tuning, LoRA with three-pipeline TTS synthetic augmentation, and GRPO reinforcement learning with WER/CER rewards and penalties. tcpMER reaches 23.70 on development and 17.97 on evaluation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

System framework and training strategy

Stage-wise inference pipeline

Speaker diarization

Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech Hao Wu 1,∗, RongQi Han 1,∗, Zhen Wang 1, Wei Liang2, Wei Xu 1 1 Shanghai Qi Zhi Institute, Shanghai, China 2Megatronix (Beijing) Technology Co., Ltd [wuhao@sqz.ac.cn](wuhao@sqz.ac.cn) arXiv :2607 .08208v 1 [ cs .CL] 9 Jul 2026 Abstract This paper describes our self-designed system for Task 1 of the MLC-SLM 2026 Challenge for multilingual two-speaker conversational speech. The system combines a modular speaker diarization front end with a challenge-adapted Qwen3-ASR-1.7B recognizer. The diarization front end performs voice activity detection, subsegment generation, CAMPPlus speaker embedding extraction, two-speaker spectral clustering, and RTTMbased audio segmentation. The resulting speaker-attributed segments are grouped by language or region and decoded by the adapted ASR model. For ASR adaptation, we ﬁrst perform supervised full ﬁne-tuning on the ofﬁcial training data, then apply LoRA ﬁne-tuning with synthetic speech generated by a threepipeline TTS-based synthetic speech augmentation framework, and ﬁnally reﬁne the model using GRPO reinforcement learning with rewards based on WER/CER and penalties for hallucination, repetition, and length deviation. On the ofﬁcial development set, the full system achieves an average tcpMER of 23.70, reducing the error rate by 6.83 absolute points relative to the released Qwen-ASR-1.7B performance. On the ﬁnal evaluation set, the system achieves an average tcpMER of 17.97 . Ablation results show that supervised ﬁne-tuning provides the largest gain, while synthetic-speech LoRA adaptation and reinforcement learning further improve robustness. Key Words: multilingual speech recognition, speaker diarization, speech language model, LoRA, TTS synthetic speech augmentation 1. Introduction Transcribing two-speaker conversational recordings into speaker-attributed text is a demanding multilingual task: a single system must simultaneously decide who spoke when, segment the audio accordingly, and recognize what was said in each language. The difﬁculty is compounded by short turns, overlapping speech, frequent speaker switches, and the heterogeneous orthographies of the 21 language/region conditions covered by the MLC-SLM 2026 Challenge Task 1 . Because diarization errors are inherited by the recognizer and recognition errors are inherited by the ﬁnal score, treating speaker diarization (SD) and automatic speech recognition (ASR) as independent modules leaves substantial accuracy on the table; a carefully coupled SD+ASR pipeline is therefore central to a competitive system. The challenge evaluates submissions with the timeconstrained permutation mixed error rate (tcpMER), in which a language-adaptive mixed error rate (MER) is ﬁrst computed per language—word error rate for space-separated scripts and character error rate for Japanese, Korean, and Thai—and then ∗Hao Wu and RongQi Han contributed equally to this work. minimized over speaker permutations under time constraints, with diarization quality reported separately through the diarization error rate (DER) . The 2026 Task 1 setting is deﬁned by the ofﬁcial challenge page as multilingual conversational speech diarization and recognition [1] . Our SD ASR system, denoted SQZ-Qwen-ASR-1.7B, targets Task 1 only. On the diarization side, we build a modular front-end within the 3D-Speaker framework [2] that performs voice activity detection (VAD), CAMPPlus speaker embedding extraction, and spectral clustering with the speaker count ﬁxed to two, followed by RTTM-driven audio cutting. On the recognition side, we start from the Qwen3-ASR-1.7B speech LLM [3] and adapt it to the MLC-SLM task setting through a threestage recipe: full supervised ﬁne-tuning (SFT) on the ofﬁcial training set, low-rank adaptation (LoRA) [4] on synthetic speech, and Group Relative Policy Optimization (GRPO) reinforcement learning that penalizes hallucinated and repeti

FAQs

How does the system combine diarization and ASR for better accuracy?

It treats speaker diarization and ASR as a coupled pipeline: diarization outputs speaker-attributed segments that guide how audio is segmented and transcribed by the adapted Qwen-ASR model.

What are the main steps in the diarization front end?

It performs VAD, generates short subsegments, extracts CAMPPlus speaker embeddings, runs two-speaker spectral clustering with the speaker count fixed to two, and produces RTTM-based audio segmentation.

How is the ASR model adapted during training?

Training follows three stages: supervised full fine-tuning on the official data, LoRA fine-tuning using synthetic speech from a three-pipeline TTS augmentation framework, and GRPO reinforcement learning using rewards tied to WER/CER and penalties for hallucination, repetition, and length deviation.

Jul 16, 2026

Upload by Theodora