NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task

Jul 16, 2026

Upload by Noah

Re-implements the NAVER LABS IWSLT 2025 instruction-following pipeline for the IWSLT 2026 Shared Task under constrained, short-audio conditions, replacing mandated components with SeamlessM4Tv2-large as the speech encoder and Qwen3-4B-Instruct as the LLM backbone. Retains the three-stage strategy of projector alignment, text-only LoRA pretraining, and multimodal merging. Builds 100k synthetic instruction-following examples across ten speech-centric task types for further Stage 3 fine-tuning, and reports COMET 0.781 for EN–ZH speech translation plus BERTScore-F1 0.346 for English SQA on MCIF.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Task and Data

Synthetic Instruction-Following Data

System

Architecture

Stage 1—Projector Alignment

Stage 2—Text-Only LoRA

Stage 3—Multimodal Merge

NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task Anand Kamble Florida State University [amk23j@fsu.edu](amk23j@fsu.edu) Aniket Tathe University of Illinois Urbana-Champaign [atathe@illinois.edu](atathe@illinois.edu) arXiv :2607 .05623v 1 [ cs .CL] 6 Jul 2026 Abstract We re-implement the NAVER LABS IWSLT 2025 instruction-following pipeline (Lee et al., 2025) for the IWSLT 2026 Shared Task (constrained condition, short audio track), adapting it to the mandated components: SeamlessM4Tv2-large (Barrault et al., 2023) as the speech encoder and Qwen3-4B-Instruct (Team, 2025) asthe LLM backbone. The three-stage approach—projector alignment, text-only LoRA pretraining, and multimodal merging—is preserved from the original design. We additionally construct 100k synthetic instructionfollowing examples across ten speech-centric task types (10k per task) from the provided corpora, suitable for further Stage 3 fine-tuning. Our primary model achieves COMET 0.781 on EN–ZH speech translation and BERTScore-F1 0.346 on English SQA on theMCIF benchmark. Code, training scripts, and generated data are publicly available.1 1 Introduction Multimodal speech LLMs such as SALMONN (Tang et al., 2024), Qwen-Audio (Chu et al., 2023), SpeechGPT (Zhang et al., 2023), and WavLLM (Hu et al., 2024) couple a frozen speech encoder with an instruction-tuned LLM (Ouyang et al., 2022) via a lightweight connector, enabling flexible multi-task inference through natural language prompts. The IWSLT 2026 Instruction-Following Shared Task (Organizers, 2026) formalizes this paradigm with the MCIF benchmark, evaluating unified models on ASR, multilingual ST (EN→{DE,IT,ZH}), and SQA. The NAVER LABS 2025 system (Lee et al., 2025) demonstrated a competitive three-stage pipeline in the IWSLT 2025 constrained setting (Abdulmuminet al., 2025), but was not publicly released. We provide the first open-source reimplementation, adapted to the IWSLT 2026 1 [https://github.com/anand-kamble/](https://github.com/anand-kamble/)[ ](https://github.com/anand-kamble/)iwslt2026-instruction-following | Dataset | Task | Lang. | Stage(s) | | --- | --- | --- | --- | | CoVoST 2 | ASR, ST | EN→{DE,IT,ZH} | 1, 2∗ , 3 | | EuroParlST | ASR, ST | EN→{DE,IT,ZH} | 1, 2∗ , 3 | | LibriSQA | SQA | EN (+ DE,IT,ZH‡) | 1†, 2∗ , 3 | | NUTSHELL | S2TSum | EN | 3 | | YTSeg | AChap | EN | 3 | Table 1: Training corpora per stage. ∗Text-only. †A.2 variant only. ‡Machine-translated. constraints (SeamlessM4T-v2-large encoder, Qwen3-4B-Instruct LLM—replacing the LLaMA- 3.1-8B (Dubey et al., 2024) backbone used in 2025) . We further construct 100k synthetic instruction-following examples across ten speechcentric task types (Section 2.1) and ablate LoRA rank and learning rate configurations for Stage 2 text pre-training. 2 Task and Data Shared Task. We participate in the constrained condition, short audio track (Organizers, 2026) . Evaluation on MCIF uses WER (↓) for ASR, COMET (Rei et al., 2020) (↑) for ST, and BERTScore-F1 (Zhang et al., 2020) (↑) for SQA. Task instructions follow the natural-language prompt format of Lee et al. (2025) . Training Corpora. Core speech data is from CoVoST 2 (Wang et al., 2021) and EuroParlST (Iranzo-Sánchez et al., 2020) (ASR/ST) and LibriSQA (Huang et al., 2024) (SQA); multilingual SQA pairs in DE, IT, ZH are obtained by machine-translating LibriSQA via SeamlessM4Tv2 . Stage 3 additionally draws on NUTSHELL (Maikezu et al., 2024) for speech summarization and YTSeg (Retkowski et al., 2024) for audio chapter detection. Table 1 summarizes corpora per stage. Figure 1: Three-stage training pipeline. Frozen modules: dashed border. Trainable: solid. Stage 3 jointly fine-tunes both projector and LoRA adapters. 2.1 Synthetic Instruction-Following Data We construct 100k synthetic examples (10k per task) from the provided corpora using open-weight Gemma models (Gemma Team, 2025) . Seven text-grounded tasks are generated by

FAQs

How does the model train across the three stages?

Stage 1 trains only the projector with the encoder and LLM frozen; Stage 2 performs text-only LoRA adaptation with the projector frozen; Stage 3 fine-tunes the projector together with the LoRA adapters under multimodal merging.

Jul 16, 2026

Upload by Noah