MCAT - Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages

Sep 4, 2026

Multimodal Large Language Models (MLLMs) have advanced speech-to-text translation, yet research remains limited by two constraints: insufficient language coverage and low inference efficiency. Many existing datasets are English-centric, which weakens many-to-many multilingual translation scaling, and long speech-to-sequence conversion (e.g., 750 tokens) severely slows generation. MCAT proposes curriculum-based language scaling with data balancing to support mutual translation across 70 languages, plus an optimized speech adapter that compresses speech to 30 tokens.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

Index Terms

Introduction

Speech-to-Text Translation Pipeline

Challenges: Language Coverage and Efficiency

Key Innovations of MCAT

Language Scaling Strategy

Efficient Speech Adapter Design

MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Keqi Deng, Xie Chen, Yang Xiang, Ming Liu, Bing Qin, YaoWei Wang arXiv :2512 .01512v2 [ cs .CL] 13 Apr 2026 Abstract—Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs’ many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy are introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-theart end-to-end models on the FLEURS dataset across 70 × 69 directions but also enhances inference efficiency. 1 Index Terms—Speech-to-Text Translation, Multimodal Large Language Models, Curriculum Learning. I. INTRODUCTION Speech-to-Text Translation (S2TT) involves converting speech from a source language into text in a target language. Traditionally, S2TT tasks have relied on a cascaded system, where an Automatic Speech Recognition (ASR) model first transcribes the speech into text [1], followed by a Machine Translation (MT) model that translates the text into the target Manuscript received 2 December 2025; revised 24 February 2026; accepted 9 April 2026 . Date of publication xx xxxx 2026; date of current version xxxxxx 2026 . The research is supported by the National Key Research and Development Program of China (2025YFE0200500), the National Science Foundation of China (U22B2059, 62276083, 62506182), National Science and Technology Major Program (Grant No. 2024ZD01NL00101), the Key Research and Development Program of Heilongjiang Province (2022ZX01A28) and the 5G Application Innovation Joint Research Institute’s Project (A003), and the Major Key Project of PCL (Grant No. PCL2025A12, PCL2025A03) .(Corresponding authors: Ming Liu; Yang Xiang.) Yexing Du, Kaiyuan Liu and Yaowei Wang are with Harbin Institute of Technology, Shenzhen, China, and also with Pengcheng Laboratory, Shenzhen, China (e-mail: [yxdu@ir.hit.edu.cn](yxdu@ir.hit.edu.cn); [1171000408@stu.hit.edu.cn](1171000408@stu.hit.edu.cn); [wangyaowei@hit.edu.cn](wangyaowei@hit.edu.cn)). Ming Liu and Bing Qin are with Harbin Institute of Technology, Harbin, China and also with Pengcheng Laboratory, Shenzhen, China (e-mail: [mliu@ir.hit.edu.cn](mliu@ir.hit.edu.cn); [qinb@ir.hit.edu.cn](qinb@ir.hit.edu.cn)). Youcheng Pan, Bo Yang and Yang Xiang are with Pengcheng Laboratory, Shenzhen, China (e-mail: [panych@pcl.ac.cn](panych@pcl.ac.cn); [yangb05@pcl.ac.cn](yangb05@pcl.ac.cn); [xiangy@pcl.ac.cn](xiangy@pcl.ac.cn)). Keqi Deng is with University of Cambridge, CB2 1TN Cambridge, U.K (e-mail: [kd502@cam.ac.uk](kd502@cam.ac.uk)) . Xie Chen is with Shanghai Jiao Tong University, Shanghai, China (e-mail: [chenxie95@sjtu.edu.cn](chenxie95@sjtu.edu.cn)). 1 The code and models are released at [https://github.com/yxduir/m2m-70](https://github.com/yxduir/m2m-70) . Source Speech (Will it rain tomorrow?) Fig. 1. Comparison of S2TT MLLMs. (a) compresses speech to 750 tokens, has limited language support, and directly generates translated text; (b) generates transcriptions and tr

FAQs

What problems does MCAT target in many-to-many speech-to-text translation?

MCAT targets limited language coverage caused by English-centric datasets and degraded inference efficiency caused by converting speech into long token sequences.

How does MCAT improve multilingual coverage across languages?

MCAT uses curriculum learning and a data balancing strategy to scale language coverage and enable mutual translation among 70 languages.

How does MCAT increase inference efficiency?

MCAT designs an efficient speech adapter that extracts features and compresses the speech representation so the speech token input is reduced to about 30 tokens.

Sep 4, 2026

Uploaded by นรินทร์