MultiSynt/MT Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

Jul 16, 2026

Upload by Anda

Open web-scale pre-training corpora remain heavily concentrated in English, slowing the development of multilingual LLMs. MultiSynt/MT introduces an open synthetic parallel corpus with about 4.8T target-language tokens across 36 languages, generated by translating 100B high-quality Nemotron-CC tokens using TOWER+ and OPUS-MT/HPLTMT systems. For many European low- and medium-resource languages, it is among the largest openly available resources. On a multilingual benchmark suite, models trained on MultiSynt/MT match the final score of a native-data baseline using ~72% fewer tokens and exceed it by ~15% under a matched 100B budget. Analyses reveal evaluation blind spots where standard multiple-choice benchmarks miss translation-quality differences, and Norwegian idiomatic, culturally grounded tasks still benefit from native data. The released corpus includes row-aligned translations from multiple systems under a permissive open license, enabling controlled multilingual pretraining research and evaluation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Abstract

Introduction

Resource gap in multilingual pretraining

MultiSynt/MT corpus construction

Empirical results and evaluation observations

MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages arXiv :2607 .00890v 1 [ cs .CL] 1 Jul 2026 Maximilian Idahl1,2 , Jörg Tiedemann3 , Sampo Pyysalo4 , David Salinas5,6 , Tomasz Galica4 , Shenbin Qian7 , Tudor Nicolae Mateiu8 , Zihao Li3 , Anna Lokrantz9 , Fedor Vitiugin4 , André Martins10,11,12 , Jenna Kanerva4 , Filip Ginter4 , Matthias Lindemann10 , Tim Isbister9 , Birger Moëll9 , Jonas Lindh9 , Jan Haji13 , Jenia Jitsev14,15,16,17 , Andrey Kutuzov7 , Stephan Oepen7 , Gema Ramírez-Sánchez8 1 ellamind 2Leibniz University Hannover 3University of Helsinki 4University of Turku 5ELLIS Institute Tübingen 6Prior Labs 7University of Oslo 8Prompsit Language Engineering 9AI Sweden 10Instituto de Telecomunicações 11Instituto Superior Técnico 12TransPerfect 13 Charles University 14 Ontocord 15LAION 16 Open-Ψ (Open-Sci) Collective 17Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ) Abstract Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.8 trillion target-language tokens across 36 languages, produced by translating 100 billion high-quality Nemotron-CC tokens with TOWER+ and OPUS-MT/HPLTMT systems. For many medium-and lowerresource European languages, this is the largest openly available pre-training resource. On abroad multilingual benchmark suite, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2.0, a native-data baseline, using roughly 72% fewer pre-training tokens, and outperform it by approximately 15% relative at a matched 100B-token training budget. Our analyses also identify evaluation blind spots: standard multiple-choice benchmarks miss translation-quality differences that a fluency-sensitive LLM-as-judge evaluation cleanly recovers on the trained LLMs (with no fluency deficit in MultiSynt itself), and Norwegian idiomatic and culturally grounded tasks, for example, remain better served by native data. We release the corpus, including rowaligned translations from multiple systems, to support controlled research on multilingual pretraining data and evaluation. 1 Introduction Openly available pre-training data at scale exist primarily for English (Penedo et al., 2024 ; Su et al., 2025), with most other languages lacking volume or quality (see Section 2 for details) . This shortfall has motivated growing interest in machine translation as a means of producing multilingual pretraining data at scale, but the practice raises legitimate concerns: translated text can carry stylistic artifacts known as “translationese” and inherits the cultural reference frame of the source language, with names, places, idioms and culturally grounded knowledge in the English source remaining English-anchored after translation (Gellerstam, 1986 ; Riley et al., 2020) . We address this resource limitation by introducing MultiSynt/MT, an open multilingual synthetic parallel corpus of approximately 4.8 trillion target-language tokens covering 36 languages, produced by translating a 100B-token sample of high-quality web data from Nemotron-CC with open translation models, including TOWER+ 9Band 72B (Rei et al., 2026) and OPUS-MT (Tiedemann et al., 2024) . MultiSynt/MT is the largest openly available pre-training corpus to date for many European low- and medium-resource languages, exceeding the largest comparable native resource (HPLT 3.0; Oepen et al., 2025) by more than an order of magnitude in the lowest-resource ones (Figure 1) . For a subset of languages, we release row-aligned parallel translations from multiple systems, enabling controlled comparison under matched source data, and the corpus is released under a permissive open license. Alongside the corpus, we present a balanced empirical characterization reporting both where reference LLMs trained on MultiSynt/MT outperform native multilingual baselines and w

FAQs

What problem does MultiSynt/MT address in multilingual LLM development?

Openly available large-scale pre-training data is concentrated in English, leaving many other languages with insufficient volume or quality. MultiSynt/MT targets this gap by providing a large synthetic parallel corpus across 36 languages.

How is MultiSynt/MT created and what is its scale?

A 100B-token sample of high-quality Nemotron-CC web data is translated into target languages using open translation models including TOWER+ and OPUS-MT/HPLTMT. The corpus contains roughly 4.8 trillion target-language tokens across 36 languages.

What do results on multilingual benchmarks show about models trained on MultiSynt/MT?

Reference LLMs trained on MultiSynt/MT reach the final score of a native-data baseline (HPLT 2.0) using about 72% fewer tokens, and they outperform it by around 15% when training at a matched 100B-token budget.

Jul 16, 2026

Upload by Anda