Dirt Cheap Web-Scale Parallel Text from the Common Crawl - Research Paper

Oct 1, 2026

Parallel text is essential for modern machine translation, but broad web crawling is infeasible for most researchers and companies. This work mines web-scale parallel data from the public Common Crawl, using an open-source extension of the STRAND algorithm. Starting only from two-letter language codes, the authors process 32TB in under a day at low cost, extracting parallel corpora across many languages and domains. The mined data boosts translation quality across multiple pairs, with improvements up to 5 BLEU, and is released for further research.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

Introduction

Mining the Common Crawl

Candidate pair selection

Dirt Cheap Web-Scale Parallel Text from the Common Crawl Jason R. Smith 1 ,2 [jsmith@cs.jhu.edu](jsmith@cs.jhu.edu) Philipp Koehn3 [pkoehn@inf.ed.ac.uk](pkoehn@inf.ed.ac.uk) Herve Saint-Amand3 [herve@saintamh.org](herve@saintamh.org) Chris Callison-Burch 1 ,2 ,5 [ccb@cs.jhu.edu](ccb@cs.jhu.edu) ⇤ Magdalena Plamada4 [plamada@cl.uzh.ch](plamada@cl.uzh.ch)[ ](plamada@cl.uzh.ch)Adam Lopez 1 ,2 [alopez@cs.jhu.edu](alopez@cs.jhu.edu) 1Department of Computer Science, Johns Hopkins University 2Human Language Technology Center of Excellence, Johns Hopkins University 3 School of Informatics, University of Edinburgh 4Institute of Computational Linguistics, University of Zurich 5 Computer and Information Science Department, University of Pennsylvania Abstract Parallel text is the fuel that drives modern machine translation systems. The Web is a comprehensive source of preexisting parallel text, but crawling the entire web is impossible for all but the largest companies. We bring web-scale parallel text to the masses by mining the Common Crawl, a public Web crawl hosted on Amazon’s Elastic Cloud. Starting from nothing more than a set of common two-letter language codes, our open-source extension of the STRAND algorithm mined 32 terabytes of the crawl in just under a day, at a cost of about $500 . Our large-scale experiment uncovers large amounts of parallel text in dozens of language pairs across a variety of domains and genres, some previously unavailable in curated datasets. Even with minimal cleaning and ﬁltering, the resulting data boosts translation performance across the board for ﬁve different language pairs in the news domain, and on open domain test sets we see improvements of up to 5 BLEU. We make our code and data available for other researchers seeking to mine this rich new data resource.1 1 Introduction A key bottleneck in porting statistical machine translation (SMT) technology to new languages and domains is the lack of readily available parallel corpora beyond curated datasets. For a handful of language pairs, large amounts of parallel data ⇤ This research was conducted while Chris CallisonBurch was at Johns Hopkins University. [1](1github.com/jrs026/CommonCrawlMiner)[github.com/jrs026/CommonCrawlMiner](1github.com/jrs026/CommonCrawlMiner) are readily available, ordering in the hundreds of millions of words for Chinese-English and ArabicEnglish, and in tens of millions of words for many European languages (Koehn, 2005) . In each case, much of this data consists of government and news text. However, for most language pairs and domains there is little to no curated parallel data available. Hence discovery of parallel data is an important ﬁrst step for translation between most of the world’s languages. The Web is an important source of parallel text. Many websites are available in multiple languages, and unlike other potential sources—such as multilingual news feeds (Munteanu and Marcu, 2005) or Wikipedia (Smith et al., 2010)—it is common to ﬁnd document pairs that are direct translations of one another. This natural parallelism simpliﬁes the mining task, since few resources or existing corpora are needed at the outset to bootstrap the extraction process. Parallel text mining from the Web was originally explored by individuals or small groups of academic researchers using search engines (Nie et al., 1999; Chen and Nie, 2000; Resnik, 1999; Resnik and Smith, 2003) . However, anything more sophisticated generally requires direct access to web-crawled documents themselves along with the computing power to process them. For most researchers, this is prohibitively expensive. As a consequence, web-mined parallel text has become the exclusive purview of large companies with the computational resources to crawl, store, and process the entire Web. To put web-mined parallel text back in the hands of individual researchers, we mine parallel text from the Common Crawl, a regularly updated 81-terabyte

FAQs

How do the authors find candidate parallel document pairs?

They use an extension of the STRAND algorithm that identifies potentially parallel documents using cues from URLs and document content.

Oct 1, 2026

Upload by xXChrisXx