INDICQE-APE - A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

Sep 15, 2026

Indic quality estimation (QE) and automatic post-editing (APE) datasets are fragmented across separate releases, limiting unified training and evaluation across tasks and language pairs. INDICQE-APE consolidates the WMT 2020–2024 shared-task lineage with an extended English–Malayalam resource into 126,754 instances across nine directional pairs. The benchmark aligns up to four label types per segment, including direct assessment, post-edit, word-level tags, and error explanations, with test stratification on four difficulty axes. Baselines evaluate six prompted LLMs and three COMET metrics for segment-level QE, and three systems for APE.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Abstract

Introduction

The INDICQE-APE Benchmark

Dataset consolidation and label alignment

Difficulty axes and measurement design

INDICQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages Diptesh Kanojia 1 Archchana Sindhujan 1 Sourabh Deoghare2 Daria Sokova 1 Shenbin Qian3 Girish Koushik 1 Tharindu Ranasinghe4 Constantin Orăsan 1 Chrysoula Zerva5 Ricardo Rei6 ,7 Frederic Blain8 André F. T. Martins5 Marco Turchi9 Matteo Negri 10 Anoop Kunchukuttan 11 , 12 Mitesh M. Khapra 11 , 12 Pushpak Bhattacharyya2 1University of Surrey 2IIT Bombay 3University of Oslo 4Lancaster University 5Instituto de Telecomunicações & Instituto Superior Técnico, University of Lisbon 6 Sword Health 7INESC-ID 8Tilburg University 9Zoom Communications 10Fondazione Bruno Kessler 11Bodhan AI 12AI4Bharat, IIT Madras [d.kanojia@surrey.ac.uk](d.kanojia@surrey.ac.uk) arXiv : 2608 . 16344v 3 [cs .CL] 12 Sep 2026 Abstract Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020–2024 shared-task lineage with an extended English-Malayalam resource into INDICQE-APE: 126 ,754 instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level tags and an error explanation, and a test set stratified over four diﬀiculty axes. We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on direct assessment and select a compressed slice of it. Segments whose segment-level and tokenlevel signals disagree are ranked below equally scored segments of the same language. Fourshot prompting costs every model at or below 3.4B both correlation and output-format compliance. Unedited MT beats every APE system we run on three of the four pairs. The benchmark1 and code2 are released. 1 Introduction Reference-free evaluation of machine translation, scoring a translation without a human reference (QE) and repairing it (APE), remains an open problem for Indic and other low-resource languages for two compounding reasons. The data is fragmented: direct assessments, post-edits, word-level tags and explanations live in separate shared-task releases with incompatible schemas, splits and provenance, 1[https://huggingface.co/datasets/surrey-nlp/](https://huggingface.co/datasets/surrey-nlp/) IndicQE-APE 2[https://github.com/surrey-nlp/IndicQE-APE](https://github.com/surrey-nlp/IndicQE-APE) so training and evaluating across tasks and pairs means rebuilding the corpus first. In addition, evaluation on the data that does exist falls back on a single learned scalar whose behaviour across languages is rarely checked. We address the first with a resource and the second with a measurement on it. INDICQEAPE unifies the WMT 2020–2024 QE and APE shared-task data for Indic pairs with an extended English→ Malayalam resource into one release of 126 ,754 instances over nine directional pairs, sliceable by task and by language pair (Table 3) . The label types are aligned on the same items: one instance may carry a direct assessment, a post-edit, word-level tags and an error explanation at once, so QE, APE and explainable QE can be studied without re-aligning corpora. Every test segment is flagged on each of four diﬀiculty axes its labels can express. Appendix A places the release against the shared-task and metric literature it consolidates. We report baselines for segment-level QE over all nine pairs and for APE over four. CometKiwi-XL (Rei et al., 2023) has the highest within-language segment correlation of the models we evaluate and the weakest cross-lingual agreement of the three trained metrics: its per-language offset moves against quality, where the offsets of the other two move with it, and pooling pairs costs it more than either. Within-language skill and cross-lingual comparability are separate properties, and a

FAQs

What problem does INDICQE-APE address in Indic machine translation evaluation?

It addresses the fragmentation of QE and APE data across incompatible releases, which otherwise forces rebuilding corpora to train and evaluate across tasks and language pairs. It also supports a more reliable measurement than a single learned scalar across languages.

What information is aligned in each INDICQE-APE instance?

Each instance can include a direct assessment, a post-edit, word-level tags, and an error explanation, aligned on the same segment so QE, APE, and explainable QE can be studied without re-aligning corpora.

How is the benchmark evaluated and what models are used?

The study benchmarks six prompted LLMs and three COMET metrics for segment-level QE, and evaluates three systems on APE. It reports correlation and output-format compliance and analyzes how unedited machine translation compares with APE systems across pairs.

Sep 15, 2026

Uploaded by Ava Thompson