From scores to insights - Predicting MT errors using reliable metrics and linguistic typology in slavic languages - MethodsX 15 (2025) 103613

Sep 29, 2026

Upload by Stanley

Machine Translation (MT) evaluation plays a crucial role in advancing systems translating into morphologically rich, low-resource languages such as Slovak. Existing automatic evaluation approaches often output a single quality score without revealing which specific error categories occur. A linguistically informed method is proposed to predict the probability of MT error categories by combining manual annotation with automatic evaluation metrics, using a modified MQM framework and an English-to-Slovak dataset. Reliability and uncertainty-aware modeling are applied to assess metric consistency and generate probability forecasts, improving explainability and reducing human effort while preserving linguistic relevance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Article Information

Keywords

Abstract

Related Work

Specifications Table

Background

Proposed Method

MethodsX 15 (2025) 103613 Contents lists available at ScienceDirect MethodsX journal [homepage:](homepage: www.elsevier.com/locate/methodsx)[ www.elsevier.com/locate/methodsx](homepage: www.elsevier.com/locate/methodsx) | From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages | | | --- | --- | | Dasa Munkovaa, Lucia Benkovaa, Michal Munka,b, Ľubomír Benko a,* , Petr Hajekb
a Constantine the Philosopher University in Nitra, Nitra, Slovakia b University of Pardubice, Pardubice, Czech Republic | | | G R A P H I C A L A B S T R A C T | | | | | A R T I C L E I N F O Keywords: Machine translation Machine translation evaluation Machine translation error prediction Error analysis Machine Translation error types A B S T R A C T Machine Translation (MT) evaluation plays a crucial role in advancing systems translating into morphologically rich, low-resource languages such as Slovak. Existing automatic evaluation methods typically offer a single quality score, lacking insight into specific error types. A novel linguistically informed methodology that predicts the probability of MT error categories by integrating manual annotation with automatic evaluation metrics is proposed. The method buildson a modified MQM framework adapted for Slovak and employs a dataset of English-to-Slovak Related research articleD. Munkova, L. Benkova, M. Munk, Ľ. Benko, P. Hajek, Predictive Modeling of Error Categories in English-Slovak Machine Translation Using Automatic Evaluation Metrics. (under review) * Corresponding author. E-mail address: [lbenko@ukf.sk](lbenko@ukf.sk) (Ľ. Benko). [https://doi.org/10.1016/j.mex.2025.103613](https://doi.org/10.1016/j.mex.2025.103613) Received 14 July 2025; Accepted 7 September 2025 Available online 8 September 2025 2215-0161/© 2025 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license ([http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)). D. Munkova et al. MethodsX 15 (2025) 103613 translations, combining outputs from statistical and neural MT systems with human reference translations. Manual annotations identified five linguistically motivated error categories. Reliability of 68 automatic metrics was assessed using Cronbach’s alpha, correlation coefficients, coefficient of determination (R²), and entropy. Bootstrapped logistic regression models were then developed to predict error occurrence probabilities. The proposed methodology improves the explainability and reliability of automatic MT evaluation by bridging the gap between holistic scoring and detailed error categorization. It significantly reduces the human effort required for quality assessment while maintaining a high degree of linguistic relevance, particularly for complex target languages like Slovak. • Predicts probabilities of specific MT error categories • Integrates linguistic expertise with statistical reliability analysis • Reduces human effort in MT evaluation while preserving linguistic precision | Specifications table | | | --- | --- | | Subject area | Computer Science | | More specific subject area | Machine Translation evaluation | | Name of your method | Prediction of machine translation errors | | Name and reference of original method | None | | Resource availability | Python 3
Python libraries for machine translation evaluation metrics (e.g., NLTK or PyTorch)
Segment alignment (e.g., HunAlign, LF Aligner or Python libraries)
Statistical software (e.g., R, STATISTICA Data Miner or IBM SPSS Modeler) | Background Machine translation (MT) evaluation plays a crucial role in improving MT systems, especially when dealing with morphologically rich, low-resource, and highly inflectional languages such as Slovak [1,2]. Existing automatic MT evaluation methods primarily focus on measuring the similarity or edit distance between MT outputs and reference t

FAQs

Why are existing automatic MT evaluation methods insufficient for Slovak?

They typically produce a single overall quality score, which does not expose the specific types of errors produced. For Slovak, rich inflection and flexible word order further complicate detailed automatic identification and classification.

How does the proposed methodology predict MT error categories?

It integrates manual annotation with automatic evaluation metrics within a modified MQM framework adapted for Slovak. It then assesses the reliability of many automatic metrics and uses bootstrapped logistic regression to predict the probability of error occurrence.

What linguistic information and metric reliability analysis are used in the approach?

Manual annotations define linguistically motivated error categories, while metric reliability is evaluated using measures including Cronbach’s alpha, correlation coefficients, coefficient of determination (R²), and entropy. Statistical modeling uses these metric signals to forecast error probabilities.

Sep 29, 2026

Upload by Stanley