Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set - A Common-Reference Staged Normalization Analysis

Sep 15, 2026

Evaluating speech recognition for a Kurdish variety written in Latin orthography using a system that outputs Arabic script highlights a measurement difficulty distinct from modeling. Direct scoring conflates writing-system differences, while common-reference normalization changes reference tokenization alongside hypotheses, altering how agreement is measured. Results analyze MMS-1B-all with a Central Kurdish adapter on 1,722 Garrusi segments, using a fixed reference folding design and staged normalization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Introduction

Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis Hiwa Asadpour Saarland University and Goethe University Frankfurt [asadpour@lingua.uni-frankfurt.de](asadpour@lingua.uni-frankfurt.de) Abstract Evaluating speech recognition for a Kurdish variety written in a Latin field orthography, using a model that outputs Arabic script, raises a measurement problem before it raises a modelling one. A direct comparison scores writing-system difference as recognition error. The usual remedy, normalizing reference and hypothesis together before scoring, changes the reference tokenization at the same time as the hypothesis, so the resulting improvement mixes an increase in agreement with a change in the denominator against which agreement is measured. I report an evaluation of MMS-1B-all with the Central Kurdish (ckb) adapter, used as released and without any adaptation on Garrusi data, on 1,722 segments of Phase 1 elicited questionnaire speech from five Garrusi speakers (9,763 reference word tokens, 117.9 minutes) . Scoring uses a common-reference design: the reference is folded once and held fixed at 9,763 tokens across all conditions, and only the hypothesis representation varies. Scoring the unmodified Arabic-script hypothesis against the fixed Latin reference gives 111.70% WER and 100.92% CER with zero word-level matches. Transliterating the hypothesis to Latin gives 102.36% and 57.89%. Additionally folding it into the reference’s reduced orthography gives 97.85% WER and 51.20% CER. Under this scoring design, the RAW-to-FOLDED transformation reduces the measured WER by 13.85 percentage points and the measured CER by 49.72 percentage points, of which the folding step contributes 4.51 and 6.69 respectively. Substantial measured error remains after both transformations: 14.53% of reference tokens are aligned as exact matches, the edits are substitution-dominated, and per-segment WER is higher for segments with fewer reference tokens. A Southern Kurdish fine-tuned system (aranemini/southern-kurdish-asr) scored under the same design performs worse on every speaker, with WER of 109.56% and CER of 55.85% over 1,703 segments, though 12,330 characters of its output fall outside the folding table and the run must be repeated against the corrected reference before its rates are final. Part of that residual is a scoring-pipeline artefact of unmeasured size rather than recognition error: the transliterator leaves 613 characters of the MMS hypothesis unconverted or unmapped, and 12,330 characters of the Southern Kurdish system’s output, before folding (§6) . I will release the fixed reference and the segment-level results, subject to the data-sharing terms of the source corpus, so that the scoring design can be checked independently. The Southern Kurdish system was scored against a pre-correction reference; its rates will be recomputed against the fixed reference before final publication. 1 Introduction Kurdish speech technology has extended beyond its original concentration on Central Kurdish, with dedicated evaluation sets now available for Northern Kurdish, Badini, and Southern Kurdish (Mohammadamini & Tahon, 2026; Mohammadamini et al., 2026), and with recent work addressing variation within Central Kurdish itself (Ahmadi et al., 2024) . For Southern Kurdish the most substantial of these is the resource of Mohammadamini and Tahon (2026), which provides 30 hours of validated read speech from 208 speakers together with a multi-domain benchmark of 100 sentences, translated from an existing Central Kurdish test set and read by eight speakers drawn from five vernaculars, and which releases the corpus, a fine-tuned MP32Vec2-BERT CTC model and inference code publicly. That study also reports a result of direct relevance to the configuration evaluated here: a Whisper model fine-tuned for Central Kurdish reaches 84.23 WER on their Southern Kurdish benchmark against 13.40 WER on the

FAQs

Why does scoring Kurdish ASR with different writing systems create a measurement problem?

A direct comparison treats orthography differences as recognition errors. This can inflate error rates even when the underlying recognition is comparable.

What is the common-reference staged normalization design used for scoring?

The reference is folded once and held fixed, while only the hypothesis representation changes across conditions. This isolates how transformations affect measured agreement.

How do staged folding and transliteration affect WER and CER?

Scoring the unmodified Arabic-script hypothesis against the fixed Latin reference yields very high WER and CER. Transliteration and additional folding reduce the measured errors, with folding contributing a substantial portion of the CER reduction.

Sep 15, 2026

Uploaded by Emma Wilson