Fit for our purpose, not yours - Benchmark for a low-resource, Indigenous language

Oct 17, 2026

Uploaded by fresh2021

Influential AI benchmarks often fail low-resource, Indigenous language communities by focusing on general-purpose performance rather than language-specific, cultural, and ethical needs. Many datasets contain grammatical and orthographic errors, poor pronunciation, limited vocabulary, and lack cultural relevance, making them unsuitable for developing NLP tools in these contexts. The paper presents a Mori-language dataset designed as a “fit-for-our-purpose” benchmark, enabling tailored evaluation, reflecting community speaker diversity, and supporting language revitalisation aspirations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

1 Introduction

2 Part one: Fit for their purpose

Fit for our purpose, not yours: Benchmark for a low-resource, Indigenous language Suzanne Duncan Te Reo Irirangi o Te Hiku o Te Ika Kaitia, NZ 0410 [suzanne@tehiku.co.nz](suzanne@tehiku.co.nz) Lee Steven Te Reo Irirangi o Te Hiku o te Ika Kaitia, NZ 0410 [lee@tehiku.co.nz](lee@tehiku.co.nz) Gianna Leoni Te Reo Irirangi o Te Hiku o te Ika Kaitia, NZ 0410 [gianna@tehiku.co.nz](gianna@tehiku.co.nz) Keoni Mahelona Te Reo Irirangi o Te Hiku o te Ika Kaitia, NZ 0410 [keoni@tehiku.co.nz](keoni@tehiku.co.nz) Peter-Lucas Jones Te Reo Irirangi o Te Hiku o te Ika Kaitia, NZ 0410 [peterlucas@tehiku.co.nz](peterlucas@tehiku.co.nz) Abstract Influential and popular benchmarks in AI are largely irrelevant to developing NLP tools for low-resource, Indigenous languages. With the primary goal of measuring the performance of general-purpose AI systems, these benchmarks fail to give due consideration and care to individual language communities, especially low-resource languages. The datasets contain numerous grammatical and orthographic errors, poor pronunciation, limited vocabulary, and the content lacks cultural relevance to the language community. To overcome the issues with these benchmarks, we have created a dataset for the Mori language (the Indigenous language of Aotearoa/New Zealand) to pursue NLP tools that are ‘fit-for-our-purpose’. This paper demonstrates how low-resourced, Indigenous languages can develop tailored, high-quality benchmarks that; i. Reflect the unique characteristics of their language, ii. Reflect the diversity of speakers in the language community, iii. Support the aspirations for the tools they are developing and their language revitalisation efforts. All of which sit within a broader understanding of the impact of colonisation on their language. 1 Introduction Benchmarking is used in natural language processing (NLP) and machine learning (ML) as an objective lens to measure or track the progress and accuracy of a model. However, standardised open-source datasets do not always provide appropriate forms of benchmarking as they rely on large datasets and favour a small group of high-resource languages, like English (Mdhaffar et al., 2024) . This means that they fall short on low-resource, under-resourced, minority, colonised and Indigenous languages. It has been generally agreed that the development of NLP tools requires large, annotated datasets. Yet, scarcity of data is a common issue in the endeavour to create quality tools because NLP research concentrates on 20 of the more widely spoken languages of the more than 7000 languages spoken throughout the world (Magueresse et al., 2020; Mdhaffar et al., 2024; Joshi et al., 2019; Joshi et al., 2020; Bird, 2022; Markl et al., 2024; Nicholas & Bhatia, 2023) . This has culminated in a 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks. limited amount of research conducted on low-resource languages, referred to as such because of the lack of data available (Doumboya et al., 2021) . For low-resource languages, and in particular Indigenous languages, the valorised benchmarks are often laden with linguistic errors, perpetuate poor pronunciation, and have limited vocabulary because the content lacks cultural and linguistic intelligence. This paper provides an overview of benchmarking for both high-and low-resource languages. It highlights how‘cutting-edge’, popular benchmarks are fit for some peoples’ purposes, but not fit for ours. In particular, we discuss FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark, and the issues we encountered, using the Mori language (the Indigenous language of Aotearoa/New Zealand) as a case study. The paper then introduces Te Reo Irirangi o Te Hiku o te Ika’s specifically curated benchmark, the first of its kind for the Mori language. The high-quality benchmark was created to address the resource gap that is fit-for-OU

FAQs

Why are popular AI benchmarks often unsuitable for low-resource, Indigenous languages?

They prioritize measuring general-purpose system performance and rely on datasets that neglect language-community-specific cultural and linguistic needs, resulting in frequent grammatical/orthographic errors, poor pronunciation, and limited vocabulary.

What dataset and benchmark does the paper introduce?

It introduces a curated “fit-for-our-purpose” dataset and benchmark for the Mori language, designed to address the resource gap and provide higher-quality, culturally and ethically appropriate evaluation.

What criteria does the proposed benchmark aim to satisfy?

It is intended to reflect the unique characteristics of the language, represent the diversity of speakers in the community, and support the goals of tool development and language revitalisation efforts.

Oct 17, 2026

Uploaded by fresh2021