TRepLiNa: Layer-wise CKA + REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B

Sep 4, 2026

Upload by Lute

The MMLoSo 2025 Language Challenge targets the shortage of resources for India’s low-resource languages by developing translation between high-resource languages (Hindi/English) and LRLs such as Bhili, Mundari, Santali, and Gondi. This study tests whether enforcing cross-lingual similarity in selected internal layers of a decoder-only multilingual LLM improves translation quality. A joint alignment method, TRepLiNa (CKA + REPINA), combines layer-wise centered kernel alignment with REPINA regularization to stabilize updates. Results show mid-layer alignment provides low-cost gains in data-scarce settings, with public code and models.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

1 Introduction

2 Related Work

TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B Toshiki Nakai 1 , Ravi Kiran Chikkala 1 , Lena Sophie Oberkircher 1 , Nicholas Jennings 1 , Natalia Skachkova2 , Tatiana Anikina2 , Jesujoba O. Alabi 1 1 Saarland University 2 German Research Center for Artiﬁcial Intelligence (DFKI) ftoshiki3738,[lenaoberkircher](lenaoberkircherg@gmail.com)[g](lenaoberkircherg@gmail.com)[@gmail.com](lenaoberkircherg@gmail.com)[ ](lenaoberkircherg@gmail.com)frach00004@teams,s8nijenn@stud,[jalabi@cs](jalabi@csg.uni-saarland.de)[g](jalabi@csg.uni-saarland.de)[.uni-saarland.de](jalabi@csg.uni-saarland.de) Abstract The 2025 Multimodal Models for LowResource Contexts and Social Impact (MMLoSo) Language Challenge addresses one of India's most pressing linguistic gaps: the shortage of resources for its diverse low-resource languages (LRLs) . The challenge focuses on developing a translation model capable of translating between High resource languages (HRLs)(Hindi/English) and LRLs (Bhili, Mundari, Santali, and Gondi) . In this study, we use the MMLoSo 2025 challenge dataset to investigate whether enforcing cross-lingual similarity in speciﬁc internal layers of a decoderonly multilingual large language model (LLM) can improve translation quality from LRLsto HRLs. Speciﬁcally, we combine Centered Kernel Alignment (CKA), a similarity metric that encourages representations of different languages to align with Representation Projection Invariance (REPINA), a regularization method that constrains parameter updates to remain close to the pretrained model, into a joint method, we call TRepLiNa (CKA + REPINA) . Our results 1 show that aligning mid-level layers with TRepLiNa is a low-cost and practical way to improve LRL translation in data-scarce settings. We make our code and models public. 1 Introduction Many multilingual LLMs share parameters across languages, yet transfer to low-resource languages (LRLs) often lags behind their performance on high-resource languages (HRLs) (Conneau et al., 2020 ; Zhang et al., 2020) . Recent analysis of Aya-23 8B (Aryabumi et al., 2024), a multilingual decoder-only model, shows strong neuron overlap across related languages in the embedding layer, perhaps due to token overlap, but it exhibits a marked drop in overlap at intermediate and higher 1 [https : / / github . com / konta3738 /](https : / / github . com / konta3738 /)[ ](https : / / github . com / konta3738 /)cka-repina-aya23 layers (Trinley et al., 2025) . This suggests a simple hypothesis: selectively increasing cross-lingual similarity where it is weakest (mid/high layers) may lead to better transfer for LRLs. We focus only on the LRL!HRL translation, based on the intuition that models generally ﬁnd it easier to understand anew language than to generate it (Lin et al., 2025) . We operationalize this via a lightweight alignment loss between hidden representations of parallel sentences, which is applied at a chosen layer ℓ . We use centered kernel alignment (CKA) (Kornblith et al., 2019), which can robustly compare representations across networks and layers, together with representation projection invariance (REPINA) (Razdaibiedina et al., 2023) to stabilize HRL features against representation drift. We perform experiments, using zero-shot (Zhao et al., 2023), few-shot (Karimi Mahabadi et al., 2022) and QLoRA-based ﬁne-tuning (Zhang et al., 2023) on Aya-23 8B, using the MMLoSo benchmark (lrl, 2025) pairs, Hindi/English pivots as HRLs; Bhili (Indo-Aryan), Mundari (Austro-asiatic), Santali (Austro-asiatic) and Gondi (Dravidian) as LRLs. Our work makes the following contributions: • We present, to the best of our knowledge, the ﬁrst systematic study of layer-wise alignment in a decoder-only LLM for low-resource machine translation (MT), comparing CKA and TRepLiNa (CKA+REPINA) across layers. • We demonstrate that mid-layer alignment (roughly layers 10–15) is most effective, with TRepLiNa consiste

FAQs

What problem does the MMLoSo 2025 challenge address for machine translation?

It addresses the lack of resources for India’s diverse low-resource languages by building translation capabilities between high-resource languages (Hindi/English) and low-resource languages such as Bhili, Mundari, Santali, and Gondi.

How does TRepLiNa improve low-resource translation?

TRepLiNa combines centered kernel alignment (CKA) with REPINA, enforcing cross-lingual similarity in chosen internal layers while constraining updates to reduce representation drift.

Which layers are most effective according to the study?

Mid-layer alignment is reported as most effective, roughly layers 10–15, with TRepLiNa consistently favoring layer 15 in limited-data settings.

Sep 4, 2026

Upload by Lute