Star at PalmX 2025 Shared Task - Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning

Sep 15, 2026

Uploaded by Violet

A two-stage framework enhances Arabic cultural understanding in small language models for PalmX 2025 Shared Task 1 (General Culture Evaluation). Continuous pretraining on a culturally enriched Arabic corpus covering 10 Arab countries and multiple cultural domains is followed by supervised fine-tuning on cultural question answering data. Parameter-Efficient Fine-Tuning using PEFT with LoRA on Qwen3-4B yields 74% development accuracy and 64% on the blind test set, ranking ninth. Results confirm targeted cultural pretraining improves cultural competency while preserving computational efficiency.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

AccessFree to download

Abstract

Introduction

Literature Review

Dataset Details

Related Work

System Overview

Architecture

Two-Stage Training Framework

Stage 1: Cultural Pretraining

Stage 2: Supervised Fine-tuning

Star at PalmX 2025 Shared Task: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning Eman Elrefai1 1Alexandria University [eman.lotfy.elrefai@gmail.com](eman.lotfy.elrefai@gmail.com) Esraa Khaled2 2 Cairo University [esraa.k.fouad@gmail.com](esraa.k.fouad@gmail.com) Alhassan Ehab3 3Minia University [alhassanehab186@gmail.com](alhassanehab186@gmail.com) Abstract We present a two-stage framework for enhancing Arabic cultural understanding in small language models, specifically designed for PalmX 2025(Alwajih et al., 2025) Shared Task 1: General Culture Evaluation. Our approach combines continuous pretraining on a culturallyenriched Arabic corpus spanning 10 Arab countries and different cultural domains, followed by supervised fine-tuning on cultural questionanswering data. Using Parameter-Efficient FineTuning (PEFT) (Zhang et al., 2025) with LoRA on the Qwen3-4B base model, we achieve 74% accuracy on the development set and 64% on the blind test set, ranking our team ninth in the competition. Our system demonstrates the effectiveness of targeted cultural pretraining for improving Arabic language models’ cultural competency while maintaining computational efficiency. 1 Introduction Arabic cultural understanding represents a critical challenge in natural language processing, as existing large language models often lack the nuanced cultural knowledge necessary to serve Arabicspeaking communities effectively. The PalmX 2025 Shared Task 1 focuses on evaluating models’ ability to understand and reason about Arabic cultural concepts, traditions, and knowledge across diverse Arab regions. Our main system strategy employs a two-stage training paradigm: (1) continuous pretraining on culturally-diverse Arabic content to build foundational cultural knowledge. (2) supervised finetuning on structured cultural question-answering data to enhance reasoning capabilities. This approach addresses the fundamental challenge of cultural representation in language models while maintaining computational efficiency through parameterefficient techniques. Key findings from our work include achieving competitive performance (74% development accuracy, 64% test accuracy) while using only 4B parameters, demonstrating that targeted cultural pretraining significantly improves performance over baseline models, and identifying that multi-domain cultural coverage is essential for robust cultural understanding. The main challenge discovered was balancing broad cultural coverage with deep domain-specific knowledge within computational constraints. 2 Literature Review The PalmX 2025 Subtask 1 presents a multiplechoice question-answering challenge focused on Arabic cultural knowledge. The input consists of cultural questions in Modern Standard Arabic (MSA) with four possible answers (A, B, C, D), and the output is the correct answer choice. Example: Question: ﻣﺎ ﻫﻲ اﻟﻌﺎﺻﻤﺔ اﻟﺘﺎرﻳﺨﻴﺔ ﻟﻠﺪوﻟﺔ اﻷﻣﻮﻳﺔ؟ Choices: ﻣﻜﺔD. اﻟﻘﺎﻫﺮةC. دﻣﺸﻖB. ﺑﻐﺪادA. Answer: B 2.1 Dataset Details Our pretraining corpus was constructed from Arabic Wikipedia articles covering 10 Arab countries (Bahrain, Egypt, UAE, Iraq, Kuwait, Jordan, Lebanon, Palestine, Syria, Saudi Arabia) and cultural domains like (Media, Sport, Transport, Healthcare, Education, Religion, Economy, History, Festivals, Tourism) . The coverage was limited to these 10 countries due to data availability and quality constraints: some Arab countries had very limited or incomplete Wikipedia content across the chosen domains, which would have introduced imbalance and sparsity into the corpus. Focusing on countries with richer and more representative cultural data ensured 825 Proceedings of The Third Arabic Natural Language Processing Conference, pages 825–829 November 8-9, 2025 ©2025 Association for Computational Linguistics both consistency and reliability of the pretraining resource. The dataset for instruction fine tuning stage comprises cultura

FAQs

What is the main approach for improving Arabic cultural understanding in small language models?

The system uses a two-stage training paradigm: continuous cultural pretraining on a culturally diverse corpus, then supervised fine-tuning on structured cultural question answering data.

What model and fine-tuning method are used?

The method builds on the Qwen3-4B base model and applies PEFT with LoRA to selected attention and feed-forward layers to adapt efficiently.

How is the training data organized and what regions does it cover?

Pretraining uses Arabic Wikipedia articles spanning 10 Arab countries and multiple cultural domains, while fine-tuning uses a cultural question-answering dataset in MSA with history, literature, traditions, geography, and social customs.

Sep 15, 2026

Uploaded by Violet