[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-296277-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-296277-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","the-qiyas-benchmark-measuring-chatgpt-mathematical-and-language-understanding-in-arabic","The Qiyas Benchmark - Measuring ChatGPT Mathematical and Language Understanding in Arabic","","Despite the growing importance of Arabic as a global language, robust pretrained language models dedicated exclusively to Arabic remain scarce, leading to limited benchmarks for evaluating Arabic model performance. This paper proposes two new Arabic benchmarks to assess mathematical reasoning and Arabic language understanding, derived from the Saudi university admissions Qiyas exam. Experiments evaluate ChatGPT-3.5-turbo and ChatGPT-4, showing challenging difficulty and overall accuracies of 49% and 64%, respectively, across question types.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/the-qiyas-benchmark-measuring-chatgpt-mathematical-and-language-understanding-in-arabic/296277/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/the-qiyas-benchmark-measuring-chatgpt-mathematical-and-language-understanding-in-arabic/296277.png","ImageObject",300,407,{"name":42,"@type":43},"Sophia Brooks","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-05","2026-09-18",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",17,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address about Arabic language models?","Question",{"text":63,"@type":64},"It addresses the lack of Arabic-native pretrained language resources and the shortage of credible benchmarks for evaluating Arabic model performance, especially for math reasoning and language understanding.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How are the proposed benchmarks constructed?",{"text":68,"@type":64},"They are derived from the standardized Qiyas exam used for university admissions in Saudi Arabia, split into quantitative (math skills) and verbal (Arabic language understanding) components.",{"name":70,"@type":61,"acceptedAnswer":71},"Which ChatGPT models are evaluated and what accuracy results are reported?",{"text":72,"@type":64},"The paper evaluates ChatGPT-3.5-turbo and ChatGPT-4 on the benchmarks. Reported overall accuracies are 49% for ChatGPT-3.5-turbo and 64% for ChatGPT-4 across question types.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},296277,1789730471,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":107,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},962084925636,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in  \nArabic  \nShahad Al-Khalifa  \niWAN Research Group, College of Computer and Information Sciences  \nKing Saud University  \nRiyadh, Saudi Arabia [shahadalkhalifa90@gmail.com](shahadalkhalifa90@gmail.com)  \nHend Al-Khalifa  \nDepartment of Information Technology, College of Computer and Information Sciences  \nKing Saud University  \nRiyadh, Saudi Arabia  \n[hendk@ksu.edu.sa](hendk@ksu.edu.sa)  \nAbstract— Despite the growing importance of Arabic as a global language, there is a notable lack of language models pretrained exclusively on Arabic data. This shortage has led to limited benchmarks available for assessing language model performance in Arabic. To address this gap, we introduce two novel benchmarks designed to evaluate models' mathematical reasoning and language understanding abilities in Arabic. These benchmarks are derived from a General Aptitude Test (GAT) called Qiyas exam, a standardized test widely used for university admissions in Saudi Arabia. For validation purposes, we assess the performance of ChatGPT-3.5-trubo and ChatGPT-4 on our benchmarks. Our findings reveal that these benchmarks pose a significant challenge, with ChatGPT-4 achieving an overall average accuracy of 64%, while ChatGPT-3.5-trubo achieved an overall accuracy of 49% across the various question types in the Qiyas benchmark. We believe the release of these benchmarks will pave the way for enhancing the mathematical reasoning and language understanding capabilities of future models tailored for the low-resource Arabic language.  \nKeywords—Benchmark, Large Language Models, Verbal test, Mathematical reasoning, ChatGPT, GPT, Arabic Language.  \nI. INTRODUCTION  \nEvaluating the capabilities of large language models (LLMs) across different tasks like mathematical reasoning and natural language understanding is a critical area of research as these general-purpose AI systems become more widely used. Developing comprehensive evaluation benchmarks, especially for languages beyond English, is crucial for driving model improvement.  \nArabic, a Semitic language with complex morphology and written from right-to-left, is spoken by over 400 million people across the Arab world [1]. Despite its status as a global language of importance, Arabic is considered low-resource in the field of natural language processing [2] . There is currently a shortage of LLMs pre-trained exclusively on large Arabic datasets. This has resulted in limited benchmarks available for robustly assessing Arabic LLM performance. While some prior work has translated English benchmarks, there is a need for high-quality, natively developed Arabic evaluation resources.  \nThe lack of robust, Arabic-native benchmarks focused on key capabilities like math reasoning and language understanding represents a significant gap in the field. Having professionally designed evaluation resources in this domain could accelerate the development of higherperforming Arabic language models. This work makes two key contributions to address this need:  \n1) It introduces Qiyas, a benchmark suite for comprehensively evaluating LLM performance on mathematical and language tasks in the Arabic language. Qiyas consists of two components - a quantitative section assessing math skills and a verbal section evaluating Arabic language understanding abilities.  \n2) Using the Qiyas benchmark, the performance of the latest ChatGPT models (versions 3.5-turbo and 4) is extensively evaluated across zero-shot, one-shot, and fewshot settings to establish strong Arabic language baselines.  \nThese benchmarks are derived from the Qiyas exam, a standardized exam widely used for university admissions in Saudi Arabia, ensuring their quality has been validated by educational experts. The results shed light on current LLM limitations for the Arabic language and highlight the impact of varied training data and prompting approaches. The release of Qiyas paves the way fo","cbCailkMJRka7tyv","https://ap.wps.com/l/cbCailkMJRka7tyv","pdf",550251,"English","# Introduction\n# Background and Literature Review\n## Qiyas as an assessment institution","[{\"question\":\"What problem does the paper address about Arabic language models?\",\"answer\":\"It addresses the lack of Arabic-native pretrained language resources and the shortage of credible benchmarks for evaluating Arabic model performance, especially for math reasoning and language understanding.\"},{\"question\":\"How are the proposed benchmarks constructed?\",\"answer\":\"They are derived from the standardized Qiyas exam used for university admissions in Saudi Arabia, split into quantitative (math skills) and verbal (Arabic language understanding) components.\"},{\"question\":\"Which ChatGPT models are evaluated and what accuracy results are reported?\",\"answer\":\"The paper evaluates ChatGPT-3.5-turbo and ChatGPT-4 on the benchmarks. Reported overall accuracies are 49% for ChatGPT-3.5-turbo and 64% for ChatGPT-4 across question types.\"}]","The Qiyas Benchmark - Measuring ChatGPT Mathematical and Language Understanding in Arabic | PDF",18]