YOMI-Bench：用于评估日语汉字阅读与语音理解的基准

Jul 16, 2026

Upload by Patrick

YOMI-Bench is introduced as a dedicated benchmark for evaluating kanji reading and phonological understanding in large language models for Japanese. Japanese kanji often have multiple possible readings that cannot be reliably determined from surface text alone, which leads to empirically low kanji-reading performance. The benchmark defines four targeted tasks and evaluates one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs, showing limited accuracy even for Japanese-focused models and poor results on reading-dependent generation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Related Work

Grapheme-to-Phoneme (G2P) Benchmarks

LLM Benchmarks in Japanese

YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese Ryota Mibayashi 1 , Hiroya Takamura2 , Hitomi Yanaka3 ,4 ,5 1 Kobe University 2 National Institute of Advanced Industrial Science and Technology (AIST) 3 The University of Tokyo 4 RIKEN 5 Tohoku University arXiv :2607 .00664v 1 [ cs .CL] 1 Jul 2026 Abstract We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMIBench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMIBench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings. 1 Introduction The multilingual abilities of large language models (LLMs) have been analyzed from various perspectives (Zhu et al., 2024) . Among these, we focus on the ability of LLMs to recognize and utilize the phonological readings of text, which is related to tasks such as grapheme-to-phoneme (G2P) estimation and phoneme-aware text generation. The relationship between written forms and their pronunciations varies substantially across languages, and even among languages that share the same writing system, notable differences can be observed. For example, although Chinese and Japanese share many kanji characters, nearly all kanji characters in Chinese correspond to a single pronunciation, except for approximately 10% of cases (Matsuo et al., 2010) . In contrast, approximately 60% of Japanese kanji have multiple possible readings (see details in Sec. 3.1) , requiring more complex linguistic information for correct interpretation. For instance, Figure 1: The overview of YOMI-Bench. the kanji character “覚” has three possible readings:“kaku,” “obo,” and “sa.” These readings vary depending on the word in which the character appears, such as “覚醒(kakusei/Awakening),” “覚える(oboeru/Memorize),” and “覚める(sameru/Wake up).” Thus, in order to correctly predict the readings of kanji characters appearing within individual words, models should not only possess the knowledge of the multiple readings of each kanji character, but also predict the correct reading based on the word. For practical situations, the ability to correctly understand the readings of kanji is important for LLMs to solve classification and generation tasks that take rhyming into account, such as proofreading and generating lyrics (Potash et al., 2015 ; Nikolov et al., 2020), rap verses (Xue et al., 2021 ; Mibayashi et al., 2023), and advertising texts (Lei et al., 2022) . However, the reading ability of LLMs that considers such linguistic characteristics of Japanese remains largely unexplored. Therefore, as illustrated in Figure 1, we construct YOMI-Bench, a benchmark for evaluating Japanese reading performance of LLMs. YOMI-Bench is a multi-task evaluation set involving seven types of binary/multiple QA and text generation tasks. For example, the question “Generate the reading of the word覚醒” asks LLMs to answer the correct kanji reading as a text generation task, where the gold answer is “kakusei.”We also use YOMI-Bench to evaluate the reading abilities of representative LLMs. The contributions of this study include: • We construct a challenging benchmark involving seven tasks that requires correct understanding of phonological readings in Japanese and release them on GitHub 1 as publicly available linguistic resources. • Using the

FAQs

Why is kanji reading in Japanese difficult for LLMs?

A single kanji character can have multiple readings, and the correct one often depends on the word context. Surface text alone is insufficient to infer the reading reliably.

What tasks does YOMI-Bench include?

YOMI-Bench is a multi-task benchmark covering seven types of binary/multiple QA and text generation tasks designed to evaluate kanji reading and phonological understanding.

How did different LLM categories perform on YOMI-Bench?

Evaluation includes one multilingual open model, four Japanese-specific open models, and five commercial models. Results indicate even Japanese-specific models show low performance, and commercial models also perform poorly on generation tasks requiring consideration of kanji readings.

Jul 16, 2026

Upload by Patrick