Building Japanese Creativity Benchmarks and Applying them to Enhance LLM Creativity

Sep 17, 2026

To evaluate the creativity of large language models (LLMs) in Japanese, this work builds three benchmarks: Japanese Creativity Questions (JCQ), Divergent Association Task (DAT), and Story Alteration Task (SAT). JCQ offers comprehensive creativity assessment using LLM-based evaluation, while DAT and SAT target specific aspects via embedding-based measurements with faster, lower-cost scoring. Correlations among the benchmarks are analyzed, and training with DAT is examined for potential creativity gains.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

Introduction

Related Work

Torrance Test of Creative Thinking (TTCT)

JCQ task definitions and criteria

Building Japanese Creativity Benchmarks and Applying them to Enhance LLM Creativity So Fukuda1 , Hayato Ogawa1 , Kaito Horio1 , Daisuke Kawahara1 , Tomohide Shibata2 1Waseda University, 2 SB Intuitions Corp. {so.fukuda@akane. , cookie3120@ruri. , kakakakakakaito@akane. , [dkw@}waseda.jp](dkw@}waseda.jp) [tomohide.shibata@sbintuitions.co.jp](tomohide.shibata@sbintuitions.co.jp) Abstract To evaluate the creativity of large language models (LLMs) in Japanese, we construct three benchmarks: Japanese Creativity Questions (JCQ), Divergent Association Task (DAT), and Story Alteration Task (SAT) . JCQ comprehensively evaluates creativity using LLMs. Meanwhile, DAT and SAT measure specific aspects of creative ability using embeddings. We also analyze correlations between JCQ and DAT, JCQ and SAT, and DAT and SAT. While JCQ provides comprehensive evaluation, it is relatively time and resource intensive. In contrast, DAT and SAT offer lower comprehensiveness but enable quick, low-cost assessment. Additionally, we investigate whether training with DAT contributes to enhancing LLM creativity. 1 Introduction Creativity is a crucial ability that has supported human progress and development. Creative thinking has been central to human activities, from artistic expression and scientific discovery to solving social problems. In recent years, with the development of large language models (LLMs), AI systems have shown potential to support and extend human creative activities in text generation and problem-solving, leading to active research in this area (Franceschelli and Musolesi, 2024 ; Tanaka et al., 2024 ; Watanabe et al., 2024 ; Li et al., 2024) . For both humans and LLMs, creativity has become an essential element for addressing the challenges of our increasingly complex society and creating new value. Previous research on LLM creativity has primarily focused on English, but there are differences in how creativity manifests and is evaluated across languages and cultures. Japanese, in particular, has different grammatical structures and expressive styles from English, with unique linguistic characteristics such as abundant homonyms and high context-dependency. These characteristics may uniquely influence LLMs’ creative expression, highlighting the importance of cross-linguistic creativity research. In this study, we construct three benchmarks to measure LLM creativity in Japanese either comprehensively or efficiently depending on the purpose, and evaluate several LLMs. The first is Japanese Creativity Questions (JCQ), developed based on the verbal tasks of the Torrance Test of Creative Thinking (TTCT) (Torrance, 1966), which is widely used to evaluate human creativity. This follows the approach of previous research (Zhao et al., 2024) . It consists of seven tasks and uses four criteria for evaluation. The second is the Divergent Association Task (DAT) (Olson et al., 2021), which requires listing words that are as semantically distant from each other as possible. The third is the Story Alteration Task (SAT), which measures how mucha story differs from the original after being altered. JCQ evaluation uses a powerful LLM as LLM-as-ajudge, while DAT and SAT evaluations use embeddings. JCQ can comprehensively evaluate creativity but requires time and resources for assessment. DAT and SAT, on the other hand, can quickly and easily measure specific aspects of creativity by using embeddings. This allows for choosing between comprehensive or rapid evaluation methods to measure LLM creativity according to specific needs. Furthermore, we investigate whether training LLMs using DAT improves creativity through generalization ability, potentially enhancing scores on JCQ and SAT. 2 Related Work The Torrance Test of Creative Thinking (TTCT) is widely known as a test for evaluating human creativity. It consists of verbal and figural tests with free-response questions, such as “List as many unusual uses for a light bulb as possible

FAQs

What are the three benchmarks proposed to evaluate Japanese LLM creativity?

The benchmarks are Japanese Creativity Questions (JCQ), Divergent Association Task (DAT), and Story Alteration Task (SAT). JCQ assesses creativity comprehensively, while DAT and SAT measure specific creative abilities more efficiently.

How does JCQ evaluation differ from DAT and SAT evaluation?

JCQ uses a powerful LLM as an evaluator (LLM-as-a-judge). DAT and SAT use embeddings to evaluate divergent associations and story alteration differences.

What is investigated about training and creativity improvement?

The study investigates whether training LLMs using DAT improves creativity, with potential generalization effects that could raise scores on JCQ and SAT.

Sep 17, 2026

Uploaded by Riley West