Probing the “Creativity” of Large Language Models - Can Models Produce Divergent Semantic Association?

Sep 17, 2026

Large language models possess remarkable capacity for processing language, but it remains unclear whether these models can further generate creative content. This study investigates creative thinking from a cognitive perspective using the divergent association task (DAT), an objective measure that generates unrelated words and computes semantic distance. Results compare models and decoding strategies, showing strong performance under greedy search and improvements from stochastic sampling and temperature scaling alongside a creativity–stability trade-off.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Abstract

Introduction

Probing the “Creativity” of Large Language Models: Can Models Produce Divergent Semantic Association? Honghua Chen and Nai Ding∗ Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Sciences, Zhejiang University / Hangzhou, China {honghuachen,[ding_nai}@zju.edu.cn](ding_nai}@zju.edu.cn) Abstract Large language models possess remarkable capacity for processing language, but it remains unclear whether these models can further generate creative content. The present study aims to investigate the creative thinking of large language models through a cognitive perspective. We utilize the divergent association task (DAT), an objective measurement of creativity that asks models to generate unrelated words and calculates the semantic distance between them. We compare the results across different models and decoding strategies. Our findings indicate that: (1) When using the greedy search strategy, GPT-4 outperforms 96% of humans, while GPT-3 .5-turbo exceeds the average human level. (2) Stochastic sampling and temperature scaling are effective to obtain higher DAT scores for models except GPT-4, but face a trade-off between creativity and stability. These results imply that advanced large language models have divergent semantic associations, which is a fundamental process underlying creativity.1 1 Introduction Large language models (LLMs) have exhibited unparalleled mastery of natural language (Bubecket al., 2023) . The primary capacity of producing the most probable next word is broadly generalizable to many language tasks, suggesting underlying cognitive abilities beyond specialized linguistic rules and patterns. There is observation that LLMs may possess reasoning abilities which is a core aspect of intelligence, including decision-making (Binz and Schulz, 2023) and theory of mind (Moghaddam and Honey, 2023) . Meanwhile, there is also increasing interest in exploring LLMs’ creativity, which is closely related to intelligence (Frith et al., 2021) . Creative use of language, such as metaphor and ∗ Corresponding author 1We release our code at [https://github.com/](https://github.com/)[ ](https://github.com/)DingNLab/probing_creativity Figure 1: Creativity from the perspective of language distribution. Creative thoughts need to be novel and valuable, which need cognitive control to inhibit common tokens and remote association to find valuable tokens. humor, is important during communication. OpenAI (2023) has reported GPT-4’s ability to understand jokes, while subsequent works show limited capacity for LLMs to generate or explain humor (Jentzsch and Kersting, 2023 ; Hessel et al., 2023) . As creativity is essential to the development of art, science, and everyday life for human (Gabora and Kaufman, 2010), it is non-trivial if models could produce creative content. Regarding to the curse of recursion for LLMs that training on generated data makes models collapse, one promising solution might be the novel language distribution through creative generation (Shumailov et al., 2023) . But since LLMs represent word meaning and predict the next word in context, it seems paradoxical that such models could create ideas not seen in training. Here, we empirically investigate the creativity of LLMs by examining models’ ability to generate divergent concepts. A general definition of creativity is the ability to create something both novel and valuable (Runcoand Jaeger, 2012) . According to the dual-process theory of creativity (Beaty et al., 2014), creative thinking relies on remote association while inhibiting common ideas (Figure 1) . Because of the intrinsic complexity, creativity is universally accepted to be unique to human beings, while models are 12881 Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 12881–12888 December 6-10, 2023 ©2023 Association for Computational Linguistics regarded as great predictors to master the e

FAQs

What problem does the study address about large language models?

It examines whether large language models can generate creative content rather than only processing language.

How is creativity measured in this research?

The study uses the divergent association task (DAT), generating unrelated words and computing pairwise semantic distance.

Which decoding strategies affect DAT performance in the results?

Greedy search shows GPT-4 outperforming most humans, while stochastic sampling and temperature scaling improve DAT scores for models other than GPT-4 but create a trade-off between creativity and stability.

Sep 17, 2026

Uploaded by Bill Black