Surprisal and Crossword Clues Difficulty - Evaluating Linguistic Processing between LLMs and Humans

Sep 19, 2026

Upload by Aladdin

Crossword clue difficulty is traditionally determined by human setters, leaving automated generators without an objective yardstick. The study models difficulty as the Surprisal of the intended answer given a clue, estimating Surprisal using token probabilities from large language models. It compares three causal LLMs—Llama-3-8B, Llama-2-7B, and Ita-GPT-2-121M—with 60 human solvers on 160 hand-balanced clues, finding negative correlation between Surprisal and accuracy.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages14

Reading Time5 min

AccessFree to download

Abstract

Introduction

Surprisal and Crossword Clues difficulty: Evaluating Linguistic Processing between LLMs and Humans Tommaso Iaquinta1, *,†, Asya Zanollo2,3,†, Achille Fusco3,4,†, Kamyar Zeinalipour1,† and Cristiano Chesi2,3,† 1 Università degli Studi di Siena (UNISI), Via Roma 56, 53100 Siena, Italy 2 University School for Advanced Studies IUSS Pavia, Piazza della Vittoria 15, 27100 Pavia, Italy 3 Laboratory for Neurocognition, Epistemology, and Theoretical Syntax -NeTS-IUSS Pavia 4 Università degli Studi di Firenze, Piazza S. Marco 4, 50121 Firenze, Italy Abstract Crossword clue difficulty is traditionally judged by human setters, leaving automated puzzle generators without an objective yard-stick. We model difficulty as the Surprisal of the answer given the clue, estimating it with token probabilities from large language models. Comparing three models three causal LLMs-Llama-3-8B, Llama-2-7B, and Ita-GPT-2-121M. with 60 human solvers on 160 hand-balanced clues, Surprisal correlates negatively with accuracy (r = –0.62 for nominal clues) . These results show that language-model Surprisal captures some of the cognitive load humans experience and that language-specific training and model scale both matter; the metric therefore enables adaptive crossword generation and provides a new test-bed for probing the alignment between human and model linguistic processing. Keywords surprisal, llm, gpt, crossword, education, linguistic games, puzzle, Crossword difficulty 1. Introduction Crossword (CW) puzzles are among the most popular language games, captivating millions through newspapers, mobile apps, voice assistants, and even televised competitions [1, 2] . The enduring appeal of crosswords across formats stems from the careful calibration of clue difficulty, which can range from accessible, beginner-friendly prompts to highly intricate, expert-level challenges. Despite advancements in automated puzzle generation, state-of-the-art systems like Dr. Fill [3] and the Berkeley Crossword Solver [1], while capable of outperforming many human solvers, still lack a reliable, objective measure to assess the challenge posed by the clues they generate. Traditional heuristics, such as clue length, grid density, historical solve statistics, and letter CLiC-it 2025: Eleventh Italian Conference on Computational Linguistics, September 24 — 26, 2025, Cagliari, Italy * Corresponding author. † † These authors contributed equally. $ [tommaso.iaquinta@unisi.it](tommaso.iaquinta@unisi.it) (T. Iaquinta); [asya.zanollo@iusspavia.it](asya.zanollo@iusspavia.it) (A. Zanollo); [achille.fusco@iusspavia.it](achille.fusco@iusspavia.it)[ ](achille.fusco@iusspavia.it)(A. Fusco); [kamyar.zeinalipour2@unisi.it](kamyar.zeinalipour2@unisi.it) (K. Zeinalipour); [cristiano.chesi@iusspavia.it](cristiano.chesi@iusspavia.it) (C. Chesi) 􀂀 https://tommyiaq.me (T. Iaquinta); [https://www.iusspavia.it/it/rubrica/asya-zanollo/](https://www.iusspavia.it/it/rubrica/asya-zanollo/) (A. Zanollo); [https://github.com/achille-fusco](https://github.com/achille-fusco) (A. Fusco); [https://kamyarzeinalipour.github.io/](https://kamyarzeinalipour.github.io/) (K. Zeinalipour); [https://github.com/cristianochesi](https://github.com/cristianochesi) (C. Chesi) 􀀚 0009-0009-1262-6768 (T. Iaquinta); 0009-0001-3987-4843 (A. Zanollo); 0000-0002-5389-8884 (A. Fusco); 0009-0006-3014-2511 (K. Zeinalipour); 0000-0003-1935-1348 (C. Chesi) © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4 .0 International (CC BY 4 .0) . Table 1 Linguistic properties for “piante che forniscono frutti per spremute, aranci” (plants that provide fruits for juice – orange trees) . | Microcategory | bareNP:rel | | --- | --- | | Macrocategory | nominal | | Accuracy | 0.526 | | RTs (log 10) | 4.214 | | Surprisal | 5.207 | Table 2 Linguistic properties for “i mobili con le grucce, armadi”(the furniture with hangers – wardrobes) . | Microca

FAQs

How is crossword clue difficulty measured in this paper?

Difficulty is modeled as the Surprisal of the answer given the clue, computed from token probabilities produced by large language models.

Which language models are compared, and how do results relate to human solving?

The study compares Llama-3-8B, Llama-2-7B, and Ita-GPT-2-121M against 60 human solvers on 160 balanced clues, showing that higher Surprisal corresponds to lower accuracy.

What are the proposed applications of the Surprisal metric for crossword puzzles?

The metric can support adaptive crossword generation, tutoring tools, fairness in online competitions, and psycholinguistic experimentation focused on alignment between human and model linguistic processing.

Sep 19, 2026

Upload by Aladdin