ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

Jul 16, 2026

Upload by Liam

Compositional generalization—the ability to understand and produce novel combinations of known components—remains a core challenge in modern AI. ClassicLogic introduces a benchmark suite built from four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Each game includes a hierarchical, explicit knowledge base where multi-step solving strategies are formally composed from simpler foundational strategies. This enables fine-grained evaluation of reasoning from basic rule learning to mathematically validated, increasing difficulty, supporting neuro-symbolic and advanced AI systems.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Introduction

Benchmarks for Compositional Generalization

Related Work

ClassicLogic: A KNOWLEDGE-DRIVEN BENCHMARK OF CLASSIC PUZZLE GAMES FOR EVALUATING COMPOSITIONAL GENERALIZATION A PREPRINT arXiv :2607 .05 185v 1 [ cs .AI] 6 Jul 2026 Mahnoor Shahid Universität Duisburg-Essen Germany [mahnoor.shahid@uni-due.de](mahnoor.shahid@uni-due.de) Hannes Rothe Universität Duisburg-Essen Germany [hannes.rothe@uni-due.de](hannes.rothe@uni-due.de) ABSTRACT Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent’s ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent’s reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems. The benchmark is open-source and available at: [https:](https:)//[github.com/Place-Beyond-Bytes/classic_games_benchmark.git](github.com/Place-Beyond-Bytes/classic_games_benchmark.git). Keywords Compositional Generalization, Benchmark, Neuro-Symbolic AI, Logic Puzzles, Multi-Step Reasoning 1 Introduction The ability to flexibly combine existing knowledge to solve new problems is a hallmark of human intelligence (Sternberg, 1984; Cosmides and Tooby, 1997) . This capacity, termed as compositional generalization Keysers et al.(2019); Wiedemer et al. (2023), enables us to generate a potentially infinite range of complex ideas and behaviors from a finite set of known components (Fodor and Pylyshyn, 1988) . Modern artificial intelligence, particularly large-scale models, has demonstrated remarkable capabilities in processing and generating human-like data(Zhao et al., 2023) . Yet, a critical gap persists between their pattern-matching prowess and the robust, systematic reasoning characteristic of human intelligence(Marcus, 2018) . These models often fail at tasks requiring logical deduction and multi-step planning, revealing a fundamental weakness in their ability to generalize systematically (Patil and Jadon, 2025; Cao et al., 2025) . To address this, the community has developed benchmarks such as SCAN (Lake and Baroni, 2018) and COGS (Kim and Linzen, 2020) to test for compositional generalization, primarily in the domain of natural language processing. However, a gap exists for benchmarks that test compositional reasoning in more structured, symbolic problem-solving domains (Wang et al., 2024) . Such domains require not only recognizing patterns but also building and executing explicit, multi-step strategies. In this paper, we introduce ClassicLogic, a novel benchmark designed to provide a transparent and challenging testbed for compositional reasoning. Our contribution is a suite of four classic logic puzzles—Sudoku, KenKen, Kakuro, and Futoshiki—that are procedurally generated and comes with a hierarchical knowledge base (KB) of solving strategies, where complex solving strategies are explicitly defined as compositions of simpler, atomic rules. For instance, advanced strategies are explicitly defined as compositions of more fundamental ones (e.g., identifying a ‘naked_pair’ in Sudoku is a composition of the ‘naked_single’ and ‘constraint_propagation’ strategies) . This structure allows us to dissect the reasoning process and evaluate three distinct and crucial forms of comp

FAQs

What problem does ClassicLogic address in AI research?

ClassicLogic targets compositional generalization: enabling AI systems to combine known components into novel, multi-step reasoning for new puzzle instances.

Which classic logic games are included in the ClassicLogic benchmark?

The benchmark contains four puzzles: Sudoku, KenKen, Kakuro, and Futoshiki.

How does the benchmark represent solving strategies and difficulty?

It provides a hierarchical, explicit knowledge base where complex strategies are defined as compositions of simpler atomic strategies, supporting fine-grained assessment across increasing, validated difficulty levels.

Jul 16, 2026

Upload by Liam