Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment

Jul 16, 2026

Upload by Aurora

Large language models confront urban planning with an urgent epistemological test: which dimensions of professional planning knowledge can be replicated by AI, and which remain irreducibly human. The paper proposes Urban Planning Bench (UPBench), a domain-specific evaluation framework using a 4×5 matrix across four knowledge pillars and five Bloom-adapted cognitive levels. Results from 25 LLMs reveal non-monotonic reasoning, where higher-order analytical tasks outperform lower-order factual recall and integrative judgment due to context-, value-, and institution-bound planning knowledge.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages32

Reading Time1 h 21 min

Abstract

Key Takeaway for Practice

Keywords

Introduction

arXiv :2606 . 11678v1 [ cs .CL] 10 Jun 2026 Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment Yijie Deng 1,2,* , He Zhu 1,2,* , Wen Wang 1,2 , Junyou Su 1,2 , Minxin Chen 1,2 , and Wenjia Zhang 1,3,† 1 Behavioral and Spatial AI Lab, Tongji University; 2 Behavioral and Spatial AI Lab, Peking University; 3 College of Architecture and Urban Planning, Tongji University *Yijie Deng and He Zhu contributed equally to this work. †Corresponding author: Wenjia Zhang. ABSTRACT Problem, Research Strategy, and Findings The emergence of large language models (LLMs) confronts urban planning with an urgent epistemological question: what dimensions of professional planning knowledge can artificial intelligence replicate, and what remains irreducibly human? Despite growing deployment of AI tools in planning practice, we lack systematic frameworks for evaluating whether these systems can reason with the contextual sensitivity, value awareness, and institutional literacy that characterize professional planning judgment. This paper introduces Urban Planning Bench (UPBench), a domainspecific evaluative framework that assesses LLM reasoning across a 4 ×5 matrix encompassing four knowledge pillars (Principles of Urban Planning, Cross-Disciplinary Integration, Planning Governance, and Planning Practice) and five cognitive levels adapted from Bloom’s revised taxonomy. Evaluating 25 LLMs through a dual-track protocol combining automated scoring with expert panel assessment, we identify an non-monotonic cognitive curve: models perform more robustly on higher-order analytical tasks than on ostensibly lower-order factual recall and integrative judgment. This counterintuitive finding reveals that planning’s “lower-order” knowledge is infact deeply embedded in institutional, jurisdictional, and temporal context—making it resistant to pattern-generalization strategies. We codify these limitations into four epistemic diagnostics—regulatory hallucination, conceptual conflation, wickedness paralysis, and phronetic deficit—each illuminating a specific dimension of planning expertise that resists computational replication. Takeaway for Practice These findings provide planning practitioners and educators with an evidencebased framework for differential delegation—determining which tasks can be responsibly augmented by AI and which require irreducibly human professional judgment. LLMs demonstrate competence in cross-disciplinary synthesis and broad analytical reasoning, suggesting productive augmentation potential for literature review, scenario generation, and preliminary policy analysis. However, they exhibit persistent incapacity in jurisdiction-specific regulatory interpretation, normative conflict resolution, and context-sensitive procedural application—tasks that constitute the core of planning’s phronetic expertise. Planning agencies should implement structured verification protocols for any AI-assisted regulatory analysis, while planning education should reorient from knowledge transmission toward cultivating the institutional literacy, normative judgment, and contextual sensitivity that constitute planning’s distinctive professional contribution. KEYWORDS artificial intelligence; planning knowledge; professional judgment; phronesis; benchmark; large language models; planning education 1. Introduction Planning has long grappled with a foundational disciplinary question: what constitutes distinctively professional planning knowledge? Friedmann (1987) framed this as the problem of linking knowledge to action—identifying what planners know that enables them to intervene meaningfully in the trajectory of cities and regions. Schön (1992a) recast professional knowledge not as applied science but as reflection-in-action, a form of knowing embedded in practice that resists codification into rules. More recently, Flyvbjerg (2001) argued that planning’s core intellectual contribution lies

FAQs

What problem does the paper address about AI and urban planning?

It examines which aspects of professional planning knowledge LLMs can replicate and which aspects remain irreducibly human, focusing on contextual sensitivity, value awareness, and institutional literacy.

How does Urban Planning Bench (UPBench) evaluate LLM reasoning?

UPBench assesses LLMs across four planning knowledge pillars and five cognitive levels, using a dual-track protocol with automated scoring plus expert panel assessment.

What key finding challenges expectations about LLM performance?

The study finds a non-monotonic cognitive curve: models perform more robustly on higher-order analytical tasks than on lower-order factual recall and integrative judgment, implying resistance to pattern-generalization for planning’s contextual knowledge.

Jul 16, 2026

Upload by Aurora