Thinking with Images for Multimodal Reasoning - Foundations, Methods, and Future Frontiers

Aug 23, 2026

Recent progress in multimodal reasoning has been driven by textual Chain-of-Thought, where models reason in language while treating vision as a fixed starting context. This creates a semantic gap between rich perceptual signals and discrete symbolic thinking. Humans instead use vision as a dynamic mental sketchpad, and the same shift is emerging in AI. The survey formalizes this paradigm shift toward models that think with images, leveraging visual information as intermediate cognitive steps and assessing the resulting evaluations, benchmarks, and transformative applications across three stages.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages57

Reading Time2 h 24 min

AccessFree to download

Introduction

The Position of Our Survey

Foundations ofthe Thinking with Images Paradigm

The Cognitive Imperative: Why We Need “Thinking with Images”?

How Does It Work? The Three-Stage Evolution

What is Thinking with Images? A Formal Definition

Unique Challenges

Stage 1: Tool-Driven Visual Exploration

Formulation

Categories of Visual Tools

Implementation Approaches

Conclusion and Future Frontiers

Stage 2: Programmatic Visual Manipulation

The Programmatic Leap: From Selection to Creation

Formulation

Categories of Composable Operations

Implementation Approaches

Conclusion and Future Frontiers

Stage 3: Intrinsic Visual Imagination

The Architectural Leap: From Execution to Imagination

Formulation

Categories of Generative Thoughts

Implementation Approaches

Conclusion and Future Frontiers

Evaluations & Frameworks for Thinking with Images

Evaluations Benchmarks

Implementation Frameworks

Conclusion and Future Frontiers

Applications

Interactive Systems and User Interfaces

Embodied AI and Robotics

AI for Science

Healthcare and Medicine

Education and Training

Future Directions

Computational and Cognitive Efficiency

Safe and Trustworthy Visual Cognition

Novel Benchmarks and Evaluation Methodologies

Thinking with Audio, Video, and the World

Open Questions and Outlook

Conclusion

arXiv :2506 .239 18v 3 [ cs .CV] 3 Jul 2025 Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers Zhaochen Su 1 , Peng Xia2 , Hangyu Guo 1 , Zhenhua Liu 1 , Yan Ma, Xiaoye Qu, Jiaqi Liu2 , Yanshu Li 1 , Kaide Zeng2 , Zhengyuan Yang3 , Linjie Li3 , Yu Cheng4 , Heng Ji5 , Junxian He 1 , Yi R. (May) Fung 1 1The Hong Kong University of Science and Technology 2UNC-Chapel Hill, 3Microsoft, 4The Chinese University of Hong Kong, 5UIUC Abstract Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-centric approach, however, treats vision as a static, initial context, creating a fundamental semantic gap between rich perceptual data and discrete symbolic thought. Human cognition often transcends language, utilizing vision as a dynamic mental sketchpad. A similar evolution is now unfolding in AI, marking a fundamental paradigm shift from models that merely think about images to those that can truly think with images. This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process, transforming vision from a passive input into a dynamic, manipulable cognitive workspace. In this survey, we chart this evolution of intelligence along a spectrum of increasing cognitive autonomy, which unfolds across three key stages: from external tool exploration, through programmatic manipulation, to intrinsic imagination. To structure this rapidly evolving field, our survey makes four key contributions. (1) We establish the foundational principles of the “Thinking with Images” paradigm and its three-stage framework. (2) We provide a comprehensive review of the core methods that characterize each stage of this roadmap. (3) We analyze the critical landscape of evaluation benchmarks and transformative applications. (4) We identify significant challenges and outline promising future directions. Through this structured overview, we aim to offer a clear roadmap for future research towards more powerful and human-aligned multimodal AI.1 The Paradigm Shift in Multimodal Reasoning Think about Images Think with Images Think with Text Textual reasoning over a static visual context. Intelligence Increasing Stage 1: Tool-Driven Visual Exploration Commanding a fixed toolkit for visual analysis. Stage 2: Programmatic Stage 3: Intrinsic Visual Imagination Visual Manipulation Creating bespoke visual Internally generating visual thoughts within a cognitive loop. operations via code generation. 1We maintain a real-time GitHub repository tracking progress at: [https://github.com/zhaochen0110/](https://github.com/zhaochen0110/)[ ](https://github.com/zhaochen0110/)Awesome_Think_With_Images. Contents 1 Introduction 4 1.1 The Position of Our Survey .............................. 5 2 Foundations ofthe Thinking with Images Paradigm 6 2.1 The Cognitive Imperative: Why We Need “Thinking with Images”? ........ 6 2.2 How Does It Work? The Three-Stage Evolution ................... 7 2.3 What is Thinking with Images? A Formal Definition ................ 8 2.4 Unique Challenges .................................. 8 3 Stage 1: Tool-Driven Visual Exploration 9 3.1 Formulation ...................................... 9 3.2 Categories of Visual Tools .............................. 10 3.3 Implementation Approaches ............................. 11 3.3.1 Prompt-Based Approaches .......................... 11 3.3.2 SFT-Based Approaches ............................ 13 3.3.3 RL-Based Approaches ............................ 14 3.4 Conclusion and Future Frontiers ........................... 14 4 Stage 2: Programmatic Visual Manipulation 15 4.1 The Programmatic Leap: From Selection to Creation ................ 16 4.2 Formulation ...................................... 16 4.3 Categories of Composable Operations

FAQs

Why does textual Chain-of-Thought create a limitation for multimodal reasoning?

Textual CoT treats vision as a static initial context, which leaves a semantic gap between perceptual information and symbolic reasoning steps. The survey motivates a paradigm where vision contributes dynamically to the reasoning process.

What are the three stages of the “Thinking with Images” paradigm?

The survey organizes the evolution into three stages: Stage 1 tool-driven visual exploration, Stage 2 programmatic visual manipulation, and Stage 3 intrinsic visual imagination. Each stage increases cognitive autonomy in how visual information is used.

What does the survey cover regarding evaluation and applications?

It reviews evaluation benchmarks and implementation frameworks, then analyzes transformative applications across interactive systems, embodied AI/robotics, AI for science, healthcare and medicine, and education and training.

Aug 23, 2026

Uploaded by Oliver Hayes