TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding - Research Summary

Sep 10, 2026

Upload by Stanley

Text-rich images are a key medium for communicating information and improving accessibility, yet current multimodal benchmarks remain limited in scale, scenario coverage, and evaluation design for holistic understanding. TIU-Bench is introduced as a large-scale, multilingual benchmark with 100,000+ full-image annotations and 22,000 validated question-answer pairs spanning 18 subtasks. It adds a structured full-image output format and a two-stage framework (T2TIU) that first represents entire images then performs reasoning, supported by extensive experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Text-rich Image Understanding (TIU)

Dataset Limitations

Task Setting and Evaluation Constraints

TIU-Bench Design and Capabilities

TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding Kun Zhang, Liqiang Niu , Zhen Cao, Fandong Meng* , Jie Zhou 1Pattern Recognition Center, WeChat AI, Tencent Inc {peterkzhang, [zhenzcao}@tencent.com](zhenzcao}@tencent.com) Abstract Text-rich images are ubiquitous in real-world applications, serving as a critical medium for conveying complex information and facilitating accessibility. Despite recent advances driven by Multimodal Large Language Models (MLLMs), existing benchmarks suffer from limited scale, fragmented scenarios, and evaluation protocols that fail to fully capture holistic image understanding. To address these gaps, we present TIU-Bench, a large-scale, multilingual benchmark comprising over 100,000 full-image annotations and 22,000 rigorously validated question-answer (QA) pairs that span 18 subtasks across diverse real-world scenarios. TIU-Bench introduces a novel full-image structured output format that jointly models geometric, textual, and relational information, enabling fine-grained evaluation of perception and reasoning capabilities. Furthermore, we propose a two-stage understanding framework named T2TIU, which first generates a structured representation of the entire image and subsequently conducts reasoning on this representation in order to address complex visualtextual queries. Extensive experiments on 10 state-of-the-art generative models highlight the challenges and opportunities in advancing textrich image understanding. Our benchmark and framework provide a comprehensive platform for developing and evaluating next-generation multimodal AI systems. 1 Introduction Text-rich images play a pivotal role in real-world scenarios by efficiently conveying complex information and improving accessibility (Biten et al., 2019) . Accurate interpretation of such images is essential for automating information extraction, advancing AI systems, and optimizing user interactions. To formalize this research domain, we define Text-rich Image Understanding (TIU) as consist- * Corresponding author. 0 20 40 60 80 100 120 Instruction (k) Figure 1: Comparison of the number of images and instructions between our dataset and existing datasets. ing of two core capabilities: perception and understanding. The perception dimension encompasses visual recognition tasks such as text detection (Liao et al., 2022), text recognition (Guan et al., 2025), formula recognition (Truong et al., 2024 ; Guan et al., 2024), and document layout analysis (Yupan et al., 2022) . In contrast, the understanding dimension involves semantic reasoning for downstream applications such as key information extraction and document-based visual question answering (e.g., DocVQA (Mathew et al., 2021), ChartQA (Masryet al., 2022), and TextVQA (Singh et al., 2019)) . Recently, MLLMs have been proposed, which integrate large language models (LLMs) with visual encoders to jointly process visual tokens and linguistic elements through unified attention mechanisms, enabling end-to-end sequence modeling. Within the TIU domain, MLLMs have demonstrated impressive results in both perception and understanding. Nevertheless, despite recent advances, there are still two key challenges in current TIU research paradigms. Dataset Limitations. TIU tasks currently face challenges related to data diversity, scale, and quality. Existing datasets such as DocVQA and ChartQA focus on isolated scenarios (e.g., documents, tables, or charts), and their fragmented objectives and scenario-specific designs hinder the 24286 Findings of the Association for Computational Linguistics: EMNLP 2025 , pages 24286–24295 November 4-9, 2025 ©2025 Association for Computational Linguistics Figure 2: Overview of sub-tasks in TIU-Bench datasets. comprehensive evaluation of perceptual and interpretative capabilities. Recent benchmarks such as OCRbench and its v2 version assess line-level text recognition in MLLMs but overlook

FAQs

What problem does TIU-Bench target in text-rich image understanding benchmarks?

It targets limited benchmark scale, fragmented scenarios, and evaluation protocols that do not capture holistic full-image perception and reasoning.

How does TIU-Bench structure evaluation for complex text-rich images?

It uses a full-image structured output format and evaluates multiple capabilities including text recognition, relation extraction, and visual-textual reasoning across 18 subtasks.

What is the role of the T2TIU framework in TIU-Bench?

T2TIU performs reasoning in two stages: generating a structured representation of the entire image first, then conducting reasoning on that representation for complex visual-textual queries.

Sep 10, 2026

Upload by Stanley