Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

Jul 16, 2026

Upload by Ophelia

Vision–language models (VLMs) have advanced rapidly in visual reasoning, yet many evaluations rely on simple datasets such as MSCOCO, use limited non-curated human descriptions, and rarely analyze the model’s error types. This work introduces the Complex Social Behavior (CSB) dataset with 100 images of complex social interactions. Results across 2017–2025 show strong gains in scene-description accuracy, with MLLMs nearly eliminating most error categories except spatial dependence, and revealing which errors most affect caption quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Abstract

Introduction

Motivation: limitations of standard benchmarks

Model progression across years and datasets

arXiv :2607 .09654v1 [ cs .CV] 10 Jul 2026 Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models Shravan Murlidarana,1,∗, Miguel P. Ecksteina,b,c,1 a Psychological & Brain Sciences, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA b Department of Computer Science, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA c Department of Electrical and Computer Engineering, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA Abstract Vision–language models (VLMs) have made remarkable progress in visual reasoning during the last decade. Most evaluations have used simple scenes (MSCOCO) that do not showcase complex human interactions or behaviors, only a handful of non-curated human descriptions as a benchmark, and have not focused on understanding the model’s error types. Here, we introduce the Complex Social Behavior (CSB) dataset, containing 100 images depicting complex social interactions/behaviors. We analyze the progression of scene descriptions over a decade (2017-2025) of VLMs (four pre-Multimodal Large Language Models, MLLMs, and five MLLMs) . We evaluate the accuracy of the models and 20 human descriptions relative to a gold standard on the CSB dataset and on a sample from MS-COCO. We analyzed five visual-cognitive error types: object detection, recognition, hallucination, scene understanding, and spatial dependence. The CSB dataset showed a more pronounced improvement than MS-COCO in scene description accuracy, with pre-MLLMs achieving much lower accuracy than the bottom-ranked human descriptions and MLLMs attaining accuracies similar to the top-ranked human descriptions. We show that MLLMs have eliminated the gap in scene description accuracy between simpler MS-COCO scenes and scenes depicting complex behaviors (CSB) . MLLMs have almost eliminated all error types in our tested datasets, except for occasionally relying on different image regions for scene descriptions than humans do (spatial dependence error) . We also show that detection, recognition, and hallucination errors have the highest impact on scene description accuracy. Together, our findings provide a more thorough evaluation of how visual language models have advanced over the last decade. ∗ Corresponding author. Email addresses: [smurlidaran@ucsb.edu](smurlidaran@ucsb.edu) (Shravan Murlidaran), [eckstein@psych.ucsb.edu](eckstein@psych.ucsb.edu) (Miguel P. Eckstein) 1 These authors contributed equally to this work. Keywords: Multi-Modal Large Language Model (MLLMs), Image Understanding, Captioning, Description Error Analysis,, Human and MLLM performance 1. Introduction Figure 1: (a) The reported performance of DNNs over the years in object recognition compared to human performance. The performance of the then state-of-the-art models exceeded human performance. (b) Reported performance of Multi-Modal Large-Language Models (MLLMs) in the scene description task on the MS-COCO dataset. The model’s performance is comparable to human performance. (c) Model descriptions for scenes with and without social interaction. The image on the left depicts a simple scene in which the models’ descriptions are similar to those observed in humans. The image on the right depicts a scene of complex social interactions among humans. We can clearly see that the human description captures the social interaction in the scene. In contrast, the pre-MLLM’s description does not capture the social interactions, while MLLMs do. Artificial Neural Networks have advanced significantly in visual reasoning since the development of deep Convolutional Neural Networks (CNNs) over a decade ago [1] . Figure 1a shows a glimpse of the early years of deep CNNs, with models that could outperform humans in tasks like object recognition. Deep Recurrent Neural Networks [2], which process and generate sequential data like text, combined with deep CNNs

FAQs

Which visual-cognitive error types most impact scene-description accuracy?

Detection, recognition, and hallucination errors have the highest impact on scene-description accuracy, while other categories improve substantially across tested models.

Jul 16, 2026

Upload by Ophelia