Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

Jul 16, 2026

Upload by Ivy

Multimodal GUI agents interpret an interface through two redundant channels: screenshot pixels and a serialized representation such as a DOM or accessibility tree. Benchmarks evaluate task success and robustness but rarely test whether the agent’s state belief is actually sourced from pixels. The study formalizes visual state reliance, attributes state beliefs to pixels, structure, or priors, and measures it via paired single-channel interventions on 310 real GUI probes using deterministic scoring. A key metric, Perception–Fusion Gap, reveals that textual state beliefs defer to structure under conflict while image-only accuracy remains near ceiling.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Findings

Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure Guijia Zhang 1 Harry Yang2 1 Shenzhen University 2The Hong Kong University of Science and Technology arXiv :2607 .04334v 1 [ cs .AI ] 5 Jul 2026 Abstract Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a DOM or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 310 real web, mobile, and desktop probes. Every probe is scored by deterministic forced choice, with no modelgenerated item and no model judge. Our central metric is the Perception-Fusion Gap pfg, the fraction of probes a model perceives correctly yet resolves toward structure under conflict. Across five models from three vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and pfg is positive for every model; non-text identity, by contrast, stays largely pixel-bound. The substitution is specific to the serialized-text and indexed-action channel, and coordinate-action agents are largely immune. For textual conflicts a white-box ablation traces the effect to a single copied structural value, and in two live environments the conflict drives wrong actions and real task failure. Visual state reliance therefore gives a measurable diagnostic of whether agent state beliefs are visually grounded, and the errors it exposes propagate to actions. Introduction Multimodal GUI agents now operate computers, browsers, and phones by consuming two redundant views of the same interface: the rendered pixels of a screenshot and a serialized structure such as a DOM or accessibility tree (Deng et al. 2023; Koh et al. 2024) . Before an agent can plan or act, it must form a belief about the current state of the interface: whether a toggle is on, which tab is selected, or what text afield contains. This belief precedes every action, and a wrong belief propagates into the decisions that follow even when the planner is otherwise competent. Existing benchmarks score the end of that loop, namely task completion, element grounding, or robustness to injected attacks (Koh et al. 2024; Cheng et al. 2024; Cao et al. 2025), and so leave the origin of the belief unmeasured. The two channels are usually consistent, yet they diverge often enough to matter: accessibility trees go stale, con- pfg: perceive correctly yet defer Figure 1: The core scenario, on a real probe from Multimodal-Mind2Web (Deng et al. 2023) . The same interface state reaches the agent through two channels: the rendered pixels P, a screenshot whose highlighted control reads Reservations, and the serialized structure S, a DOM node we edit to read Budget Truck. Read from the pixels alone the agent answers Reservations correctly, yet its fused belief follows the edited structure. We report the fraction of such perceive-correctly-yet-defer probes as the Perception– Fusion Gap pfg, our central metric. tent renders after the structure is captured, and localized strings disagree with displayed labels. Such divergence is documented at scale: human CLAY (Li et al. 2022) annotations mark 10.6% of RICO structure nodes as having no valid visual representation, and such a ghost node appears on 37.4% of screens. A model that reports the correct state by copying a structural string it never visually verified matches the screen only as long as the structure happens to agree, and it fails without warning once structure and reality diverge. This divergence motivates the question but does not by itself reveal which channel a belief follows, and provenance i

FAQs

What two channels do multimodal GUI agents use to understand an interface state?

They consume rendered screenshot pixels and a serialized structure such as a DOM or accessibility tree. The agent forms its interface-state belief before planning or acting based on these channels.

Why do existing GUI-agent benchmarks not fully answer whether beliefs come from pixels?

They typically score end results like task completion, element grounding, or robustness to attacks, without isolating where the belief underlying decisions originates. As a result, belief provenance is left unmeasured.

What is the Perception–Fusion Gap and what does it diagnose?

Perception–Fusion Gap is the fraction of probes where a model perceives the correct visible state yet resolves toward structure when the channels conflict. It provides a measurable diagnostic of whether state beliefs are visually grounded and how those grounding errors propagate to actions.

Jul 16, 2026

Upload by Ivy