When Depth Is Better Told Than Shown Depth-Ordinal Prompting for Vision-Language Spatial Reasoning

Jul 16, 2026

Vision-language models are expected to reason about physical space, yet relative depth and 3D arrangement judgments remain weak even when depth maps are available. Providing depth as an image can worsen performance because frozen models struggle to access and regulate the rendered pseudodepth input. Depth-Ordinal Prompting (DOP) converts monocular depth into a targeted ordinal text cue for queried objects, without adding a depth image, training modules, feature injection, or labels. Across benchmarks, DOP improves spatial reasoning when pseudo-depth yields reliable object-level ordering and stays competitive with training-free depth prompting while remaining simpler.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Abstract

Introduction

Spatial reasoning limitations in VLMs

Why showing depth can hurt

Probing and causal evidence

Depth interface diagnosis

Proposed method: Depth-Ordinal Prompting (DOP)

When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning Quynh Vo Phuc Dao Cong-Duy Nguyen Thong Nguyen* National University of Singapore Center of AI Research, VinUniversity [thong.nguyen@u.nus.edu](thong.nguyen@u.nus.edu) arXiv :2607 . 11173v1 [ cs .CV] 13 Jul 2026 Abstract Vision-language models (VLMs) are expected to reason about physical space—which object is closer, what lies behind what, and how objects are arranged in 3D—yet they still struggle with such spatial judgments. A natural remedy is to show the model a depth map, but we find that this can make performance worse. We show that depth is not absent: it reaches the language model, but becomes difficult to access for downstream reasoning, while rendered pseudodepth maps act as noisy auxiliary images that frozen VLMs cannot easily regulate. We propose Depth-Ordinal Prompting (DOP), a training-free method that converts monocular depth into a single question-targeted ordinal text cue at the queried objects, without adding a depth image, training a module, injecting features, or using labels. Our key finding is form dependence: the same depth signal can hurt when shown as an image but help when told as text.Across benchmarks, models, and depth estimators, DOP improves spatial reasoning when pseudo-depth provides reliable object-level ordering and remains largely neutral in strong originalimage regimes. It is also competitive with the strongest training-free depth-prompting alternative while being simpler and more targeted. 1. Introduction Vision-language models (VLMs) are increasingly expected to reason about physical space: which object is closer, what lies behind what, and how objects are arranged in 3D. Such capabilities are central to robotics, embodied AI, and assistive perception [2, 3, 5] . Yet spatial reasoning remains a persistent weakness. Recent benchmarks show that even strong VLMs struggle with relative depth, direction, and objectlevel spatial relations [6, 8, 19, 22, 30] . A natural remedy is to provide geometry explicitly, for example by appending a monocular depth map to the original color image or by training models with depth-and 3D-aware supervision [2– *Corresponding author Figure 1 . Tell, don’t show. The same monocular depth signal can hurt when shown as an additional image but help when converted into a targeted ordinal text cue. DOP keeps the original color image and communicates only the queried-object depth relation, allowing the frozen VLM to weigh the cue against its original-image evidence. 5, 20, 37] . Surprisingly, we find that the simplest version of this remedy often makes performance worse: when pseudodepth is rendered as an additional image, frozen VLMs can degrade substantially on point-and object-level spatial reasoning. Fig. 1 previews this form contrast: rendering depth as another image can mislead the frozen VLM, whereas expressing the queried-object relation as a short text cue can make the same geometric information easier to use. This raises a basic question: if depth is useful, why does showing depth hurt? This contrast suggests that the issue is not simply the absence of depth, but the interface through which depth is made usable by a frozen VLM. We therefore ask whether the model fails because it never encodes depth, or because depth becomes difficult to use later in the computation. To answer this, we conduct layer-wise probing of the model’s hidden states, as summarized in Fig. 2. We find that early visual features contain a clear depth signal, and that this signal reaches the first language-model layers with little loss. However, as computation proceeds, the same signal becomes hard to read with a simple linear probe, even though a non-linear probe can still recover it. In other words, depth is not erased; it becomes entangled in a form that is less accessible to the model’s downstream reasoning. A causal intervention further supports this view: when

FAQs

Why can adding a depth map as an image make vision-language spatial reasoning worse?

When rendered as pseudodepth, the extra visual input is hard for frozen VLMs to align, interpret, and selectively ignore, leading to degraded object-level spatial judgments.

What is Depth-Ordinal Prompting (DOP)?

DOP is a training-free interface that reads monocular pseudo-depth at the queried object regions and converts it into a single ordinal text cue (e.g., which object appears closer), appending it to the original prompt without adding a depth image or labels.

What is the key finding behind DOP’s effectiveness?

The same depth signal can hurt when shown as an image but help when expressed as a targeted text cue, suggesting a form-dependent accessibility gap for downstream reasoning.

Jul 16, 2026

Upload by Seraphina