Exposing Contextual Amnesia in Vision–Language Segmentation - A Diagnostic Benchmark of Compositional, Relational, and Absence-Aware Referring Expressions

Aug 26, 2026

Uploaded by Sarah

Vision–language segmentation models are increasingly deployed as if they understand referring expressions like humans, but evaluation on photographic benchmarks can hide prompt parsing failures behind lexical shortcuts. The study isolates a failure mode called contextual amnesia, showing systematic collapse on relational reasoning, multi-target selection, instruction negation, and empty-target rejection despite solid attribute grounding. A controlled synthetic benchmark with 10,000 images and 52,791 objects evaluates six open-source pipelines across five linguistic regimes using 105 metrics, revealing large attribute-versus-relational gaps and near-total IoU collapse for negation and empty targets, alongside strong synthetic-curriculum training and targeted adaptation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Contextual Amnesia and Failure Characterization

Synthetic Diagnostic Benchmark Design

Dataset Composition and Rendering Setup

Evaluation Protocol and Metrics

Experimental Results Across Linguistic Regimes

Attribute vs Relational Prompt Analysis

Negation and Empty-Target Failure Analysis

Model Training, Adaptation, and Ensemble Methods

Synthetic Curriculum and From-Scratch Training

Targeted Adapters and Late-Fusion Ensemble

Exposing Contextual Amnesia in Vision–Language Segmentation: A Diagnostic Benchmark of Compositional, Relational, and Absence-Aware Referring Expressions Yağız Ekrem Dalar Ethosoft [https://orcid.org/0009-0009-3522-4403](https://orcid.org/0009-0009-3522-4403) Nedim Mutlu Sezer Ethosoft [https://orcid.org/0009-0001-3499-6067](https://orcid.org/0009-0001-3499-6067) Ömer Faruk Aksoy Ethosoft [https://orcid.org/0009-0003-3557-9358](https://orcid.org/0009-0003-3557-9358) Feyzi Arda Salihoğlu Ethosoft [https://orcid.org/0009-0006-5841-1664](https://orcid.org/0009-0006-5841-1664) Ahmet Rıfat Öztürk Ethosoft [https://orcid.org/0009-0001-2941-0412](https://orcid.org/0009-0001-2941-0412) Research Article Keywords: referring expression segmentation, vision–language models, synthetic benchmarks, compositional generalisation, relational reasoning, CLIP, SAM, contextual amnesia Posted Date: August 4th, 2026 DOI: [https://doi.org/10.21203/rs.3.rs-10558897/v1](https://doi.org/10.21203/rs.3.rs-10558897/v1) License: 􀁱 􀅏 This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License Additional Declarations: The authors declare no competing interests. Exposing Contextual Amnesia in Vision–Language Segmentation: A Diagnostic Benchmark of Compositional, Relational, and Absence-Aware Referring Expressions Research Article / Ara¸stırma Makalesi AUTHOR INFORMATION WITHHELD FOR DOUBLE-BLIND PEER REVIEW (Yazar bilgileri çift-kör hakemlik gere˘gi gizlenmi¸stir; kabul sonrası eklenecektir.) (Geli¸s/Received: –.–.20XX; Kabul/Accepted: –.–.20XX) DOI: 10.17671/gazibtd.XXXXXXX Abstract—Vision–language segmentation models are increasingly deployed as if they understood referring expressions the way humans do. Yet their evaluation is dominated by photographic benchmarks where lexical shortcuts can mask whether a model has parsed the prompt. We isolate, measure, and characterise a failure mode we call contextual amnesia: the systematic collapse of state-of-the-art referring segmentation systems on prompts requiring relational reasoning, multi-target selection, instruction negation, or empty-target rejection, despite competent attribute grounding. We design a controlled synthetic benchmark of 10000 images and 52791 rendered objects, evaluate six open-source pipelines across ﬁve linguistic regimes on 105 metrics, and report a 4–6× gap between attribute and relational prompts and a complete 0 .00 IoU collapse on negation and empty-target prompts for all detector-cascade pipelines. Three ≈9M-parameter randomly-initialised architectures trained from scratch surpass the strongest billion-parameter cascade by +24 to +25 IoU points. Five targeted adaptation techniques, and a late-fusion ensemble of two specialists, reach 0.687 IoU—2 .00× the strongest zero-shot pipeline. Keywords—referring expression segmentation, vision–language models, synthetic benchmarks, compositional generalisation, relational reasoning, CLIP, SAM, contextual amnesia. Highlights—• A controlled, fully-synthetic referring-segmentation benchmark (10000 images, 52791 objects, ﬁve linguistic regimes) . • All six tested zero-shot pipelines collapse 4–6× on relational prompts and to 0 .00 IoU on negation / empty-target prompts.• A 9M-parameter network trained from scratch on the synthetic curriculum surpasses every billion-parameter pipeline by +24 IoU.• Five targeted adapters and a two-specialist ensemble lift SigLIP+SAM to 0 .687 IoU, 2.00× the strongest zero-shot pipeline. Görü–Dil Bölütleme Modellerinde Ba˘glamsal Amnezi: Bile¸simsel, ˙Ili¸skisel ve Yokluk-Farkında Gönderim ˙Ifadeleri için Tanılayıcı Bir Kar¸sıla¸stırma Kümesi Özet—Görü–dil bölütleme modelleri, gönderim ifadelerini insanlar gibi anladı˘gı varsayılarak giderek yaygın biçimde konu¸slandırılmaktadır. Ancak de˘gerlendirmeleri, sözcüksel kısayolların—nadir nesne kategorileri, kendine özgü görsel dokular, veri kümesine özgü önyargılar—modelin i

FAQs

What problem does the paper identify in referring expression segmentation models?

It identifies contextual amnesia: a systematic collapse when prompts require relational reasoning, multi-target selection, instruction negation, or empty-target rejection, even if attribute grounding remains competent.

How is the diagnostic benchmark constructed to avoid shortcut-based evaluation?

It uses a controlled, fully synthetic benchmark with 10,000 images and 52,791 rendered objects, evaluated across multiple linguistic regimes and measured with 105 metrics.

What performance patterns are reported between attribute-focused and relational prompts?

The results report a 4–6× gap between attribute and relational prompts, and detector-cascade pipelines show a complete near-zero IoU collapse on negation and empty-target prompts.

Which approaches improve results, according to the paper?

From-scratch training of small randomly initialized architectures on the synthetic curriculum outperforms stronger billion-parameter cascades, and targeted adaptation plus a late-fusion ensemble further improves IoU.

Aug 26, 2026

Uploaded by Sarah