Promptable Concept Segmentation from Above Evaluating SAM 3’s Zero-Shot and One-Shot Capabilities in Remote Sensing

Jul 16, 2026

Deployment of large-scale foundation models such as Segment Anything Model 3 (SAM 3) enables open-vocabulary, training-free computer vision, yet their ability to generalize to out-of-distribution Earth Observation imagery remains insufficiently measured. A multi-task, empirical evaluation is conducted for remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. A structural adaptation converts SAM 3’s decoupled presence head into a standalone zero-shot classifier and isolates prompt modalities to diagnose crossmodal alignment. Results show severe crossmodal interference, training-free proxy evaluation, strong harmonic-mean segmentation performance, and remaining limits tied to resolution and semantic blind spots, motivating parameter-efficient geospatial fine-tuning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Introduction

Promptable Concept Segmentation (PCS) and SAM 3

Domain gaps in Earth Observation imagery

Limitations of supervised fine-tuning

Study objective and evaluation under zero fine-tuning

Promptable Concept Segmentation from Above: Evaluating SAM 3’s Zero-Shot and One-Shot Capabilities in Remote Sensing Mohammad Dabaja, Turgay Celik arXiv :2607 .09583v1 [ cs .CV] 10 Jul 2026 Abstract—The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3’s performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model’s multimodal decoder. Our findings reveal severe crossmodal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder. Index Terms—Cross-Modal Alignment, Earth Observation, Generalized Zero-Shot Learning, Scene Classification, Instance Segmentation, Open-Vocabulary Object Detection, Promptable Concept Segmentation, Remote Sensing, Segment Anything with Concepts (SAM 3), Vision-Language Models. I. INTRODUCTION THE emergence of massive vision-language foundation models has transitioned the computer vision field away from dataset-specific, closed-set architectures toward openvocabulary architectures. Central to this evolution is the concept of promptable segmentation, introduced by the Segment Anything Model (SAM) [1] . SAM shifted from class-specific segmentation to class-agnostic segmentation guided by spatial prompts, a capability subsequently extended to temporal video tracking by SAM 2 [2] . The deployment of SAM 3 [3] advances this class-agnostic foundation toward a unified task defined as Promptable Concept Segmentation (PCS) . By integrating a multimodal decoder and a decoupled presence Both authors are affiliated with the Department of ICT, University of Agder, Grimstad, Norway ([e-mail: mohammadkd@uia.no](e-mail: mohammadkd@uia.no); [turgay.celik@uia.no](turgay.celik@uia.no)). head, SAM 3 separates global semantic recognition from localized spatial bounding. This decoupled architecture unifies zero-shot conceptual reasoning with precise mask generation. However, initial benchmark data reveal a critical limitation. While SAM 3 achieves 56.4 AP on general-domain datasets like COCO, its zero-shot performance drops to 15.2 AP on the RF-100VL benchmark [3], which comprises 100 specialized visual domains. This performance gap indicates that while SAM 3 acts as an effective generalist, its direct application to highly specialized fields, such as Earth Observation (EO) and Remote Sensing (RS), requires objective evaluation to determine if costly domain-specific fine-tuning is necessary. The direct application of general-domain foundation models to remote sensing image

FAQs

What problem does the study target when evaluating SAM 3 for remote sensing?

It targets SAM 3’s poorly quantified ability to generalize out-of-distribution to Earth Observation imagery, where top-down geometry and semantic differences create cross-modal alignment failures.

How is SAM 3 structurally adapted to support zero-shot evaluation?

The presence head is repurposed into a standalone zero-shot classifier, separating global semantic recognition from localized spatial mask generation.

What main finding emerges from isolating textual versus visual prompt modalities?

Visual prompts align the decoder to complex remote sensing geometry, while textual prompts introduce misaligned ground-level semantic bias that degrades coordinate regression via crossmodal interference.

Jul 16, 2026

Upload by Valentina