Scene-Graph ViT - End-to-End Open-Vocabulary Visual Relationship Detection

Nov 4, 2026

Visual relationship detection extracts object instances and the semantic relationships linking them, often requiring separate relationship modules or decoders that complicate optimization and limit end-to-end performance. An encoder-only, decoder-free Transformer approach is introduced for open-vocabulary VRD, representing objects as tokens and modeling relationships implicitly. A Relationship Attention mechanism selects likely object pairs, enabling a single-stage training recipe mixing object and relationship annotations. The method reports state-of-the-art results on Visual Genome and large-vocabulary GQA, with real-time inference plus ablations, qualitative examples, and zero-shot analyses.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Related Work

Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection Tim Salzmanny1 ;2, Markus Ryll2, Alex Bewley?1, and Matthias Minderer?1 1 Google DeepMind {tsal, bewley, [mjlm}@google.com](mjlm}@google.com) 2 Technical University Munich {tim.salzmann, [markus.ryll}@tum.de](markus.ryll}@tum.de) Figure 1: Relationships detected by our method on an unseen image. The top relationships by conﬁdence score are shown. Photo by Spacejoy on Unsplash. Abstract. Visual relationship detection aims to identify objects and their relationships in images. Prior methods approach this task by adding separate relationship modules or decoders to existing object detection architectures. This separation increases complexity and hinders end-to-end training, which limits performance. We propose a simple and highly efﬁcient decoder-free architecture for open-vocabulary visual relationship detection. Our model consists of a Transformer-based image encoder that represents objects as tokens and models their relationships implicitly. To extract relationship information, we introduce an attention mechanism that selects object pairs likely to form a relationship. We provide a single-stage recipe to train this model on a mixture of object and relationship detection data. Our approach achieves state-of-the-art relationship detection performance on Visual Genome and on the large-vocabulary GQA benchmark at real-time inference speeds. We provide ablations, real-world qualitative examples, and analyses of zero-shot performance. yWork done while at Google DeepMind.? Advising project leads in alphabetical order. 2 T. Salzmann et al. 1 Introduction A fundamental goal of computer vision is to decompose visual scenes into structured semantic representations. A commonly studied task towards this goal is object detection, in which objects in an image are localized by bounding boxes and classiﬁed into semantic categories. However, a scene description also includes the semantic relationships between objects. This gives rise to the task of visual relationship detection (VRD,[4, 6, 32, 48, 49, 55]) . In VRD, the model detects objects and infers pairwise relationships between them in the form of triplets3 [32] . Detecting both objects and their relationships allows the construction of a scene graph [19, 30, 48] in which objects are represented as nodes and their relationships as edges. Scene graph generation (SGG) has wide-ranging applications in robotics [1,5,11,17,38,42] and image retrieval [19,27] . Increasingly, structured scene representations are also employed to provide grounding and explainability to multimodal large language models [15, 36, 59] . Prior work typically draws a distinction between object detection and relationship prediction. Detection is performed either as a wholly separate step before relationship prediction [8, 32, 48, 52, 55, 57], or by separate model parts such as “relationship decoders” that are responsible for modeling the interactions between objects [7,49,61] . This distinction makes it hard to optimize such models end-to-end for VRD [61] . In contrast, we propose an encoder-only architecture that models objects and relationships jointly, directly in the image encoder. Our architecture performs open-vocabulary relationship detection and can be trained end-to-end on arbitrary mixtures of object detection and relationship annotations. Our model builds on a Transformer-based encoder-only object detector [34] in which the output tokens ofthe image encoder directly represent object proposals. From these tokens, class embeddings and bounding boxes are decoded with lightweight heads. Our insight is that this architecture is perfectly set up to learn relationships between objects directly in the image encoder, without the need for additional relationship-speciﬁc stages. This is because the existing self-attention in the encoder already models all-to-all pairwise interactions between th

FAQs

What limitation do prior visual relationship detection methods have for end-to-end training?

They often separate object detection from relationship prediction using extra modules or decoders, which increases complexity and makes end-to-end optimization for VRD difficult.

How does Scene-Graph ViT model relationships without a decoder for relationships?

It uses a Transformer-based encoder-only architecture where self-attention over object proposal tokens implicitly models all pairwise interactions, and then applies a Relationship Attention layer to obtain relationship embeddings for selected token pairs.

How are candidate subject-object pairs selected, and how is this trained efficiently?

A self-supervised hard attention mechanism selects highest-confidence subject-object pairs, reducing pairwise combinations. The attention scores are directly supervised without propagating gradients through the hard selection.

Nov 4, 2026

Uploaded by Dewi_Sukmawati