Knowledge-augmented Few-shot Visual Relation Detection - arXiv 2303.05342v1

Aug 22, 2026

Upload by Rizky

Visual Relation Detection (VRD) seeks to recognize Subject–Predicate–Object relationships to support image understanding. Existing VRD systems often require thousands of labeled samples per relationship, while few-shot approaches still suffer from limited generalization due to the large semantic diversity and compositional variability of visual relations. The proposed knowledge-augmented few-shot VRD framework integrates textual knowledge from a pre-trained language model and visual relation knowledge from an automatically built knowledge graph, improving coverage and transfer. Experiments on three Visual Genome benchmarks show substantial gains over state-of-the-art models.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Visual Relation Detection and the Few-shot Challenge

Knowledge-augmented Few-shot VRD Framework

Textual Knowledge

Visual Relation Knowledge

arXiv :2303 .05342v1 [ cs .CV] 9 Mar 2023 Knowledge-augmented Few-shot Visual Relation Detection Tianyu Yu 1 , Yangning Li 1 , Jiaoyan Chen2 , Yinghui Li 1 , Hai-Tao Zhengy1, Xi Cheny3, Qingbin Liu3 , Wenqiang Liu3 , Dongxiao Huang3 , Bei Wu3 , and Yexin Wang3 1 Shenzhen International Graduate School, Tsinghua University, Shenzhen, China 2Department of Computer Science, The University of Manchester, Mancester, UK 3Tencent, Shenzhen, China [yiranytianyu@gmail.com](yiranytianyu@gmail.com) Abstract Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. Some recent papers tackle this problem by few-shot learning with elaborately designed pipelines and pre-trained word vectors. However, the performance of existing few-shot VRD models is severely hampered by the poor generalization capability, as they struggle to handle the vast semantic diversity of visual relationships. Nonetheless, humans have the ability to learn new relationships with just few examples based on their knowledge. Inspired by this, we devisea knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD. The textual knowledge and visual relation knowledge are acquired from a pre-trained language model and an automatically constructed visual relation knowledge graph, respectively. We extensively validate the effectiveness of our framework. Experiments conducted on three benchmarks from the commonly used Visual Genome dataset show that our performance surpasses existing state-of-the-art models with a large improvement. 1. Introduction Visual Relation Detection (VRD) targets at the detection of relations, i.e. Subject-Predicate-Object triplets, which capture a wide variety of interactions (relationships) between objects in images. For example, the input image in Figure 1 shows the eating relationship between the y Corresponding authors: [zheng.haitao@sz.tsinghua.edu.cn](zheng.haitao@sz.tsinghua.edu.cn), jasonx[chen@tencent.com](chen@tencent.com) Figure 1: Leveraging external knowledge to augment fewshot VRD. Due to the large semantic diversity of eating, a model trained with the few-shot examples, such as horse-eating-grass or child-eating-hotdog cannot be easily generalized to new examples, such as dog-eating-apple. While external knowledge can be leveraged to improve the model's generalization ability. For instance, textual knowledge from large textual corpora can describe meanings of dog, apple, and eating. Visual relation knowledge learnt from the visual relation knowledge graph can show common relationships between objects, such as the fact that eating is a common relationship between home pets (e.g., dog, rabbits) and fruits (e.g., apple) . dog and the apple. The detected relations provide a deep and comprehensive understanding of the semantic content of images and have facilitated state-of-the-art models in numerous applications in computer vision such as visual question answering[18, 39], image retrieval [19], and image captioning [45, 23] . Most existing VRD methods [29, 22, 10] rely on a large amount of training data for each relationship to achieve satisfactory performance. However, the distribution of relationships in real-world images is extremely long-tail since many relationships are inherently uncommon. Speciﬁcally, 92.3% of relationships in the Visual Genome dataset [21] have only ten or fewer samples (e.g., packing, spouting), while wearing has about 12,400 times more samples. As a result, these methods can only handle those frequent relationships which have many labeled samples. In order to enable VRD with limited samples, some recent works try to solve this task in the few-shot setting with either visual relation detection pre-training on frequent relatio

FAQs

What problem does Visual Relation Detection (VRD) address?

VRD detects relationships in images in the form of Subject–Predicate–Object triplets, enabling deeper semantic understanding of visual interactions.

Why do existing few-shot VRD models generalize poorly?

They are constrained by the poor generalization capability caused by the vast semantic diversity and compositional nature of visual relationships.

How does the proposed framework improve few-shot VRD generalization?

It augments few-shot VRD with textual knowledge from a pre-trained language model and visual relation knowledge learned from an automatically constructed visual relation knowledge graph.

What evidence is provided to validate the effectiveness of the framework?

Experiments on three benchmarks from the Visual Genome dataset show performance surpassing existing state-of-the-art models with a large improvement.

Aug 22, 2026

Upload by Rizky