Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification

Jul 16, 2026

Upload by Skyler

Growing image availability increases the need for effective organization, yet labeling large-scale datasets is costly and time-consuming, motivating semi-supervised learning that exploits labeled and unlabeled samples. Graph Convolutional Networks (GCNs) can propagate label information on data graphs, but image graphs lack predefined structure and are usually built from noisy visual feature proximity. This work refines kNN and reciprocal kNN edges by using a VLM to generate textual descriptions and an LLM to score semantic similarity for pruning. Experiments on Corel5k show improved classification accuracy and release accompanying code.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Proposed Approach

Dataset and Feature Extraction

Graph Modeling

arXiv :2607 .09104v1 [ cs .CV] 10 Jul 2026 Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification Camila Piscioneri Magalhães 1 , Lucas Pascotti Valem 1 1 Institute of Mathematics and Computer Science (ICMC) University of São Paulo (USP) São Carlos – SP – Brazil [piscioneri@usp.br](piscioneri@usp.br) , [lucas@icmc.usp.br](lucas@icmc.usp.br) Preprint. Accepted at the Workshop de Trabalhos de Alunos de Graduação (WTAG) of the Brazilian Symposium on Databases (SBBD 2026) . Abstract. While the growing availability of image data has driven significant advances, labeling datasets remains costly and time-consuming. Therefore, semi-supervised approaches such as Graph Convolutional Networks (GCNs), which learn from both labeled and unlabeled data, have emerged as a promis ing solution. One of the primary challenges in applying GCNs to image clas sification is graph construction, since, unlike in citation networks or similar domains, images typically do not come with a predefined structural represen tation. For visual data, most studies construct graphs based on the similar ity between feature vectors from pretrained deep learning backbones, typically by employing kNN or reciprocal kNN algorithms. Although Large Language Models (LLMs) have shown remarkable capability in capturing high-level semantics, their integration with GCNs for image classification remains under explored. Aiming to fill this gap, our approach uses a Vision Language Model (VLM) to generate textual image descriptions, which are then processed by an LLM to estimate semantic similarity scores between connected images. These scores guide the pruning of edges in kNN and reciprocal kNN graphs, filtering out semantically irrelevant neighbors. Experimental results reveal that leveraging LLMs for graph refinement can improve classification accuracy, particularly for kNN graphs and some backbones. The source code is publicly available at [gcnllm.lucasvalem.com](gcnllm.lucasvalem.com). 1. Introduction The volume of image data being produced and stored has grown dramatically in recent years, driven by applications in areas such as healthcare, security, and social media [Uelwer et al. 2025, Ghosh et al. 2024] . Organizing, indexing, and classifying these large image collections is a central problem for modern data management systems. While collecting images has become inexpensive, labeling them remains costly and timeconsuming, as it typically depends on specialized human effort [Uelwer et al. 2025] . This limitation motivates methods that learn effectively from collections in which only a small fraction of the samples is labeled. In this scenario, semi-supervised methods, which learn jointly from labeled and unlabeled data, have emerged as a promising direction. Among them, Graph Convolutional Networks (GCNs) [Wu et al. 2019] stand out by representing samples as nodes of a graph and propagating label information through the connections between them, allowing the few available labels to guide the classification of the unlabeled ones [Müller et al. 2024] . A key obstacle when applying GCNs to image classification is that, unlike citation networks and other naturally graph-structured domains, image collections usually do not come with a predefined graph: it must be constructed from the data itself. Most approaches build this graph from the similarity between feature vectors extracted by pretrained deep networks, typically using kNN or reciprocal kNN graphs [Müller et al. 2024] . However, how to model an effective graph remains an open research challenge, since proximity in the visual feature space does not always reflect semantic relationships and may introduce noisy connections that degrade classification. Recently, Large Language Models (LLMs) have shown a remarkable ability to capture high-level semantics from text [Li et al. 2024, Ghosh et al. 2024], yet their use to support graph construction for

FAQs

Why is semi-supervised image classification needed in this work?

Labeling images at scale is expensive and time-consuming, so the work focuses on methods that learn from both labeled and unlabeled data to reduce labeling requirements.

What is the main challenge in applying GCNs to image classification?

Unlike citation networks, image collections do not come with an inherent graph structure, so the graph must be constructed from the data and naive similarity connections can be noisy.

How do the proposed VLM and LLM improve the graph used by the GCN?

A Vision Language Model generates textual descriptions for images, and an LLM estimates semantic similarity between connected images. These scores prune edges in kNN and reciprocal kNN graphs to remove semantically irrelevant neighbors.

Jul 16, 2026

Upload by Skyler