AeroRAG Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning

Jul 16, 2026

Upload by Himbo

AeroRAG targets reliable visual question answering in aerial imagery where critical evidence is often carried by small objects, explicit quantities, coarse locations, and inter-object relations. Conventional dense visual-token representations lack alignment with these structured semantics. The method introduces a scene-graph-guided multimodal retrieval-augmented generation framework: it converts images into structured visual knowledge and retrieves query-relevant semantic chunks to build compact prompts. Experiments on AUG and VG-150 show consistent gains over strong MLLM baselines, with larger improvements in dense and relation-sensitive scenes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

I. Introduction

Motivation: limitations of existing MLLMs in aerial VQA

Problem framing: interface mismatch in perception-to-reasoning

Proposed approach: AeroRAG framework

AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning Junxiao Xue 1†, Quan Deng2 , 1†, Tingqi Hu3 , Meicong Si3 , Xinyi Yin3 , Yunyun Shi4 , and Xuecheng Wu4∗‡ arXiv :2604 . 17889v2 [ cs .CV] 12 Jul 2026 Abstract—Recent progress in multimodal large language models (MLLMs) has yet to resolve the challenge of reliable visual question answering in aerial imagery. In such scenes, task-critical evidence is often carried by small objects, explicit quantities, coarse locations, and inter-object relations, whereas conventional dense visual-token representations are not well aligned with these structured semantics. To address this interface mismatch, we propose AeroRAG, a scene-graphguided multimodal retrieval-augmented generation framework for visual question answering. The framework first convertsan input image into structured visual knowledge, including object categories, quantities, spatial locations, and semantic relations, and then retrieves query-relevant semantic chunks to construct compact prompts for a text-based large language model. Rather than relying on direct reasoning over dense visual tokens, our method introduces a more explicit intermediate interface between perception and language reasoning. Experiments on the AUG aerial dataset and the generaldomain VG-150 benchmark show consistent improvements over six strong MLLM baselines, with the largest gains observed in dense aerial scenes and relation-sensitive reasoning. We further evaluate the framework on VQAv2 to demonstrate that the proposed interface remains compatible with standard visual reasoning settings. These results suggest that structured retrieval is a practical design direction for deployment-oriented and grounded visual reasoning systems. I. INTRODUCTION Multimodal large language models (MLLMs) have become essential frameworks for merging visual and linguistic understanding, fueled by various advanced models [1]–[3] . These approaches connect pre-trained vision encoders with large language models (LLMs), achieving outstanding performance in general visual question answering (VQA) [4],[5] and image captioning [6] . At the same time, there is increasing interest in deploying such models in mission critical visual analysis scenarios, especially in aerospace *This work was supported by the Central Government Guiding Local Science and Technology Development Fund under Grant No. 2026ZY04001 . 1Research Center for Space Computing System, Zhejiang Lab, Hangzhou 311100, China (E-mail: [xuejx@zhejianglab.cn](xuejx@zhejianglab.cn)); 2Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 311000, China (E-mail: [dengquan23@mails.ucas.ac.cn](dengquan23@mails.ucas.ac.cn)); 3 School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China (E-mail: {htqiserendipity, smc ggjxb, [yinxinyi](yinxinyi}@stu.zzu.edu.cn)[}](yinxinyi}@stu.zzu.edu.cn)[@stu.zzu.edu.cn](yinxinyi}@stu.zzu.edu.cn)). 4 School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China (E-mail: [yunyunshi@stu.xjtu.edu.cn](yunyunshi@stu.xjtu.edu.cn); [wuxc3@ieee.org](wuxc3@ieee.org)); Work done during Quan Deng’s research internship at Zhejiang Lab. ∗ Corresponding author: Xuecheng Wu. †Equal contributions. ‡Project lead. Fig. 1. The qualitative comparison on challenging aerial VQA examples. Existing MLLMs often miss small objects or confuse counts and spatial relations, whereas our proposed multimodal RAG framework provides more accurate and visually grounded answers. and aerial imagery applications such as remote sensing, environmental monitoring, urban planning, and autonomous inspection. In these settings, the requirement is not only strong perception, but also reliable and grounded reasoning over complex scenes. However, directly applying existing MLLMs to aerial VQA remains challenging. As illustrated in Fig. 1, even advanced models such as GPT-4o

FAQs

What problem does AeroRAG address in aerial visual question answering?

AeroRAG addresses unreliable visual QA caused by small objects, counting, coarse localization, and relation reasoning that conventional MLLMs struggle with in aerial scenes.

How does AeroRAG differ from directly prompting multimodal LLMs with dense visual tokens?

AeroRAG converts the image into scene-graph-based structured knowledge (categories, quantities, locations, relations) and then performs query-conditioned retrieval to build compact text prompts.

Which datasets and evaluation setups are used to demonstrate the effectiveness of AeroRAG?

Experiments are conducted on the AUG aerial dataset and the VG-150 benchmark, and compatibility with standard visual reasoning settings is evaluated using VQAv2.

Jul 16, 2026

Upload by Himbo