What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge

Aug 30, 2026

Upload by Bintang

Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) aims to combine large language model reasoning with the structured evidence of knowledge graphs, but evaluation practice often fails to isolate genuine reasoning. Existing benchmarks frequently include questions answerable from a single existing triple, while inconsistent metrics and permissive matching obscure fair comparison. This work proposes BRINK, a benchmark with a protocol targeting reasoning under knowledge incompleteness by removing direct supporting facts and requiring inference through alternative graph paths. Experiments show limited reasoning under missing knowledge, reliance on memorization, and design-dependent generalization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Evaluation gaps in KG-RAG benchmarks

Related Work

Knowledge Graph Question Answering (KGQA)

BRINK Benchmarking for Reasoning under Incomplete Knowledge

Construction method using rule mining and grounding removal

Empirical Findings

What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge Dongzhuoran Zhou1,2 , Yuqicheng Zhu2,3 , Xiaxia Wang5 , Hongkuan Zhou2,3 , Yuan He4,5 , Jiaoyan Chen6 , Steffen Staab3,7 , Evgeny Kharlamov1,2 1University of Oslo, 2Bosch Center for AI, 3University of Stuttgart, 4Amazon, 5University of Oxford, 6The University of Manchester, 7University of Southampton [dongzhuoran.zhou@de.bosch.com](dongzhuoran.zhou@de.bosch.com) Abstract Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) is an increasingly explored approach for combining the reasoning capabilities of large language models with the structured evidence of knowledge graphs. However, current evaluation practices fall short: existing benchmarks often include questions that can be directly answered using existing triplesin KG, making it unclear whether models perform reasoning or simply retrieve answers directly. Moreover, inconsistent evaluation metrics and lenient answer matching criteria further obscure meaningful comparisons. In this work, we introduce a general method for constructing benchmarks and present BRINK (Benchmark for Reasoning under Incomplete Knowledge) to systematically assess KG-RAG methods under knowledge incompleteness. Our empirical results show that current KG-RAG methods have limited reasoning ability under missing knowledge, often rely on internal memorization, and exhibit varying degrees of generalization depending on their design. 1 Introduction Retrieval-Augmented Generation (RAG) has become a widely adopted framework for enhancing large language models (LLMs) by incorporating external knowledge through a retrieve-then-generate paradigm (Khandelwal et al., 2020 ; Izacard and Grave, 2021 ; Borgeaud et al., 2022 ; Ram et al., 2023 ; Zhu et al., 2025b) . By conditioning the generation on the retrieved documents, RAG enables LLMs to answer questions or perform tasks using more comprehensive and up-to-date knowledge than what is stored in their parameters. To improve retrieval accuracy, enable structured reasoning and support explanation, recent research has pivoted toward RAG methods based on Knowledge Graph (KG-RAG) (Han et al., 2024 ; Peng et al., 2024) . Most of them directly use existing knowledge graphs (KGs) (Luo et al., 2024 ; Sun et al., 2024 ; Zhou et al., 2025b), while some of them construct and extend structured knowledge from unstructured documents (Fang et al., 2024) . Such KG-RAG systems are expected to be well-suited for structured reasoning, where answers require synthesizing information from multiple connected facts. Despite growing interest, current evaluation practices for KG-RAG fall short in two key ways. First, most existing benchmarks (Yihet al., 2016 ; Talmor and Berant, 2018a) are constructed on top of complete KGs, where direct evidence supporting the answer is readily available. For example, given the question“Who is the brother of Justin Bieber?”, the KG contains the triple hasBrother(JustinBieber, JaxonBieber), allowing the system to answer the question without reasoning. However, realworld KGs are often incomplete, and answering such questions in practice may require reasoning over alternative paths, e.g., combining hasParent(JustinBieber, JeremyBieber) and hasChild(JeremyBieber, JaxonBieber) to infer the sibling relationship. As a result, current benchmarks do not assess whether KG-RAG methods can reason over missing knowledge or simply retrieve answers directly from explicit evidence. Second, evaluation protocols across KG-RAG studies lack standardization and rigor. We identify two pervasive issues: (1) ambiguous definitions, where metrics like “accuracy” fluctuate arbitrarily between Exact Match and permissive substring inclusion; and (2) implementation discrepancies, where official codebases frequently contradict their paper descriptions. For instance, several benchmarks describe using ranking-based metrics (e.g., Hits@1) but im

FAQs

Why do current KG-RAG benchmarks fail to measure reasoning reliably?

Many benchmarks are built on complete knowledge graphs where the answer can be supported by an explicit triple, allowing systems to retrieve answers without multi-step reasoning. In addition, evaluation metrics and answer-matching criteria are often inconsistent or overly lenient.

What is BRINK and what problem does it address?

BRINK is a benchmark for reasoning under incomplete knowledge. It is designed so each question cannot be answered using a single explicit triple and instead requires inferring the answer by reasoning over alternative paths in an incomplete KG.

How are BRINK questions constructed?

BRINK uses a two-step process: it mines high-confidence logical rules from the KG with a rule mining algorithm, then generates natural language questions from rule groundings while removing the directly supporting triple.

What do the experiments reveal about existing KG-RAG methods?

Most models perform poorly when direct supporting facts are removed, indicating limited reasoning capability under missing knowledge. Training-based methods show stronger robustness, and textual entity labels significantly improve performance, suggesting reliance on memorization.

Aug 30, 2026

Upload by Bintang