Evaluating RAG Metrics in Applied Contexts An Experiment Its Findings and Its Limitations

Jul 16, 2026

Upload by Quinn

Empirical evaluation of relevance-focused RAG metrics is conducted using a French business-domain question answering dataset built by human annotators. A RAG system’s generated responses and retrieved spans are scored with metrics from four libraries: Ragas, DeepEval, RAGChecker, and Opik, then compared against human evaluator judgments and recall for retrieval quality. Correlations are analyzed, methodology limitations are discussed, and comparisons to prior literature motivate future research directions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Introduction

Context and Data

The Question-Answering Dataset

Methodology

Results and Analysis

Limitations

Comparison With Literature

Future Research Directions

arXiv :2607 .07302v 1 [ cs .CL] 8 Jul 2026 Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations Quentin Brabant Orange Research, Lannion, France [quentin. brabant@orange. com](quentin. brabant@orange. com) Abstract This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) . These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM 2026 (Brabant, 2026) . 1 Introduction Evaluating and comparing RAG (Retrieval Augmented Generation) systems remains a challenging task today: even when a sufficiently large set of test questions with reference answers is available, automatically evaluating a system’s responses against these references is far from trivial. A popular approach is to use so-called LLMas-a-judge metrics to perform this evaluation. Although these metrics generally seem more relevant than classical metrics such as BLEU, it is difficult to know in advance what the relevance of a specific metric will be on the dataset under consideration, especially since evaluation criteria can vary (relevance, factuality, completeness of responses, etc.) . It is therefore useful, when evaluating a RAG system under development, to conduct an evaluation of available metrics in order to verify that they provide acceptable approximations of the criterion considered, and that they will thus enable reliable comparison of different iterations of the RAG system being developed. Generally, metrics are evaluated by measuring their correlation with scores given by humans. This article reports an experiment of this type. This experiment is based on a question-answering dataset created by annotators from business data. The responses produced and the documents retrieved by a RAG system are scored using RAG metrics from four libraries: Ragas 1 (Es et al. , 2024), DeepEval2 , RAGChecker3 (Ru et al. , 2024), and Opik4 . These metrics are compared to reference evaluations: human evaluations for assessing generated responses, and recall for evaluating retrieval. Note that our objective is not to compare the different metrics evaluated, asthe study results are strongly dependent on the choices made during its design. The reported experiments rather aim to apply a methodology in order to test its advantages and limitations. Unfortunately, the question-answering dataset used cannot be made public; however, we share the raw scores and the code used for the statistical analyses5 . The article is organized as follows. Section 2 describes the application context and the data used, including manual annotation and evaluation processes. Section 3 describes the methodology applied to compare reference evaluations with evaluations produced by metrics from the tested libraries. Section 4 reports and analyzes the results of this experiment. Certain limitations of our methodology are highlighted in Section 5 . In Section 6, our methodology is compared to those employed in the literature. We finally propose some research perspectives aimed at facilitating the application of reliable methodologies for evaluating RAG metrics, in Section 7 . 2 Context and Data Our company is developing a RAG solution designed to answer questions in French about a business domain. The developed system processes each given question via two key modules: first, a retriever, whose role is to retrieve releva

FAQs

What does the experiment evaluate in RAG systems?

It evaluates the relevance of multiple RAG metrics applied to a question-answering task, scoring both generated responses and retrieved spans.

How is the dataset created for the study?

A question-answering dataset is built from business documents and includes 96 questions, each linked to reference answers and one or more reference spans annotated by company employees.

How are metric outputs validated in the experiment?

Metric scores are compared to reference evaluations from two evaluators for response quality and to recall for retrieval evaluation, followed by correlation analysis.

Jul 16, 2026

Upload by Quinn