Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

Jul 16, 2026

LLM-as-a-judge evaluation is widely used for retrieval-augmented generation (RAG), yet reusing the same model family as both generator and judge makes self-leniency hard to diagnose. Eval-Pair Matrix presents a controlled metaevaluation protocol for source-grounded RAG. It induces hidden answer-causal contradictions from GaRAGe grounding, generates answers from perturbed passages using GPT, Grok, and Gemini, and has the same models judge against original evidence. Results on validated records show minimal same-model F1 and recall differences, with a limited robust paired gap related to answers avoiding an induced claim; targeted human review finds no genuine false alarms.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Introduction

Eval-Pair Matrix pipeline

Experimental setup and matrix design

Paired vs diagonal/off-diagonal analysis

Results and human evaluation

Methodological lessons

Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG Sriram Selvam [selvamsriram@gmail.com](selvamsriram@gmail.com) Anneswa Ghosh [anneswaghosh@gmail.com](anneswaghosh@gmail.com) arXiv :2607 . 10626v 1 [ cs .CL] 12 Jul 2026 Abstract LLM-as-a-judge evaluation is widely used for retrieval-augmented generation (RAG), but reusing the same model family as both generator and judge makes self-leniency difficult to identify. We introduce Eval-Pair Matrix, a controlled metaevaluation protocol for source-grounded RAG. Starting from GaRAGe questions and grounding passages, we induce one hidden answer-causal contradiction per record, generate answers from perturbed passages with GPT, Grok, and Gemini models, and then use the same models as blind judges to evaluate each answer against the original passages. The experiment contains 300 core records, 897 labeled generator outputs, and 2,683 judge verdicts in a crossed 3 × 3 matrix; the primary analysis uses 275 fully validated records. Instead of comparing diagonal and off-diagonal cells across different answers, we estimate same-model effects by pairing judges on the exact same candidate answer. This changes the interpretation: diagonal and offdiagonal F1 are similar, and the paired same-model recall effect is near zero (−0 .5 pp; 95% clusterbootstrap CI [−2 .7 , +1 .7]) . The only robust paired gap is lower matching-judge flagging for answers that avoided the induced claim (−4 .3 pp) . A targeted human evaluation finds that reviewed apparent false positives are alternate source-error detections, mistakes in labeling whether the induced claim was adopted, or unclear cases; none were adjudicated as genuine false alarms. The lesson is methodological: RAG judge studies should report full matrices, answer-paired effects, behavior strata, and label-task alignment. 1 Introduction Retrieval-augmented generation (RAG) systems combine a language model’s parametric knowledge with external passages supplied at inference time (Lewis et al., 2020) . This makes them attractive for knowledge-intensive applications, but it also creates a difficult evaluation problem: an answer maybe fluent, cited, and still contradict the reference passages. In practice, many RAG evaluations now use another LLM as the judge because LLM judges are cheap, fast, and can emit structured rationales (Liu et al., 2023 ; Zheng et al., 2023 ; Kim et al., 2023) . When the generator and judge are from the same model family, evaluation may become circular. Existing LLM-as-a-judge work documents position, verbosity, length, style, familiarity, and selfpreference effects (Dubois et al., 2024 ; Shi et al., 2024 ; Liu et al., 2024 ; Wataoka et al., 2024 ; Ye et al., 2024 ; Spiliopoulou et al., 2025) . This literature also shows that self-preference is an identification problem: a judge may favor its own model’s output because it is better, clearer, more familiar, or more stylistically aligned, rather than because the judge is lenient toward its own model. We ask how this problem changes in source-grounded RAG. If we induce a known source-relative contradiction hidden from the judge, then the judge is not choosing a favorite answer; it is checking whether a candidate answer contradicts the original evidence. We build the task from GaRAGe, a benchmark with human-written answers and passage-level grounding annotations (Sorodoc et al., 2025) . For each selected example, an LLM perturber chooses one answer-causal target proposition and rewrites every passage that states or entails that claim so the evidence consistently supports a plausible but false replacement value. Generators answer the original question using the perturbed grounding. Judges then evaluate the answer against the unmodified grounding only. Figure 1 summarizes the end-toend design. A raw diagonal analysis compares each judge’s matching-model cell with that judge’s two offdiagonal generator cells. That contrast is useful

FAQs

What problem does Eval-Pair Matrix address in LLM-as-a-judge RAG evaluation?

It addresses the difficulty of detecting self-leniency when the same model family is used for both the generator and the judge, where the judge may favor outputs that are more familiar or aligned.

How does Eval-Pair Matrix induce and control evaluation contradictions?

Starting from GaRAGe questions and grounded passages, it induces one hidden answer-causal contradiction per record by perturbing passages so the evidence supports a plausible but false replacement, then generates answers from these perturbed passages.

What does the answer-paired analysis change compared with diagonal vs off-diagonal matrix comparisons?

Instead of comparing different answers across diagonal and off-diagonal cells, it pairs judges on the exact same candidate answer to estimate same-model effects, which alters the interpretation of the matrix outcomes.

What do the experimental and human evaluation findings indicate about false alarms?

Automatic metrics show small same-model effects for error-detection recall, and targeted human review of apparent false positives concludes none were genuine false alarms; they correspond to source-error detection differences, labeling mistakes, or unclear cases.

Jul 16, 2026

Upload by Evangeline