JuryProbe - An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification

Sep 15, 2026

Model-based evaluation systems increasingly rely on panels of inexpensive LLM judges to make accept-or-escalate decisions for factuality. In reference-free settings, majority agreement can hide dangerous consensus failures when shared false-negative blind spots drive coincident errors. JuryProbe introduces an empirical consensus-risk diagnostic paired with a calibration-based routing policy that estimates panel-level risk and activates grounded verification when high risk is detected. Experiments on frozen FEVER claim pools and additional audited corruption families quantify correlated false negatives and show that routed verification eliminates unanimous false consensus in trusted-reference best case.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Consensus risk in reference-free judge panels

JuryProbe diagnostic and calibration-based routing

Evaluation and results

Audited FEVER corruptions and additional claim families

Routed policy vs grounded verification

Negative control and stress tests

arXiv : 2608 .20607v2 [cs .CL] 27 Aug 2026 JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification Tianxin Zhou [zhoutx0@gmail. com](zhoutx0@gmail. com) Independent Researcher Ruixi Lin [ruix.lin@northeastern. edu](ruix.lin@northeastern. edu) Independent Researcher Reviewed on OpenReview: [https: // openreview. net/ forum? id= dgBczhxcZY](https: // openreview. net/ forum? id= dgBczhxcZY) Abstract Model-based evaluation systems increasingly use panels of inexpensive LLM judges to make accept-or-escalate decisions. In factuality settings, accepting a claim because several reference-free judges agree can create a hidden risk: agreement may arise from shared false-negative blind spots rather than independent evidence. We introduce JuryProbe, an empirical consensus-risk diagnostic for reference-free factuality judge panels, paired with a calibration-based routing policy. JuryProbe estimates panel-level consensus risk from a labeled calibration probe using false-negative-only (FN-only) judge correlation and falseconsensus lift; when a panel is flagged high-risk, reference-free majority accepts are routed to the same judge panel with trusted references. Using frozen FEVER-supported claim pools, fixed-seed construction, and audited Number and Entity corruption families, we show that reference-free panels exhibit substantial correlated false negatives (FN-only correlations of 0.402 and 0.368; false-consensus lifts of 3.13 × and 18.13×) . Under a trusted-reference bestcase diagnostic, unanimous false consensus drops to zero both on minimal-pair FEVER corruptions and on non-minimal-pair scientific evidence. In flagged settings, the routed policy is, by construction, equivalent to grounding every reference-free majority accept (verified in 34/34 flagged splits) . Its improvement therefore comes from accept-conditioned grounding, while the diagnostic’s operational role is to determine whether to activate this policy. A fixed, pre-specified rule produces both labels out of sample: it flags synthetic, benchmark-authored, and scientific claim families in 8–10 of 10 splits each, while standing down on the negative control in 0 of 10 splits. There, standing down avoids grounding roughly 28% of deployment claims at a 0 .004 absolute increase in false-accept rate. Stress tests with BM25-retrieved references and distribution-shifted deployment streams show that false-accept reduction persists under weak retrieval, while true-accept coverage degrades and stale stand-down labels require periodic labeled recalibration. JuryProbe provides no formal risk guarantee, nor do we establish reliable stand-down on natural panels; the supported contribution is an empirical diagnostic of high-risk panel error dependence. 1 Introduction In many model-evaluation pipelines, the most consequential action is not producing a score, but deciding whether an output should be accepted without further verification. A cheap reference-free judge panel makes this decision attractive: several models can be queried quickly, and majority or unanimous agreement appears to provide redundancy. Yet this deployment logic relies on an assumption that is rarely tested: agreement is informative only when judge errors are sufficiently independent. If the judges share the same false-negative blind spots, the panel may accept a corrupted claim precisely when it appears most confident. In this setting, agreement is not merely noisy evidence; it can become the event that triggers unsafe acceptance. This paper studies that failure mode as a measurement-and-routing problem. We introduce JuryProbe, an empirical consensus-risk diagnostic with calibration-based routing for reference-free factuality judge panels. Prior work has established LLM-as-a-judge evaluation as a practical paradigm (Zheng et al., 2023; Wanget al., 2024; Zhu et al., 2025), and recent work has explored replacing single judges with p

FAQs

What failure mode does JuryProbe target in reference-free factuality judging?

It targets the risk that reference-free judges can agree on corrupted claims due to shared false-negative blind spots, making agreement an unsafe trigger rather than reliable evidence.

How does JuryProbe estimate consensus risk for a judge panel?

It estimates panel-level consensus risk from a labeled calibration probe using FN-only judge correlation and false-consensus lift, combining them into a risk regime rather than a per-sample classifier.

What does the calibration-based routing policy do when the panel is flagged high-risk?

When calibration indicates a high-risk consensus regime, routed accept decisions are sent to the same judge panel with trusted references to activate grounded verification for the relevant majority accepts.

Sep 15, 2026

Uploaded by Clementine