When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

Jul 16, 2026

Upload by Rowan

LLM-as-judge assumes that consistency—judge agreement or a model’s self-samples—indicates correctness, but agreement is not accuracy: models can concur due to shared bias, memorized heuristics, option-position priors, or systematic hallucination. A large cross-runner audit (265,000 samples across GPQA Diamond and AIME) finds agreement a positive yet weak predictor (ρ 0.20–0.59) that varies by regime, best for unsaturated mid-tier models and compute allocation, but worst for an over-confident frontier model. Deidentified per-run rows are released.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Setup

Data provenance

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals Kaihua Ding* University of Pennsylvania [dkaihua@upenn.edu](dkaihua@upenn.edu) arXiv :2607 .08065v 1 [ cs .AI] 9 Jul 2026 Abstract LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or ‘mixture-of-experts’(Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency—agreement among judges, or among a model’s own samples—indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with eachother, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME—265,000 samples. Using majoritycorrectness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (ρ 0.20– 0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst—over-confident yet no more accurate—for the most consistent frontier model (agreement ≥ 0.8 on 77% of GPQA case-result entries, 48% of those wrong) . An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginalpreserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the deidentified per-run rows and answer distributions. *The author’s prior work spans output-based error estimation (Ding, 2018) and AI-system evaluation and assessment design (Ding, 2025a) . 1 Introduction Large language models are increasingly used to evaluate AI systems—as LLM-as-judge, and in ensemble or multi-judge (‘mixture-of-experts’) panels now common in enterprise evaluation pipelines (Zheng et al., 2023) . These pipelines rest on a pervasive intuition: that agreement signals correctness—if several judges concur, or if a model returns the same answer across many stochastic samples, the answer is trusted, and if they disagree it is treated as a guess. But agreement-with-itself isnot the same as being right: a model can repeat an answer because it is genuinely certain, or because every sample passes through the same memorized heuristic, shared misconception, option-position prior, or systematic hallucination. This is the selfreferential analogue of the self-preference bias documented when models judge their own outputs (Zheng et al., 2023 ; Panickssery et al., 2024)—agreement with oneself can encode shared bias rather than correctness. Self-consistency decoding operationalizes this by returning the majority answer over K samples (Wang et al., 2023), building on chain-of-thought prompting (Wei et al., 2022) . The same agreement signal is now reused well beyond decoding—as a confidence estimate for costbased routing and cascades (Chen et al., 2023), for adaptive sample-budget allocation (Aggarwal et al., 2023), and for selective prediction and abstention (Geifman and El-Yaniv, 2017 ; Kamath et al., 2020) . That self-consistency raises accuracy via majority voting is well established, as is the broader finding that elicited LLM confidence is often miscalibrated and can even worsen after post-training (Kadavath et al., 2022 ; Lin et al., 2022 ; Tian et al., 2023 ; OpenAI, 2023) . Less systematically examined is how reliably the agreement signal itself functions as the confidence proxy that routing and abstention systems already assume it to be—and when it fails—under a controlled, cross-replicate

FAQs

What additional checks and resources does the study provide?

An option-shuffle control shows part of the “confidence” is positional, and a cross-family check across multiple Claude tiers indicates recurring confident errors shared across providers. The study also releases a deidentified, schema-validated dataset and analysis pipeline.

Jul 16, 2026

Upload by Rowan