EPC A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems

Jul 16, 2026

Upload by Logic

EPC (Evaluator Preference Coupling) introduces a standardized, RFC-style protocol for measuring how evaluator biases propagate through LLM agent closed-loop adaptation via evaluator feedback. The protocol specifies a four-phase isolation paradigm, including executor/evaluator configuration, text and visual strategy/task design, the TTRL update rule, and metric computation (γ, JSD, ECE, Brier) plus an output schema. It is paired with a timebound aversioned reference snapshot that version-tags coupling measurements across evaluator conditions and documents expected measurement decay as evaluators update.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Protocol Specification

Overview

Agent Configuration

TTRL Algorithm

arXiv :2607 .00297v 1 [ cs .LG] 1 Jul 2026 EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems Anonymous authors Paper under double-blind review Abstract When LLM agents use evaluator feedback to adapt their behavior in closed loops, evaluator biases propagate through the agent’s strategy distribution—a phenomenon known as evaluator preference coupling. Prior work has documented coupling across multiple evaluator families and model versions, but the field lacks a standardized protocol that enables third-party researchers to (i) reproduce coupling measurements,(ii) compare results across evaluators and time points, and (iii) detect measurement decay as proprietary evaluators silently update. This paper provides the protocol. We specify EPC (Evaluator Preference Coupling)—a detailed, RFC-style protocol specification for the four-phase isolation paradigm, covering executor and evaluator configuration, strategy and task design, the TTRL update rule, metric computation (γ, JSD, ECE, Brier), and output schema. We accompany the protocol with aversioned Reference Snapshot v1.0: coupling measurements for eight evaluator conditions (N = 122 unique experimental repetitions across GPT-4o, Qwen, DeepSeek, and others) derived from five independent studies, annotated with evaluator version identifiers, API endpoints, and measurement dates. The snapshot is explicitly timebound: all values are conditional on specific model versions and are expected to decay as proprietary evaluators update. We define a versioning convention (vX .Y-Z , encoding protocol version, snapshot version, and evaluator generation) and provide a usage guide covering adoption, interpretation, and known pitfalls. The protocol, reference snapshot, and implementation code are released as open infrastructure. 1 Introduction Evaluator-driven preference dynamics have been documented across multiple LLM agent configurations Liu (2026a;b;c) . In the standard setup, an agent maintains a strategy weight distribution, receives pairwise feedback from an evaluator, and adapts via test-time reinforcement learning (TTRL) . The coupling coefficient γ and Jensen-Shannon divergence (JSD) quantify how strongly the evaluator’s preferences transfer across task domains and how concentrated the agent’s strategy distribution becomes. However, the field currently operates without a standardized protocol. Each study uses slightly different protocol variants, task sets, strategy definitions, and metric implementations. Crossstudy comparison is impossible. More critically, proprietary evaluators silently update, causing measurements to decay within weeks Liu (2026a) . Without versioned baselines and explicit expiration dates, the literature accumulates measurements that are no longer valid for current model versions. This problem is not unique to evaluator coupling: a recent audit of 26 AI benchmarks found that the median benchmark has a longevity score of just 5 out of 100 BenchRisk (2026), and the ML community is shifting toward continuous, versioned, community-governed evaluation infrastructure MLCommons (2026); SWE-rebench (2025); HF Community (2026) . This paper provides the protocol, the reference snapshot, and the versioning convention. This paper is not a claim of new empirical findings. It is a protocol specification paper—analogous to an RFC in the networking community or a measurement standard in the physical sciences. The coupling measurements in the reference snapshot have been previously reported in domain-specific studies Liu (2026a) . Our contribution is the standardization, versioning, and community infrastructure that transforms these measurements from one-off observations into a reproducible, comparable, and auditable measurement system. We explicitly do not introduce new metrics, new experimental conditions, or new scientific claims. We introduce a discipline—a protocol that enables the community to collectively ma

FAQs

What problem does EPC address in evaluator-driven LLM agent experiments?

Evaluator preference coupling causes evaluator biases to propagate through the agent’s strategy distribution during closed-loop adaptation. EPC standardizes how to measure this coupling so results can be reproduced and compared across evaluators and time points.

How does the EPC protocol measure evaluator preference coupling?

It uses a four-phase isolation paradigm: pure-text, pure-visual, T→V coupling, and V→T coupling. The coupling coefficient γA→B quantifies how the agent’s weight distribution shifts relative to the pure-domain reference.

Why does the reference snapshot emphasize versioning and measurement decay?

Proprietary evaluators can silently update, making earlier coupling measurements no longer valid for current evaluator and model versions. EPC therefore requires explicit evaluator identity recording and timebound, version-tagged baselines to keep results auditable and current.

Jul 16, 2026

Upload by Logic