PERSONAJUDGE Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data

Jul 16, 2026

Large language models increasingly act as judges in AI evaluation, yet common methods rely on consensus preferences that overlook how individual evaluators differ. PERSONAJUDGE introduces a simulation approach that uses evaluator-specific auxiliary signals—retrospective reasoning traces and interface telemetry—combined with categorical judgments to enable LLM-based individual simulation via in-context learning. A systematic study uses 32 trained annotators and 4,200 judgments in a 4×4×4 factorial design, showing up to 9.9-point gains over a base judge, with reasoning traces helping most and telemetry often reducing performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Abstract

Introduction

Related Work

PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data Zeyu He1* , Xuan Qi2 , Subramanian Chidambaram2 , Zhichao Xu2 , Vinayak Arannil2 , Lydia Chilton2,3 , Alex C. Williams2 1Pennsylvania State University, 2AWS AI Fundamental Research, 3 Columbia University Correspondence: [zeyuhe@psu.edu](zeyuhe@psu.edu) arXiv :2607 .05742v 1 [ cs .HC] 7 Jul 2026 Abstract Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data—retrospective reasoning traces and interface telemetry—to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 × 4 × 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator’s neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency—rather than simulatability itself—is the cross-task-stable property (r = 0 .728) . These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individualaware AI assessment. 1 Introduction Large Language Models (LLMs) are increasingly used as judges for model comparison, reward modeling, and benchmark construction because they offer a scalable alternative to human evaluation (Bai et al., 2022b ; Liu et al., 2023 ; Zheng et al., 2023 ; Lambert et al., 2025) . However, most LLM-asJudge pipelines target an aggregate signal: models are trained, aligned, or evaluated against pooled * Work completed during ZH’s Amazon internship. Figure 1: PERSONAJUDGE workflow. A human evaluator first completes judgment tasks, producing three complementary signals: a categorical judgment, interface telemetry, and retrospective reasoning. PERSONAJUDGE organizes these signals into evaluator-specific demonstrations and combines them with the task instruction and a new task instance. An LLM then uses this information to simulate the target evaluator’s judgment. preferences that collapse across annotators (Stiennon et al., 2020 ; Ouyang et al., 2022 ; Bai et al., 2022a ; Zhang et al., 2025), so they simulate crowdlevel consensus rather than any specific evaluator. This is a key limitation for open-ended evaluation, where a single shared standard may not exist and annotators apply different criteria to the same outputs (Van der Lee et al., 2021 ; Davani et al., 2022) . In such settings disagreement is often meaningful rather than noise (Basile et al., 2021 ; Fleisiget al., 2023 ; Frenda et al., 2025) . Even LLM judges that match average ratings can miss group- and person-level differences (Movva et al., 2024) . Current models struggle to infer individual preferences from value-laden statements (Jiang et al., 2025) . Matching consensus thus does not guarantee faithful simulation of any individual evaluator. This motivates a different target: individual evaluator simulation, which can support evaluation auditing, per-person reward modeling, and fairness analysis of whose perspectives evaluation pipelines underrepresent. Prior personalized methods condition on persona descriptions or compact preference profiles (Dong et al., 2024b ; Wang et al., 2024a), but represent people through summaries rather than their own decision traces. We instead ask: given an evaluator’s historical judgm

FAQs

What problem does PERSONAJUDGE address in LLM-as-judge evaluation?

Most pipelines target aggregate consensus, which can fail to reflect individual evaluator criteria in open-ended settings where annotators disagree meaningfully. PERSONAJUDGE targets individual evaluator simulation instead of crowd-level consensus.

How does PERSONAJUDGE simulate an individual evaluator’s judgments?

It constructs evaluator-specific demonstrations from multiple signals: a categorical judgment, interface telemetry, and retrospective reasoning. These are combined with task instructions and a new instance so an LLM can simulate the target evaluator via in-context learning.

Which auxiliary data source improves simulation the most, and which can hurt performance?

Retrospective reasoning traces provide the largest gains, especially with higher collection effort. Interface telemetry often hurts performance despite being cheaper to collect, according to the empirical findings.

Jul 16, 2026

Upload by Theodore