A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

Jul 16, 2026

Upload by Riley

Empirical reliability is assessed for Gemini models used as audio judges that score full-duplex agent conversations directly from raw stereo waveforms. Evidence is built on Gemini 2.5 Flash as a ground-truth reference, validated against three calibrated human raters over 209 stereo sessions. Ratings cover eight production dimensions across 152 conversations spanning accent-and-condition strata, plus 57 adversarial defect-injected clips, with agreement and sensitivity analyzed per dimension to support substitution where reliability holds.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Background

Voice-agent audio judges

Study purpose

Related Work

LALM-as-judge for speech

Full-duplex evaluation

Reliability versus validity

Study Design

Dataset and dimensions

Human annotation protocol

LALM scoring approach

arXiv :2607 .07985v 1 [ cs .CL] 8 Jul 2026 A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents A. Sayyad, J. Emmons, S. Jones, T. Lin, H. Krishnan Salesforce Applied AI Research, e Verse team Abstract We report the empirical reliability of Gemini models as audio judges that score full-duplex agent conversations directly from the raw stereo waveform, tested across three models in the Gemini family: 2.5 Flash, 3.5 Flash, and 3.1 Pro. Our primary evidence base uses Gemini 2.5 Flash as the ground-truth model, validated against three calibrated human raters on 209 stereo sessions, scored on 8 production dimensions: 152 full-duplex conversations across 13 accent-and-condition strata, together with 57 adversarial defect-injected clips. The evidence for Gemini 2.5 Flash is consistent across three tests. (i) On 5 of 8 dimensions the LALM-human Spearman ρ departs from the pairwise human-human ρ by at most 0.07, and on 7 of 8 dimensions the two quantities’ 95 percent bootstrap confidence intervals overlap. (ii) The LALM agrees with the three-rater human mean within 1 point on 60 to 92 percent of sessions on 6 of 8 dimensions. (iii) On 45 of 48 (defect, dimension) cells the LALM is as sensitive as humans or better under Newcombe-Wilson 95 percent confidence intervals, though most of these are underpowered nulls rather than demonstrated parity. Rank-ordering ability transfers across the Gemini family: 3.5 Flash improves simple agreement to 8 of 8 dimensions, while 3.1 Pro rates several dimensions markedly lower than humans despite comparable rank correlation. A model swap should be re-validated on calibration specifically, not assumed from rank-correlation alone. We identify four areas where deployment requires care, and we estimate that human rating alone for our current evaluation cadence costs roughly two orders of magnitude more than the equivalent LALM workload. The data presented here provides a defensible empirical basis for deploying the LALMas a substitute or fourth rater on the dimensions where the evidence supports it. Keywords: LALM-as-judge, audio language models, validation, voice agents, production deployment, full-duplex audio. 1 Background Our voice-agent product surface relies on two production audio judges, AgentSpeechFidelity and ConversationalAudioQuality, which between them emit ratings on eight dimensions for every evaluated session. To date these judges have been validated informally against single raters on small samples. This study addresses a single question: whether LALM-generated ratings agree with human ratings closely enough to substitute for one or more human raters in a production audioevaluation pipeline. We frame substitutability as an empirical, per-dimension property rather thana global one, and quantify it against a three-rater human reference standard. Recent work in the LALM-as-judge literature reports strong correlations between LALM judges and human Mean Opinion Score (MOS) on isolated text-to-speech utterances [1, 2, 3] . To our knowledge, this is the first study to evaluate a LALM as a judge of raw audio from enterprise fullduplex agent conversations against a multi-rater human reference standard. We do so over stereo agent-client conversations where channel content, turn-taking dynamics, and conversational disfluencies all influence the rating distribution. We extend this line of work to full-duplex conversational audio using a production evaluation stack. 2 Related Work LALM-as-judge for speech. AudioJudge [1] judges speech attributes (pronunciation, rate, quality) with high system-level human correlation; ALLD [2] produces descriptive MOS judgements; and SpeechQualityLLM [3] predicts dimension-wise MOS. Broad benchmarks such as AIR-Bench [10] score free-form outputs with a text LLM rather than validating an audio judge against humans. All operate on isolated utterances, and none reports multi-rater human validation over full-duplex convers

FAQs

What does the study evaluate about LALM audio judges?

It evaluates whether LALM-generated ratings for full-duplex, raw stereo agent-client conversations agree with human ratings closely enough to substitute for one or more human raters in an evaluation pipeline.

How was reliability evidence collected and validated?

Gemini 2.5 Flash is used as the ground-truth model and is validated against three calibrated human raters across 209 stereo sessions. Ratings are computed on eight production dimensions, including clean accent strata and codec-degraded conditions, plus adversarial defect-injected clips.

Does agreement transfer across different Gemini models?

Rank-ordering ability transfers across the Gemini family, but model swaps require re-validation. The study reports different behavior across models: one model improves agreement across all dimensions, while another rates some dimensions markedly lower than humans.

Jul 16, 2026

Upload by Riley