JADE Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

Jul 16, 2026

Upload by Finn

Evaluating agentic AI on open-ended professional tasks faces a core trade-off between rigor and flexibility. Static rubrics ensure reproducibility but cannot cover diverse valid strategies, while LLM-based judging adapts to individual answers yet risks instability and bias. JADE addresses this with a two-layer framework: Layer 1 encodes expert knowledge as reusable evaluation skills for stable criteria, and Layer 2 performs claim-level, report-specific verification with evidence-dependency gating to invalidate refuted claims. Experiments on BizBench show improved stability and uncovered agent failure modes, with transfer to HealthBench and DR.BENCH.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

Abstract

Introduction

Stability–Adaptivity Dilemma

JADE Two-Layer Evaluation Framework

Main Contributions

JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks Lanbo Lin * 1 Jiayao Liu * 1 Tianyuan Yang * 1 Li Cai 1 2 Yuanwu Xu 1 Lei Wei 1 3 Sicong Xie 1 Guannan Zhang 1 arXiv :2602 .06486v 3 [ cs .AI] 13 Jul 2026 Abstract Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-asa-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domaingrounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at [https:](https:)//[github.com/smiling-world/JADE](github.com/smiling-world/JADE). 1. Introduction The evolution of Large Language Models (LLMs) from chat-bots to reasoning engines has catalyzed the rise of autonomous agentic systems (Naveed et al., 2025 ; Wanget al., 2024a) . By integrating multi-step reasoning with tooluse capabilities, these agents are increasingly deployed in *Equal contribution 1Alibaba International Digital Commerce Group 2Zhejiang University 3Peking University. Correspondence to: Lanbo Lin <[linlanbo.llb@alibaba-inc.com](linlanbo.llb@alibaba-inc.com) > . Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . complex, long-horizon professional workflows (Luo et al., 2025 ; Jimenez et al., 2024) . As capabilities advance, robust evaluation (Yehudai et al., 2025 ; Du et al., 2026) becomes critical for guiding research and ensuring reliability in highstakes applications. Most existing agent benchmarks focus on tasks with verifiable outcomes, such as multi-hop web search (Wei et al., 2025 ; Zhou et al., 2025b ; Chen et al., 2025b), code generation (Jimenez et al., 2024 ; Quan et al., 2025), or toolmediated reasoning (Yao et al., 2025a ; Wang et al., 2026 ; Barres et al., 2025), enabling objective and reproducible evaluation. However, such proxy tasks capture only a limited subset of real-world deployments. In real-world applications, agents must solve open-ended problems involving strategic analysis, evidence synthesis, and decision-making under evolving conditions (e.g., strategic procurement, market analysis, and professional consulting) . These tasks rarely admit a single “gold” answer, making realistic evaluation an unresolved challenge. For example, a query for “FDA-certified suppliers for stainless steel tumblers with low shipping cost to the USA” admits multiple valid response strategies, whose quality cannot be reliably assessed without consistent professional principles and real-time evidence verification. Recent benchmarks (Du et al., 2026 ; Yao et al., 2025b) explore expert-authored rubrics and LLM-generated checkliststo approximate professional judgment. While promising, these approaches expose a fundamental dilemma: (i) Static checklists ensure stability but lack adaptivity to diverse valid solutions (Arora et al., 2025 ; Ruan et al., 2026 ; Zhu et al., 2025); (ii) LLM-as-a-judge methods (Zheng et al., 2023 ; Du et al., 2026) provide adaptivity through direct scoring or LLM-generate

FAQs

What problem does JADE address in evaluating agentic AI for open-ended professional tasks?

JADE targets the stability–adaptivity dilemma: static rubrics lack flexibility for multiple valid strategies, while LLM-as-judge approaches can be unstable and biased without expert-grounded, claim-level principles.

How does JADE achieve both stable evaluation and adaptive flexibility?

JADE uses two layers: Layer 1 encodes expert knowledge as predefined evaluation skills to provide stable criteria, and Layer 2 performs report-specific, claim-level evaluation with evidence-dependency gating to reject conclusions built on refuted claims.

What benchmarks and domains are used to validate JADE?

Experiments use BizBench, and the method is further demonstrated with alignment to expert-authored rubrics and transfer to HealthBench and DR.BENCH, covering medical and multiple-domain professional evaluation settings.

Jul 16, 2026

Upload by Finn