SynthAVE Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

Jul 16, 2026

Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires large, multilingual labeled datasets covering thousands of product types and attributes. Human annotation at this combinatorial scale is prohibitively expensive. SynthAVE introduces a large-scale human-validated benchmark with 12,726 products, 229 product categories, 792 attributes, and 4 languages. Synthetic label quality is validated using a multi-LLM arena: 21 judge configurations (7 model families × 3 prompts) with majority voting, matching human experts at Cohen’s κ = 0.92.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Abstract

Introduction

Related Work

SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation Andrea Scarinci1 , Virginia Negri1 , Brayan Impata1 , Suleiman Khan1 , Victor Martinez1 , Marcello Federico1 1Amazon {andscar, vrgngr, biimpata, suleimkh, vicmg, [marcfede}@amazon.com](marcfede}@amazon.com) arXiv :2607 .07469v 1 [ cs .CL] 8 Jul 2026 Abstract Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs (Negri et al., 2025), deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German) . To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families × 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohen’s κ = 0 .92 (95.2% agreement), while individual judges show substantial inter-model agreement (Fleiss’ κ = 0 .76) . This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review. 1 Introduction Fine-tuning and evaluating large language models (LLMs) for e-commerce applications demands massive volumes of high-quality labeled data. Specifically, predicting and evaluating product attribute values from unstructured catalog text (e.g., product titles, descriptions, bullet points) requires labeled examples that maintain representativeness across three key dimensions: product categories (e.g., electronics, clothing, furniture), attributes (e.g., color, material, dimensions), and languages. At catalog scale—thousands of product categories, thousands of attributes, and multiple languages—achieving reliable model performance requires hundreds of labeled examples per (product category × attribute × language) triplet. This combinatorial requirement translates to millions of annotations, making human labeling prohibitively costly. Prior work demonstrates the feasibility of using LLMs to generate synthetic labels for e-commerce attribute value extraction (Negri et al., 2025) . However, deploying such approaches at industrial scale introduces a critical challenge: how to validate label quality efficiently across heterogeneous attributes and languages without exhaustive manual review. A scalable solution must integrate synthetic label generation with automated quality control mechanisms. This paper presents SynthAVE (Synthetic data for Attribute Value Extraction), a large-scale human-validated benchmark augmenting prior generation methodology (Negri et al., 2025) with scalable quality assurance. We introduce a multi-LLM auditing framework where each generated label is independently evaluated by 21 judge configurations (7 model families × 3 prompts), with final labels determined through majority voting. To mitigate systematic biases, we employ diverse models from different providers alongside varied prompting strategies. Ground truth for calibrating this system is established through human validation of 12,726 products. Empirical evaluation demonstrates that diverse models with different biases aggregate into highly reliable predictions: the ensemble achieves 95.2% agreement with human experts (Cohen’s κ = 0 .92) while enabling cost-effective validation at scale. Our main contributions are: • SynthAVE: A human-validated benchmark of 12,726 products spanning 229 pr

FAQs

Why is human labeling expensive for e-commerce attribute extraction?

Because reliable performance requires many labeled examples across combinations of product categories, attributes, and languages, which produces millions of annotations.

What is SynthAVE and what scale does it cover?

SynthAVE is a human-validated benchmark for attribute value extraction covering 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German).

How does the LLM-Arena framework validate synthetic labels at scale?

Each synthetic label is evaluated independently by 21 judge configurations (7 model families × 3 prompts), and final labels are selected via majority voting, reaching 95.2% agreement with human experts.

Jul 16, 2026

Upload by Seraphina