3D-DefectBench A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

Jul 16, 2026

Upload by Sage

Automated evaluation is essential for scaling generative 3D systems, yet judge reliability depends on the entire evaluation pipeline rather than the vision-language model alone. A full pipeline includes rendering choices, provided visual evidence, task specification, and reference-label construction. 3D-DefectBench introduces a rigorous large-scale benchmark with nine fine-grained binary defect types across geometry, texture, and prompt adherence, and studies pipeline factors via balanced factorial designs over 84 inference setups.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages43

Reading Time1 h 48 min

Introduction

Evaluation pipelines as a measurement problem

Reference label regimes (silver vs. expert)

Limitations of holistic or preference-only signals

arXiv :2607 . 10826v1 [ cs .CV] 12 Jul 2026 3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects Zhenyu Zhao 1,†, Nanshan Jia 1,2,* , Jihyeon Je 1,3,* , Yifu Tang 1,2,* , Alvin Chan 1 , Michael Spedden 1 , Michael V. Palleschi 1 , Sui Huang 1 , Jingshen Wang4 , Zeyu Zheng2 1Roblox Corporation 2Berkeley AI Research Lab & Department of Industrial Engineering and Operations Research, University of California, Berkeley 3Computer Science Department, Stanford University 4Division of Biostatistics, University of California, Berkeley Abstract Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of an automated judge depends on the full evaluation pipeline—not only the underlying vision-language model (VLM), but also how the asset is rendered, which visual evidence is provided, how the task is specified, and how human reference labels are constructed. We introduce 3D-DefectBench, a large-scale instantiation of a methodology for rigorous evaluation-pipeline analysis. The benchmark complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and prompt adherence, providing actionable diagnostics for generator development and detailed visibility into judge behavior. Using a balanced factorial design, we vary the VLM, camera protocol, visual input, and prompt schema across 84 inference designs, then use a cost-aware staged study to validate the resulting design conclusions on a broader frontier-model set. Model choice is the dominant source of variation in agreement with human labels, but the remaining pipeline factors also influence agreement, interact with the model, and can alter the best configuration for a given judge. Within the tested design space, a compact six-view RGB protocol performs comparably to denser view sets and configurations augmented with depth or normal channels, making it a strong cost-effective default. Under this fixed design, the best of 12 VLMs still trail trained human labelers, while texture agreement drops sharply when moving from expert-agreement labels to noisier silver labels. These results show that automated judges should be evaluated as complete pipelines and calibrated across human reference regimes, rather than benchmarked solely as standalone models. We release labels, prompts, predictions, and Croissant metadata on HuggingFace. Keywords vision-language models · 3D generation · evaluation · LLM-as-judge · benchmark 1 Introduction Evaluating generative 3D systems at scale is difficult. Both the input and the output are open ended: a text prompt can describe an effectively unbounded set of objects, while the resulting textured mesh may vary in geometry, appearance, composition, and adherence to the prompt. In production and model-development settings, evaluation may cover large volumes of generations, repeated checkpoint comparisons, online failure analysis, and even annotation of training data for evaluators or downstream models. Exhaustive human review at this scale is costly and slow, motivating the use of vision-language models (VLMs) as automated judges that inspect multi-view renders together with the generation prompt [Duggal et al., 2025, Wu et al., 2024b, Zhang et al., 2025b] . However, evaluating VLM judges is fundamentally a measurement problem, not simply a model benchmarking problem. Using a VLM as a judge involves more than choosing a VLM model. Human labelers can inspect a 3D asset interactively by rotating and zooming the mesh, whereas a VLM receives a constructed representation of that asset. Its * This work was done while Nanshan Jia, Jihyeon Je, and Yifu Tang were interns at Roblox Corporation. † Correspondence: [zzhao@roblox.com](zzhao@roblox.com). agreement with human labels may therefore depend not only on the VLM’s capability, but

FAQs

Why must automated 3D evaluation treat a VLM judge as a full pipeline rather than a standalone model?

A VLM receives a constructed representation, so agreement with human labels depends on rendering, visual channels, task/output formatting, and reference-label construction. Different settings can confound model quality with the evaluation setup itself.

What new benchmark is introduced, and what kinds of defects does it measure?

3D-DefectBench is a large-scale benchmark for rigorous evaluation-pipeline analysis. It uses nine fine-grained binary defects covering geometry, texture, and prompt adherence to provide actionable diagnostics for generator development.

How does the study validate conclusions across model and pipeline variations?

It applies a balanced factorial design that varies the VLM, camera protocol, visual input, and prompt schema across 84 inference designs, then uses a cost-aware staged study to validate findings on a broader frontier-model set.

Jul 16, 2026

Upload by Sage