[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273692-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273692-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","oenobench-a-wine-domain-benchmark-for-knowledge-grounded-evaluation-of-large-language-models","OenoBench - A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models","","OenoBench introduces a wine-domain knowledge benchmark with 3,266 multiple-choice questions across six pillars—regions, grape varieties, viticulture, winemaking, producers, and business—and four difficulty tiers. The corpus is built from 38,104 provenance-verified, source-anchored facts extracted by scrapers from government registries, peer-reviewed journals, and Wikipedia/Wikidata. A fact-grounded LLM pipeline reformats and audits claims with URL-level traceability, while a nine-agent evaluation calibrated to a human gold sheet via Cohen’s κ measures accuracy, reasoning lift, self-preference, and cost-efficiency. ",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/oenobench-a-wine-domain-benchmark-for-knowledge-grounded-evaluation-of-large-language-models/273692/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/oenobench-a-wine-domain-benchmark-for-knowledge-grounded-evaluation-of-large-language-models/273692.png","ImageObject",300,407,{"name":42,"@type":43},"Sage","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-27","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is OenoBench and what does it measure?","Question",{"text":63,"@type":64},"OenoBench is a wine-domain benchmark that evaluates large language models using 3,266 multiple-choice questions spanning six knowledge pillars and four difficulty tiers. It measures performance on knowledge-grounded questions under controlled construction and audit conditions.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How is the benchmark’s knowledge grounded and traceable?",{"text":68,"@type":64},"The dataset is built from 38,104 atomic, provenance-verified facts extracted from structured sources such as government registries, peer-reviewed journals, and Wikipedia/Wikidata. Every question and claim traces back to a URL, and the LLM pipeline reformats verified facts without treating the model as the source of truth.",{"name":70,"@type":61,"acceptedAnswer":71},"How are questions generated and audited to reduce bias or leakage?",{"text":72,"@type":64},"Questions are generated using five strategies across five generator families, then assessed by a nine-agent audit calibrated against a human gold sheet using Cohen’s κ. The audit also reports removed items and difficulty relabeling, including a closed-book solvability signal that helps isolate parametric recall from contextual reasoning.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273692,1789472506,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},687197207057,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","arXiv : 2608 .20 106v 1 [cs .CL] 20 Aug 2026  \nOenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models  \nNikita Khudov, DipWSET  \nStrategAI  \n[nikitahudov@gmail.com](nikitahudov@gmail.com)  \nAbstract  \nWe introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built from  \n38,104 atomic, source-anchored facts extracted by 35 provenance-verified scrapers from government registries (INAO, TTB, OIV), peer-reviewed journals, and Wikipedia/Wikidata. Our methodological contribution is an LLM-driven pipeline in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine-agent audit calibrated against a human gold sheet via Cohen’s κ . Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%–84%, led by o3 at 83.6% ; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8 pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows +9 pp selfpreference on its own questions while Google shows −8 pp inverse preference;  \n(iv) frontier open-weight models share the cost-vs-accuracy Pareto frontier with proprietary reasoning models; and (v) every config gains ≈33 pp on closed-book solvable items, revealing a parametric-recall ceiling that only the contextual slice avoids. We release corpus, audit findings, human-review app, and construction code under CC-BY-SA-4 .0.  \n1 Introduction  \nLarge language models are increasingly evaluated on knowledge-intensive benchmarks, but the most widely-used resources—MMLU [6], GPQA [16], and their successors [19, 21, 15]—share three weaknesses: they aggregate breadth at the expense of depth within any single discipline; they are typically authored by a small team using one generation pipeline, exposing systematic blind-spots and stylistic regularities that frontier models can learn to exploit [14, 20]; and the questions are increasingly suspected of contamination through web-scale pre-training corpora [17, 13, 3] . As models grow more capable, the marginal value of a benchmark depends increasingly on how it was constructed.  \nWe argue that domain-specialised benchmarks built around bodies of knowledge that are externally validated by humans—through certification, regulation, or peer-reviewed practice—offer a useful complement to broad knowledge benchmarks. Examples already exist in medicine [9, 10], law [5], and finance [8] . We extend this line of work to a domain that has so far received little attention from the benchmarking community, despite being unusually well-defined: wine.  \nWhy wine. Wine integrates plant biology, geology and climate science, organic chemistry, food microbiology, sensory neuroscience, regulatory law, and business economics. It is one of very few  \nPreprint.  \ndomains for which inter-governmental and governmental sources publish structured taxonomic data (INAO, OIV, TTB, regional consortia), university research groups (UC Davis, Geisenheim, Bordeaux Sciences Agro) produce open scholarly literature, and professional certification bodies (WSET Levels 1–4, Court of Master Sommeliers, Institute of Masters of Wine) define a graded competence ladder. These properties make it possible to build a benchmark whose every question is traceable to a verifiable source while still spanning the recall–reasoning–application spectrum that distinguishes a strong taster, winemaker, or buyer from a memorised glossary.  \nContributions.  \n1. A 3,266-question wine benchmark across six pillars and four difficulty tiers, built from  \n38,104 atomic provenance-tagged facts (Section 3) .  \n2. An LLM-driven, fact-grounded generation pipeline with five strategies × five g","cbCainbVy5VZxuDC","https://ap.wps.com/l/cbCainbVy5VZxuDC","pdf",1248631,29,"English","# Abstract\n# 1 Introduction\n# 2 Related Work\n## Broad knowledge benchmarks\n## Domain-specific benchmarks\n## LLM-as-judge bias and self-preference","[{\"question\":\"What is OenoBench and what does it measure?\",\"answer\":\"OenoBench is a wine-domain benchmark that evaluates large language models using 3,266 multiple-choice questions spanning six knowledge pillars and four difficulty tiers. It measures performance on knowledge-grounded questions under controlled construction and audit conditions.\"},{\"question\":\"How is the benchmark’s knowledge grounded and traceable?\",\"answer\":\"The dataset is built from 38,104 atomic, provenance-verified facts extracted from structured sources such as government registries, peer-reviewed journals, and Wikipedia/Wikidata. Every question and claim traces back to a URL, and the LLM pipeline reformats verified facts without treating the model as the source of truth.\"},{\"question\":\"How are questions generated and audited to reduce bias or leakage?\",\"answer\":\"Questions are generated using five strategies across five generator families, then assessed by a nine-agent audit calibrated against a human gold sheet using Cohen’s κ. The audit also reports removed items and difficulty relabeling, including a closed-book solvability signal that helps isolate parametric recall from contextual reasoning.\"}]","OenoBench - A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models | PDF",73]