[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-327448-105":59,"doc-detail-327448-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","hermes-a-multi-granularity-labeling-substrate-for-pre-training-data-abstract","HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data - Abstract","","Most data-mixing methods presume corpora are already partitioned, and the chosen groups constrain what a mixer can express. Existing label schemes commit to a single semantic axis at one granularity, so changing resolution requires rebuilding labels. HERMES argues the bottleneck lies in the label system and introduces a hierarchical, data-derived labeling substrate. A Learned Semantic Transform plus 3-stage residual vector quantization assigns each document once with a coarse-to-fine code, where prefix length controls granularity up to about 130k cells.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/hermes-a-multi-granularity-labeling-substrate-for-pre-training-data-abstract/327448/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/hermes-a-multi-granularity-labeling-substrate-for-pre-training-data-abstract/327448.png","ImageObject",300,407,{"name":92,"@type":93},"Ivy","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-04","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":52},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"Why do existing data-mixing methods struggle when granularity changes?","Question",{"text":112,"@type":113},"They rely on fixed label schemes tied to one semantic axis and one granularity. When resolution changes, the labels must be rebuilt because the taxonomy does not automatically adapt.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What is HERMES and how does it represent granularity?",{"text":117,"@type":113},"HERMES is a hierarchical data-derived labeling substrate. It uses a Learned Semantic Transform followed by a 3-stage residual vector quantizer to assign each document a single coarse-to-fine code, where granularity is controlled by the code prefix length.",{"name":119,"@type":110,"acceptedAnswer":120},"What does the Stage-2 sampler study show at different granularity levels?",{"text":121,"@type":113},"At fixed L12 granularity, switching Stage-2 sampling improves a 16-task capability macro-average by about +0.0253. At the finer level L123, the advantage collapses under a quota-preserving comparison, becoming numerically tied.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},327448,1790521609,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":52,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":56,"language":139,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":67,"update_tm":143,"read_time":144,"preview_page_count":4},549758252649,"https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819","HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data  \nMixtures  \nZiyun Qiao1,2,*,† Yue Min1,* Ruining Chen3 Yujun Li1,†  \n1 Wizard Quant, Beijing, China 2 Peking University, Beijing, China 3 University of  \nScience and Technology of China, Hefei, China  \n*Equal contribution.  \n†Correspondence to: [qiaozy@stu.pku.edu.cn](qiaozy@stu.pku.edu.cn), [liyujun@wizardquant.com](liyujun@wizardquant.com)  \narXiv :2607 .02266v 1 [ cs .LG] 2 Jul 2026  \nAbstract  \nMost data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels (provenance, topic or format taxonomies, flat embedding clusters) commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to ∼ 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: atone prefix length, a combined Stage-2 rule contrast (equal-subbucket coverage vs. sizeproportional within-bucket quality top-30%) lifts a 16-task capability macro-average by +0 .0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract ∼5 × . HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.  \n1 Introduction  \nData curation has become a decisive factor in LLMpre-training (Penedo et al., 2023, 2024), shifting attention from parameter scaling alone toward the mixing of heterogeneous sources. Pre-training data mixing has two separable layers: a label system that partitions the corpus, and a mixer or sampler that consumes those labels. What has constrained group-level pipelines is the label layer, not the mixer machinery: provenance is coarse, taxonomies commit to a single semantic axis and  \nFigure 1: Three corpus-control paradigms. Top: fixed taxonomies (source, topic, format) scale but fix one semantic axis at one granularity. Bottom: per-sample selection (DSIR, MATES, LESS) reaches the document at per-document compute. Middle: HERMES exposes a data-derived hierarchy whose prefix reads (L 1 , L 12 , L 123 ) deliver multiple granularities from one offline annotation, with no re-clustering.  \ngranularity, and flat clusterings require recomputation to change K. We address this with HERMES (Hierarchical Embedding-based RVQ for Mixture Selection), a one-shot hierarchical residual vector quantizer that annotates each document with a coarse-to-fine code (c1 , c2 , c3 ) and exposes granularity as a prefix length on a single trained RVQ codebook stack (Figure 1) . The detailed comparison to existing provenance, distilled-taxonomy, and embedding-cluster label families is deferred to Section 2.  \nAt the K = 256–130k label scale, regressionstyle mixture optimizers (RegMix (Liu et al., 2025b), Data Mixing Laws (Ye et al., 2025)) sit outside the proxy budget, since their fitting bases scale at least linearly in cell count; we therefore instantiate only O(1)-proxy outer weights, Uniform and  \nDoReMi. Per-sample selectors that avoid labels pay per-document compute at web scale (Section 4.3) . HERMES meets these constraints. A Learned  \nSemantic Transform, in the spirit of optimized product quantization (Ge et al., 2013), feeds a 3-stage residual vector quantizer (Liu et al., 2015 ; Huijbenet al., 2024); the prefix length on the resulting code defines granularity over a single trained RVQ codebook stack: L 1 = c 1 gives 256 buckets,","cbCair2fY66lqzei","https://ap.wps.com/l/cbCair2fY66lqzei","pdf",2675600,"English","# Abstract\n# 1 Introduction\n## Label system vs. mixer layer\n## Hierarchical RVQ substrate and granularity via prefix length\n## Experimental findings and Stage-2 sampler interactions","[{\"question\":\"Why do existing data-mixing methods struggle when granularity changes?\",\"answer\":\"They rely on fixed label schemes tied to one semantic axis and one granularity. When resolution changes, the labels must be rebuilt because the taxonomy does not automatically adapt.\"},{\"question\":\"What is HERMES and how does it represent granularity?\",\"answer\":\"HERMES is a hierarchical data-derived labeling substrate. It uses a Learned Semantic Transform followed by a 3-stage residual vector quantizer to assign each document a single coarse-to-fine code, where granularity is controlled by the code prefix length.\"},{\"question\":\"What does the Stage-2 sampler study show at different granularity levels?\",\"answer\":\"At fixed L12 granularity, switching Stage-2 sampling improves a 16-task capability macro-average by about +0.0253. At the finer level L123, the advantage collapses under a quota-preserving comparison, becoming numerically tied.\"}]","HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data - Abstract | PDF",1789977223,48]