[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81836-en":3,"doc-seo-81836-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":20,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},81836,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","SemHash-LLM Multi-Granularity Semantic Hashing Framework for Document Deduplication","Large-scale document deduplication needs semantic equivalence without sacrificing efficiency across massive corpora. SemHash LLM introduces a multi-granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM adjudication. Character, token, and document-level signals are fused via gated mechanisms, then cascaded filtering reduces candidates. Semantic projection hashing learns compact binary codes from distilled LLM embeddings, while attention weighted MinHash suppresses boilerplate. Adaptive boundaries and uncertainty routing improve robustness under template pollution, short perturbations, containment, and viral fragments, achieving strong duplicate detection with under 1% neural verification cost.","SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication  \nXinyi Fang * Independent Researcher  \nShanghai, China [fangxy.carol@gmail.com](fangxy.carol@gmail.com)  \nKejian Tong  \nIndependent Researcher Mukilteo, USA [tongcs2021@gmail.com](tongcs2021@gmail.com)  \nJiabei Liu  \nNortheastern University Oakland, USA liu.jiabe@northeastern.ed  \nTao Ning  \nSyracuse University San Jose, USA[ntgd1102@gmail.com](ntgd1102@gmail.com)  \nYuhang He  \nIndependent Researcher Chicago, USA [yuhang.he@outlook.com](yuhang.he@outlook.com)  \narXiv :2607 .0 160 1v 1 [ cs .AI] 2 Jul 2026  \nAbstract—Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted MinHash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.  \nIndex Terms—document deduplication, semantic hashing, MinHash, large language models, contrastive learning, uncertainty estimation  \nI. INTRODUCTION  \nModern data pipelines face a persistent dilemma, large web corpora demand aggressive deduplication for efficiency and data quality, yet strict lexical filtering often removes useful semantic variants while semantic methods are usually too expensive at scale. This tension has become more important as recent language model studies show that data redundancy strongly affects memorization, optimization, and downstream quality [1].Similar concerns about resilience under dynamic operating conditions have also been studied in logistics and routing systems [2] . At the same time, open pretraining corpora continue to grow in size and heterogeneity, making scalable curation a first order systems problem [3] .  \nCurrent solutions struggle because they optimize only one side of the trade off. Exact and near exact fingerprinting methods are fast, but they are brittle under paraphrase, template wrapping, and small character perturbations. Pure embedding based retrieval is more semantically aware, but its computational and thresholding costs are difficult to control across billions of documents.Related efforts on efficient large model inference also explore multi-stage compression and  \nacceleration strategies that balance accuracy with deployment cost, including adaptive quantization, token pruning, and decode-time optimization [4] . These challenges are amplified in modern corpora that mix long documents, short snippets, boilerplate heavy pages, and containment relationships within the same pipeline [5] .  \nWe address this problem with SemHash LLM, a unified framework that combines learned semantic hashing, attention weighted lexical sketching, adaptive contrastive decision boundaries, and selective LLM based refinement. Our design breaks the conventional efficiency versus accuracy compromise by using a cascaded pipeline for pruning, multi granularity fusion for robustness, and uncertainty aware routing so that expensive reasoning is applied only to rare borderline cases.  \nII. RELATED WORK  \nRecent work on retrieval and matching has shown that contextualized token interaction can improve fine grained similarity estimation beyond bag of words fingerprints. COIL demonstrated that exact lexical matching can be strengthened with contextualized token repres","cbCail8v7yXOEM0p","https://ap.wps.com/l/cbCail8v7yXOEM0p","pdf",45082255,7,1,"English","en",105,"# Introduction\n# Related Work\n# Methodology","[{\"question\":\"What problem does SemHash LLM target in large-scale document deduplication?\",\"answer\":\"It targets efficient deduplication that preserves semantic equivalence at web-corpus scale, avoiding the trade-off where lexical methods are brittle and embedding methods are too costly to control over billions of documents.\"},{\"question\":\"How does SemHash LLM combine multiple similarity signals?\",\"answer\":\"It fuses character, token, and document-level signals using gated fusion, integrating semantic projection hashing with attention weighted MinHash to emphasize informative content and suppress boilerplate.\"},{\"question\":\"How does the framework keep computation low while improving robustness?\",\"answer\":\"A cascaded filtering pipeline reduces candidate pairs, and uncertainty-aware routing applies selective LLM-based refinement only to rare borderline cases, improving robustness with less than 1% neural verification cost.\"}]","SemHash-LLM Multi-Granularity Semantic Hashing Framework for Document Deduplication | PDF",1784176523,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"semhash-llm-multi-granularity-semantic-hashing-framework-for-document-deduplication","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/semhash-llm-multi-granularity-semantic-hashing-framework-for-document-deduplication/81836/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does SemHash LLM target in large-scale document deduplication?","Question",{"text":76,"@type":77},"It targets efficient deduplication that preserves semantic equivalence at web-corpus scale, avoiding the trade-off where lexical methods are brittle and embedding methods are too costly to control over billions of documents.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does SemHash LLM combine multiple similarity signals?",{"text":81,"@type":77},"It fuses character, token, and document-level signals using gated fusion, integrating semantic projection hashing with attention weighted MinHash to emphasize informative content and suppress boilerplate.",{"name":83,"@type":74,"acceptedAnswer":84},"How does the framework keep computation low while improving robustness?",{"text":85,"@type":77},"A cascaded filtering pipeline reduces candidate pairs, and uncertainty-aware routing applies selective LLM-based refinement only to rare borderline cases, improving robustness with less than 1% neural verification cost.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]