[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85081-en":3,"doc-seo-85081-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85081,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","H3D Benchmarking Unsupervised Text Hashing for Fine-Grained Document Deduplication","Document hashing enables compact representations for efficient similarity search and document deduplication, yet existing work seldom compares hashing pipelines under a unified protocol for fine-grained scientific documents. H3D provides an unsupervised text hashing benchmark for fine-grained deduplication, evaluating non-learning fingerprints such as MinHash, SimHash, Winnowing, FuzzyHash, and FlyHash, plus semantic-sensitive methods using frozen BGE embeddings with two quantization strategies. Benchmarks on CSFCube and RELISH report ranking quality, efficiency, and robustness under controlled compression, revealing lexical/structural trade-offs versus semantic preservation.","arXiv :2607 .08382v 1 [ cs .IR] 9 Jul 2026  \nH3D: Benchmarking Unsupervised Text Hashing for FineGrained Document Deduplication  \nQianren Mao∗  \nZhongguancun Laboratory Jiaxun Lyu  \nBeihang University  \nJunnan Liu  \nMonash University Australia Zhijun Chen  \nHong Kong Polytechnic University Jingzheng Li  \nZhongguancun Laboratory Hanwen Hao  \nBeihang University  \nBo Li  \nBeihang University  \n∗Corresponding author.  \n[maogr@zgclab.edu.cn](maogr@zgclab.edu.cn)  \n[19373403@buaa.edu.cn](19373403@buaa.edu.cn)  \n[junnan.liu@monash.edu](junnan.liu@monash.edu)  \n[zhijun.chen@polyu.edu.hk](zhijun.chen@polyu.edu.hk)  \n[ljz@zgclab.edu.cn](ljz@zgclab.edu.cn)  \n[20373190@buaa.edu.cn](20373190@buaa.edu.cn)  \n[libo@act.buaa.edu.cn](libo@act.buaa.edu.cn)  \nAbstract  \nDocument hashing provides compact representations for efficient similarity search and document deduplication, but existing studies rarely compare hashing pipelines under a unified protocol for fine-grained scientific documents. H3D is an unsupervised text hashing benchmark for fine-grained document deduplication. It evaluates representative unsupervised non-learning hashing approaches (MinHash, SimHash, Winnowing, FuzzyHash, FlyHash) together with semantic-sensitive methods built from frozen BGE embeddings and two quantization strategies (BGE-BIHash and BGELSHash) . The non-learning methods generate hash fingerprints through manually designed mathematical rules without training or labeled similarity pairs, which distinguishes them from neural semantic hashing models. We benchmark all methods on CSFCube and RELISH, two datasets that provide complementary evaluation settings: facet-level analysis for scientific-document similarity and larger-scale split-level evaluation for biomedical similarity search. H3D jointly reports ranking quality (MAP, NDCG@20), efficiency, and robustness under controlled text compression. The results show a consistent trade-off: lexical and structural fingerprints are competitive for near-duplicate matching, while semantic-sensitive representations better preserve similarity under content rewriting, at higher computational cost. We further analyze when different similarity measures become rank-equivalent for specific hash representations, improving the interpretability and reproducibility of method comparisons.  \n DocAILab/Document-Fingerprints  Hugging Face Dataset  \n1 Introduction  \nDocument hashing is a fundamental technique for efficient textual similarity search Huang et al. (2024) and document deduplication Lee et al. (2022) . It enables the transformation of high-dimensional text representations into compact hash codes while preserving semantic or structural properties, allowing for fast and scalable document comparison.  \nAt web and pretraining-corpus scale, a representative non-learning pipeline is MinHashLSH: documents are first converted into token or n-gram sets, MinHash signatures approximate their Jaccard overlap, and locality-sensitive hashing splits the signature into bands so that highly overlapping documents are likely to collide in at least one bucket Rao & Zhu (2016); Alibaba Cloud (2026) . This design turns exhaustive pairwise comparison into candidate generation followed by lightweight verification, which explains its continued use in large-scale text-cleaning systems. Recent systems further optimize this pipeline for extreme-scale deduplication. For example, LSHBloom replaces the conventional MinHashLSH index with Bloom filters, preserving MinHashLSH-style duplicate detection while substantially reducing storage requirements for billion-document settings Khan et al. (2024) . These developments show that practical deduplication depends not only on the hash signature itself, but also on the indexing and candidategeneration mechanism wrapped around it. In recent years, learning-based hashing methods Chaidaroon & Fang (2017); Shen et al. (2018); Dong et al. (2019); Zheng et al. (2020); Doan & Reddy (2020); Qiu et al. (2021); Ou et al. (20","cbCaitY6ofSQwUnT","https://ap.wps.com/l/cbCaitY6ofSQwUnT","pdf",744954,2,1,29,"English","en",105,"# Introduction\n## Document hashing for similarity search and deduplication\n## Non-learning unsupervised hashing pipelines\n## Indexing and candidate generation mechanisms\n## Learning-based hashing and the need for unsupervised methods","[{\"question\":\"What is H3D and what problem does it target?\",\"answer\":\"H3D is an unsupervised text hashing benchmark designed for fine-grained document deduplication, especially for scientific documents where prior comparisons lack a unified evaluation protocol.\"},{\"question\":\"Which hashing approaches does H3D evaluate?\",\"answer\":\"H3D evaluates representative unsupervised non-learning methods (MinHash, SimHash, Winnowing, FuzzyHash, FlyHash) and semantic-sensitive methods built from frozen BGE embeddings with two quantization strategies (BGE-BIHash and BGELSHash).\"},{\"question\":\"How are methods compared and what trade-off does the benchmark reveal?\",\"answer\":\"H3D reports ranking quality (MAP, NDCG@20), efficiency, and robustness under controlled text compression, showing a consistent trade-off: lexical/structural fingerprints work well for near-duplicate matching, while semantic-sensitive representations better preserve similarity under content rewriting but at higher computational cost.\"}]",1784200925,73,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"h3d-benchmarking-unsupervised-text-hashing-for-fine-grained-document-deduplication","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/h3d-benchmarking-unsupervised-text-hashing-for-fine-grained-document-deduplication/85081/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is H3D and what problem does it target?","Question",{"text":75,"@type":76},"H3D is an unsupervised text hashing benchmark designed for fine-grained document deduplication, especially for scientific documents where prior comparisons lack a unified evaluation protocol.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which hashing approaches does H3D evaluate?",{"text":80,"@type":76},"H3D evaluates representative unsupervised non-learning methods (MinHash, SimHash, Winnowing, FuzzyHash, FlyHash) and semantic-sensitive methods built from frozen BGE embeddings with two quantization strategies (BGE-BIHash and BGELSHash).",{"name":82,"@type":73,"acceptedAnswer":83},"How are methods compared and what trade-off does the benchmark reveal?",{"text":84,"@type":76},"H3D reports ranking quality (MAP, NDCG@20), efficiency, and robustness under controlled text compression, showing a consistent trade-off: lexical/structural fingerprints work well for near-duplicate matching, while semantic-sensitive representations better preserve similarity under content rewriting but at higher computational cost.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]