[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86460-en":3,"doc-seo-86460-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86460,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",6,"Technology","Robust Scalable Detection of Text Containment in Large Web-Crawled Corpora","FindMyText is an open-source Python package for efficiently determining whether an input text is fully or partially contained in a target corpus. Building on document fingerprinting, it introduces a chain-based mechanism that links matching fingerprints to localize shared text fragments more reliably than similarity-only methods. This improves detection of near-verbatim copies and supports tasks such as checking potential copyrighted material presence. A distributed disk-based index enables scaling to large web-crawled datasets, and a new benchmark shows strong results on arXiv, Wikipedia, and generic web content.","FindMyText: Robust, Scalable Detection of Text Containment  \nin Large Web-Crawled Corpora  \nLars Henry Berge Olsen1 , Pierre Lison1 , Martin Jullum1 and Mark Anderson1  \n1Norwegian Computing Center, Oslo, Norway.  \n{lhbolsen, plison, jullum, [anderson](anderson}@nr.no)[}@nr.no](anderson}@nr.no)  \n§ [https://github.com/NorskRegnesentral/FindMyText](https://github.com/NorskRegnesentral/FindMyText) 􀂓 [https://findmytext.nr.no](https://findmytext.nr.no)  \n(password: EMNLP2026)  \narXiv :2607 . 10020v 1 [ cs .CL] 10 Jul 2026  \nAbstract  \nWe present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus. The tool builds on prior techniques for document fingerprinting, but extends them with a novel mechanism to explicitly capture sequences of matching fingerprints. By identifying such chains, the tool can more reliably detect near-verbatim copies of a given text rather than mere textual similarities. This makes FindMyText particularly suited for verifying the presence of copyrighted material in a corpus. Leveraging a distributed, disk-based indexing framework, the system scales to large web-crawled datasets. Using a new benchmark for evaluating text containment methods, we show that FindMyText outperforms alternative approaches across three datasets (ArXiv papers, Wikipedia, and generic web content) .  \n1 Introduction  \nLarge Language Models (LLMs) depend on colossal amounts of data for their pre-training, much of it gathered through web crawling. Understanding the composition of this pre-training data is crucial for multiple NLP tasks, such as data selection and curation (Albalak et al., 2024 ; Parmar et al., 2024), enhancing model transparency and interpretability (Wang et al., 2023 ; Chang et al., 2024), and assessing potential copyright infringements or licensing violations (Karamolegkou et al., 2023 ; Longpre et al., 2024 ; Scharrenberg and Sun, 2025) .  \nPinpointing the exact texts that were part of an LLM’s pre-training data is, however, a non-trivial problem. Commercial LLM providers have so far been reluctant to disclose the exact content of their training data, often in fear of lawsuits. This led to the development of various techniques to conduct membership inference attacks on black-box LLMs (Oren et al., 2023 ; Shi et al., 2023), notably by taking advantage of LLMs’ memorisation abilities (Hartmann et al., 2023 ; Ahmed et al., 2026) .  \nExtracted fingerprints (winnowing)  \nFingerprint  \nInverted index  \nFingerprints mapped to lists of (doc id, offset) pairs  \nFigure 1: General sketch of the FindMyText approach.  \nThe reliability of those attacks have, however, been called into question (Meeus et al., 2024 ; Zhang et al., 2024a ; Liu et al., 2025), particularly for production LLMs that are carefully designed to avoid generating copyrighted content.  \nTransparency obligations introduced by recent regulations such as the EU AI Act (European Parliament and Council, 2024) may in the future compel LLM providers to disclose more details about their data sources. Nevertheless, even if the full training corpus were made publicly available, establishing whether a specific text (say, a copyrighted book chapter) is part of that corpus remains technically difficult. A text may indeed be spread across numerous websites, each with its own format, and mixed together with other pieces of content. Furthermore, pre-training datasets typically undergo multiple preprocessing steps, including text extraction (e.g. OCR), boilerplate removal, document segmentation, and various types of text normalisation and reformatting. Due to this extensive curation process, techniques for near-duplicate detection (Manku et al., 2007) may fail to reliably establish whether a text is part of a pre-training corpus.  \nThis paper presents a novel method for the efficient detection of text containment, defined as  \nthe task of determining whether an input text is wholly or par","cbCaibvmjOVssoQg","https://ap.wps.com/l/cbCaibvmjOVssoQg","pdf",889907,1,12,"English","en",105,"# Introduction\n# Related Work\n## String alignment\n## Global and local alignment\n# FindMyText Methods\n## Chain-based fingerprint matching\n# System Design and Implementation\n# Experimental Results\n## Benchmark and datasets\n# Conclusion","[{\"question\":\"What problem does FindMyText solve?\",\"answer\":\"FindMyText determines whether an input text is wholly or partially included in a target corpus. It focuses on locating shared text fragments rather than only measuring overall similarity.\"},{\"question\":\"How does FindMyText improve over near-duplicate detection based on shared fingerprint ratios?\",\"answer\":\"It explicitly identifies chains of matching fingerprints, enabling more precise detection and localization of shared text fragments. This improves reliability for detecting near-verbatim copies.\"},{\"question\":\"Why is the approach suitable for large web-crawled datasets?\",\"answer\":\"It uses a distributed, disk-based indexing framework that scales to large corpora. It is also designed to be robust to discrepancies between user-provided text and corpus text.\"}]",1784211860,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"robust-scalable-detection-of-text-containment-in-large-web-crawled-corpora","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/robust-scalable-detection-of-text-containment-in-large-web-crawled-corpora/86460/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does FindMyText solve?","Question",{"text":75,"@type":76},"FindMyText determines whether an input text is wholly or partially included in a target corpus. It focuses on locating shared text fragments rather than only measuring overall similarity.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does FindMyText improve over near-duplicate detection based on shared fingerprint ratios?",{"text":80,"@type":76},"It explicitly identifies chains of matching fingerprints, enabling more precise detection and localization of shared text fragments. This improves reliability for detecting near-verbatim copies.",{"name":82,"@type":73,"acceptedAnswer":83},"Why is the approach suitable for large web-crawled datasets?",{"text":84,"@type":76},"It uses a distributed, disk-based indexing framework that scales to large corpora. It is also designed to be robust to discrepancies between user-provided text and corpus text.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":28,"slug":121},8,"Research & Report","research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]