[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-277914-105":3,"detail-sidebar-cat-1-en-105":81,"doc-detail-277914-en":126},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","documentnet-bridging-the-data-gap-in-document-pre-training","DocumentNet - Bridging the Data Gap in Document Pre-Training","","Document understanding tasks, especially Visually-rich Document Entity Retrieval (VDER), face data scarcity due to strict privacy constraints and high annotation costs. Non-overlapping entity spaces across datasets further limit knowledge transfer. DocumentNet addresses this by collecting massive-scale, weakly labeled web data under a universal four-level ontology spanning nearly 400 document types. Pre-training with DocumentNet yields strong gains on classic and few-shot VDER benchmarks and extends LLM multimodal capabilities for document understanding.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/template/","Template",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/template/presentations/","Presentations",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/template/documentnet-bridging-the-data-gap-in-document-pre-training/277914/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/documentnet-bridging-the-data-gap-in-document-pre-training/277914.png","ImageObject",442,249,{"name":42,"@type":43},"Lute","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-19","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is training data scarce for VDER tasks?","Question",{"text":63,"@type":64},"Raw document images are limited by privacy constraints, and detailed field-level annotations require costly, expert human labeling.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does DocumentNet improve knowledge transfer across document types?",{"text":68,"@type":64},"It uses a predefined four-level ontology and a universal entity-space design, enabling weakly labeled web-scale data collection that is not tied to specific document types.",{"name":70,"@type":61,"acceptedAnswer":71},"What benefits does DocumentNet provide during pre-training for VDER and LLMs?",{"text":72,"@type":64},"Incorporating DocumentNet into pre-training improves VDER performance in both classic and few-shot settings and provides multimodal data to enhance LLM document understanding capabilities.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},277914,1791745021,{"code":4,"msg":82,"data":83},"success",[84,87,92,97,102,107,112,117,122],{"id":55,"doc_module":22,"doc_module_name":25,"category_name":29,"show_sort_weight":85,"slug":86},90,"presentations",{"id":88,"doc_module":22,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},12,"Resumes",80,"resumes",{"id":93,"doc_module":22,"doc_module_name":25,"category_name":94,"show_sort_weight":95,"slug":96},14,"Invoices",70,"invoices",{"id":98,"doc_module":22,"doc_module_name":25,"category_name":99,"show_sort_weight":100,"slug":101},15,"Posters",60,"posters",{"id":103,"doc_module":22,"doc_module_name":25,"category_name":104,"show_sort_weight":105,"slug":106},16,"Social Media",50,"social-media",{"id":108,"doc_module":22,"doc_module_name":25,"category_name":109,"show_sort_weight":110,"slug":111},17,"Forms",40,"forms",{"id":113,"doc_module":22,"doc_module_name":25,"category_name":114,"show_sort_weight":115,"slug":116},18,"Letters",30,"letters",{"id":118,"doc_module":22,"doc_module_name":25,"category_name":119,"show_sort_weight":120,"slug":121},21,"Paper Templates",5,"papers-templates",{"id":123,"doc_module":22,"doc_module_name":25,"category_name":124,"show_sort_weight":4,"slug":125},158,"General","general-158",{"code":4,"msg":82,"data":127},{"doc_id":79,"user_id":128,"nickname":42,"user_avatar":129,"doc_module":22,"category_id":55,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":103,"language":135,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":136,"faqs":137,"seo_title":138,"seo_description":12,"update_tm":139,"read_time":140},137454149569,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","DocumentNet: Bridging the Data Gap in Document Pre-Training  \nLijun Yu‡⋄ , Jin Miao†, Xiaoyu Sun†, Jiayi Chen∤ , Alexander G. Hauptmann‡,  \nHanjun Dai†, Wei Wei†⋄  \n‡Carnegie Mellon University, †Google, ∤University of Virginia  \n⋄ [lijun@cmu.edu](lijun@cmu.edu) , [wewei@google.com](wewei@google.com)  \nAbstract  \nDocument understanding tasks, in particular, Visually-rich Document Entity Retrieval (VDER), have gained significant attention in recent years thanks to their broad applications in enterprise AI. However, publicly available data have been scarce for these tasks due to strict privacy constraints and high annotation costs. To make things worse, the non-overlapping entity spaces from different datasets hinder the knowledge transfer between document types. In this paper, we propose a method to collect massivescale and weakly labeled data from the web to benefit the training of VDER models. The collected dataset, named DocumentNet, does not depend on specific document types or entity sets, making it universally applicable to all VDER tasks. The current DocumentNet consists of 30M documents spanning nearly 400 document types organized in a four-level ontology. Experiments on a set of broadly adopted VDER tasks show significant improvements when DocumentNet is incorporated into the pre-training for both classic and few-shot learning settings. With the recent emergence of large language models (LLMs), DocumentNet provides a large data source to extend their multimodal capabilities for VDER.  \n1 Introduction  \nDocument understanding is one of the most errorprone and tedious tasks many people have to handle every day. Advancements in machine learning techniques have made it possible to automate such tasks. In a typical Visually-rich Document Entity Retrieval (VDER) task, pieces of information are retrieved from the document based on a set of predefined entity types, known as the schema. For example,“amount”,“date”, and “item name” are major parts of an invoice schema.  \nThe current setup of VDER tasks presents several unique challenges for acquiring sufficient training data. First, the availability of raw document  \nimages is greatly limited due to privacy constraints. Real-world documents, such as a driver’s license ora bank statement, often contain personally identifiable information and are subject to access controls. Second, detailed annotation is costly and typically requires intensive training for experienced human annotators. E.g., it takes deep domain knowledge to correctly label different fields in complex tax forms. Finally, knowledge sharing between various types of documents is constrained by inconsistent label spaces and contextual logic. For example, the entity sets (i.e., schema) could be mutually exclusive, or the same entity type could take different semantic meanings in different contexts.  \nA number of models have been proposed for VDER tasks with various success (Huang et al., 2022 ; Lee et al., 2022 ; Appalaraju et al., 2021 ; Gu et al., 2021) . To tackle the aforementioned challenges, most prior works initialize from a language model followed by BERT-style (Devlinet al., 2019) pre-training on document datasets with additional layout and visual features. However, even the largest dataset currently in use, i.e. IITCDIP (Lewis et al., 2006) dataset, has a limited size and only reflects a subset of document types. In this paper, we introduce the method of building the DocumentNet dataset, which enables massive-scale pre-training for VDER modeling. DocumentNet is collected over the Internet using a pre-defined ontology, which spans hundreds of document types with a four-level hierarchy. Experiments demonstrated that DocumentNet is the key to advancing the performance on the commonly used FUNSD (Jaume et al., 2019), CORD (Park et al., 2019), and RVL-CDIP (Lewis et al., 2006) benchmarks in both classic and few-shot setups. More recently, LLMs (OpenAI, 2023 ; Anil et al., 2023) have shown great potential for VDER t","cbCaiaNdjL9x1rZ4","https://ap.wps.com/l/cbCaiaNdjL9x1rZ4","pdf",1918544,"English","# Introduction\n## Challenges in Visually-rich Document Entity Retrieval (VDER)\n# Related Work","[{\"question\":\"Why is training data scarce for VDER tasks?\",\"answer\":\"Raw document images are limited by privacy constraints, and detailed field-level annotations require costly, expert human labeling.\"},{\"question\":\"How does DocumentNet improve knowledge transfer across document types?\",\"answer\":\"It uses a predefined four-level ontology and a universal entity-space design, enabling weakly labeled web-scale data collection that is not tied to specific document types.\"},{\"question\":\"What benefits does DocumentNet provide during pre-training for VDER and LLMs?\",\"answer\":\"Incorporating DocumentNet into pre-training improves VDER performance in both classic and few-shot settings and provides multimodal data to enhance LLM document understanding capabilities.\"}]","DocumentNet - Bridging the Data Gap in Document Pre-Training | PDF",1789505269,6]