[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84391-en":3,"doc-seo-84391-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84391,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing","As available training data approaches its physical limit, further LLM gains from scaling data expansion diminish. UltraX shifts focus to data utilization efficiency by improving refinement quality while meeting large-scale requirements. The method introduces function-calling programmatic editing with insertion, deletion, and modification to support fine-grained instance-level edits. It uses dataset-adaptive prompt optimization, line alignment mapping, dynamic context replacement, and low-confidence filtering with ratio-controlled sampling to stabilize training distribution. Experiments with 1B models show the highest average performance across multiple corpora with >2% relative gains and strong data-efficiency.","arXiv :2607 .08646v 1 [ cs .CL] 9 Jul 2026  \nUltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing  \nXinlong Zhao1 Dongsheng Liu2 Hengyu Zhao2 , Zixuan Fu3 , Zheng Wang3 , Jie Cai2 , Jie Zhou2 , Qiang Ma3 , Xuanhe Zhou4 , Xu Han3 , Yudong Wang3†,Zhiyuan Liu3†  \n1Peking University 2ModelBest Inc. 3Tsinghua University 4 Shanghai Jiao Tong University  \n[xlzhao25@stu.pku.edu.cn {wangyudong](xlzhao25@stu.pku.edu.cn {wangyudong) , [liuzy}@tsinghua.edu.cn](liuzy}@tsinghua.edu.cn)  \n Datasets: [https://huggingface.co/datasets/openbmb/UltraX-Preview](https://huggingface.co/datasets/openbmb/UltraX-Preview)  \n Model: [https://huggingface.co/openbmb/UltraX-0.6B-Preview](https://huggingface.co/openbmb/UltraX-0.6B-Preview)  \n Code: [https://github.com/openbmb/UltraX](https://github.com/openbmb/UltraX)  \nAbstract  \nAs available training data approaches its physical limit, the performance gains derived from Scaling Laws have begun to diminish. Consequently, the key to further enhancing the performance of Large Language Models (LLMs) has shifted from mere data expansion to improving data utilization efficiency, by enhancing data quality to better exploit the latent potential of existing data. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert originalrefined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments with 1B models pretrained from scratch on multiple corpora show that UltraX achieves the highest average performance across all corpora, with relative improvements exceeding 2% on multiple datasets. UltraX also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.  \n1 Introduction  \nScaling laws (Kaplan et al., 2020) demonstrate that while the performance of Large Language Models (LLMs) (Achiam et al., 2023 ; Comanici et al., 2025 ; Hu et al., 2024 ; Team et al., 2025 ; Qwen Team, 2026 ; GLM-5-Team, 2026 ; DeepSeek-AI, 2026) continues to improve with the expansion of model parameters and training data, this growth potential has begun to reach a point of diminishing returns as available training data approaches its physical limit. Consequently, the focus for further enhancing LLM performance should shift from mere data scaling to the systematic improvement of training data quality (Wang et al., 2026) .  \nIn large-scale corpora, existing data quality improvement methods mainly fall into rule-based filtering and cleaning (Raffel et al., 2020 ; Penedo et al., 2024 ; Weber et al., 2025 ; Rae et al., 2021), and model-based  \n∗Equal contribution.†Corresponding authors.  \nselection (Wenzek et al., 2020 ; Wang et al., 2025) and refinement (Gunasekar et al.","cbCaijMDSKcxE1n3","https://ap.wps.com/l/cbCaijMDSKcxE1n3","pdf",922422,1,35,"English","en",105,"# Introduction\n## Scaling laws and diminishing returns\n## Limitations of existing refinement methods\n## UltraX approach and key design choices","[{\"question\":\"Why does improving LLM performance require more than simply adding training data?\",\"answer\":\"As available training data nears its physical limit, the performance gains predicted by scaling laws begin to diminish, so improvements must come from better data utilization through higher-quality refinement.\"},{\"question\":\"What core idea does UltraX use for scalable data refinement?\",\"answer\":\"UltraX uses a function-calling refinement framework rather than end-to-end text generation, enabling structured instance-level editing via predefined operations including insertion along with deletion and modification.\"},{\"question\":\"How does UltraX maintain quality and training stability at scale?\",\"answer\":\"UltraX generates reliable program supervision with dataset-adaptive prompt optimization, converts refined pairs into structured supervision using line alignment mapping and dynamic context replacement, and stabilizes training with low-confidence example filtering and ratio-controlled sampling.\"}]",1784195268,88,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"ultrax-refining-pre-training-data-at-scale-with-adaptive-programmatic-editing","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/ultrax-refining-pre-training-data-at-scale-with-adaptive-programmatic-editing/84391/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does improving LLM performance require more than simply adding training data?","Question",{"text":75,"@type":76},"As available training data nears its physical limit, the performance gains predicted by scaling laws begin to diminish, so improvements must come from better data utilization through higher-quality refinement.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What core idea does UltraX use for scalable data refinement?",{"text":80,"@type":76},"UltraX uses a function-calling refinement framework rather than end-to-end text generation, enabling structured instance-level editing via predefined operations including insertion along with deletion and modification.",{"name":82,"@type":73,"acceptedAnswer":83},"How does UltraX maintain quality and training stability at scale?",{"text":84,"@type":76},"UltraX generates reliable program supervision with dataset-adaptive prompt optimization, converts refined pairs into structured supervision using line alignment mapping and dynamic context replacement, and stabilizes training with low-confidence example filtering and ratio-controlled sampling.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]