[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85335-en":3,"doc-seo-85335-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85335,13056703020460,"Valentina","https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923",8,"Research & Report","Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals","Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), but many reward optimization and distribution matching approaches tightly couple exploration with distribution alignment. This coupling forces costly exploration on the policy model and prevents asynchronous generation, caching, reuse, and cross-model transfer of optimization signals. Proxy-guided Update Signal Transfer (PUST) decouples update-signal exploration from alignment by using a lightweight proxy to discover high-reward behaviors, extracting relative improvement signals, and transferring them to guide primary model alignment. Evaluations on Qwen3-family models show robust, adjustable gains and seamless weak-to-strong and cross-model transfer.","arXiv :2607 . 1 1505v 1 [ cs .LG] 13 Jul 2026  \nProxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals  \nDaocheng Fu∗ , 1 ,2 , Rong Wu∗ , 1 ,3 , Yu Yang∗ , 1 ,3 , Xuemeng Yang 1 , Jianbiao Mei 1 ,3 , Licheng Wen 1 ,4 ,5 , Pinlong Cai 1 , Yong Liu3 , Botian Shi†, 1 , Yu Qiao 1  \n1 Shanghai AI Laboratory, 2 Fudan University, 3 Zhejiang University,  \n4 Shanghai Innovation Institute, 5 Shanghai Jiao Tong University  \n∗ Equal Contribution, † Corresponding Author  \nAbstract  \nPost-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy’s initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weakto-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.  \n Date: July 14, 2026  \n Github Repo: [https://github.com/KnowledgeXLab/PUST](https://github.com/KnowledgeXLab/PUST)  \n Huggingface Page: [https://huggingface.co/KnowledgeXLab/PUST-Experiments](https://huggingface.co/KnowledgeXLab/PUST-Experiments)  \n1 Introduction  \nPost-training is essential for refining the usability and domain-specific capabilities of large language models (LLMs) [1], yet increasing model scale and domain diversity demand higher training efficiency and lower computational costs. Existing PPO-like methods (e.g., PPO [2], GRPO [3]) optimize models via on-policy rollouts and reward feedback. However, adapting them to multiple domains typically necessitates a sequential and costly optimization process across different settings (Fig. 1(a)) . The emergence of OPD [4, 5] introduces a new paradigm: instead of generating reward signals directly, it uses distribution matching to align a student model with optimized expert teachers, enabling cross-domain data-level parallelization (Fig. 1(b)) . This shift implies that reward-oriented optimization and distribution alignment serve distinct roles, motivating our investigation into how policy-improvement signals are explored and applied for optimization.  \nAs depicted in Fig. 1(b), this parallel pipeline comprises two distinct stages: policy exploration and policy alignment. It initially utilizes reward-oriented exploration to discover high-reward behaviors and construct expert  \n\n| \u003Cbr> |  |\n| --- | --- |\n|  |  |\n\nFigure 1: Comparison of post-training pipelines. (a) Serial: Sequential domain exploration, risking catastrophic forgetting. (b) Parallel: Parallel domain exploration followed by unified policy alignment. (c) Proxy Asynchronous (Ours): A proxy model conducts","cbCaibNFF0BDodBH","https://ap.wps.com/l/cbCaibNFF0BDodBH","pdf",3753311,4,1,17,"English","en",105,"# Introduction\n## Background and Motivation\n## Proposed Approach (PUST)\n## Key Advantages","[{\"question\":\"PUST的核心思想是什么？\",\"answer\":\"PUST将“更新信号的探索”和“分布对齐”解耦：用轻量代理模型进行低成本探索，提取代理模型初态到优化态的相对改进信号，再将该更新信号传递给主模型以引导对齐更新。\"},{\"question\":\"为什么需要解耦探索与分布对齐？\",\"answer\":\"现有方法往往把探索与对齐绑在同一主模型中，导致昂贵的探索开销，并限制信号的异步生成、缓存、复用以及跨模型迁移。解耦后可以更高效、可扩展地完成后训练。\"},{\"question\":\"PUST如何支持弱到强改进与跨模型迁移？\",\"answer\":\"PUST传递的是相对改进（相对提升方向）而非绝对策略分布，因此能自然支持弱到强的性能提升，并实现不同模型之间的无缝迁移。\"}]",1784202573,43,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"proxy-exploration-and-reusable-guidance-a-modular-llm-post-training-paradigm-via-proxy-guided-update-signals","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/proxy-exploration-and-reusable-guidance-a-modular-llm-post-training-paradigm-via-proxy-guided-update-signals/85335/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"PUST的核心思想是什么？","Question",{"text":75,"@type":76},"PUST将“更新信号的探索”和“分布对齐”解耦：用轻量代理模型进行低成本探索，提取代理模型初态到优化态的相对改进信号，再将该更新信号传递给主模型以引导对齐更新。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"为什么需要解耦探索与分布对齐？",{"text":80,"@type":76},"现有方法往往把探索与对齐绑在同一主模型中，导致昂贵的探索开销，并限制信号的异步生成、缓存、复用以及跨模型迁移。解耦后可以更高效、可扩展地完成后训练。",{"name":82,"@type":73,"acceptedAnswer":83},"PUST如何支持弱到强改进与跨模型迁移？",{"text":84,"@type":76},"PUST传递的是相对改进（相对提升方向）而非绝对策略分布，因此能自然支持弱到强的性能提升，并实现不同模型之间的无缝迁移。","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]