[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83938-en":3,"doc-seo-83938-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83938,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","TREK: Distill to Explore, Reinforce to Refine","Group Relative Policy Optimization (GRPO) works when the current policy samples useful reasoning trajectories, but it stalls on hard prompts where correct solution modes fall outside the student’s on-policy support. TREK (Teacher-Routed Exploration via Forward KL) expands exploration by using distillation for support growth rather than imitation. TREK selects low pass-rate prompts, queries a proposal source for verified candidates, pulls top-ranked verified modes into the student via a short forward-KL phase, then resumes standard on-policy GRPO. On AIME 2024/2025 and ALFWorld/ScienceWorld, TREK improves accuracy and success rates and reaches high performance earlier on hardest task types.","arXiv :2607 .05339v 1 [ cs .LG] 6 Jul 2026  \nTREK: Distill to Explore, Reinforce to Refine  \nYuanda Xu 1†∗ Zhengze Zhou 1 ∗ Kayhan Behdin 1 Jelena Markovic-Voronov 1 Hejian Sang 1‡ Xiaomin Li2 Wenhui Zhu 1 Xinchen Du 1 ,3§ Aida Rahmattalabi 1 Ran He 1 Sen Na3 Zhipeng Wang 1‡ Alborz Geramifard 1  \n1LinkedIn Corporation 2Harvard University 3 Georgia Institute of Technology  \nAbstract  \nGroup Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student’s on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-r proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student’s support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the selfcontext variant reaches 38.5 and 49 .6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.  \n1 Introduction  \nDistillation, curriculum learning, and reinforcement learning from verifiable rewards have become central tools for improving language-model reasoning [Hinton et al., 2015, Kim and Rush, 2016, Bengio et al., 2009, Kumar et al., 2010, Shao et al., 2024, Guo et al., 2025] . In Group Relative Policy Optimization (GRPO), a policy samples a group of candidate responses for each prompt and receives relative advantages from outcome rewards or programmatic verifiers. The appeal of GRPO is that it directly optimizes the behaviors that the deployed model can sample, avoiding a learned value model and keeping training closely tied to test-time behavior.  \nThe same on-policy property also creates a limitation. GRPO is most effective when the current student already assigns nontrivial probability mass to useful solution modes. On harder prompts, however, the student may repeatedly sample plausible but structurally similar wrong trajectories. In  \n∗ Equal contribution.  \n†[Correspondence to](Correspondence to yuanda@math.princeton.edu)[ yuanda@math.princeton.edu](Correspondence to yuanda@math.princeton.edu)[ ](Correspondence to yuanda@math.princeton.edu)‡Work done while at LinkedIn Corporation.  \n§ Work done while an intern at LinkedIn Corporation.  \nPreprint.  \n\n|  Direct GRPO  | TREK (DeepSeek-V4)  | TREK (self-context) |\n| --- | --- | --- |\n\nAccuracy (%)  \n50  \n40  \n30  \n20  \n(a) AIME 2025  \n1 . 7B 8B 14BQwen3 scale  \nAccuracy (%)  \n60  \n50  \n40  \n30  \n20  \n(b) AIME 2024  \n1 . 7B 8B 14BQwen3 scale  \n(c) ALFWorld · Qwen2.5-7B-Instruct  \nSuccess Rate (%)  \n84  \n82  \n80  \n78  \n76  \n74  \n\n| 75.8 |  |  |  | 80.4 |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |\n\n82.8  \nGRPO TREK TREK  \n(DeepSeek-V4) (self-ctx)  \nFigure 1: Summary of main results. (a)–(b): Math reasoning accuracy (avg@16,%) ","cbCaih3uXB6WjPHe","https://ap.wps.com/l/cbCaih3uXB6WjPHe","pdf",802830,1,18,"English","en",105,"# Abstract\n# Introduction\n## Distillation as exploration for GRPO\n## Limitations of on-policy GRPO","[{\"question\":\"What problem does TREK address in GRPO training?\",\"answer\":\"GRPO can stall on hard prompts because the student repeatedly samples structurally similar wrong trajectories, leaving correct solution modes outside its on-policy support. TREK targets the exploration coverage bottleneck rather than reward assignment among sampled trajectories.\"},{\"question\":\"How does TREK expand the student’s support for hard prompts?\",\"answer\":\"TREK first finds prompts where the unaided student has a very low pass rate, generates verified candidate solutions from a proposal source, keeps the top-r candidates ranked by the student’s likelihood, applies a short forward-KL phase to move verified modes into the student’s support, and then continues standard on-policy GRPO refinement.\"},{\"question\":\"What experimental results does TREK report on math and agentic tasks?\",\"answer\":\"On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models on AIME 2024 and AIME 2025, including Qwen3-8B gains. On agentic tasks, TREK increases ALFWorld and ScienceWorld success rates and achieves high success early in training on the hardest task types compared with unaided GRPO.\"}]",1784191549,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"trek-distill-to-explore-reinforce-to-refine","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/trek-distill-to-explore-reinforce-to-refine/83938/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does TREK address in GRPO training?","Question",{"text":74,"@type":75},"GRPO can stall on hard prompts because the student repeatedly samples structurally similar wrong trajectories, leaving correct solution modes outside its on-policy support. TREK targets the exploration coverage bottleneck rather than reward assignment among sampled trajectories.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does TREK expand the student’s support for hard prompts?",{"text":79,"@type":75},"TREK first finds prompts where the unaided student has a very low pass rate, generates verified candidate solutions from a proposal source, keeps the top-r candidates ranked by the student’s likelihood, applies a short forward-KL phase to move verified modes into the student’s support, and then continues standard on-policy GRPO refinement.",{"name":81,"@type":72,"acceptedAnswer":82},"What experimental results does TREK report on math and agentic tasks?",{"text":83,"@type":75},"On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models on AIME 2024 and AIME 2025, including Qwen3-8B gains. On agentic tasks, TREK increases ALFWorld and ScienceWorld success rates and achieves high success early in training on the hardest task types compared with unaided GRPO.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]