[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82813-en":3,"doc-seo-82813-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82813,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL","Reinforcement learning for non-verifiable instruction following increasingly uses LLM judges with prompt-specific rubrics as reward signals, but training often relies on static prompt corpora. Static prompts create a mismatch between prompt difficulty and policy capability, causing rubric judgments to lack discriminative reward when rollouts show no quality variance. LLM-as-a-Tutor shifts the LLM from judge to tutor by pairwise examining rollout quality and appending atomic constraints to monotonically raise difficulty, producing self-calibrating training signals without external schedules. Evaluations on three benchmarks show consistent improvements over policy-unaware and prior policy-adaptive methods.","arXiv :2607 .044 12v 1 [ cs .AI ] 5 Jul 2026  \nLLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL  \nYujin Kim 1 ∗ Namgyu Ho 1∗ Sangmin Hwang 1∗ Joonkee Kim2 Yongjin Yang3 Sangmin Bae 1 Seungone Kim4 Jaehun Jung5 Se-Young Yun 1† Hwanjun Song 1†  \n1 KAIST 2Upstage 3University of Toronto 4 Carnegie Mellon University 5NVIDIA  \n{yujin399,itsnamgyu,[sangmin.hwang}@kaist.ac.kr](sangmin.hwang}@kaist.ac.kr)  \nAbstract  \nReinforcement learning (RL) for non-verifiable instruction following increasingly  \nrelies on LLM judges with prompt-specific rubrics as reward signals. While  \nrecent methods adapt these rubrics to the evolving policy during training, the  \ntraining prompts themselves remain static, drawn from fixed corpora. This static  \napproach often results in a critical misalignment between prompt difficulty and  \npolicy capability, leaving the judge unable to recover a discriminative reward  \nsignal when prompts fail to elicit quality variance among rollouts. To address  \nthis misalignment, we introduce LLM-as-a-Tutor, a framework that extends the  \nLLM’s role from judge to tutor: a single model serves as an examiner that pairwise  \ncompares policy rollouts to detect non-challenging prompts, and as a generator that  \nappends atomic constraints to them. This append-only design monotonically raises  \ndifficulty in step with the policy’s capability, producing a self-calibrating training  \nsignal without external difficulty schedules. On three complex instruction-following  \nbenchmarks, our method consistently outperforms both policy-unaware baselines  \nand prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting  \nprompt adaptation as a missing axis of policy-awareness in non-verifiable RL.  \n1 Introduction  \nReinforcement learning depends on the reward signal’s ability to discriminate among policy outputs of varying quality [1, 25, 30, 31, 33] . In verifiable domains such as math and code, programmatic checkers provide this signal directly [6, 27] but open-ended instruction following has no analogous checker. Bradley-Terry reward models [4, 22, 29] provide only coarse discrimination, leading to reward hacking and plateaued performance on complex benchmarks [3, 7, 23, 32] . Recent work conditions LLM judges on instance-specific rubrics [5, 26, 32], recovering fine-grained discrimination and generalizing zero-shot to novel domains [13] .  \nThe discriminative power of the LLM judge, however, comes with a precondition: the prompt itself must induce rollouts that differ in quality. We refer to such prompts as challenging for the current policy. When a prompt is too easy for the current policy, all rollouts succeed; when too hard, all rollouts fail. In either case, the rubric judge has nothing to discriminate, and the reward signal collapses regardless of judge quality. This is a property of the policy-prompt pair, not of the prompt alone. Yet existing pipelines for non-verifiable RL with LLM judges draw training prompts from static corpora [5, 26, 32], leaving the supply of challenging prompts to chance.  \nA natural response is to adapt prompts to the current policy so they reliably induce discriminative rollouts, a property we call policy-adaptiveness. Recent work in non-verifiable RL pursues this but  \n∗ Equal contribution. †Corresponding authors.  \nPreprint.  \nFigure 1: Overview of LLM-as-a-Tutor: when the policy’s answers to a prompt are indistinguishable in quality, a tutor LLM adds a constraint to make the prompt more challenging.(Left) Static RL training corpora contain prompts that are non-challenging for the current policy and provide little learning signal. (Right) The tutor examines a pair of policy rollouts; if their quality is indistinguishable, it appends a constraint that turns the non-challenging prompt into a challenging one and elicits variance among the new rollouts.  \ndoes not fully exploit the discriminative capabilities of LLMs. One line uses scalar reward-model scores [36],","cbCaiq1PkrkVcwrl","https://ap.wps.com/l/cbCaiq1PkrkVcwrl","pdf",1831610,1,29,"English","en",105,"# Abstract\n# Introduction\n## Reward discrimination in non-verifiable domains\n## Policy-aware prompt adaptation\n## LLM-as-a-Tutor framework\n## Training procedure and difficulty control\n# Evaluation","[{\"question\":\"Why do static training prompts hurt non-verifiable RL with LLM judges?\",\"answer\":\"Static prompts may be too easy or too hard for the current policy, so all rollouts end up with indistinguishable quality. When that happens, even a high-quality rubric judge cannot discriminate and the reward signal collapses.\"},{\"question\":\"How does LLM-as-a-Tutor extend the LLM’s role during training?\",\"answer\":\"It turns the LLM from a judge into a tutor with two functions: an examiner that compares pairs of policy rollouts to detect non-challenging prompts, and a generator that appends atomic constraints to make those prompts more challenging.\"},{\"question\":\"What is the key mechanism that makes the training signal self-calibrating?\",\"answer\":\"The tutor uses an append-only design: it increases prompt difficulty monotonically by adding constraints only when rollout pairs are indistinguishable in quality. This avoids relying on external difficulty schedules while keeping the seed prompt and base rubric otherwise unchanged.\"}]",1784183132,73,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"llm-as-a-tutor-policy-aware-prompt-adaptation-for-non-verifiable-rl","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/llm-as-a-tutor-policy-aware-prompt-adaptation-for-non-verifiable-rl/82813/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do static training prompts hurt non-verifiable RL with LLM judges?","Question",{"text":75,"@type":76},"Static prompts may be too easy or too hard for the current policy, so all rollouts end up with indistinguishable quality. When that happens, even a high-quality rubric judge cannot discriminate and the reward signal collapses.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does LLM-as-a-Tutor extend the LLM’s role during training?",{"text":80,"@type":76},"It turns the LLM from a judge into a tutor with two functions: an examiner that compares pairs of policy rollouts to detect non-challenging prompts, and a generator that appends atomic constraints to make those prompts more challenging.",{"name":82,"@type":73,"acceptedAnswer":83},"What is the key mechanism that makes the training signal self-calibrating?",{"text":84,"@type":76},"The tutor uses an append-only design: it increases prompt difficulty monotonically by adding constraints only when rollout pairs are indistinguishable in quality. This avoids relying on external difficulty schedules while keeping the seed prompt and base rubric otherwise unchanged.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]