[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-573805-105":53,"doc-detail-573805-en":120},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":113,"head_meta":115,"extra_data":117,"updated_unix":119},105,"en","becoming-experienced-judges-selective-test-time-learning-for-evaluators","Becoming Experienced Judges - Selective Test-Time Learning for Evaluators","","Automatic evaluation with large language models (LLMs), often called LLM-as-a-judge, is widely used in reasoning and alignment tasks. Existing evaluators typically treat each test case independently and reuse a single fixed prompt, preventing accumulation of evaluation insights and sample-specific criteria. Learning While Evaluating (LWE) enables sequential improvement at inference time without extra training or external signals. Selective LWE updates its metaprompt only on self-inconsistent cases to reduce cost while maintaining gains, outperforming strong baselines on two benchmarks.",{"@graph":63,"@context":112},[64,80,95],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":11,"@type":70,"position":76},"https://docshare.wps.com/template/presentations/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/becoming-experienced-judges-selective-test-time-learning-for-evaluators/573805/",4,{"url":78,"name":59,"@type":81,"author":82,"headline":59,"publisher":85,"fileFormat":88,"inLanguage":57,"description":61,"dateModified":89,"datePublished":89,"encodingFormat":88,"isAccessibleForFree":90,"interactionStatistic":91},"DigitalDocument",{"name":83,"@type":84},"Jacob","Person",{"url":68,"name":86,"@type":87},"DocShare","Organization","application/pdf","2026-10-14",true,{"@type":92,"interactionType":93,"userInteractionCount":4},"InteractionCounter",{"@type":94},"ViewAction",{"@type":96,"mainEntity":97},"FAQPage",[98,104,108],{"name":99,"@type":100,"acceptedAnswer":101},"What limitation do existing LLM-as-a-judge evaluators have?","Question",{"text":102,"@type":103},"They evaluate each case independently using a single fixed prompt, so they cannot accumulate and reuse insights across cases or tailor evaluation criteria per sample.","Answer",{"name":105,"@type":100,"acceptedAnswer":106},"How does Learning While Evaluating (LWE) improve evaluators during inference?",{"text":107,"@type":103},"LWE evolves a metaprompt over sequential test cases by storing self-generated feedback as evaluation insights and using them to produce sample-specific evaluation instructions.",{"name":109,"@type":100,"acceptedAnswer":110},"What is Selective LWE and why is it more cost-effective?",{"text":111,"@type":103},"Selective LWE updates the metaprompt only on self-inconsistent or challenging cases, focusing computation where it matters while preserving performance improvements.","https://schema.org",{"og:url":78,"og:type":114,"og:title":59,"og:site_name":86,"og:description":61},"article",{"robots":116,"canonical":78},"index,follow",{"doc_id":118,"site_id":56},573805,1791946989,{"code":4,"msg":5,"data":121},{"doc_id":118,"user_id":122,"nickname":83,"user_avatar":123,"doc_module":9,"category_id":8,"category_name":11,"doc_title":59,"doc_description":61,"doc_content":124,"file_id":125,"file_url":126,"file_type":127,"file_size":128,"view_count":4,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":129,"language":130,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":131,"faqs":132,"seo_title":133,"seo_description":61,"update_tm":119,"read_time":134},962084931830,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","Becoming Experienced Judges: Selective Test-Time Learning for Evaluators  \nSeungyeon Jwa 1 Daechul Ahn 1 Reokyoung Kim 1 Dongyeop Kang2 Jonghyun Choi 1†  \n1 Seoul National University 2University of Minnesota  \n{amyj97,daechulahn,reokyoungkim,[jonghyunchoi}@snu.ac.kr](jonghyunchoi}@snu.ac.kr) [dongyeop@umn.edu](dongyeop@umn.edu)  \nAbstract  \nAutomatic evaluation with large language models, commonly known as LLM-as-a-judge, is now standard across reasoning and alignment tasks. Despite evaluating many samples in deployment, these evaluators typically (i) treat each case independently, missing the opportunity to accumulate and reuse evaluation insights across cases, and (ii) rely on a single fixed prompt for all cases, neglecting the need for sample-specific evaluation criteria. We introduce Learning While Evaluating (LWE), a framework that allows evaluators to improve sequentially at inference time without requiring additional training or external signals. LWE maintains an evolving metaprompt that (i) stores evaluation insights derived from self-generated feedback during sequential testing, and (ii) allows evaluators to leverage these insights to generate samplespecific evaluation instructions for accurate and consistent judgments. Furthermore, we propose Selective LWE, which updates the metaprompt only on self-inconsistent cases, focusing computation where it matters most. This selective approach retains the benefits of sequential learning while being far more cost-effective. Across two pairwise comparison benchmarks, Selective LWE outperforms strong baselines, empirically demonstrating that evaluators can improve during sequential testing with a simple selective update—learning most from the cases they struggle with.*  \n1 Introduction  \nLarge language models (LLMs) and visionlanguage models (VLMs) are increasingly used as automatic evaluators, commonly referred to as (V)LLM-as-a-judge (Zheng et al., 2023 ; Chen et al.,  \n†JC is with ECE, ASRI and IPAI in Seoul National University and a corresponding author.  \n* Code is available at [https://github.com/snumprlab/](https://github.com/snumprlab/)[ ](https://github.com/snumprlab/)lwe  \nFigure 1: Comparison of three evaluation approaches. A vanilla judge evaluates each test case (TC) independently using a fixed prompt throughout evaluation (eval.) . LWE (ours) employs a meta-prompt (M) that evolves sequentially as the evaluator progresses through test cases, enabling sample-specific tailoring and continual improvement during evaluation. Selective LWE (ours) further enhances efficiency by updating the meta-prompt only on challenging cases (e.g., the redhighlighted TCs 2, 5, 8, 14), preserving performance gains while substantially reducing computational overhead. The color gradient illustrates progressive improvement of the judge’s performance over time.  \n2024 ; Lambert et al., 2025) . Despite their growing importance, current evaluators remain surprisingly rigid: they typically apply a single evaluation prompt for all cases and evaluate each case in isolation, as shown in Fig. 1 (“Vanilla Judge”) . This rigidity limits evaluators in two key ways: it prevents tailoring evaluation criteria to individual testcases and the reuse of insights gained from earlier decisions. This motivates our research question: can evaluators adapt and improve during testing?  \nHumans routinely exhibit such improvement in test-time. A student taking an exam refines their strategy as they progress, and a judge becomes more experienced as they encounter diverse cases (Flavell, 1979) . Analogously, recent advances in LLMs show that allocating more inference computation per sample improves perfor-  \n697  \nProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics  \nVolume 2: Short Papers, pages 697–721  \nMarch 24-29, 2026 ©2026 Association for Computational Linguistics  \nFigure 2: Overview of proposed LWE. Given a test case xt , the meta-prompt Mt−1 generates a ","cbCaih0HK84aWhp6","https://ap.wps.com/l/cbCaih0HK84aWhp6","pdf",1687812,25,"English","# Introduction\n## Learning While Evaluating (LWE)\n## Selective LWE and Efficiency\n## Empirical Evaluation","[{\"question\":\"What limitation do existing LLM-as-a-judge evaluators have?\",\"answer\":\"They evaluate each case independently using a single fixed prompt, so they cannot accumulate and reuse insights across cases or tailor evaluation criteria per sample.\"},{\"question\":\"How does Learning While Evaluating (LWE) improve evaluators during inference?\",\"answer\":\"LWE evolves a metaprompt over sequential test cases by storing self-generated feedback as evaluation insights and using them to produce sample-specific evaluation instructions.\"},{\"question\":\"What is Selective LWE and why is it more cost-effective?\",\"answer\":\"Selective LWE updates the metaprompt only on self-inconsistent or challenging cases, focusing computation where it matters while preserving performance improvements.\"}]","Becoming Experienced Judges - Selective Test-Time Learning for Evaluators | PDF",9]