[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82501-en":3,"doc-seo-82501-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82501,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",8,"Research & Report","Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training","Large language model test-time training (TTT) is commonly assessed with local proxy metrics such as lower perplexity, improved future-token loss, long-context gains, or reward increases after updates on recent tokens, retrieved context, or targeted attempts. These metrics match stream or domain adaptation claims, yet provide weaker evidence for deployed assistant memory and sparse post-deployment learning. The framework calibrates TTT memory claims to behavioral evidence, using a claim-calibrated evidence ladder and an evaluation protocol with explicit-memory baselines and disjoint failure categories.","arXiv :2607 .00368v 1 [ cs .CL] 1 Jul 2026  \nBeyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM  \nTest-Time Training  \nXiangchen Song 1 Zhenhao Chen2 Lingjing Kong 1 Shaoan Xie 1  \nXinshuai Dong 1 Guangyi Chen 1 ,2 Kun Zhang 1 ,2  \n1 Carnegie Mellon University 2MBZUAI  \n[xiangchs@cs.cmu.edu](xiangchs@cs.cmu.edu) , [kunz1@cmu.edu](kunz1@cmu.edu)  \nAbstract  \nLarge language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, targetdomain data, or verifiable task attempts, and then judged by perplexity, futuretoken loss, long-context performance, or reward. These metrics are well matched to claims about stream adaptation, domain adaptation, context compression, and reward-backed test-time improvement. They are weaker evidence, however, for a capability that TTT results are increasingly used to motivate: deployed assistant memory, personalization, or sparse post-deployment learning, which instead requires behavioral evidence such as later recall, paraphrase robustness, retention, locality, conflict handling, and use in downstream actions after the original support context is removed. We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them. It has two components: a claim-calibrated evidence ladder that separates stream/domain adaptation, bridge internalization, and deployment-time behavioral learning; and an evaluation protocol with matched explicit-memory baselines and mutually exclusive failure categories.  \nWe validate the framework by auditing recent TTT and memory-adjacent work and by instantiating it as a controlled diagnostic in which, in a sparse nonce-fact setting, one-step LoRA updates lower support and answer loss across three Qwen3 model scales while generated free-form recall stays at zero, exposing a measurable gap between proxy improvement and deployment behavior. The framework gives authors and evaluators a concrete standard for aligning TTT memory claims with the evidence actually reported.  \n1 Introduction  \nTest-time training (TTT) challenges the conventional “train, then deploy” boundary by allowing model states or parameters to change during inference. In large language models, recent work has made this idea technically concrete: models may update from retrieved neighbors [11], learn online hidden-state updates through fast weights [25], perform large-chunk updates for throughput and state capacity [35], or align online updates with next-token prediction [9] . Related work studies targeted context-specific updates [4], meta-learned long-context learning [28], parameter-efficient context memories [7], locally supported parametric memories [17], input-perplexity minimization [13], label-free uncertainty signals [32], unlabeled reinforcement learning [40], and self-directed update data [41, 1] .  \nAcross these settings, the evaluation recipe is relatively stable. A model is updated at test time on recently observed tokens, retrieved examples, task attempts, or generated data, and performance is then reported through lower perplexity, better future-token prediction, improved long-context  \nPreprint.  \naccuracy, or higher reward. These results show real progress on in-sequence adaptation and compact use of recent context, often addressing practical limits of transformer-based systems such as finite context windows or static parameters. They also explain why TTT is attractive for LLM systems: online updates may allow a model to adapt to new evidence rather than relying only on fixed parameters or the current prompt.  \nThe central observation of this paper is that this evidence is not always calibrated to the claims it is used to support. In real-world assistant settings, the relevant question is often not whether an update improves prediction on a nearby continuation. It is whether a deployed model can hear a sparse user preference, acquir","cbCaiiTmGi9UJmyw","https://ap.wps.com/l/cbCaiiTmGi9UJmyw","pdf",842255,1,23,"English","en",105,"# Abstract\n# 1 Introduction","[{\"question\":\"Why are proxy metrics like perplexity insufficient for deployment-memory claims in test-time training?\",\"answer\":\"Proxy improvements on local likelihood often do not verify that a deployed model can later recall, retain, paraphrase robustly, or resolve conflicts after the original support context is removed.\"},{\"question\":\"What core idea does the paper introduce for evaluating TTT memory claims?\",\"answer\":\"It proposes a behavioral evaluation framework that calibrates memory or personalization claims to behavioral evidence beyond perplexity, including later recall, locality, retention, and conflict handling.\"},{\"question\":\"How does the framework distinguish between in-sequence adaptation and deployment-time behavioral learning?\",\"answer\":\"In-sequence adaptation updates on a support chunk and is tested on future tokens from the same stream, while deployment-time behavioral learning requires using sparse, high-value interactions later in a stable and personalized way.\"}]",1784180959,58,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"beyond-perplexity-a-behavioral-evaluation-framework-for-deployment-memory-claims-in-llm-test-time-training","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/beyond-perplexity-a-behavioral-evaluation-framework-for-deployment-memory-claims-in-llm-test-time-training/82501/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why are proxy metrics like perplexity insufficient for deployment-memory claims in test-time training?","Question",{"text":74,"@type":75},"Proxy improvements on local likelihood often do not verify that a deployed model can later recall, retain, paraphrase robustly, or resolve conflicts after the original support context is removed.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What core idea does the paper introduce for evaluating TTT memory claims?",{"text":79,"@type":75},"It proposes a behavioral evaluation framework that calibrates memory or personalization claims to behavioral evidence beyond perplexity, including later recall, locality, retention, and conflict handling.",{"name":81,"@type":72,"acceptedAnswer":82},"How does the framework distinguish between in-sequence adaptation and deployment-time behavioral learning?",{"text":83,"@type":75},"In-sequence adaptation updates on a support chunk and is tested on future tokens from the same stream, while deployment-time behavioral learning requires using sparse, high-value interactions later in a stable and personalized way.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]