[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81627-en":3,"doc-seo-81627-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},81627,8796095360427,"Lucas Martin","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","RELISH: LLM Regression with a Latent Iterative State Head","RELISH (Regression with a Latent Iterative State Head) introduces a lightweight architecture for text regression using large language models. Instead of decoding numeric targets as text or combining multiple generations, it predicts scalar values directly from frozen LLM representations. A learned latent state is iteratively refined via cross-attention over token-level features, then mapped to a point estimate with a linear regressor. Across six datasets, four backbones, and two training regimes, RELISH consistently surpasses autoregressive, regression-aware inference, and predictive-head baselines while requiring only ~3.4–3.7M trainable parameters.","RELISH: LLM REgression with a Latent Iterative State Head  \nYiheng Su  \nSchool of Information The University of Texas at Austin [sam.su@utexas.edu](sam.su@utexas.edu)  \nMatthew Lease  \nSchool of Information The University of Texas at Austin [ml@utexas.edu](ml@utexas.edu)  \narXiv :2604 .0 1206v2 [ cs .CL] 10 Jul 2026  \nAbstract  \nWe present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-attention over token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across six datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only ∼3.4–3.7M trainable parameters across frozen LLM backbones (only 0.01–0.04% additional overhead), far less than LoRA-based alternatives that grow with model size (0.26–0.42%) . Our code is available at [https://github.com/SamSoup/RELISH](https://github.com/SamSoup/RELISH).  \n1 Introduction  \nIn the era of large language models (LLMs), most natural language processing (NLP) tasks are unified under the text-to-text paradigm, in which models consume text inputs and produce text outputs (Raffel et al., 2020; Anthropic, 2025; Singh et al., 2025; Yang et al., 2025; Google DeepMind, 2026) . Theoretically, this paradigm is appealing because the same pretrained model, linguistic interface, and decoding procedure can be applied across diverse applications that traditionally required different architectures. Empirically, this paradigm has been particularly successful for intrinsically generative tasks such as question answering, summarization, translation, and code generation (Clark et al., 2018; Hendrycks et al., 2020; Chen et al., 2021; Srivastava et al., 2023; Wang et al., 2024b; Kazemi et al., 2025; Li et al., 2025b; Center for AI Safety et al., 2026) . Here, the need to produce coherent, contextually appropriate text aligns naturally with the language modeling objectives used to pre-train LLMs, such as autoregressive next-token prediction (Radford et al., 2019; Brown et al., 2020) .  \nHowever, the text-to-text paradigm is less suited to predictive tasks such as classification and regression (Lukasik et al., 2025) . Unlike generative tasks whose outputs are naturally text, predictive tasks seek discrete labels or continuous scalars. In regression, this mismatch makes text generation suboptimal because predictive quality depends on numerical error, whereas language modeling penalizes token mismatches. For example, if the target is 1.0, a regression objective like MSE prefers 0.9 to 0.1, whereas a language modeling objective like cross-entropy treats them as discrete tokens and remains agnostic to numerical proximity.  \nRecent work has explored how to adapt LLMs to regression. Three major families have emerged (see Table 1): autoregressive decoding (Vacareanu et al., 2024; Song et al., 2024; Song & Bahri, 2025; Akhauri et al., 2025), regression-aware inference (Lukasik et al., 2024; 2025; Chiang et al., 2025), and predictive heads (Xin et al., 2021; Wang et al., 2022; Zhuang et al., 2023; Fernandes et al., 2023; Zhang & Li, 2024; Tang et al., 2025; Nguyen et al., 2024) . Autoregressive methods simply prompt the LLM. Regression-aware methods complement next-token generation with regression-aware decision rules that explicitly minimize numerical error.  \nPredictive heads avoid text generation entirely by finetuning a predictive head (e.g., a linear layer) attached to the LLM to direct","cbCaiaAyn0249wce","https://ap.wps.com/l/cbCaiaAyn0249wce","pdf",2499388,4,1,50,"English","en",105,"# Introduction\n## Why text-to-text is suboptimal for regression\n## Prior families of LLM regression methods\n## The RELISH approach and motivation","[{\"question\":\"What problem does RELISH address in LLM regression?\",\"answer\":\"RELISH addresses the mismatch between text generation objectives and numeric targets in regression, where token-based losses do not reflect numerical proximity and naive predictive heads may miss regression-relevant information.\"},{\"question\":\"How does RELISH produce scalar regression outputs?\",\"answer\":\"RELISH iteratively refines a learned latent state using cross-attention over token-level representations, then maps the final latent state to a point estimate using a linear regressor.\"},{\"question\":\"How does RELISH compare to prior regression approaches in performance and efficiency?\",\"answer\":\"RELISH outperforms prior baselines across multiple datasets, LLM backbones, and training regimes while remaining highly parameter-efficient, using only about 3.4–3.7M trainable parameters on frozen LLMs.\"}]",1784174954,126,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"relish-llm-regression-with-a-latent-iterative-state-head","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/relish-llm-regression-with-a-latent-iterative-state-head/81627/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does RELISH address in LLM regression?","Question",{"text":75,"@type":76},"RELISH addresses the mismatch between text generation objectives and numeric targets in regression, where token-based losses do not reflect numerical proximity and naive predictive heads may miss regression-relevant information.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does RELISH produce scalar regression outputs?",{"text":80,"@type":76},"RELISH iteratively refines a learned latent state using cross-attention over token-level representations, then maps the final latent state to a point estimate using a linear regressor.",{"name":82,"@type":73,"acceptedAnswer":83},"How does RELISH compare to prior regression approaches in performance and efficiency?",{"text":84,"@type":76},"RELISH outperforms prior baselines across multiple datasets, LLM backbones, and training regimes while remaining highly parameter-efficient, using only about 3.4–3.7M trainable parameters on frozen LLMs.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":22,"slug":113},6,"Technology","technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]