[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82147-en":3,"doc-seo-82147-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82147,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing","Correlation-aware contextual bandit algorithms are developed for LLM routing where arm rewards are correlated and auxiliary surrogate signals come from an ML predictor. The work addresses shortcomings of classical contextual bandits that assume conditional independence and only use selected-arm bandit feedback. Two designs are proposed: coupled reward-mixing for faster learning under reliable surrogates, and decoupled prediction-mixing for robustness. Regret analyses and evaluations on LLM routing benchmarks quantify improved sample efficiency and better accuracy–cost trade-offs under varying accuracy versus cost.","arXiv :2607 .090 15v 1 [ cs .LG] 10 Jul 2026  \nCorrelation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing  \nAjay Narayanan Sridhar, Ronak Singh, Mehrdad Mahdavi, Vijaykrishnan Narayanan  \nThe Pennsylvania State University  \n{afs6372,rjs7006,mzm616,[vxn9}@psu. edu](vxn9}@psu. edu)  \nAbstract  \nWe study contextual bandit problems with correlated arms and access to surrogate reward signals produced by a machine learning model, motivated by applications such as large language model (LLM) routing. Unlike classical contextual bandits that rely solely on bandit feedback and assume conditional independence across arms, our setting allows context-dependent inter-arm correlations and auxiliary reward information that may be noisy or misspecified. We propose algorithms that leverage such surrogate rewards through two complementary designs. A coupled reward-mixing approach pools true and surrogate rewards to accelerate learning when surrogate signals are reliable, while a decoupled prediction-mixing approach maintains separate estimators for bandit feedback and surrogate rewards and adaptively combines their predictions. This decoupling yields robustness to surrogate misspecification, recovering regret guarantees comparable to reward-only bandit methods in the worst case, while achieving improved regret when surrogate predictions are sufficiently informative. We provide theoretical regret analyses for both approaches and evaluate them on LLM routing benchmarks under varying accuracy versus cost trade-offs. The results demonstrate improved sample efficiency and consistently better accuracy–cost trade-offs compared to standard contextual bandit baselines and strong static routing methods.  \n1 Introduction  \nThe rapid proliferation of Large Language Models (LLMs) has made it increasingly challenging for end-users to keep track of advancements and optimally select models for their specific needs. Presently, numerous proprietary LLM providers exist [38, 23 , 51], and thousands more are openly available through repositories such as Hugging Face [28] . This abundance creates a practical systems problem: for a given query, which model should be called? The answer is rarely the most accurate model, since inference incurs nontrivial cost (e.g., pricing, latency, compute) . In many applications, the objective is an accuracy–cost trade-off: a slightly less accurate model may be preferable if it is significantly cheaper or faster, and the optimal choice varies with context.  \nExisting unified interfaces address this challenge through model cascading [11 , 26], ensembling [30], and routing [36, 37 , 55 , 42 , 33] . We focus on routing: selecting one model per query to maximize user-defined utility. A common reduction models LLM routing as a contextual multi-armed bandit, where each LLM is treated as a conditionally independent arm given the incoming query context, and only the selected arm’s reward is observed, requiring costly exploration. In the d-dimensional linear setting, where the context is represented by a d-dimensional embedding and each arm’s expected reward is assumed to be a linear function of this embedding, standard methods achieve regret RS = O ( pdKT log(T/d)) [19] given K arms (LLMs) for T stream of queries, where regret is the cumulative loss relative to the best context-dependent action. However, this conditional-independence abstraction is often too coarse: models exhibit query-dependent correlations due to shared pretraining data, alignment pipelines, and architectures. Exploiting such correlations can reduce uncertainty about unplayed arms and lower the effective exploration burden.  \nLLM routing also provides a source of side information largely absent from standard bandit settings: offline performance data. Benchmarks or historical logs can train machine learning (ML) predictors that map query  \ncontexts to per-arm reward estimates. During online routing, these estimates can serve as surrogate rewards for unplay","cbCaib92VuG5ySlc","https://ap.wps.com/l/cbCaib92VuG5ySlc","pdf",3397647,2,1,44,"English","en",105,"# Introduction\n## Problem setting: LLM routing as contextual bandits\n## Correlation-aware and surrogate-reward motivation\n## Proposed approaches: coupled and decoupled designs","[{\"question\":\"How does the method use surrogate rewards in LLM routing?\",\"answer\":\"After selecting an LLM arm and observing the realized reward, the algorithm receives ML-predicted surrogate rewards for additional arms identified by a feedback graph. These surrogates provide partial information beyond the chosen arm.\"},{\"question\":\"What is the key difference between coupled and decoupled reward-mixing approaches?\",\"answer\":\"The coupled approach pools de-biased true rewards with surrogate rewards to fit a single contextual model, accelerating learning when surrogates are reliable. The decoupled prediction-mixing approach keeps separate estimators for bandit feedback and surrogate rewards, then adaptively combines them for robustness.\"},{\"question\":\"Why do inter-arm correlations matter in this setting?\",\"answer\":\"Because LLMs exhibit query-dependent correlations caused by shared training, alignment, and architectures, exploiting these correlations can reduce uncertainty about unplayed arms and lower the effective exploration burden.\"}]",1784178442,111,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"correlation-aware-contextual-bandits-with-surrogate-rewards-for-llm-routing","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/correlation-aware-contextual-bandits-with-surrogate-rewards-for-llm-routing/82147/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-20","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the method use surrogate rewards in LLM routing?","Question",{"text":75,"@type":76},"After selecting an LLM arm and observing the realized reward, the algorithm receives ML-predicted surrogate rewards for additional arms identified by a feedback graph. These surrogates provide partial information beyond the chosen arm.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the key difference between coupled and decoupled reward-mixing approaches?",{"text":80,"@type":76},"The coupled approach pools de-biased true rewards with surrogate rewards to fit a single contextual model, accelerating learning when surrogates are reliable. The decoupled prediction-mixing approach keeps separate estimators for bandit feedback and surrogate rewards, then adaptively combines them for robustness.",{"name":82,"@type":73,"acceptedAnswer":83},"Why do inter-arm correlations matter in this setting?",{"text":84,"@type":76},"Because LLMs exhibit query-dependent correlations caused by shared training, alignment, and architectures, exploiting these correlations can reduce uncertainty about unplayed arms and lower the effective exploration burden.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]