Semantic Bandits - In-Context Exploration-Exploitation is Biased by Semantic Priors - arXiv

Sep 15, 2026

Uploaded by Aurora

Large language models increasingly serve as decisionmaking agents in environments that demand careful exploration and exploitation. Existing studies raise questions about how these agents truly balance both behaviors. Unlike classical agents, LLM agents interact through natural language, creating semantic cues without explicit task counterparts. We introduce the semantic bandit framework to analyze how language-derived semantic priors from pre-training shape exploration. Semantically informative action labels reduce exploration, while negative rewards provoke substantially more exploration, revealing reliability and robustness implications.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages34

Reading Time1 h 26 min

AccessFree to download

Introduction

Motivation: limitations of classical RL evaluation

Semantic multi-armed bandit framework

Empirical findings: action labels and reward polarity

arXiv :2608 . 16707v1 [ cs .CL] 17 Aug 2026 Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors David Eric Austin1,2 Kaheer Suleman3 Jackie Chi Kit Cheung1,2,4 1McGill University 2Mila – Quebec AI Institute 3Skyfall AI 4Canada CIFAR AI Chair, Mila Abstract Large language models (LLMs) are increasingly deployed as decisionmaking agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors—inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings. 1 Introduction Large language models (LLMs) exhibit sophisticated capabilities on a wide variety of tasks (Bubeck et al., 2023; Srivastava et al., 2023) . This has led to widespread deployment of LLMbased decision-making agents in in-context reinforcement learning (ICRL) tasks, which require acting on incomplete information and adapting across sequential turns based on feedback (Shinn et al., 2023; Zhou et al., 2023; Drouin et al., 2024). Any such decision-making agent must balance between exploring new actions to gather information and exploiting past actions known to yield high reward (Sutton & Barto, 2020) . While prior work (Krishnamurthy et al., 2024; Monea et al., 2024) evaluates ICRL using frameworks from classical reinforcement learning (RL), we argue that these frameworks are incomplete. Classical decision-making agents operate on symbolic problem representations. These representations are deliberately minimal: they preserve the formal structure of the task (states, actions, rewards) and discard everything else. In contrast, LLMs capture additional semantic information beyond the formal task structure. The name assigned to an action (Monea et al., 2024; Min et al., 2022), the magnitude and sign of numeric values (Mirzadehet al., 2025), and the domain in which a problem is situated (Kambhampati et al., 2024) are all semantically meaningful to a language model in ways they are not to a classical agent. This additional information may reflect genuine regularities in the world or it may consist of inscrutable associations that are not human-interpretable, learned as word co-occurrence statistics during pre-training. We refer to the inductive biases that arise from these learned associations as semantic priors. These biases have no formal counterpart in classical RL. Due to this fundamental difference, evaluating ICRL using frameworks from classical RL leaves out crucial explanatory variables. To address this shortcoming, we introduce the semantic multi-armed bandit (Figure 1), a novel extension of the canonical bandit problem that explicitly parametrizes key aspects of the textual representation alongside the standard reward distributions, and use it to demonstrate how ICRL exploration is biase

FAQs

What is the semantic bandit introduced in the paper?

The semantic bandit extends the multi-armed bandit by explicitly parameterizing textual labels for actions alongside standard reward distributions, enabling analysis of language-driven exploration behavior.

How do semantic priors affect exploration versus exploitation?

Semantically informative action labels bias the model toward exploitation, reducing exploration when labels align with the reward structure and worsening performance when misaligned.

Why do negative rewards lead to more exploration?

Negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions learned during pre-training.

Sep 15, 2026

Uploaded by Aurora