ASK in the Dark - Uncertainty-Gated LLM Assistance under Partial Observability

Sep 21, 2026

Uploaded by Maeve

Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) with broad reasoning priors. However, integrating SLM guidance has been difficult: standard uncertainty-gated methods yield near-zero overwrite across environments, indicating the model rarely changes the policy. The work attributes this to a prompt context bottleneck and introduces ASK+, which adds trajectory-aware context and structured reasoning. It shows strong gains across multiple POMDP-style benchmarks, while entropy-based gating targets action uncertainty rather than state uncertainty, keeping assistance effective beyond fully observable settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Introduction

ASK and uncertainty-gated guidance

Evaluation setup under partial observability

Contributions

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability Juarez Monteiro1, *,†, Nathan Gavenski2,†, Guilherme Lima1 , Francisco Galuppo 1 , Odinaldo Rodrigues2 and Adriano Veloso1 1 Kunumi Institute, Belo Horizonte, MG, Brazil 2 King’s College London, London, United Kingdom Abstract Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) that carry broad reasoning priors. Yet integrating SLM guidance into this setting has proven difficult: across all test environments, vanilla uncertaintygated approaches achieve an overwrite rate at or near zero, meaning the SLM almost never contributes an independent action. We trace this failure to the bare egocentric prompt, which provides insufficient context for genuine reasoning, and identify it as a context problem rather than a capacity problem. We propose ASK+, which supplies the SLM with trajectory-aware context (a partially revealed map, visited positions, and action history) and structured chain-of-thought reasoning, converting it from a passive redundancy check into a more informative consultant that occasionally corrects the policy. We further establish that the predictive entropy signal used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings. The stateful prompt drives substantial gains: on DoorKey, where vanilla ASK matches PPO (both 89%), ASK+ reaches 93% success; on FourRooms, success climbs from 53% to 70%; on HigherLower, accuracy reaches 73. 7%, matching the SLM-only upper bound. Across all environments, Qwen3.5-2B matches or exceeds Qwen3.5-4B, confirming that prompt design and selective gating dominate the impact of model scale, enabling guidance without large models. Keywords partial observability, reinforcement learning, language models, uncertainty estimation, Monte Carlo Dropout 1. Introduction Agents operating under partial observability must act on incomplete information, relying on egocentric observations that may be insufficient to distinguish between meaningfully different underlying states. This setting is particularly challenging for reinforcement learning (RL) agents, which learn during training to approximate a value function that reflects the specific structure of the training environment. When that structure changes, such as the repositioning of a door, the agent has no mechanism to detect the mismatch. Its observation looks familiar, and it follows its value function confidently toward failure. Recent work [1, 2] addresses this challenge by combining external guidance from language models (LMs) with the RL agent. LMs are a natural candidate for this role: trained on vast amounts of human knowledge, they can reason about goals, infer structure, and suggest sensible actions even in novel situations. However, this reasoning ability is contingent on having sufficient context. Under partial observability, the agent’s view of the world is inherently limited, and an LM consulting only a narrow, egocentric observation may lack the global context needed to reason reliably, producing guidance that is confidently wrong rather than helpfully corrective [3] . The ASK framework [1] addresses this by selectively gating small LM (SLM) queries, consulting the language model only when the policy’s uncertainty exceeds a threshold. In its original evaluation on FrozenLake, a fully observable environment, this selective strategy allowed ASK to approach PPO-level PRL+CAIPI Workshop @ IJCAI-ECAI 2026 * Corresponding author. † These authors contributed equally. $ [juarez@kunumi.com](juarez@kunumi.com) (J. Monteiro); [nathan.schneider_gavenski@kcl.ac.uk](nathan.schneider_gavenski@kcl.ac.uk) (N. Gavenski); [adriano@kunumi.com](adriano@kunumi.com) (G. Lima); [francisc

FAQs

Why does vanilla uncertainty-gated LLM assistance fail under partial observability?

It is traced to an egocentric prompt context bottleneck that provides insufficient information for genuine reasoning. As a result, the SLM rarely contributes an independent action rather than correcting the policy.

What does ASK+ add to make guidance work in POMDP settings?

ASK+ supplies trajectory-aware, stateful context to the SLM, including a partially revealed map, visited positions, action history, and structured chain-of-thought reasoning.

What uncertainty signal is used for selective querying, and what does it measure?

The method uses predictive entropy for selective querying. The paper establishes that this signal measures action uncertainty rather than state uncertainty, and it remains informative in POMDPs.

Sep 21, 2026

Uploaded by Maeve