Bayesian Uncertainty Propagation for Agentic RAG Pipelines Proof-of-Concept Study on Multi-Hop Question Answering

Jul 16, 2026

Upload by Ivy

Trustworthy deployment of Agentic Retrieval-Augmented Generation (RAG) systems requires mechanisms for estimating when multistage reasoning pipelines may fail. This work introduces an uncertainty-aware Agentic RAG framework in which planner, evaluator, and generator stages emit uncertainty signals from semantic divergence and generator self-evaluation. Signals are propagated through a Bayesian Network to produce system-level uncertainty and node-level failure indicators across the workflow. Experiments on StrategyQA and HotpotQA evaluate discrimination, selective prediction, calibration, and expected error behavior.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Background and Motivation

Methodology on Uncertainty Quantification

Uncertainty Monitoring for LLM Agents

arXiv :2607 .00972v 1 [ cs .AI] 1 Jul 2026 Bayesian Uncertainty Propagation for Agentic RAG Pipelines: A Proof-of-Concept Study on Multi-Hop Question Answering Louis Donaldson 1 [0009-0000-3862-8061], Connor Walker 1 [0009-0008-8181-1644], Koorosh Aslansefat1 [0000-0001-9318-8177], and Yiannis Papadopoulos 1 [0000-0001-7007-5153] University of Hull, Hull HU6 7RX, UK [l.donaldson-2020@hull.ac.uk](l.donaldson-2020@hull.ac.uk) Abstract. Trustworthy deployment of Agentic Retrieval-Augmented Generation (RAG) systems requires mechanisms for estimating when multistage reasoning pipelines may fail. This paper presents an uncertaintyaware Agentic Retrieval-Augmented Generation (RAG) framework in which planner, evaluator and generator stages produce uncertainty signals derived from semantic divergence and generator self-evaluation. These signals are propagated through a Bayesian Network (BN) to estimate system-level uncertainty and provide node-level indicators of potential failure points across the workflow. The approach is evaluated on StrategyQA and HotpotQA using GPT-3.5-Turbo and GPT-4.1-Nano, with Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Accuracy-Rejection Curve (AUARC), Expected Calibration Error (ECE), and Brier Score used to assess discrimination, selective prediction and calibration. Results show that Bayesian propagation is more effective on HotpotQA, where uncertainty accumulates across multi-hop reasoning stages, while StrategyQA exposes limitations caused by miscalibration and unreliable upstream signals. The study positions Bayesian uncertainty propagation as a promising but preliminary mechanism for monitoring Agentic RAG systems, with future validation required in industrial domains such as Offshore Wind (OSW) maintenance decision support. Keywords: Multi-Agent Systems · Uncertainty Quantification · Runtime Monitoring · AI-Enabled Industrial Decision Support Systems 1 Introduction As Large Language Models (LLMs) become increasingly integrated into industrial workflows, their ability to automate routine tasks and provide humanunderstandable interpretations of complex data has the potential to improve operational efficiency across a wide range of sectors. As confidence in these systems 2 L. Donaldson et al. grows, they are expected to play increasingly important roles in business-critical processes. One potential application area is Offshore Wind (OSW) maintenance scheduling, where increasing turbine complexity and expanding wind farm capacity require the interpretation of large volumes of operational data to support maintenance planning. Recent work has demonstrated the potential of LLMbased decision-support and safety monitoring within this domain, while also highlighting the need for reliable run-time monitoring and trustworthy uncertainty estimation [16] . 1.1 Background and Motivation The adoption of LLMs within specialised domains is accompanied by significant challenges regarding transparency, reliability and trust. In healthcare, the inability to inspect the reasoning process behind generated recommendations may affect clinical decision making and patient safety, while privacy considerations further complicate deployment [13, 1] . Similar concerns arise in finance and legal applications, where opaque reasoning makes regulatory compliance and decision auditing difficult [1, 13] . In scheduling, manufacturing and Predictive Maintenance (PdM), operators often require understandable explanations before trusting automated decisions, yet conventional black-box models provide limited insight into the reasoning behind their outputs [13, 20, 9] . Retrieval-Augmented Generation (RAG) partially addresses these limitations by grounding generated responses in retrieved knowledge sources, allowing supporting evidence to be inspected and verified by human operators [2, 3, 15] . Hybrid and Multi-Agent Architectures (MAAs) further improve reasoning by decomposing comp

FAQs

What uncertainty sources are used in the Agentic RAG framework?

The planner, evaluator, and generator stages generate uncertainty signals based on semantic divergence and generator self-evaluation, which are then used for propagation.

How does the method estimate both system-level and stage-level uncertainty?

A Bayesian Network combines node-level uncertainty estimates to produce an interpretable system-level confidence score and indicates which stages contribute most to overall uncertainty.

What benchmarks and models are used for the proof-of-concept evaluation?

The approach is evaluated on StrategyQA and HotpotQA multi-hop question answering benchmarks using GPT-3.5-Turbo and GPT-4.1-Nano.

Jul 16, 2026

Upload by Ivy