Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

Jul 16, 2026

Bayesian Linguistic Forecaster (BLF) is an agentic system for binary forecasting that delivers state-of-the-art results on the ForecastBench benchmark. BLF combines a semi-structured linguistic belief state updated by an LLM at each tool-use step, multi-trial aggregation via logit-space averaging with shrinkage, and hierarchical calibration using Platt scaling with a hierarchical prior. Experiments on 400 backtesting questions show consistent superiority across question types, supported by ablations and mixed-effects analysis controlling for variability, plus a robust back-testing framework with low leakage.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages64

Reading Time2 h 41 min

Abstract

Introduction

Experimental Setup

Problem definition

arXiv :2604 . 18576v4 [ cs .AI] 12 Jul 2026 Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs Kevin Murphy Department of Computer Science University of British Columbia Vancouver, BC, Canada [murphyk@cs.ubc.ca](murphyk@cs.ubc.ca) Abstract We present the Bayesian Linguistic Forecaster (BLF), an agentic system for binary forecasting that achieves state-of-the-art performance on the ForecastBench benchmark. The system is built on three ideas. (1) Linguistic belief state: a semi-structured representation combining numerical probability estimates with natural-language evidence summaries, updated by the LLM at each step of an iterative tool-use loop. This contrasts with the common approach of appending all retrieved evidence to an ever-growing, unstructured context. (2) Hierarchical multi-trial aggregation: running K independent trials and combining them using logit-space averaging shrinkage with a data-dependent prior. (3) Hierarchical calibration: Platt scaling with a hierarchical prior, which avoids over-shrinking extreme predictions for sources with skewed base rates. On 400 questions from the ForecastBench leaderboard, BLF outperforms all the top public methods, including Cassi, GPT-5, Grok 4.20, and Foresight-32B. Careful ablation studies, using mixed effects analysis to control for question variability (which accounts for 62% of the variance in performance), reveals that all 3 components contribute to the overall gains, but some components matter more than others, depending on the base LLM, and the setting (e.g. with or without a crowd prior) . All our experiments are based on a robust back-testing framework which we develop, which has a leakage rate below 1.5%, and may be of independent interest. 1 Introduction Forecasting the probability of future events is a fundamental challenge with applications in geopolitics, finance, and public health [Tetlock and Gardner, 2015, Spiegelhalter, 2025] . Recent work has shown that LLMs can approach human-level forecasting when given web search access [Halawi et al., 2024], and benchmarks such as ForecastBench [Karger et al., 2025] provide standardized evaluation with online leaderboards. We present BLF (Bayesian Linguistic Forecaster), an agentic system that achieves a new state-of-the-art (SOTA) performance on ForecastBench. Our approach is organized around three key ideas: 1. Linguistic belief states. Most forecasting agents either search in parallel then reason once, or sequentially accumulate raw search results in context. BLF instead maintains a semi-structured belief state—a probability estimate paired with natural-language evidence summaries—updated by the LLM at each step. We refer to this loosely as “Bayesian-style” updating: the slots are designed to mirror the form of a sequential Bayesian update (prior + evidence → posterior), but the actual update is an LLM forward pass, which may not satisfy the consistency criteria required to correspond to proper Bayesian inference Qiu et al. [2025], Falck et al. [2024] . (In Sec. J, we evaluated a more Preprint. traditional Bayesian approach based on sequential updating with explicit LLM-estimated likelihoods, but it was much worse.) 2. Multi-trial aggregation. LLM forecasting exhibits high variance across runs. We run K=5 independent trials and aggregate by averaging in logit space. We also explore hierarchical shrinkage toward the empirical or uniform prior (inspired by James-Stein / empirical Bayes), which further helps on datasets with high trial variance. 3. Hierarchical calibration. To ensure the forecasts are calibrated, we use Platt scaling [Platt, 1999] . However, global Platt scaling can over-shrink well-calibrated extreme predictions. We use hierarchical Platt scaling with per-source intercept offsets, which is critical when empirical priors produce source-specific biases, especially in the zero-shot setting. On 400 backtesting questions from ForecastBench (FB), BLF ac

FAQs

What is the Bayesian Linguistic Forecaster (BLF) designed to do?

BLF is an agentic system for binary forecasting, estimating event probabilities in a sequential interaction loop and achieving state-of-the-art performance on ForecastBench.

How does BLF represent and update forecasting information over time?

BLF maintains a semi-structured linguistic belief state that pairs probability estimates with natural-language evidence summaries, which the LLM updates at each step of an iterative tool-use loop.

What techniques does BLF use to improve forecast accuracy and calibration?

BLF uses hierarchical multi-trial aggregation in logit space with shrinkage and performs hierarchical calibration via Platt scaling with a hierarchical prior to avoid over-shrinking extreme predictions.

Jul 16, 2026

Upload by Ava Thompson