MafiaScope: 非侵入式、时间分辨的信念探测，用于社交推理游戏中的LLM智能体

Jul 16, 2026

Upload by Riley

LLM agents’ public actions provide limited insight into their social reasoning, since correct voting can be guesswork and effective lying may hide internal beliefs. MafiaScope provides an open testbed that converts Mafia into a measurement instrument for machine Theory of Mind: after each public utterance, every agent answers structured private probe questions scored against engine ground truth. An interactive visualizer tracks belief trajectories, supports impersonation and calibration panels, and enables counterfactual replay via forked steps. A 32-game DeepSeek study with 13,815 probe answers finds poor confidence calibration and includes a 30-fork replay workflow end-to-end, with released engine, viewer, and 200+ cross-model games.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Related Work

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games* Ilia Karpov HSE University [karpovilia@gmail.com](karpovilia@gmail.com) arXiv :2607 . 10645v 1 [ cs .CL] 12 Jul 2026 Abstract An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. After every public utterance, every agent privately answers a configurable set of structured probe questions; the answers never re-enter the game and are scored automatically against the ground truth the engine knows. An interactive visualizer renders the belief trajectories: impersonate mode shows the game as one agent sees it, panels chart timeline-aligned accuracy and calibration, and counterfactual replay forks any recorded step. In a 32-game DeepSeek case study with 13,815 parsed probe answers, stated confidence is poorly calibrated, with expected calibration error 0.17, agents over-predict being suspected 1.5 times, and a 30-fork replay experiment walks the counterfactual replay workflow end to end. Engine, viewer and a corpus of 200+ cross-model games are released under an open licence; live demo: [https://karpovilia](https://karpovilia). [github.io/mafiascope/](github.io/mafiascope/) ; screencast: [https:](https:)//[vimeo.com/1208920221](vimeo.com/1208920221) . 1 Introduction Multi-agent LLM systems increasingly operate where success depends on modelling other minds: negotiation, collaboration, multi-party dialogue (Tan et al., 2023 ; Wei et al., 2023) . Static Theoryof-Mind (ToM) benchmarks probe such abilities with self-contained vignettes or scripted multiparty conversations (Le et al., 2019 ; Kim et al., 2023), but cannot show how an agent’s model of other agents evolves inside an adversarial interaction it partially causes (Ma et al., 2023 ; Riemer et al., * Paper has been submitted to the EMNLP 2026 System Demonstrations track. Figure 1: The viewer on an English demo game: groundtruth graph and log in the centre, side panels are each agent’s private beliefs; node colour = assigned role, edge = suspicion. The ring on a living Mafioso,“hid N %”, is the share of the informed crowd missing it. Okabe-Ito palette with pattern redundancy. 2025) . Conversely, work that places LLMs inside social deduction games (Xu et al., 2023 ; O’Gara, 2023) typically evaluates only outcomes such as win rates and vote accuracy, leaving the agents’ beliefs a black box, or draws out reasoning inside the game context, where the questioning itself changes subsequent behaviour. Mafia 1 distils this challenge: an informed minority, the Mafia, who know eachother, covertly kills one player each night; the uninformed majority must unmask them through public discussion and daytime votes. Success on both sides hinges on modelling, and for the Mafia also managing, what the others believe. We demonstrate MafiaScope, a testbed built around one central idea, non-invasive, timeresolved belief probing: after every public utterance, privately question every agent with structured questions whose answers never re-enter the game, 1 [https://en.wikipedia.org/wiki/Mafia_](https://en.wikipedia.org/wiki/Mafia_) (party_game) and score every answer against the ground truth the engine already knows. Measurement is therefore dense in time: every agent is probed after every public message, hundreds of belief snapshots per game, locating belief revision at the level of a single utterance. It is noninvasive in a checkable sense: probe answers never enter any agent’s persistent context; §4 spells out this guarantee and its limits. And it is self-scoring: Mafia’s hidden roles are known to the engine, so first-order beliefs (“Casey is Mafia”) are directly gradeable, and second-order beli

FAQs

What were key findings from the DeepSeek case study?

Across 32 games and 13,815 parsed probe answers, stated confidence was poorly calibrated (ECE 0.17), with agents over-predict being suspected by about 1.5 times, and the counterfactual replay workflow was demonstrated end-to-end with 30 forks.

Jul 16, 2026

Upload by Riley