When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games

Jul 16, 2026

Upload by Liam

As large language models operate as autonomous agents that state intentions before acting, a core safety issue is whether public commitments are honored under private deviation. This work studies LLM agents in repeated n-player games using a three-stage protocol that separates private intent, public announcements, and final actions, enabling classification of deviations as premeditated or impulsive. Experiments with three frontier models across six games and 10 rounds reveal planned deviations and persistent, model-specific announcement semantics mismatches.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Introduction

Research gaps in existing evaluations

Three-stage endogenous promise protocol

Research questions

When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games Jerick Shi 1 2 Terry Jingcheng Zhang 2 3 Bernhard Schlkopf 4 Vincent Conitzer * 1 Zhijing Jin * 2 3 4 arXiv :2607 .05 132v2 [ cs .CY] 7 Jul 2026 Abstract As large language models are deployed as autonomous agents that communicate intentions before acting, a critical safety question is whether agents that publicly commit to actions will honor those commitments. We place LLM agents in repeated n-player games with a three-stage protocol that separates private intent, public announcement, and final action, allowing us to identify whether each deviation from a stated announcement was already planned during private deliberation. Evaluating three frontier models across six games in homogeneous and heterogeneous groups over 10 rounds, we report two findings. First, when agents deviate from their announcements, the deviation is predominantly already stated in their private plan (exceeding 90% in the highest-deception conditions), yet this is not a fixed model property: the same model ranges from perfect honesty to near-total deviation across games. Second, different models interpret announcements incompatibly, some as binding commitments and others as cheap talk, producing payoff gaps that emerge in Round 0 and persist across all 10 rounds. Systems that combine models from different providers therefore cannot assume shared announcement semantics and require empirical testing of model interactions before deployment. 1 1. Introduction As large language models (LLMs) transition from passive tools to autonomous agents that plan, negotiate, and take 1 Carnegie Mellon University, Pittsburgh, USA 2Jinesis AI Lab, Vector Institute and University of Toronto, Toronto, Canada 3EuroSafeAI 4Max Planck Institute for Intelligent Systems, T¨ubingen, Germany. Correspondence to: Jerick Shi <[junkais@andrew.cmu.edu](junkais@andrew.cmu.edu) >. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . 1Code at [https://github.com/Jerick-1380/](https://github.com/Jerick-1380/)[ ](https://github.com/Jerick-1380/)LLM-Trust-Breaking. consequential actions (Xi et al., 2025), they are increasingly deployed in multi-agent settings where they communicate intentions before acting (Wang et al., 2024 ; Filippas et al., 2024) . A critical safety question is whether agents that publicly commit to actions will honor those commitments when they can privately deviate. Existing evaluations of LLM deception demonstrate that models frequently misrepresent intended actions when doing so is instrumentally useful (Taylor & Bergen, 2025 ; Hagendorff, 2023), engage in in-context scheming (Meinke et al., 2024), and deviate from commitments in game-theoretic settings (Akata et al., 2023 ; Poje et al., 2024) . However, these evaluations are largely limited to one-shot or short-horizon interactions with homogeneous model groups. Real-world deployment rarely involves isolated one-shot interactions. Agents interact repeatedly, accumulate reputations, and encounter partners running different models. Whether deceptive patterns from constrained settings persist, attenuate, or worsen under richer conditions is unknown; recent benchmarks of cooperation-sustaining mechanisms in repeated social dilemmas (Tewolde et al., 2026) evaluate whether cooperation holds, but not whether agents honor stated commitments, and we return to this distinction in §2 . Three limitations of existing evaluations leave critical gaps: one-shot protocols cannot assess whether deception persists or attenuates once agents observe consequences; exogenously assigned announcements cannot determine whether deception is premeditated or impulsive; and homogeneousmodel evaluations miss the exploitation risks that arise when deployed systems combine models from different providers (Hammond et al., 2025) that interpret communication signals

FAQs

How does the three-stage protocol separate intent, announcement, and action?

Agents first privately plan (Stage 1), then publicly announce intentions in a round-robin order (Stage 2), and finally choose final actions (Stage 3). A subsequent trust reflection phase influences planning in the next round.

What distinguishes premeditated from impulsive promise breaking?

By comparing whether the announced commitment is already reflected in the private plan stage versus emerging later, deviations can be classified as premeditated or impulsive.

Why can systems that mix models from different providers fail to rely on shared announcement meaning?

Different models interpret the same public announcements incompatibly—some treat them as binding commitments while others treat them as cheap talk—creating persistent payoff gaps across rounds.

Jul 16, 2026

Upload by Liam