CONTRA: 红队测试个性化智能体的配置树搜索方法

Jul 16, 2026

Upload by Eliana

Recent LLM-based agent systems extend beyond dialogue into autonomous, customizable workflows through modifiable internal files and installable skills. This added capability increases the risk of unintended harmful actions. The work studies how agent configuration interacts with the likelihood of executing dangerous behaviors without explicit instruction. It proposes CONTRA, an LLM-assisted tree-search method that discovers benign-looking configurations by simulating agent execution, building a skill–malicious action dataset, and evaluating attack success rates.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Configuration Generation and Sandboxed Execution

Large-Scale Evaluation

arXiv :2607 .03220v 1 [ cs .CR] 3 Jul 2026 CONTRA: Red-Teaming Configurations of Personalizable Agents Jonathan Nöther, Adish Singla, Goran Radanovic Max Planck Institute for Software Systems, Germany {jnoether,adishs,[gradanovic}](gradanovic}@mpi-sws.org)[@mpi-sws.org](gradanovic}@mpi-sws.org) Abstract Recent tools such as OpenClaw have extended the capabilities of LLM-based agents from simple dialog-based systems to fully autonomous agents. These systems allow personalization of the agent through modifiable internal files and the installation of skills. While this enables deployment in a wide range of settings and the automation of diverse tasks, greater capability and autonomy increases the risk of malicious actions being executed unintentionally. In this work, we explore the interplay between agent configuration and the risk of executing dangerous actions without explicit instruction. To this end, we propose CONfiguration Tree-search for Redteaming Agents (CONTRA), an LLM-assisted tree-search algorithm that discovers agent configurations resulting in the execution of malicious actions. CONTRA works by reasoning about benign yet dangerous configurations and evaluating them ina simulated environment. We construct a dataset of the 473 most popular skills from a public repository, along with 2–5 corresponding malicious target actions per skill. In a large-scale analysis, we find that 75. 1% of skills have at least one configuration resulting in the execution of a malicious action, most of which have not been detected as containing malicious content by existing scans. Overall, CONTRA successfully identifies a configuration leading to the execution of the target action in 39 .2% of all tested cases. Our findings demonstrate that current agents provide insufficient safety with respect to personalization. 1 Introduction General-purpose personal assistants, such as OpenClaw, possess the versatile reasoning required to manage complex workflows across emails, messages, and private files, as well as interacting with both humans and external agents. While these capabilities allow for novel useful automation, this same integration introduces a significant surface area for malicious actions and unintended consequences. A defining feature of these agents is deep personalization using the agent’s internal configuration files. By maintaining persistent notes of past interactions, the agent evolves a specialized memory and personality that is injected into its context. This allows both the memorization of past experiences, such as enabling learning from past failure, as well as customizing the behavior of the agent towards the user’s needs. Further, the agent’s utility can additionally be extended by installing custom skills, markdown files that instruct the agent on when and how to use external APIs and services. While these features drive utility, they also create a "black-box" of behavioral safety, since an agent’s actions are a direct byproduct of its unique configuration. In this paper, we explore the red-teaming of agent configurations with regards to agent skills. We specifically investigate whether, given a specific skill and a malicious target action, there exist a configuration that results in the agent executing that target action. The configuration itself should be benign, i.e. it should not use any adversarial tactics or directly instruct the agent to perform the target action. Systematically answering this allows us to map the latent vulnerabilities introduced by user-driven personalization. Preprint. Configuration Generation Sandboxed Execution Figure 1: Illustration of CONTRA using a simplified, yet real example. We start with the skill being evaluated, a target action and an archive of previous attempts. We sample one configurations from the archive and instruct the Orchestrator to reason about potential changes that are benign but could lead to the target action. This is then given to the relevant su

FAQs

CONTRA研究的核心问题是什么？

研究如何在给定特定技能和恶意目标动作的情况下，找到一种“看似无害”的智能体配置，使其在缺乏明确指令时仍会执行目标恶意动作。

CONTRA是如何发现能触发恶意行为的配置的？

CONTRA通过对配置文件进行LLM辅助的树搜索，推理哪些配置变更仍“良性”但可能导致目标动作，并在沙箱模拟环境中评估结果，由评审器判断动作与消息是否命中目标。

实验的主要发现有哪些？

在对ClawHub上473个最受欢迎技能进行大规模测试后，75.1%的技能至少存在一个会导致恶意动作执行的配置；39.2%的测试动作成功被触发，且多数导致触发的配置本身在评估中被认为是良性的。

Jul 16, 2026

Upload by Eliana