Beyond the Prompt Jailbreaking Function Calling LLMs via Simulated Moderation Traces

Jul 16, 2026

Upload by Himbo

Jailbreak attacks pose a serious risk to safe deployment of large language models, especially in stateful function-calling environments. The work identifies a structural weakness beyond prompt-level studies: trusted schemas, structured arguments, untrusted tool outputs, and accumulated dialog state are interleaved in one shared context, blurring control logic and data. SMT, a black-box simulated moderation-trace framework, builds a multi-turn trajectory that exploits validation feedback to progressively weaken safety constraints and induce harmful outputs. Extensive experiments on multiple commercial LLMs show consistently highest attack success and HarmScore with few queries, proving prompt sanitization is insufficient.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

Abstract

Introduction

Motivation and threat model

Limitations of prompt-centric attacks

Function-calling architectural vulnerability

Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces Disclaimer. This paper contains examples of harmful language. Reader discretion is recommended. Junlong Liu 1,+ , Haobo Wang 1,+ , Weiqi Luo 1,* , Xiaojun Jia2 1Sun Yat-sen University, 2 Nanyang Technological University {liujlong27, [wanghb69](wanghb69}@mail2.sysu.edu.cn)[}](wanghb69}@mail2.sysu.edu.cn)[@mail2.sysu.edu.cn](wanghb69}@mail2.sysu.edu.cn), [luoweiqi@mail.sysu.edu.cn](luoweiqi@mail.sysu.edu.cn), [jiaxiaojunqaq@gmail.com](jiaxiaojunqaq@gmail.com) arXiv :2607 .0048 1v 1 [ cs .CR] 1 Jul 2026 Abstract—Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses atthe prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderationauditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext toelicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model’s safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for contextaware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at [https://github.com/liujlong27/SMT](https://github.com/liujlong27/SMT). 1. Introduction Large language models (LLMs) are increasingly deployed as software assistants [1],[2], autonomous agents [3], and tool-augmented applications’ core reasoning and con trol components [4] . To reduce the risk of generating il-+Equal contribution. * Corresponding author. legal, toxic, or harmful content [5], [6], model providers employ post-training alignment techniques, such as supervised safety fine-tuning and reinforcement learning from human feedback (RLHF) [7], [8] . Nevertheless, safety-aligned LLMs remain vulnerable to jailbreak attacks [9], [10], in which adversaries use carefully crafted prompts [11], deceptive contextual framing [12], or iterative inducement strategies [13], [14] to circumvent refusal mechanisms and elicit prohibited content [15] . Understanding such failures is crucial as LLMs move beyond standalone chat interfaces to production workflows that maintain interaction histories, invoke external functions, and process multi-source data. Most existing jailbreak research adopts a prompt-centric threat model, treating the user-visible prompt as the primary attack surface. Single-turn methods manipulate the wording, representation, or structure of harmful requests. For example, ArtPrompt [17] obfuscates sensitive terms with ASCII-art representations, while CC-BOS [14] optimizes black-box jailbreak prompts using classical Chinese contexts and multi-dimensional fruit fly optimization, Do Anything Now [

FAQs

What key vulnerability does the paper highlight in function-calling LLM systems?

It shows that shared context windows interleave trusted schemas, structured arguments, tool outputs, and prior states, blurring the boundary between control logic and untrusted data across multiple turns.

How does the SMT method generate an attack trajectory?

SMT constructs a multi-turn simulated moderation auditing workflow and uses a fabricated moderation frame to obtain harmful generations under the pretext of red-team testing.

What do the evaluation results conclude about prompt-level defenses?

Across multiple commercial LLMs and two safety benchmarks, SMT achieves the highest average attack success and HarmScore with near-minimal queries, indicating prompt-level sanitization alone cannot reliably defend tool-enabled LLM systems.

Jul 16, 2026

Upload by Himbo