AMT-X: A Phase-Structured Multi-Turn Red-Teaming Framework with Checklist-Gated Dual-Metric Evaluation for LLM Safety

Jul 16, 2026

LLM safety evaluation often relies on single-turn attack datasets and single-judge scoring, which underestimates risk from adaptive adversaries that operate through extended dialogues. AMT-X (Adaptive Multi-Turn Exploitation) introduces a phase-structured multi-turn red-teaming framework as an explicit, reproducible state machine with semantic-driven transitions. It replaces single-judge evaluation with a multi-role jury and phase-conditioned checklists, yielding dual metrics that distinguish partially actionable outputs from fully operational, real, and actionable harm.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages35

Reading Time1 h 28 min

Introduction

Motivation for multi-turn evaluation

Limitations of existing benchmarks

Proposed AMT-X contributions

arXiv :2607 . 1 1 15 1v 1 [ cs .CR] 13 Jul 2026 AMT-X: A Phase-Structured Multi-Turn Red-Teaming Framework with Checklist-Gated Dual-Metric Evaluation for LLM Safety YI TING SHEN, Vulcan Research, AIFT, Singapore KENTAROH TOYODA, Vulcan Research, AIFT, Singapore ALEX LEUNG, Vulcan Research, AIFT, Singapore Safety evaluation of large language models (LLMs) relies largely on single-turn attack datasets and single-judge scoring, underestimating risk from adaptive multi-turn adversaries and reporting a single success rate that does not separate partially actionable outputs from those carrying complete operational detail. We propose AMT-X (Adaptive Multi-Turn Exploitation), a phase-structured multi-turn red-teaming framework. Unlike prior multi-turn attacks that rely on ad hoc escalation or free-form per-goal plans, AMT-X casts the attack as an explicit, reproducible multi-phase state machine driven by semantic signals from the victim, and replaces single-judge scoring with a multi-role jury whose phase-conditioned checklists gate success on actionable harm. Across six frontier victim models (queried under their default safety alignment, without added moderation layers) and seven Moderation sub-categories, AMT-X attains overall attack success rates of 97.6–100% under a lenient score threshold, but 66.7–78.6% under a stricter gate requiring complete, real, and operational detail: a gap of up to 33 percentage points between partially and fully actionable harm. CCS Concepts: • Security and privacy → Software and application security; • Computing methodologies → Natural language processing. Additional Key Words and Phrases: LLM safety, red-teaming, multi-turn jailbreak, adversarial evaluation 1 Introduction The rapid deployment of large language models in production environments (spanning medical advisory systems, customer service agents, autonomous coding assistants, and general-purpose chatbots) has created an urgent need for rigorous adversarial evaluation. Safety fine-tuning through reinforcement learning from human feedback (RLHF) [25] and constitutional AI [4] has significantly reduced single-turn harmful outputs, but the adversarial landscape has evolved in response. Attackers no longer rely on a single carefully crafted prompt; instead, they engage in extended, adaptive conversations that build context, exploit logical inconsistencies, and progressively extract policy-violating outputs. As attackers become more sophisticated, the central question becomes whether our evaluation methodology keeps pace. Mainstream safety benchmarks have not kept pace with this shift. HarmBench [22], AdvBench [41], and JailbreakBench [6] measure robustness against static single-turn datasets, providing strong comparability across models but systematically understating risk from adaptive multi-turn adversaries. Iterative attacks such as PAIR [7] and TAP [23], multi-turn methods such as Crescendo [33] and Chain of Attack (CoA) [38], and more recent agentic attackers such as GOAT [27], ActorAttack [32], and X-Teaming [31] have demonstrated that adaptive and conversational strategies can substantially bypass safety-aligned models; human multi-turn red-teamers similarly defeat defenses that report single-digit ASRs under automated single-turn attacks [15] . However, these methods still share three structural limitations that we revisit in detail in Section 3: (i) attacks are improvised turn by turn, so trajectories are neither reproducible nor structurally attributable: two runs of the “same” attack can diverge, and it remains unclear which components actually drive success; (ii) success is typically reported asa single LLM score, which does not separate nominal successes that carry complete operational detail from those that stop short of it; and (iii) the same single LLM judge that grades success also exhibits documented positional, verbosity, and self-preference biases, further undermining reported numbers. Authors’ Contact Informat

FAQs

Why do existing LLM safety benchmarks understate multi-turn attack risk?

They mainly use static single-turn datasets and single-judge scoring, which do not reflect adaptive conversations that accumulate context and progressively extract policy-violating outputs.

How does AMT-X make multi-turn red-teaming more reproducible and attributable?

AMT-X casts attacks as an explicit, reproducible multi-phase state machine with semantic-signal-driven phase transitions and a structured technique library.

What are the two evaluation metrics in AMT-X, and why are they important?

AMT-X computes a lenient success rate and a stricter full success rate, separating partially actionable outputs from fully operational detail, so the gap between them becomes measurable.

Jul 16, 2026

Upload by Seraphina