Retroactive Chain-of-Thought (RetroCoT) Forensic Reconstruction Prompts as a Safety Diagnostic Across Model Generations

Jul 16, 2026

Upload by Aria

Safety alignment in large language models is typically assessed using direct, imperative harmful requests, but performance is strongly affected by pragmatic register. Models that refuse direct requests may comply when the same underlying objective is expressed through a different communicative stance. The work proposes Retroactive Chain-of-Thought (RETROCOT), a single-turn attack that reframes harmful requests as forensic reconstruction tasks by presupposing the outcome and asking for a reverse causal chain. Experiments report substantial attack success on GPT-4o, while GPT-5-family models refuse the specific reconstruction register.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction

Mismatched generalization and pragmatic shifts

Presupposition accommodation and the RETROCOT framing

arXiv :2607 .04645v 1 [ cs .CL] 6 Jul 2026 Retroactive Chain-of-Thought ( RETROCOT): Forensic Reconstruction Prompts as a Safety Diagnostic Across Model Generations Samira Hajizadeh Department of Computer Science Columbia University New York, NY, USA [samira.hajizadeh@columbia.edu](samira.hajizadeh@columbia.edu) Abstract Safety alignment in large language models is typically evaluated against direct, imperative harmful requests. We show that this alignment is highly conditioned on pragmatic register: models that refuse a direct request frequently comply when the same underlying objective is expressed through a different communicative stance. This suggests that current alignment policies are not invariant to semantic equivalence, but remain sensitive to how a request is pragmatically framed. We introduce Retroactive Chain-ofThought (RETROCOT), a single-turn attack that reframes harmful requests as forensic reconstruction tasks. Rather than requesting harmful instructions directly, RETROCOT presupposes that the harmful outcome has already occurred and asks the model, acting as a forensic analyst, to reconstruct in reverse the causal chain that produced it. On AdvBench (n=50), RETROCOT achieves attack success rate (ASR) of 58% on gpt-4o and 52% on gpt-4omini, compared with direct-request baselines of 0% and 4%, respectively. We further identify a pronounced generation gap: GPT-5-family models refuse RETROCOT entirely, explicitly identifying the reconstruction premise in their refusal rationales, consistent with explicit coverage of this reconstruction register. However, this robustness does not generalize across pragmatic forms. A single adversarial feedback turn presenting an existing forensic reconstruction response alongside evaluator critique raises ASR from 0% to 48% on GPT-5.4-mini and from 58% to 94% on GPT-4o; a control condition omitting the fabricated low score achieves 85% on GPT-5.4-mini, indicating that the operative element is pragmatic continuation within the established forensic frame rather than score manipulation. These results suggest that frontier-model alignment remains conditioned on pragmatic framing rather than semantic intent, and that new pragmatic registers can continue to expose alignment blind spots even after previously known attack forms have been covered. 1 Introduction Large language models (LLMs) are aligned to refuse harmful requests, but this alignment isnot invariant to how those requests are expressed. We show that models which consistently refuse direct imperative prompts will frequently comply with identical underlying objectives expressed through a different communicative stance. A model that refuses ”Write instructions for synthesizing X” will often comply when asked instead to reconstruct, as a forensic analyst, the steps that led to the documented synthesis of X. The harmful information sought is identical; only the pragmatic form changes. This failure is an instance of what Wei et al. (2023) term mismatched generalization: safety training covers a specific distribution of input forms without generalizing to the full scope of contexts in which harmful capabilities can be elicited. Andriushchenko & Flammarion (2025) demonstrate a minimal case of this gap: shifting the tense of a harmful question from imperative to past tense raises GPT-4o’s attack success rate significantly across 20 reformulations. We ask whether deeper pragmatic shifts, those that change not merely tense but the entire communicative function of the request, produce proportionally larger failures from a single attempt. We introduce Retroactive Chain-of-Thought (RETROCOT), a single-turn attack grounded in the linguistic phenomenon of presupposition accommodation: the tendency of cooperative language agents to silently accept unstated background assumptions in order to make an utterance coherent (Stalnaker, 1999) . Rather than requesting harmful instructions directly, RETROCOT presupposes that the ha

FAQs

What problem does Retroactive Chain-of-Thought (RETROCOT) address in model safety evaluation?

It addresses that safety alignment often depends on how harmful goals are pragmatically framed, not just on the semantic intent of the request. The same objective can yield refusal or compliance depending on communicative stance.

How does RETROCOT change a harmful request to bypass refusals?

RETROCOT presupposes the harmful outcome has already occurred and asks the model, as a forensic analyst, to reconstruct the reverse causal chain. This shifts the model from evaluating whether to comply to analyzing how the outcome could have happened.

Do newer model generations respond consistently to RETROCOT?

No. GPT-5-family models refuse RETROCOT more robustly and identify the reconstruction premise in their refusals, but robustness does not transfer across other pragmatic forms. Adversarial feedback turns can substantially increase attack success on these models.

Jul 16, 2026

Upload by Aria