Reasoning Consistency Scanning: Auditing Chain-of-Thought Validity in AI Safety Evaluations

Jul 16, 2026

Upload by Eden

Chain-of-thought (CoT) explanations in AI safety evaluations can be unfaithful: stated reasoning may not match the process that produced the final output. This work introduces reasoning consistency scanning, a method that checks whether transcript reasoning is logically consistent with the accompanying answer, enabling transcript-only post-hoc auditing. The paper formalizes consistency separately from faithfulness, proposes a six-subtype taxonomy of inconsistency, builds a 60-transcript benchmark, implements an InspectScout scanner, and reports systematic results across models and task types.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Introduction

Faithfulness versus consistency

Construct validity and risks of inconsistency

Reasoning consistency scanning methodology

Contributions and empirical setup

arXiv :2607 .07229v 1 [ cs .AI] 8 Jul 2026 Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations Silvia Santano Abstract Prior work has shown that chain-of-thought (CoT) reasoning is often unfaithful: a model’s stated reasoning does not reliably reflect the process that produced its output [Turpin et al., 2023, Lanham et al., 2023, Chen et al., 2025, Walden and Wanner, 2026] . Detecting unfaithfulness, though, requires controlled experimental interventions, which cannot be applied to evaluation transcripts after the fact. We turn instead to a more tractable question that has received less attention: whether the stated reasoning is logically consistent with the answer it accompanies. Unlike faithfulness, consistency can be assessed from a transcript alone, with no intervention. We introduce reasoning consistency scanning, a reusable method for detecting this property in AI safety evaluation transcripts. Our contributions are fourfold. First, we formalize reasoning consistency as distinct from faithfulness and define a six-subtype taxonomy of inconsistency. Second, we build a validated benchmark of 60 transcripts, manually adapted from InstrumentalEval outputs. Third, we implement a working scanner for InspectScout, the first to target this property in safety evaluation transcripts. Fourth, we report results across four generator models and three evaluations from inspect evals, showing that reasoning inconsistency is present, detectable, and varies systematically across both models and task types.1 1 Introduction Chain-of-thought reasoning has become central to how AI safety evaluations are interpreted. When a model is tested for dangerous capability, deception, or alignment-relevant behavior, evaluators increasingly examine not just what the model outputs but how it reasons about the task. Reasoning traces are treated as evidence that a model is genuinely deliberating, recognizing constraints, or weighing consequences, which are all properties that matter for safety assessments. This practice rests on an assumption that is less examined than it might appear: that the model’s stated reasoning is actually connected to its final answer. However, they often are not. A growing body of research on CoT faithfulness has demonstrated that reasoning traces frequently do not reflect the internal computational process that produced the output [Turpin et al., 2023, Chen et al., 2025] . Models act on information they do not mention in their reasoning, and deny reliance on features that demonstrably influenced their answers [Walden and Wanner, 2026] . However, faithfulness and consistency are distinct properties requiring different methods to detect. Faithfulness detection depends on controlled interventions such as injecting hints, perturbing inputs and measuring what changes, and establishes causal claims about internal computation. These methods cannot be applied to existing transcript corpora. Consistency, by contrast, asks a simpler question: ”does the stated reasoning logically connect to and predict the final output, as observable text?” This is detectable from transcripts alone, without any experimental intervention, making it amenable to systematic post-hoc auditing. The direct consequence is that, if reasoning and answer are decoupled, any eval that uses reasoning traces as evidence of a safety-relevant property is measuring something different from what it claims. This is a direct threat to construct validity, the foundational question of whether an eval is actually measuring the property it was designed to measure. When a model’s reasoning trace does not connect to its output, two problems follow. First, the eval result cannot be trusted. Whether the inconsistency stems from confusion, post-hoc rationalisation, or strategic behavior, theeval is not measuring what it claims. And second, CoT monitoring is undermined. Safety researchers increasingly rely on reaso

FAQs

What are the paper’s main contributions?

It formalizes reasoning consistency with a six-subtype inconsistency taxonomy and decision procedure, creates a validated benchmark of 60 labeled transcripts, implements a working InspectScout scanner using an LLM-as-judge design, and reports results across multiple generator models and evaluation sets.

Jul 16, 2026

Upload by Eden