Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Jul 16, 2026

Upload by Hazel

Code language models increasingly support production software workflows for debugging, refactoring, and iterative repair, where evaluation benchmarks assume that the instructions acted upon are correct. This work tests robustness when that assumption fails by running four experiments on algorithmic Python problems with deterministic tests (RunBugRun), including single-pass and iterative repair. Results show models recognize incorrect instructions as wrong yet still obey them, injecting further errors that iterative self-repair cannot recover, leading to irrecoverable semantic collapse.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages43

Reading Time1 h 48 min

Abstract

Introduction

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse Raj Jaiswal 1∗, Anany Singh Divy 1∗, Savar Bhasin 1∗, Adi Bajpai 1∗, Tanuja Ganu3 , Rajiv Ratn Shah2 1IIIT Delhi 2IIT Kanpur 3Microsoft Research India {jaiswalp, anany23084, savar23497, [adi23035}@iiitd.ac.in](adi23035}@iiitd.ac.in) [tanuja.ganu@microsoft.com](tanuja.ganu@microsoft.com) , [rajivratn@iitk.ac.in](rajivratn@iitk.ac.in) ∗Equal contribution arXiv :2607 .04537v 1 [ cs . SE] 5 Jul 2026 Abstract Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to passrate evaluation, with direct consequences for code language models deployed in production settings. All code, prompts, and data are available in the Appendix 9. 1 Introduction Software development has undergone a fundamental shift as Large language models have moved beyond isolated code generation (Dong et al., 2025 ; Zamfirescu-Pereira et al., 2025 ; Hoda, 2026) making real modifications to real codebases with real consequences. Yet every benchmark that evaluates them assumes the instructions they act on are correct. (Dong et al., 2025 ; Zamfirescu-Pereira et al., 2025 ; Hoda, 2026) into active roles across the full GPT-5 .3 Codex Claude Sonnet 4.6 Qwen3-Coder GLM-5 Kimi-K2.5 Progressive Dataset Narrowing Across RQ Stages 0 100 200 300 400 500 600 Number of Problems Figure 1: RQ1 is the full 538-problem baseline. Problem counts across RQ2–RQ4 after eligibility filtering, each stage runs only the failure cases. RQ3 and RQ4 confirms damage through Ghost (Unknown) errors accumulation and self repairment fails to correct and reverse it. development lifecycle—debugging, refactoring, testing, and iterative repair (Dong et al., 2025) . Systems such as GitHub Copilot (Stray et al., 2026), Cursor (He et al., 2026), Devin(Li et al., 2025), and Claude Code (Li et al., 2025) are no longer experimental; they are trusted collaborators in production workflows, making real modifications to real codebases with real consequences (Jimenez et al., 2024 ; Yang et al., 2024 ; Xia et al., 2024) . This transition from code assistant to coding agents marks a critical inflection point—one where the stakes of model behavior extend far beyond benchmark performance and into the reliability of software that the world depends on. Prior work (Larbi et al., 2025 ; Wu et al., 2025 ; Agrawal et al., 2025) has studied robustness to structural noise, ambiguous prompts, and incomplete task descriptions, yet none of these settings place a model in direct conflict with a plausible but incorrect instruction while objective evidence contradicts it in real time. Code is uniquely positioned to close this gap. Unlike natural language tasks (Lin et al., 2022 ; Hendrycks et al., 2021 ; Joshi et al., 2017) where correctness is inherently subjective, program correctness

FAQs

What is the core problem studied in this paper?

The paper studies what happens when code language models receive incorrect instructions while objective tests show the conflict in real time.

How do the experiments assess model behavior?

They evaluate models across four experiments in single-pass and iterative repair settings using the RunBugRun dataset of algorithmic Python problems with deterministic test cases.

What is Blind Obedience and why is it dangerous?

Blind Obedience is the tendency to follow incorrect instructions even after correctly identifying them as wrong. This compliance introduces compounding “Ghost Errors,” and subsequent iterative self-repair fails to converge, causing irrecoverable semantic corruption.

Jul 16, 2026

Upload by Hazel