Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle

Jul 16, 2026

Upload by Eliana

AI is increasingly embedded across the security lifecycle, with the same generative substrate drafting application code, hardening it, and probing for weaknesses. The work argues that removing humans is not a harmless step toward autonomy: when builders, defenders, and testers share one training distribution, common blind spots emerge and verification independence is lost. Human removal collapses an external oracle, limits intervention time, creates a predictable target for adversaries, and erodes accountability. Evidence draws on autonomous code generation, adversarial ML, fault tolerance, and all-machine hacking tournaments, concluding humans must remain permanently in the loop.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Three Hats, One Head

A Lifecycle Under One Roof

arXiv :2607 .032 15v 1 [ cs .CR] 3 Jul 2026 Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle MOHAMED CHAHINE GHANEM∗ , School of Computer Science and Mathematics, Keele University, United Kingdom and Cybersecurity Institute, University of Liverpool, United Kingdom Artificial intelligence has spread across the whole of the security lifecycle. The same family of models now writes application code, hardens it, and probes it for weaknesses, so that a single generative substrate increasingly performs all three roles at once. Enthusiasm for this convergence tends to treat full autonomy asthe natural end point of partial assistance. This article argues that it is not. When the system that builds an artifact is drawn from the same distribution as the systems that defend and test it, the three roles inherit a common set of blind spots, and the independence that makes verification meaningful is quietly lost. Removing the human does more than raise the automation level: it collapses the external oracle against which machine output is judged, outruns the point at which a person could intervene, hands adversaries a predictable and poisonable target, and dissolves the locus of accountability when something fails. Drawing on evidence from autonomous code generation, adversarial machine learning, software fault tolerance, and the first all-machine hacking tournaments, we argue that the human belongs in the loop not as a temporary scaffold but as a permanent structural requirement, and set out what a defensible division of labour between people and machines should preserve. CCS Concepts: • Security and privacy → Systems security; • Computing methodologies → Artificial intelligence; • Social and professional topics → Computing / technology policy. Additional Key Words and Phrases: Autonomous cyber defence, human-in-the-loop, AI security lifecycle, verification, accountability, algorithmic monoculture, adversarial machine learning ACM Reference Format: Mohamed Chahine Ghanem. 2026. Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle. ACM AILett. 1, 2, Article XX (June 2026), 7 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)nnnnnnn.nnnnnnn 1 Three Hats, One Head Open a modern software pipeline and you are likely to find an AI model in at least three places. One model, or one family of models, drafts the application code. Another reviews that code, flags insecure patterns, and proposes hardening. A third generates test cases, fuzzes interfaces, and searches for exploitable flaws. The three functions correspond to the oldest division of labour insecurity practice: the builder who creates the system, the defender who protects it, and the breaker who tries to subvert it. What is new is that a single generative substrate is beginning to wear all three hats at the same time. The prevailing narrative treats this as an unambiguous advance. If AI can assist a developer, why not let it write the whole module; if it can assist a defender, why not let it run the security ∗ Corresponding author. Author’s Contact Information: Mohamed Chahine Ghanem, School of Computer Science and Mathematics, Keele University, Newcastle-Under-Lyme, United Kingdom and Cybersecurity Institute, University of Liverpool, Liverpool, United Kingdom, [mohamed.chahine.ghanem@liverpool.ac.uk](mohamed.chahine.ghanem@liverpool.ac.uk). This work is licensed under a Creative Commons Attribution 4 .0 International License. © 2026 Copyright held by the owner/author(s) . ACM 3068-8590/2026/6-ARTXX [https://doi.org/10.1145/nnnnnnn.nnnnnnn](https://doi.org/10.1145/nnnnnnn.nnnnnnn) ACM AI Lett., Vol. 1, No. 2, Article XX. Publication date: June 2026 . XX:2 M. C. Ghanem operations centre; if it can assist a red team, why not let it hunt vulnerabilities unsupervised. Each step looks like a modest extension of the last,

FAQs

为什么作者反对把人从 AI 驱动的安全生命周期中移除？

作者认为移除人并不是简单提高自动化水平，而是会破坏验证所需的独立性，并削弱干预与问责机制，从而损害安全生命周期应保证的性质。

当构建、防御和测试由同一类生成模型承担时会出现什么问题？

当这些角色来自相同分布训练，容易继承同一组盲点，导致验证独立性被悄然丢失，安全评估的有效性下降。

文中用哪些证据来支撑“人必须在环路中”的主张？

文中结合了自主代码生成、对抗式机器学习、软件容错以及首批全机黑客竞赛等证据，并指出在严格“无人工”规则下系统仍会暴露出需要人类结构性参与的问题。

Jul 16, 2026

Upload by Eliana