Is Vibe Coding Safe? - Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Sep 15, 2026

Uploaded by Logic

Vibe coding is a software development approach where engineers prompt an LLM agent to complete complex coding tasks with minimal supervision. Despite growing adoption, production safety remains unclear. To address this, SUSVIBES is introduced: a benchmark of 186 real-world, feature-request tasks from open-source projects with human-fixed vulnerable implementations. Results across 12 coding-agent settings show consistently poor security performance, even when functional correctness is high. Preliminary vulnerability-hint strategies do not remove the underlying security failures.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

AccessFree to download

Abstract

Introduction

SUSVIBES Benchmark

Task Construction and Verification

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks Songwen Zhao 1 2 Danqing Wang 1 Kexun Zhang 1 Jiaxuan Luo 3 Zhuo Li 4 Lei Li 1 arXiv : 25 12 .03262v 3 [cs .SE] 2 1 Aug 2026 Abstract Vibe coding is a new software development paradigm in which human engineers prompt a large language model (LLM) agent to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, is the generated code really safe to deploy in production? To investigate this question, we propose SUSVIBES, a benchmark consisting of 186 feature-request software engineering tasks from real-world open-source projects, for which, human programmers committed vulnerable implementations. We evaluate 12 widely used coding agentic settings with frontier models on the benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 57% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 11.8% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe coding, particularly in security-sensitive applications. The code and dataset are available at [https://github.com/LeiLiLab/susvibes](https://github.com/LeiLiLab/susvibes. The)[. The](https://github.com/LeiLiLab/susvibes. The) leaderboard is at [https://leililab.github.io/](https://leililab.github.io/)[ ](https://leililab.github.io/)susvibes-leaderboard. 1. Introduction Vibe coding is a new programming paradigm in which users use LLMs to produce software primarily by describing goals in natural language with minimal review of the generated code (Fawzy et al., 2025 ; Sarkar & Drosos, 2025) . It has 1Carnegie Mellon University, Language Technologies Institute 2 Columbia University 3Johns Hopkins University 4HydroX AI. Correspondence to: Songwen Zhao <[sz3296@columbia.edu](sz3296@columbia.edu) >, Danqing Wang <[danqingw@cs.cmu.edu](danqingw@cs.cmu.edu) >. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . Figure 1. SUSVIBES: A feature-request task consists of a task description, the codebase, and an execution environment. The vibe coding agent is asked to add the new feature to the given codebase. The agent can interact with the environment to get feedback andrefine its solution patch. The solution patch will be tested with both human-written functional and security tests. As the example shows, the solution patch cannot pass the security tests without check unsafe opts. been increasingly adopted, as indicated by the popularity of AI-based integrated development environments like Cursor and command-line interfaces like Claude Code. A recent survey shows that 75% of respondents are using vibe coding, among which 90% find it satisfactory (Perry, 2025) . Another survey suggests that beginner programmers with less than a year’s experience are much more likely to be vibe coding optimists (WIRED, 2025) . Frontier AI companies, such as Anthropic, admittedly use “vibe coding in prod[uction]”(Anthropic, 2024) . While vibe coding may have increased engineer productivity, the security of agent-generated code remains questionable, especially when vibe coding users may not have the ability or intent to examine it carefully. Various sources report security incidents such as API keys being exposed in plaintext and authentication vulnerabilities, some of which have already been exploited by malicious parties (Archibald & Kaplan, 2025) . Traditional competitive coding and NL2Code tasks, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), focus more on implementing an isolated function based on a clear description of functionality. Instead, vibe coding focuses more on real-world s

FAQs

What does vibe coding involve, and why is its safety uncertain?

Vibe coding prompts an LLM agent to complete coding tasks from natural-language goals with little supervision. Its safety is uncertain because the generated code may contain security vulnerabilities that users may not detect.

What is SUSVIBES and what does it measure?

SUSVIBES is a benchmark with 186 realistic feature-request software engineering tasks derived from real-world open-source projects. It measures how vulnerable the produced patches are in security-sensitive scenarios using separate functional and security tests.

How do coding agents perform on SUSVIBES?

Across 12 widely used coding-agent settings with frontier models, agents perform poorly on software security. Even when solutions can be functionally correct, only a small fraction are secure.

Can vulnerability-hint strategies improve security outcomes?

Experiments show that preliminary security strategies, such as augmenting feature requests with vulnerability hints, cannot mitigate the security issues in a meaningful way.

Sep 15, 2026

Uploaded by Logic