The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code

Jul 16, 2026

Upload by Jiven

Large language models increasingly generate C++, a memory-unsafe language where small oversights can become exploitable bugs. Most AI-code security evaluations rely on static analysis alone, which cannot confirm runtime violations or cover untested paths. This work asks whether AI-generated C++ is measurably less safe than human-written code and whether verification tools agree. It introduces VULBENCH-CPP with 8,918 programs, each labeled across four verification tiers, finding AI code about twice as likely to trigger confirmed runtime memory-safety violations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Problem and importance

The danger of static-only evaluation

The inadequacy of any single verification tier

Approach and benchmark construction

The Illusion of Safety: Multi-Tier Verification of AI vs . Human C++ Code Saif Mahmud∗ , Fadul Sikder∗ , Yuede Ji∗ , Zhang Haotian†, and Jeff (Yu) Lei∗ ∗ Computer Science and Engineering, The University of Texas at Arlington, Arlington, Texas, USA Email: {sxm8931, [fxs5359](fxs5359}@mavs.uta.edu)[}](fxs5359}@mavs.uta.edu)[@mavs.uta.edu](fxs5359}@mavs.uta.edu), {yuede.ji, [ylei](ylei}@uta.edu)[}](ylei}@uta.edu)[@uta.edu](ylei}@uta.edu) †Computer Science and Engineering, New Jersey Institute of Technology, Newark, New Jersey, USA Email: [haotian.zhang@njit.edu](haotian.zhang@njit.edu) arXiv :2607 .00 107v2 [ cs . SE] 5 Jul 2026 Abstract—Large language models increasingly generate C++, a memory-unsafe language where a single overlooked violation can become an exploitable bug. Yet most security evaluations of AI-generated code rely on static analysis alone, which flags warnings without confirming runtime violations or reasoning about untested paths. We ask whether AI-generated C++ is measurably less safe than human-written code, and whether common verification tools agree on the risk. We introduce VULBENCH-CPP, a benchmark of 8,918 C++ programs from three open-weight LLMs (Gemma 3 27B IT, LLaMA 3.3 70B Instruct, Qwen 2.5 Coder 32B Instruct) and human authors across 851 competitive-programming tasks. Each program is annotated by four verification tiers: functional testing, static analysis (cppcheck, clang-tidy), dynamic analysis (ASan/UBSan), and bounded model checking (ESBMC). Accounting for the correlation among solutions to a shared task, we find that AIgenerated code is roughly twice as likely as human code to trigger a confirmed runtime violation, even after controlling for code length and test pass-rate. Under static analysis the two look equally safe, but this is misleading: the apparent similarity reflects code length rather than real safety, and the tiers detect largely different classes of violation, so no single tier is sufficient. The gap is consistent across independent generations. We plan to release the benchmark, harness, and annotated results in future. Findings are scoped to short, algorithmic C++ under zero-shot generation. I. INTRODUCTION Problem and importance. Large language models (LLMs) are increasingly used to generate source code in professional software development [1] . GitHub reports that AI-assisted coding now contributes to a substantial and growing fraction of code in active repositories [2], [3], and enterprise adoption is accelerating across systems programming, embedded development, and high-performance computing, domains where C++ dominates. For memory-unsafe languages like C++, which provides no runtime exceptions or memory safety guarantees, the stakes are particularly high: a single buffer overflow or use-after-free vulnerability translates to an exploitable security breach or silent data corruption [4] . The central question is unavoidable: how secure is AI-generated C++ code, and can practitioners trust existing verification practices to catch its flaws? The danger of static-only evaluation. Prior evaluations of AIgenerated code security have employed static analysis as their primary (or exclusive) verification method [3], [5], [7] . For C++ in particular, static-only evaluation creates a dangerous illusion of security. Static tools (cppcheck, clang-tidy) report comparable warning rates for AI-generated and human-written code, superficially suggesting the two are equally safe. However, static analysis cannot confirm that warnings correspond to true runtime violations, nor can it reason about untested code paths. This methodological limitation conceals a critical vulnerability gap: AI-generated C++ code triggers confirmed runtime memory-safety violations at several times the rate of human-written code, a gap invisible to static analysis alone. Practitioners relying exclusively on static checks deploy AI code with undetected runtime vulnerabilities accumulating unnotice

FAQs

What problem does the study address about AI-generated C++ code?

The study investigates whether AI-generated C++ is less secure than human-written code, especially given that C++ is memory-unsafe and that static-only evaluation can miss real runtime vulnerabilities.

How is VULBENCH-CPP constructed and what does it cover?

VULBENCH-CPP includes 8,918 C++ programs across 851 competitive-programming tasks, combining outputs from three open-weight LLMs with human reference solutions.

What four verification tiers does the benchmark use for each program?

Each program is annotated with functional testing, static analysis (cppcheck, clang-tidy), dynamic analysis (ASan/UBSan), and bounded model checking (ESBMC).

Why is static analysis alone considered misleading in this context?

Static tools may report similar warning rates for AI and human code, but they cannot confirm that warnings correspond to true runtime violations or reason about code paths not exercised by tests.

Jul 16, 2026

Upload by Jiven