Code Execution Capability as a Metric for Machine Learning - Assisted Software Vulnerability Detection Models - Abstract

Aug 5, 2026

Upload by Ivy

This paper studies how the ability to learn code execution tasks influences accuracy on software vulnerability detection (SVD) benchmark datasets. Initial results show models can reach near state-of-the-art SVD accuracy without learning code execution tasks, yet they generalize poorly across SVD benchmarks. The findings suggest dataset bias enabling prediction of non-SVD signals. By combining SVD datasets, SVD accuracy drops while correlation with code execution task accuracy improves, supporting the need for reduced bias collection methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Abstract

Introduction

Motivation for ML-based vulnerability detection

Relation to code execution understanding

Research Questions and Experimental Setup

RQ1: Learning code execution tasks

RQ2: Correlation between SVD accuracy and code execution ability

RQ3: Effect of combining datasets on SVD accuracy

RQ4: Effect of combining datasets on correlation with code execution tasks

Main Contributions

Computer Science and Engineering Faculty Publications Computer Science & Engineering 2023 Code Execution Capability as a Metric for Machine Learning–Assisted Software Vulnerability Detection Models Daniel Grahn Lingwei Chen Junjie Zhang Follow this and additional works at: [https://corescholar.libraries.wright.edu/cse](https://corescholar.libraries.wright.edu/cse) Part of the Computer Sciences Commons This Article is brought to you for free and open access by Wright State University’s CORE Scholar. It has been accepted for inclusion in Computer Science and Engineering Faculty Publications by an authorized administrator of CORE Scholar. For more information, please contact [library-corescholar@wright.edu](library-corescholar@wright.edu). Code Execution Capability as a Metric for Machine Learning–Assisted Software Vulnerability Detection Models Daniel Grahn, Lingwei Chen, Junjie Zhang Department of Computer Science and Engineering Wright State University Dayton, USA {dan.grahn,lingwei.chen,[junjie.zhang}@wright.edu](junjie.zhang}@wright.edu) Abstract—In this paper, we consider how the ability to learn Code Execution Tasks affects a model’s accuracy on software vulnerability detection (SVD) benchmark datasets. We initially find that models can achieve near state-of-the-art accuracy on SVD benchmarks regardless of their ability to learn Code Execution Tasks. However, these models fail to generalize well across SVD benchmarks. The results indicate a bias in the datasets that allows models to predict nonSVD signals. Under the theory that different collection methods will reduce biases, we investigate combining the SVD datasets. When trained on combined datasets, SVD accuracy is reduced but correlation with Code Execution Task accuracy improves. Our contributions are (1) using a reversed curriculum learning to evaluate model capabilities, (2) demonstrating the criticality of code execution understanding to machine learning– assisted software vulnerability detection, (3) evidence that improved diversity of SVD datasets will lead to improved accuracy and generalizability,(4) and benchmarks of recent models across multiple SVD datasets. I. Introduction Detecting software vulnerabilities during development reduces risk and increases the stability of the software. If delayed, undetected vulnerabilities may be exploited by malicious actors. While most vulnerabilities are introduced and removed without incident, others have led to the loss of property, resources, and life. To prevent such damages, significant efforts are organized around the detection and mitigation of vulnerabilities. These endeavors have historically been a resource-intensive process. Pair programming, code reviews, and other best practices take precious human hours. While Source Code Analysis (SCA) may be automated as part of continuous integration, the prevalence of false positives requires programmers to review many reports. As Machine Learning (ML) becomes capable of performing evermore complex tasks, there is a push to use these capabilities to improve software vulnerability detection (SVD) . Over the last few years, much time and effort have been spent pursuing machine learning–assisted software vulnerability detection (MLAVD) techniques. Previous research has shown significant deficiencies in the datasets used for MLAVD but did not consider the role of the models themselves [10] . Unlike the long-standing application of ML to Computer Vision (CV) and Natural Language Processing (NLP), MLAVD is a relatively new field [11] . As CV and NLP techniques developed, inspiration was drawn from how humans sense and understand the world [9] . But programming languages are different from natural languages [14, 21] . Natural languages evolve across time and space, while programming languages have strict syntax and unambiguous semantics. Likewise, SVD is highly dissimilar to object detection and other CV tasks. Vulnerabilities are not defin

FAQs

How does learning code execution tasks affect SVD accuracy?

Models may achieve near state-of-the-art SVD accuracy even when they cannot learn code execution tasks, but they generalize poorly across benchmarks. Correlation improves when datasets are combined.

What evidence is there for bias in SVD benchmark datasets?

The paper reports that models lacking code execution learning still perform well on SVD benchmarks but fail to generalize well. This pattern indicates bias that allows predictions based on non-SVD signals.

What changes when SVD datasets are combined?

Training on merged datasets reduces mean SVD accuracy, yet increases the correlation between SVD performance and code execution task accuracy, suggesting better alignment with code execution understanding.

Aug 5, 2026

Upload by Ivy