JavaVulBench Java Vulnerability Benchmark with Realistic Splits and Unified Multi-Backend Harness

Jul 16, 2026

Upload by Aurora

JavaVulBench releases a benchmark dataset and evaluation harness for Java vulnerability detection, built on CVE-grounded coverage of ~30,600 Java methods across 1,740 CVEs and 700+ projects. Samples are labelled at method and line granularity and include per-CVE publication dates. Five realistic split strategies—random, project-disjoint, temporal, deduplicated (Jaccard 0.8), and unseen CWE-family—support leakage-controlled experiments. A unified multi-backend harness standardizes evaluation for encoder classifiers, Ollama-served local generative models, and OpenRouter API LLMs, with a leakage-aware pre-training contamination audit.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages4

Reading Time10 min

Abstract

Introduction

Dataset

Mining

Labelling

Splits

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode Norbert Sándor Szolnoki∗ Gábor Antal∗ [szolnoki@inf.u-szeged.hu](szolnoki@inf.u-szeged.hu) [antal@inf.u-szeged.hu](antal@inf.u-szeged.hu) University Of Szeged Szeged, Hungary arXiv :2607 .02825v 1 [ cs .CR] 2 Jul 2026 Abstract We release JavaVulBench, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains ∼30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single LlmPrediction schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3, DeepSeek-Coderv2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube ([https://www.youtube.com/watch?v=nMTX_hqkuoM](https://www.youtube.com/watch?v=nMTX_hqkuoM)) 1 . 1 Introduction Deep-learning-based vulnerability detection has been benchmarked predominantly on C/C++ corpora such as Big-Vul [9] and Devign [33]. Java is a widely-deployed enterprise language and the OWASP Top-10 categories [27] (injection, deserialisation, SSRF, path traversal, authentication/authorisation) commonly appear in disclosed Java CVEs, yet Java-focused evaluation infrastructure is comparatively thin. When Java is evaluated at all it is typically through a random split of the multilingual CVEfixes corpus [1], with no project-disjoint or temporal partition. Chakraborty et al. [5] showed on C/C++ that this single methodological choice can inflate reported F1 by 15-40 points, because near-duplicate methods drawn from the same project routinely leak between train and test. To the best of our knowledge, existing Java-focused benchmarks-Vul4J [4], VJBench [32], CWE-Bench-Java (released with the IRIS system) [20], and the Java slice of CVEfixes [1] -do not simultaneously ship multiple realistic split strategies, explicit non-vulnerable samples, line-level labels, a unified multi-backend evaluation harness, and a per-model pre-training contamination audit. Table 5 ∗ Both authors contributed equally to this research. 1[https://www.youtube.com/watch?v=nMTX_hqkuoM](https://www.youtube.com/watch?v=nMTX_hqkuoM) summarises this comparison. JavaVulBench is built to fill that combination. Intended users. Vulnerability-detection researchers who need a Java benchmark with controlled leakage, and tool builders who want a drop-in harness that evaluates their detector against existing baselines under identical conditions. Contributions. We release JavaVulBench, consisting of: (i) a CVE-grounded corpus of ∼30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-sample publication dates and five realistic split strategies (random, project-disjoint, temporal, deduplicated at Jaccard 0.8, unseen CWE-family); (ii) a unified evaluation harness with a single LlmPrediction schema spanning three backend families-HuggingFace encoder classifiers (CodeBERT [10], GraphCodeBERT [14], UniXcoder [13]), Ollama [24]-served DeepSeek-Coder-1.3B [15], and eight OpenRouter [26]-served API LLMs (GPT-4o [25], GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3 [8], DeepSeek-Coder-v2 [7], Qwen-2 . 5-Coder-14B/7B [17]

FAQs

What does JavaVulBench provide for Java vulnerability detection research?

JavaVulBench provides a benchmark dataset plus a unified evaluation harness for assessing Java vulnerability detectors under controlled, leakage-aware conditions.

Which split strategies are included in JavaVulBench?

It includes five strategies: random, project-disjoint, temporal, deduplicated (Jaccard 0.8), and unseen CWE-family, each distributed as ready-to-use JSONL.

How does the evaluation harness support different model backends?

The harness uses a single LlmPrediction schema to evaluate encoder classifiers, Ollama-served local generative models, and OpenRouter-served API LLMs via a one-command pipeline.

Jul 16, 2026

Upload by Aurora