[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83686-en":3,"doc-seo-83686-105":30,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83686,4810365810221,"Aurora","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","JavaVulBench Java Vulnerability Benchmark with Realistic Splits and Unified Multi-Backend Harness","JavaVulBench releases a benchmark dataset and evaluation harness for Java vulnerability detection, built on CVE-grounded coverage of ~30,600 Java methods across 1,740 CVEs and 700+ projects. Samples are labelled at method and line granularity and include per-CVE publication dates. Five realistic split strategies—random, project-disjoint, temporal, deduplicated (Jaccard 0.8), and unseen CWE-family—support leakage-controlled experiments. A unified multi-backend harness standardizes evaluation for encoder classifiers, Ollama-served local generative models, and OpenRouter API LLMs, with a leakage-aware pre-training contamination audit.","JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware  \nEvaluation Mode  \nNorbert Sándor Szolnoki∗  \nGábor Antal∗  \n[szolnoki@inf.u-szeged.hu](szolnoki@inf.u-szeged.hu)  \n[antal@inf.u-szeged.hu](antal@inf.u-szeged.hu)  \nUniversity Of Szeged  \nSzeged, Hungary  \narXiv :2607 .02825v 1 [ cs .CR] 2 Jul 2026  \nAbstract  \nWe release JavaVulBench, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains ∼30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single LlmPrediction schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3, DeepSeek-Coderv2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube ([https://www.youtube.com/watch?v=nMTX_hqkuoM](https://www.youtube.com/watch?v=nMTX_hqkuoM)) 1 .  \n1 Introduction  \nDeep-learning-based vulnerability detection has been benchmarked predominantly on C/C++ corpora such as Big-Vul [9] and Devign [33]. Java is a widely-deployed enterprise language and the OWASP Top-10 categories [27] (injection, deserialisation, SSRF, path traversal, authentication/authorisation) commonly appear in disclosed Java CVEs, yet Java-focused evaluation infrastructure is comparatively thin. When Java is evaluated at all it is typically through a random split of the multilingual CVEfixes corpus [1], with no project-disjoint or temporal partition. Chakraborty et al. [5] showed on C/C++ that this single methodological choice can inflate reported F1 by 15-40 points, because near-duplicate methods drawn from the same project routinely leak between train and test.  \nTo the best of our knowledge, existing Java-focused benchmarks-Vul4J [4], VJBench [32], CWE-Bench-Java (released with the IRIS system) [20], and the Java slice of CVEfixes [1] -do not simultaneously ship multiple realistic split strategies, explicit non-vulnerable samples, line-level labels, a unified multi-backend evaluation harness, and a per-model pre-training contamination audit. Table 5  \n∗ Both authors contributed equally to this research.  \n1[https://www.youtube.com/watch?v=nMTX_hqkuoM](https://www.youtube.com/watch?v=nMTX_hqkuoM)  \nsummarises this comparison. JavaVulBench is built to fill that combination.  \nIntended users. Vulnerability-detection researchers who need a Java benchmark with controlled leakage, and tool builders who want a drop-in harness that evaluates their detector against existing baselines under identical conditions.  \nContributions. We release JavaVulBench, consisting of: (i) a CVE-grounded corpus of ∼30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-sample publication dates and five realistic split strategies (random, project-disjoint, temporal, deduplicated at Jaccard 0.8, unseen CWE-family); (ii) a unified evaluation harness with a single LlmPrediction schema spanning three backend families-HuggingFace encoder classifiers (CodeBERT [10], GraphCodeBERT [14], UniXcoder [13]), Ollama [24]-served DeepSeek-Coder-1.3B [15], and eight OpenRouter [26]-served API LLMs (GPT-4o [25], GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3 [8], DeepSeek-Coder-v2 [7], Qwen-2 . 5-Coder-14B/7B [17]","cbCaijgH5igHWduw","https://ap.wps.com/l/cbCaijgH5igHWduw","pdf",532863,3,1,4,"English","en",105,"# Abstract\n# Introduction\n# Dataset\n## Mining\n## Labelling\n## Splits","[{\"question\":\"What does JavaVulBench provide for Java vulnerability detection research?\",\"answer\":\"JavaVulBench provides a benchmark dataset plus a unified evaluation harness for assessing Java vulnerability detectors under controlled, leakage-aware conditions.\"},{\"question\":\"Which split strategies are included in JavaVulBench?\",\"answer\":\"It includes five strategies: random, project-disjoint, temporal, deduplicated (Jaccard 0.8), and unseen CWE-family, each distributed as ready-to-use JSONL.\"},{\"question\":\"How does the evaluation harness support different model backends?\",\"answer\":\"The harness uses a single LlmPrediction schema to evaluate encoder classifiers, Ollama-served local generative models, and OpenRouter-served API LLMs via a one-command pipeline.\"}]",1784189735,10,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":28},"javavulbench-java-vulnerability-benchmark-with-realistic-splits-and-unified-multi-backend-harness","",{"@graph":36,"@context":84},[37,52,67],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":22},"https://docshare.wps.com/document/javavulbench-java-vulnerability-benchmark-with-realistic-splits-and-unified-multi-backend-harness/83686/",{"url":51,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":24,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":41,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-26","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What does JavaVulBench provide for Java vulnerability detection research?","Question",{"text":74,"@type":75},"JavaVulBench provides a benchmark dataset plus a unified evaluation harness for assessing Java vulnerability detectors under controlled, leakage-aware conditions.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Which split strategies are included in JavaVulBench?",{"text":79,"@type":75},"It includes five strategies: random, project-disjoint, temporal, deduplicated (Jaccard 0.8), and unseen CWE-family, each distributed as ready-to-use JSONL.",{"name":81,"@type":72,"acceptedAnswer":82},"How does the evaluation harness support different model backends?",{"text":83,"@type":75},"The harness uses a single LlmPrediction schema to evaluate encoder classifiers, Ollama-served local generative models, and OpenRouter-served API LLMs via a one-command pipeline.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,133],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":29,"doc_module":4,"doc_module_name":46,"category_name":131,"show_sort_weight":29,"slug":132},"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":46,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]