[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85771-en":3,"doc-seo-85771-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85771,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","Using LLMs to Adjudicate Static Analysis Alerts with Error Reduction Techniques","Static analysis helps detect security weaknesses before software deployment, yet it generates alert volumes far beyond what analysts can review. This work evaluates how large language models classify static-analysis alerts as true bugs or false alarms, introducing error-reduction via two complementary methods: consistency checking across repeated runs and an LLM reasoning evaluation that reconciles discordant verdicts. Across Juliet, FormAI, and SV-COMP, mistake mitigation yields at least 98% recall and at least 94.8% specificity, supported by memorization and trigger-based validity experiments.","Using LLMs to Adjudicate Static-Analysis Alerts with  \nError Reduction Techniques  \nWilliam Klieber Carnegie Mellon Univ.  \nPittsburgh, USA[weklieber@cert.org](weklieber@cert.org)  \nDavid Svoboda  \nCarnegie Mellon Univ. Pittsburgh, USA [svoboda@sei.cmu.edu](svoboda@sei.cmu.edu)  \nLori Flynn Carnegie Mellon Univ.  \nPittsburgh, USA[lflynn@cert.org](lflynn@cert.org)  \nRuben Martins  \nCarnegie Mellon Univ. Pittsburgh, USA [rubenm@andrew.cmu.edu](rubenm@andrew.cmu.edu)  \narXiv :2607 .09979v 1 [ cs . SE] 10 Jul 2026  \nAbstract  \nStatic analysis is widely used for finding security weaknesses in source code before deployment, but it often produces far more alerts than analysts can review. We study how well large language models (LLMs) can adjudicate (classify as a real bug or a false alarm) staticanalysis alerts. We use two mistake-mitigation methods:  \n(1) a consistency check (CC) that runs the LLM multiple times and checks that the verdicts are consistent with each other, and (2) an LLM reasoning evaluation (LRE) step that runs the LLM multiple times and then asks the LLM to choose a verdict after evaluating the reasoning provided by each run.  \nWe evaluated several LLMs on three test suites: Juliet, FormAI, and SV-COMP. Across all three suites, the mid-tier reasoning LLMs that we tested (o4-mini, gpt-oss-120b, gpt-oss-20b) reach high recall (percent of real bugs that the tool correctly flags as needing repair / manual attention) and specificity (percent of actually false alerts that the tool correctly dismisses as false alarms) . With mistake mitigation, they reach at least 98% recall and at least 94.8% specificity on every suite (with CC alone on Juliet and SV-COMP, and with LRE+CC on FormAI) .  \nWe probe Juliet memorization and show that o4-mini can often reconstruct sanitized test cases’ original identities, so we base our generalization claims primarily on FormAI, scored against our own unpublished manual adjudications. A complementary flipped-verdict experiment suggests that o4-mini does exercise its reasoning capabilities on Juliet rather than reciting a memorized verdict, but doesn’t definitively rule out the possibility of overfitting. We also note a few cases where the LLM disagreed with our initial manual adjudications but the LLM’s explanation of its answer convinced us that its answer was correct and our initial manual adjudication was wrong. We also report results of using the LLM to synthesize a program that dynamically triggers the flaw as independent evidence; a validity check rejected every trigger driver aimed at a false alarm, so a valid trigger proved to be strong evidence of a real flaw.  \n1 Introduction  \nIt is a standard step in software development to evaluate source code for security weaknesses before it is fielded. Static analysis (SA) is widely used and is among the best automated techniques available, but using it well requires substantial manual effort: a tool will typically report many alerts, a large fraction of which are false positives, and an analyst must adjudicate each alert (decide whether it indicates a real flaw or not) . The volume of alerts is often too large to review in its entirety, so teams triage. A common practice is to manually adjudicate only the highest-severity alerts and to leave the remainder unreviewed. Unreviewed alerts constitute unknown risk: a real vulnerability may be hiding among them.  \nRecent large language models (LLMs) change what is feasible. Unlike earlier machine-learning approaches, modern reasoning LLMs produce a detailed chain of reasoning leading to their conclusion, and that reasoning can be double-checked. LLMs can also request information they lack (e.g., the definition of a struct or macro) and a driver program can retrieve and supply it. Several groups have begun to apply LLMs to static-analysis alerts and to false-positive reduction [1, 2, 3] .  \nThis paper studies how well modern LLMs can perform this adjudication. We built LASAA (LLMs for Adjudication of","cbCais3QP3JLYcPf","https://ap.wps.com/l/cbCais3QP3JLYcPf","pdf",617609,1,22,"English","en",105,"# Introduction\n## Contributions","[{\"question\":\"What problem does the paper address in static analysis workflows?\",\"answer\":\"Static analysis produces many security alerts, and a large fraction are false positives, forcing analysts to manually adjudicate each alert. Unreviewed alerts create unknown risk because real vulnerabilities may remain hidden.\"},{\"question\":\"How do the proposed methods reduce LLM adjudication errors?\",\"answer\":\"The paper uses (1) a consistency check that runs the LLM multiple times and compares verdicts, and (2) an LLM reasoning evaluation that runs multiple times and then asks the LLM to select a verdict by weighing the provided reasoning.\"},{\"question\":\"What performance results are reported across the evaluated benchmarks?\",\"answer\":\"Across Juliet, FormAI, and SV-COMP, the evaluated mid-tier reasoning LLMs achieve high recall and specificity. With mistake mitigation, they reach at least 98% recall and at least 94.8% specificity on every suite, depending on whether CC alone or LRE+CC is used.\"}]",1784206158,55,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"using-llms-to-adjudicate-static-analysis-alerts-with-error-reduction-techniques","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/using-llms-to-adjudicate-static-analysis-alerts-with-error-reduction-techniques/85771/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address in static analysis workflows?","Question",{"text":75,"@type":76},"Static analysis produces many security alerts, and a large fraction are false positives, forcing analysts to manually adjudicate each alert. Unreviewed alerts create unknown risk because real vulnerabilities may remain hidden.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How do the proposed methods reduce LLM adjudication errors?",{"text":80,"@type":76},"The paper uses (1) a consistency check that runs the LLM multiple times and compares verdicts, and (2) an LLM reasoning evaluation that runs multiple times and then asks the LLM to select a verdict by weighing the provided reasoning.",{"name":82,"@type":73,"acceptedAnswer":83},"What performance results are reported across the evaluated benchmarks?",{"text":84,"@type":76},"Across Juliet, FormAI, and SV-COMP, the evaluated mid-tier reasoning LLMs achieve high recall and specificity. With mistake mitigation, they reach at least 98% recall and at least 94.8% specificity on every suite, depending on whether CC alone or LRE+CC is used.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]