[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82013-en":3,"doc-seo-82013-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82013,7971461740886,"Theodore","https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2",8,"Research & Report","Reforge: A Method for Benchmarking LLMs’ Reverse Engineering Capabilities in Decompiled Binary Function Naming","Large language models (LLMs) are increasingly used for reverse-engineering tasks, yet capability claims outpace measurable evidence. Existing benchmarks often treat the creation of function-level ground truth as a solved preprocessing step and report accuracy without stating how many functions are reliably evaluable. Reforge argues the key barrier is binary-to-source alignment under compiler optimization reliability, not raw model ability. It introduces a provenance-tracked pipeline and an eight-gate confidence funnel, plus uncertainty-aware evaluation showing optimization-related performance decay via survivorship bias.","REFORGE: A METHOD FOR BENCHMARKING LLMS’REVERSE ENGINEERING CAPABILITIES INDECOMPILED BINARY FUNCTION NAMING  \nNicolas Koller  \nSystrion AG  \nFlughafenstraße 52  \n22335 Hamburg, Germany  \nAndreas U. Schmidt  \nWilhelm Büchner Hochschule  \nHilpertstraße 31  \n64295 Darmstadt, Germany  \nABSTRACT  \nLarge language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-tosource alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proofof-concept evaluation of seven contemporary LLMs on function naming demonstrates the substrate and motivates uncertainty-aware benchmarking practice.  \nKEYWORDS  \nReverse engineering; LLMs; Binary analysis; Benchmarking; Decompilation; Function naming  \n1. INTRODUCTION  \nReverse engineering (RE) is the practice of recovering the structure, behaviour, and intent of a program from its compiled form. It underpins vulnerability research, malware analysis, interoperability work, and software forensics, and it is among the most labour-intensive activities in computer security, because compilation discards the names, types, and structure that make source code intelligible. Capable large language models (LLMs) trained on code have raised the prospect for automation in this domain, and a growing body of research applies LLMs to decompiled-code understanding, identifier recovery, and decompilation repair.  \nThe stakes of understanding what LLMs can and cannot do have risen sharply. In November 2025, Anthropic reported what it described as the first documented cyber-espionage campaign executed largely autonomously by an AI system, in which an agent was manipulated into performing reconnaissance, vulnerability discovery, exploitation, and post-exploitation across roughly thirty target organizations with limited human intervention (Anthropic, 2025) . Notably, this report observed that model’s hallucinations andoverstated findings remained a practical limiting factor. LLMs are therefore no longer a speculative tool in the offensive and defensive security loop but an operational one whose reliability is uncertain, which makes the rigorous measurement of LLM capability on core RE subtasks a question of immediate practical importance.  \nYet measurement is where the field is weakest. Evaluating an LLM on a task such as recovering a function name from its decompiled, stripped form requires ground truth: a reliable mapping from each binary function back to the source-level identifier it originated from. Constructing that mapping is hard. Compilation is a lossy, many-to-one transformation; optimization inlines, merges, reorders, and eliminates functions, so that the population of source functions and the population of binary functions diverge as optimization increases, and debug-derived mappings fragment under these transformations (Dramko et al., 2024; Tan et al., 2025) . Most existing evaluations treat this alignment problem as a pre-processing detail and report a single accuracy number per optimization l","cbCainhQ97rUXZAD","https://ap.wps.com/l/cbCainhQ97rUXZAD","pdf",359049,6,1,9,"English","en",105,"# Introduction\n## Motivation and measurement gap\n## Contribution overview\n# Reforge Method (pipeline and confidence funnel)\n## Ground-truth construction and alignment uncertainty\n## Stable-key paired analysis and metrics\n# Proof-of-Concept Evaluation\n# Discussion","[{\"question\":\"Why do current LLM-assisted reverse-engineering benchmarks give potentially misleading results?\",\"answer\":\"They often assume binary-to-source function alignment as a solved preprocessing step, reporting a single accuracy score without revealing how many functions are truly evaluable or how optimization changes the set of survivors. This can cause survivorship bias and overstate performance decay.\"},{\"question\":\"What is Reforge, and what problem does it target?\",\"answer\":\"Reforge is a provenance-tracked benchmarking pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation. It treats alignment quality and its uncertainty as first-class, reportable outputs.\"},{\"question\":\"How does Reforge make alignment uncertainty measurable during evaluation?\",\"answer\":\"It operationalizes alignment uncertainty using an eight-gate confidence funnel and stratifies functions into three confidence tiers. This yields uncertainty-aware reporting and auditable attrition across optimization levels.\"}]",1784177571,23,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"reforge-a-method-for-benchmarking-llms-reverse-engineering-capabilities-in-decompiled-binary-function-naming","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/reforge-a-method-for-benchmarking-llms-reverse-engineering-capabilities-in-decompiled-binary-function-naming/82013/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-29","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"Why do current LLM-assisted reverse-engineering benchmarks give potentially misleading results?","Question",{"text":76,"@type":77},"They often assume binary-to-source function alignment as a solved preprocessing step, reporting a single accuracy score without revealing how many functions are truly evaluable or how optimization changes the set of survivors. This can cause survivorship bias and overstate performance decay.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What is Reforge, and what problem does it target?",{"text":81,"@type":77},"Reforge is a provenance-tracked benchmarking pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation. It treats alignment quality and its uncertainty as first-class, reportable outputs.",{"name":83,"@type":74,"acceptedAnswer":84},"How does Reforge make alignment uncertainty measurable during evaluation?",{"text":85,"@type":77},"It operationalizes alignment uncertainty using an eight-gate confidence funnel and stratifies functions into three confidence tiers. This yields uncertainty-aware reporting and auditable attrition across optimization levels.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":107,"slug":137},19,"General","general"]