[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-573266-105":53,"doc-detail-573266-en":120},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":113,"head_meta":115,"extra_data":117,"updated_unix":119},105,"en","benchmarking-security-risk-detection-and-verification-in-open-agentic-skill-ecosystems","Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems","","Open agent platforms let community contributors publish reusable skills that agents invoke at runtime, but this extensibility creates supply-chain risk from malicious skills hidden behind seemingly benign specifications. SkillVetBench introduces a two-stage benchmark: semantic vetting over natural-language instructions to uncover hidden intent, followed by sandbox execution that collects auditable runtime evidence. Built from confirmed malicious skills in OpenClaw, including ClawHavoc samples, it shows semantic-only and signature methods miss up to 89% of threats driven by instruction-level and cross-component logic.",{"@graph":63,"@context":112},[64,80,95],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/benchmarking-security-risk-detection-and-verification-in-open-agentic-skill-ecosystems/573266/",4,{"url":78,"name":59,"@type":81,"author":82,"headline":59,"publisher":85,"fileFormat":88,"inLanguage":57,"description":61,"dateModified":89,"datePublished":89,"encodingFormat":88,"isAccessibleForFree":90,"interactionStatistic":91},"DigitalDocument",{"name":83,"@type":84},"McQueen","Person",{"url":68,"name":86,"@type":87},"DocShare","Organization","application/pdf","2026-10-13",true,{"@type":92,"interactionType":93,"userInteractionCount":4},"InteractionCounter",{"@type":94},"ViewAction",{"@type":96,"mainEntity":97},"FAQPage",[98,104,108],{"name":99,"@type":100,"acceptedAnswer":101},"What problem does SkillVetBench address in open agentic skill ecosystems?","Question",{"text":102,"@type":103},"It addresses the lack of benchmarks that evaluate both detecting malicious skills and verifying what those skills do at runtime.","Answer",{"name":105,"@type":100,"acceptedAnswer":106},"How does SkillVetBench perform its two-stage security vetting?",{"text":107,"@type":103},"First, it performs semantic vetting on each skill’s natural-language specification to detect hidden malicious intent. Second, it executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence.",{"name":109,"@type":100,"acceptedAnswer":110},"What do the experiments show about existing detection baselines?",{"text":111,"@type":103},"Semantic-only and signature-based baselines are insufficient, missing up to 89% of malicious skills, especially those whose threats come from natural-language instructions and cross-component interactions.","https://schema.org",{"og:url":78,"og:type":114,"og:title":59,"og:site_name":86,"og:description":61},"article",{"robots":116,"canonical":78},"index,follow",{"doc_id":118,"site_id":56},573266,1791935991,{"code":4,"msg":5,"data":121},{"doc_id":118,"user_id":122,"nickname":83,"user_avatar":123,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":124,"file_id":125,"file_url":126,"file_type":127,"file_size":128,"view_count":4,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":129,"language":130,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":131,"faqs":132,"seo_title":133,"seo_description":61,"update_tm":119,"read_time":40},5909890329169,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","arXiv :2606 .00925v1 [ cs .CR] 30 May 2026  \nBenchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems  \nIsmail Hossain1 , Sai Puppala2 , Zhuoran Lu3 , Sajedul Talukder1 and Nan Jiang1  \n1University of Texas at El Paso, TX, USA, 2 Southern Illinois University-Carbondale, IL, USA, 3Purdue University, IN, USA  \nOpen agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supply-chain risk: malicious contributors can hide harmful behavior inside skills that appear benign under superficial inspection. However, existing defenses are hard to evaluate because there is no benchmark that measures both malicious-skill detection and runtime verification. We present SkillVetBench, a two-stage security vetting benchmark for open agentic skill ecosystems. The first stage performs semantic vetting over each skill’s natural-language specification to detect hidden malicious intent. The second stage executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence. We build a benchmark from confirmed malicious skills in the live OpenClaw ecosystem, including samples from the recent ClawHavoc supplychain campaign. Unlike static-only methods, SkillVetBench verifies detected threats with execution traces. Our experiments show that: (1) semantic-only and signature-based baselines are insufficient, missing up to 89% of malicious skills whose threats arise from natural-language instructions, multicomponent logic, or cross-component interactions; (2) runtime attacks are concentrated in a small set of high-permission primitives, especially exec, write_file, install_skill, and spawn; and (3) SkillVetBench provides case studies in which sandbox execution directly supports malicious verdicts with concrete runtime evidence.  \n1. Introduction  \nOpen agentic platforms are rapidly evolving into large-scale ecosystems where agents can discover, install, and invoke community-authored skills at runtime [4, 62] . Examples such as OpenClaw and NanoClaw allow third-party contributors to publish reusable skills that extend an agent’s executable action space. OpenClaw alone hosts more than 60,000 skills, illustrating both the scale and practical importance of these emerging ecosystems. However, the same openness that enables rapid capability expansion also introduces a new supply-chain attack surface: malicious contributors can embed harmful behavior inside skills that appear benign to users, agents, or marketplace scanners.  \nThis risk is no longer hypothetical. The recent ClawHavoc supply-chain campaign [63] introduced 1, 184 malicious skills into the ClawHub marketplace despite the presence of official submission vetting tools. The malicious contributors weaponized multiple components of the skill artifact to enable harmful behaviors, including credential theft and cryptocurrency exfiltration. Particularly, some of these behaviors only surface at execution time and cannot be caught by official static scanners. The incident exposes a broader evaluation gap for emerging open agent platforms: current agent-skill marketplaces lack systematic benchmarks for assessing whether security vetting tools can both detect malicious skills and verify their actual runtime behavior.  \nSecurity evaluation in traditional software ecosystems has long faced the same challenge, and decades of work have converged on a clear answer: combining static reasoning with dynamic analysis [9, 15, 51] . Prior work on package registries and software supply-chain security shows that malicious packages often evade metadata-or source-level inspection and that sandboxed execution can  \nEmails: [ihossain@miners.utep.edu](ihossain@miners.utep.edu), [sai.puppala@siu.edu](sai.puppala@siu.edu), [lu800@purdue.edu](lu800@purdue.edu), [stalukder@utep.edu](stalukder@utep.edu), [njiang@utep.edu](njiang@utep.edu)  \n[Sajedul Talukder and Nan Jiang jointly su","cbCaibHE3k2q9S16","https://ap.wps.com/l/cbCaibHE3k2q9S16","pdf",1155441,51,"English","# Introduction\n## Proposed benchmark: SkillVetBench\n## Two-stage vetting design (semantic + sandbox runtime)\n## Experimental findings and risk concentration","[{\"question\":\"What problem does SkillVetBench address in open agentic skill ecosystems?\",\"answer\":\"It addresses the lack of benchmarks that evaluate both detecting malicious skills and verifying what those skills do at runtime.\"},{\"question\":\"How does SkillVetBench perform its two-stage security vetting?\",\"answer\":\"First, it performs semantic vetting on each skill’s natural-language specification to detect hidden malicious intent. Second, it executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence.\"},{\"question\":\"What do the experiments show about existing detection baselines?\",\"answer\":\"Semantic-only and signature-based baselines are insufficient, missing up to 89% of malicious skills, especially those whose threats come from natural-language instructions and cross-component interactions.\"}]","Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems | PDF"]