[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82680-en":3,"doc-seo-82680-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82680,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","TAG A Lightweight Framework for Test-Driven Agentic Artifact Generation","Large Language Models can generate structured artifacts such as database queries, threat framework mappings, and entity schemas, yet production reliability remains difficult. TAG is a lightweight framework built on LLMs generate and systems validate. It uses test-driven agentic generation with informative failure feedback, combines deterministic programmatic tests for verifiable properties and LLM-based semantic tests for nuanced correctness, and employs expert-calibrated judges to mirror expert decision distributions. Deployed at Microsoft Sentinel for security artifacts, TAG enables reliable outputs efficiently.","TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation  \nYaniv Melamed, 1 Yoni Zukerman, 1 Michal Shechter, 1 Miri Weissler, 1 Ashwin Patil, 1 Hani Neuvirth-Telem 1  \n1Microsoft  \narXiv :2607 .026 15v2 [ cs .CR] 8 Jul 2026  \nAbstract  \nGenerating structured artifacts with Large Language Models - e.g. database queries, threat framework mappings, entity schemas-is relatively straightforward; however, making them reliable enough for production deployments presents challenges. We present TAG, a lightweight framework based on a core principle: LLMs generate, we validate. This reframing shifts responsibility from generation quality to validation rigor. The framework rests on three key attributes: First, test driven generation: when tests fail, the LLM receives indicative error messages that expose why the output failed, enabling the LLM to understand its mistakes and refine subsequent attempts. Second, deterministic and LLMbased tests: deterministic tests catch heuristics that can be programmatically verified (schema, syntax, cross-reference), while LLM-based tests evaluate nuanced semantic and delicate features that resist programmatic inspection (intent alignment, logical consistency, domain correctness) . Third, expert-distilled judges: LLM-based tests are calibrated to distill and replicate human expert decision distribution, transforming manual human quality gates into scalable, reusable evaluation proxies that reflect professional-grade validation standards. We demonstrate the framework on three artifact types in the security domain-KQL query generation, MITREATT&CK mapping, and entity mapping - deployed in production at Microsoft Sentinel. We believe this framework can be applied beyond security to other artifact generation tasks, providing a path to reliable, high-quality outputs without sacrificing the efficiency gains of LLM generation.  \nIntroduction  \nAcross many professional domains, experts spend significant time crafting structured artifacts that run in production systems and serve customers at scale. A threat analyst authors detection rules that continuously monitor for advanced persistent threats across an enterprise with millions of endpoints. A compliance engineer maps regulatory requirements to executable policy rules that enforce data residency and encryption standards at scale. A data engineer designs ETL pipelines that transform raw telemetry into normalized schemas powering downstream ML models and business intelligence. An infrastructure engineer provisions cloud environments using infrastructure-as-code templates that reliably deploy multi-region systems to production. A  \nCorrespondence: [yanivmelamed@microsoft.com](yanivmelamed@microsoft.com).  \nhealthcare informaticist maps clinical observations to standardized medical coding frameworks that ensure interoperability across hospital systems. A security architect encodes threat modeling results into entity schemas that automatically classify assets and risks in security platforms.  \nThese artifacts share a common profile: they are authored once by a domain expert, validated carefully against correctness criteria, and then deployed broadly to production where they operate autonomously. The authoring process is slow and expensive - it requires deep domain knowledge, iterative refinement, and manual quality assurance-but the deployment is high-leverage: a single well-crafted artifact can serve an entire customer base.  \nLarge Language Models have made it remarkably easy to generate such artifacts. Given a natural-language specification, an LLM can produce a database query, a configuration file, a mapping to a standardized framework, or an entity schema in seconds. Making those artifacts reliable enough to deploy, however, remains hard. A single hallucinated column name, an over-broad filter, or a semantically incorrect mapping is enough to render a generated artifact useless-or worse, silently wrong in production. The dominant response has be","cbCaib0DpGHVzS2v","https://ap.wps.com/l/cbCaib0DpGHVzS2v","pdf",699080,1,7,"English","en",105,"# Abstract\n# Introduction\n## Challenge: making generated artifacts reliable\n## Test-driven reframing: acceptance criteria via tests\n# TAG framework principles\n## Test-driven agentic generation loop\n## Programmatic and semantic testing\n## Expert-calibrated LLM judges\n# Demonstration and deployment","[{\"question\":\"What core principle does TAG use for reliable artifact generation?\",\"answer\":\"TAG reframes the problem so that LLMs generate while validation defines acceptance criteria. The system uses a test suite to verify correctness and guide iterative refinement.\"},{\"question\":\"How does TAG handle correctness checks in both programmatic and semantic aspects?\",\"answer\":\"Deterministic tests verify properties such as schema conformance and syntax validity, while LLM-based tests evaluate semantic qualities that are hard to express formally, such as intent alignment and logical consistency.\"},{\"question\":\"How are expert-calibrated LLM judges created in TAG?\",\"answer\":\"TAG calibrates judge prompts using a dataset of expert-labeled examples, tuning prompts to replicate the expert decision profile and prioritize low false-positive rates to reliably reject bad artifacts.\"}]",1784182245,18,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"tag-a-lightweight-framework-for-test-driven-agentic-artifact-generation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/tag-a-lightweight-framework-for-test-driven-agentic-artifact-generation/82680/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What core principle does TAG use for reliable artifact generation?","Question",{"text":75,"@type":76},"TAG reframes the problem so that LLMs generate while validation defines acceptance criteria. The system uses a test suite to verify correctness and guide iterative refinement.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does TAG handle correctness checks in both programmatic and semantic aspects?",{"text":80,"@type":76},"Deterministic tests verify properties such as schema conformance and syntax validity, while LLM-based tests evaluate semantic qualities that are hard to express formally, such as intent alignment and logical consistency.",{"name":82,"@type":73,"acceptedAnswer":83},"How are expert-calibrated LLM judges created in TAG?",{"text":84,"@type":76},"TAG calibrates judge prompts using a dataset of expert-labeled examples, tuning prompts to replicate the expert decision profile and prioritize low false-positive rates to reliably reject bad artifacts.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]