[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84510-en":3,"doc-seo-84510-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84510,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",8,"Research & Report","The Correctness Illusion in LLM-Generated GPU Kernels","Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) can score numerical correctness using fixed-shape, small-sample allclose-style checks, which varies in input count across benchmarks. A controlled corpus of Triton and CPU stand-in kernels—containing 15 correct controls and 9 LLM-style buggy variants seeded from transcription errors—is re-evaluated with opschema-aware seeded fuzzing against an fp64 CPU reference and per-(op,dtype) absolute tolerances. The seeded oracle flags all buggy kernels and cleanly passes all controls, with identical verdicts across five GPU classes.","arXiv :2606 .20128v1 [ cs . SE] 18 Jun 2026  \nThe Correctness Illusion in LLM-Generated GPU Kernels  \nDipankar Sarkar[0000-0001-5431-6367]  \nArizona State University, USA  \n[dsarkar3@asu.edu](dsarkar3@asu.edu)  \nAbstract. Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks.  \nThe shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under opschema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL) . The verdicts are identical across all five GPUs:  \n10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed  \nLLM. Every flagged failure replays byte-for-byte from a stored seed.  \nKeywords: GPU kernel testing · Triton · fuzzing · mixed precision · LLM code generation · reproducibility  \n1 Introduction  \nLLM-generated GPU kernels are now load-bearing. KernelBench [6], TritonBenchG (the basis of TritonBench-revised referenced from [13]), and GEAK [13] generate hundreds of CUDA and Triton kernels per evaluation. Agentic systems such as KernelBand [7] and STARK [3] compose generated kernels into longer pipelines. Every published benchmark in this family scores correctness through a fixedshape, small-sample allclose-style check. KernelBench [6] draws five random inputs at the reference shape, for example. The number of samples varies. The shape, dtype, and tolerance are fixed per kernel.  \nWe test the oracle. We argue, and measure, that it is systematically optimistic in three specific ways. (i) The shape candidate set is one shape per op. Kernels with tail masking, off-by-one accumulation, or block-size assumptions pass at the chosen shape and fail elsewhere. (ii) The dtype candidate set is one dtype per op. fp16 and bf16 are rarely tested when fp32 is the listed input. Mixed-precision  \n2 D. Sarkar  \noverflow, underflow, and accumulation errors stay undetected. (iii) The tolerance is hand-picked per op. atol and rtol are typically set one to three orders of magnitude looser than the kernel’s measured error envelope, so loose tolerances absorb real wrongness.  \nWe construct a controlled corpus of 24 kernels and re-evaluate it under seeded, op-schema-aware fuzzing with a high-precision (fp64) reference. 15 of the 24 are correct controls. 9 are LLM-style buggy variants seeded with documented transcription errors (missing 0 .5 × in GELU, other=0 .0 versus-inf in softmax tail masking, missing sqrt in RMSNorm, accumulator overwrite in matmul, missing 1/ √D in attention, wrong alpha in LeakyReLU, and three others) . The full per-kernel listing is in Section 3.3, Table 1 . The contributions are four.  \n– A method. Op-schema-aware shape generation produces per-input shapes from shared symbolic dims (matmul A[M, K]·B[K, N], attention B, H, S, D) . The fuzzer covers the operator’s real domain instead of an arbitrary rank-3 cube.  \n– A faithful oracle. The validator compares outputs against an fp64 CPU reference with per-(op, dtype) absolute tolerances, records the full elementwise error distribution (max abs, max rel, ULP percentiles), and detects NaN and Inf for fp16 and bf16 . Note that the validator’s tolerance is absoluteonly per (op, dtype), in contrast to PyTorch’s allclose which combines an absolute","cbCaijma6FqGSYLE","https://ap.wps.com/l/cbCaijma6FqGSYLE","pdf",319414,1,10,"English","en",105,"# 1 Introduction\n# 2 Related Work\n# 3 Methodology\n## 3.3 Kernel Corpus and Listing","[{\"question\":\"为什么现有 LLM 生成 GPU 内核基准在正确性评分上可能产生“幻觉”？\",\"answer\":\"文中指出，基准通常只在每个算子固定一个形状与一个数据类型上做少量 allclose 风格检查，并且容差往往被人为放宽；这会让只在特定形状/类型/容差下“看似正确”的错误被漏检。\"},{\"question\":\"作者如何构造用于验证的内核语料库？\",\"answer\":\"构造了包含 24 个内核的受控语料：15 个正确控制样本以及 9 个由文档化转录错误生成的 LLM 风格错误变体，并在后续将操作扩展到 26 个。\"},{\"question\":\"如何验证并复现实验中被判定的失败？\",\"answer\":\"使用基于算子模式（op-schema）的种子模糊生成输入，用 fp64 CPU 参考与按 (op,dtype) 的绝对容差进行逐元素误差与 NaN/Inf 检测；每次失败会存储输入快照，回放脚本可逐字节重跑并确认裁定一致。\"}]",1784196217,25,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"the-correctness-illusion-in-llm-generated-gpu-kernels","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/the-correctness-illusion-in-llm-generated-gpu-kernels/84510/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么现有 LLM 生成 GPU 内核基准在正确性评分上可能产生“幻觉”？","Question",{"text":75,"@type":76},"文中指出，基准通常只在每个算子固定一个形状与一个数据类型上做少量 allclose 风格检查，并且容差往往被人为放宽；这会让只在特定形状/类型/容差下“看似正确”的错误被漏检。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"作者如何构造用于验证的内核语料库？",{"text":80,"@type":76},"构造了包含 24 个内核的受控语料：15 个正确控制样本以及 9 个由文档化转录错误生成的 LLM 风格错误变体，并在后续将操作扩展到 26 个。",{"name":82,"@type":73,"acceptedAnswer":83},"如何验证并复现实验中被判定的失败？",{"text":84,"@type":76},"使用基于算子模式（op-schema）的种子模糊生成输入，用 fp64 CPU 参考与按 (op,dtype) 的绝对容差进行逐元素误差与 NaN/Inf 检测；每次失败会存储输入快照，回放脚本可逐字节重跑并确认裁定一致。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":21,"slug":133},"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]