[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81897-en":3,"doc-seo-81897-105":31,"detail-sidebar-cat-0-en-105":93},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":28,"seo_description":14,"update_tm":29,"read_time":30},81897,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","HiFA4 Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference","HiFA4 presents a post-training operator-level design for LLM inference that executes both QKT and PV matrix multiplications in FlashAttention as 4-bit HIF4 Cube GEMMs on Ascend NPUs, while keeping the online softmax state in FP16. The method combines Smooth-QK, which applies calibration-static per-channel rescaling to Q and K after RoPE, and P-Reordering, which accumulates the softmax normalizer from the same quantized attention weights used in the PV GEMM. Experiments on multiple models reduce quantization-induced decision drift and regressions versus direct 4-bit quantization.","arXiv :2607 .04302v 1 [ cs .LG] 5 Jul 2026  \nHiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference  \nHui Dong∗ Yanzhao Li Jie Gao Chunlu Li  \nZhiyuan Zhang Yupeng Sun Zhenyuan Chen Zhiqiang Zou  \nHuawei Technologies  \n[donghui24@huawei.com](donghui24@huawei.com)  \nAbstract  \nWe present HiFA4, a post-training operator-level design that executes both the QKT and PV matrix multiplications in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining online softmax state in FP16 . To our knowledge, HiFA4 is the first such design evaluated on standard NLP benchmarks.  \nWe introduce two complementary mechanisms. Smooth-QK applies a calibration-static, per-channel equivalent rescaling to Q and K after RoPE, transferring quantization difficulty from K to Q. Because the target channels are determined by fixed model parameters rather than input data, the scale factors are computed once from a small offline calibration set and require no per-tile online reduction at inference (only a fusible element-wise scaling) . P-Reordering ensures that the softmax normalizer is accumulated from the same quantized attention weights ˆP used in the PV GEMM, rather than a higher-precision reconstruction. We prove (Theorem 1) that the conventional inconsistent formulation introduces a systematic, coherent output-scaling error, and validate empirically on a Qwen3-8B Layer-0 MMLU trace that this error is present in 100% of 3 .6 million real attention tiles with median magnitude ε¯ = −0 .064. P-Reordering additionally allows the normalizer accumulation to be fused into the PV Cube GEMM.  \nEvaluated on five LLMs, HiFA4 consistently reduces quantization-induced decision drift. On Qwen3-8B, HiFA4 recovers 37.5% of the accuracy gap introduced by direct HIF4 quantization (sample-weighted δw narrowing from 1.12pp to 0.70pp, Table 2), reduces the fraction of BF16-inconsistent predictionson MMLU from 16.3% to 8.2%, and cuts accuracy regressions (samples correctly answered by BF16 but incorrectly by the quantized model) by 57%(1 ,071 → 465 on MMLU full set) . On Gemma2-9B, mild smoothing keeps HiFA4 within 0.7 pp of BF16 while cutting MMLU accuracy regressions by 27% relative to direct quantization. On the three models where the applicability gate disables Smooth-QK (LLaMA3.1-8B, Mistral-7B, Phi-4B), the remaining HiFA4 components—P-Reordering together with the adopted Q-Mean auxiliary—reduce full-set MMLU accuracy regressions by 41–52%(and HellaSwag by 39–42%) with no K-smoothing, showing that this part of the design generalizes beyond models with concentrated K-outliers. A theoretical instruction-scheduling analysis projects a 35 .4% critical-path latency reduction relative to BF16 by fusing the softmax normalizer into the PV Cube GEMM via P-Reordering; on-hardware validation will be reported when the target Ascend NPU becomes publicly available.  \n1 Introduction  \nFlashAttention [Dao et al. , 2022 , Dao, 2024 , Shah et al. , 2024 , Zadouri et al. , 2026] is the widely-adopted tiled attention kernel for LLM inference. Its two dominant matrix multiplications, QKT and PV , constitute the majority of attention compute in the prefill stage. Executing both as 4-bit Cube GEMMs using Ascend HIF4 would substantially improve throughput, since 4-bit operands allow the Cube units to process four times as many elements per cycle compared to BF16 . The challenge is preserving accuracy: direct 4-bit quantization of attention activations causes substantial degradation in language models.  \nPrior FP4 attention work has mainly focused on diffusion and video-generation models. SageAttention3 [Zhang et al. , 2025b] reports FP4 inference results on CogVideoX, HunyuanVideo, Mochi, Flux, and SD3.5, but its NLP-related experiments do not correspond to a full post-training FP4 inference evaluation.  \nConcurrent work targets NVIDIA NVFP4 rather than Ascend HIF4 and uses different correction strategies: online scale search with mix","cbCaih61Np4e0Sb6","https://ap.wps.com/l/cbCaih61Np4e0Sb6","pdf",4401013,6,1,22,"English","en",105,"# Abstract\n# Introduction\n# Related Work\n# Background","[{\"question\":\"What is HiFA4 and what hardware does it target for LLM inference?\",\"answer\":\"HiFA4 is a post-training operator-level design that runs FlashAttention’s QKT and PV computations as 4-bit HIF4 Cube GEMMs. It targets Ascend NPUs and keeps online softmax state in FP16.\"},{\"question\":\"How does Smooth-QK improve accuracy under 4-bit quantization?\",\"answer\":\"Smooth-QK performs calibration-static, per-channel equivalent rescaling of Q and K after RoPE. It transfers quantization difficulty from K to Q using precomputed scale factors from an offline calibration set.\"},{\"question\":\"What problem does P-Reordering address and how does it affect performance?\",\"answer\":\"P-Reordering eliminates a coherent output-scaling error caused by normalizer–GEMM inconsistency by accumulating the softmax normalizer from the same quantized attention weights used in the PV GEMM. It also enables fusing the normalizer accumulation into the PV Cube GEMM, reducing the projected critical-path latency.\"}]","HiFA4 Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference | PDF",1784176927,55,{"code":4,"msg":32,"data":33},"ok",{"site_id":25,"language":24,"slug":34,"title":13,"keywords":35,"description":14,"schema_data":36,"social_meta":88,"head_meta":90,"extra_data":92,"updated_unix":29},"hifa4-training-free-4-bit-flashattention-on-ascend-hif4-npus-for-llm-inference","",{"@graph":37,"@context":87},[38,55,70],{"@type":39,"itemListElement":40},"BreadcrumbList",[41,45,49,52],{"item":42,"name":43,"@type":44,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":46,"name":47,"@type":44,"position":48},"https://docshare.wps.com/document/","Document",2,{"item":50,"name":12,"@type":44,"position":51},"https://docshare.wps.com/document/research-report/",3,{"item":53,"name":13,"@type":44,"position":54},"https://docshare.wps.com/document/hifa4-training-free-4-bit-flashattention-on-ascend-hif4-npus-for-llm-inference/81897/",4,{"url":53,"name":13,"@type":56,"author":57,"headline":13,"publisher":59,"fileFormat":62,"inLanguage":24,"description":14,"dateModified":63,"datePublished":64,"encodingFormat":62,"isAccessibleForFree":65,"interactionStatistic":66},"DigitalDocument",{"name":9,"@type":58},"Person",{"url":42,"name":60,"@type":61},"DocShare","Organization","application/pdf","2026-07-29","2026-07-16",true,{"@type":67,"interactionType":68,"userInteractionCount":20},"InteractionCounter",{"@type":69},"ViewAction",{"@type":71,"mainEntity":72},"FAQPage",[73,79,83],{"name":74,"@type":75,"acceptedAnswer":76},"What is HiFA4 and what hardware does it target for LLM inference?","Question",{"text":77,"@type":78},"HiFA4 is a post-training operator-level design that runs FlashAttention’s QKT and PV computations as 4-bit HIF4 Cube GEMMs. It targets Ascend NPUs and keeps online softmax state in FP16.","Answer",{"name":80,"@type":75,"acceptedAnswer":81},"How does Smooth-QK improve accuracy under 4-bit quantization?",{"text":82,"@type":78},"Smooth-QK performs calibration-static, per-channel equivalent rescaling of Q and K after RoPE. It transfers quantization difficulty from K to Q using precomputed scale factors from an offline calibration set.",{"name":84,"@type":75,"acceptedAnswer":85},"What problem does P-Reordering address and how does it affect performance?",{"text":86,"@type":78},"P-Reordering eliminates a coherent output-scaling error caused by normalizer–GEMM inconsistency by accumulating the softmax normalizer from the same quantized attention weights used in the PV GEMM. It also enables fusing the normalizer accumulation into the PV Cube GEMM, reducing the projected critical-path latency.","https://schema.org",{"og:url":53,"og:type":89,"og:title":13,"og:site_name":60,"og:description":14},"article",{"robots":91,"canonical":53},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":94},[95,99,103,107,112,116,121,124,129,132,136],{"id":21,"doc_module":4,"doc_module_name":47,"category_name":96,"show_sort_weight":97,"slug":98},"Story & Novel",90,"story-novel",{"id":48,"doc_module":4,"doc_module_name":47,"category_name":100,"show_sort_weight":101,"slug":102},"Literature",80,"literature",{"id":54,"doc_module":4,"doc_module_name":47,"category_name":104,"show_sort_weight":105,"slug":106},"Exam",70,"exam",{"id":108,"doc_module":4,"doc_module_name":47,"category_name":109,"show_sort_weight":110,"slug":111},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":47,"category_name":113,"show_sort_weight":114,"slug":115},"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":47,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":47,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":47,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":47,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":47,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":47,"category_name":138,"show_sort_weight":108,"slug":139},19,"General","general"]