STEEL Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD’s XDNA NPU

Jul 16, 2026

Energy-efficient inference for large language model (LLM) agents is increasingly important on laptop systems-on-chip, where cloud offloading raises latency, reliability, and privacy concerns. While NPUs improve power efficiency, mapping attention onto diverse architectures and explicit data-movement models is difficult. STEEL presents an open-source FlashAttention implementation for XDNA-like NPUs, with a prefill dataflow formulation exploiting spatial parallelism and on-chip memory. A sparsity-aware pipeline placement mitigates causal-mask imbalance, reducing synchronization overhead and improving utilization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

Motivation for energy-efficient agent inference

Challenges mapping attention to NPUs

STEEL contributions

STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD’s XDNA™ NPU Victor J.B. Jung†∗ , Gagandeep Singh∗ , Joseph Melber∗ , Kristof Denolf∗ , Francesco Conti‡, Luca Benini†‡ ∗AMD Research and Advanced Development (RAD) . †Integrated Systems Laboratory (IIS), ETH Z¨urich, Switzerland. ‡ Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy. arXiv :2607 .09385v1 [ cs .DC] 10 Jul 2026 Abstract—The growing adoption of large language model (LLM)-based agents within operating system workflows has increased the importance of energy-efficient inference on laptopclass systems-on-chip (SoCs). While cloud offloading remains common, it introduces reliability and privacy concerns that are particularly problematic for agentic workloads. Recent laptop SoCs, therefore, incorporate neural processing units (NPUs) optimized for energy efficiency; however, effectively mapping attention mechanisms onto NPUs remains challenging due to architectural diversity and explicit data-movement programming models. In this work, we present STEEL, the first open-source implementation of FlashAttention targeting XDNA-like NPUs. STEEL introduces a dataflow formulation of prefill attention, enabling efficient exploitation of spatial parallelism and on-chip memory. Furthermore, STEEL addresses the load imbalance induced by the causal mask by leveraging a sparsity-aware pipeline placement onto the NPU array, reducing synchronization overhead and improving utilization. We evaluate STEEL on the AMD Ryzen™ AI 9 HX 370 SoC and compare its performance against optimized central processing unit (CPU) and graphics processing unit (GPU) implementations. Experimental results show that STEEL reduces energy consumption by an average of 9.17 × and 1.75 × relative to CPU and GPU baselines, respectively. On XDNA™ 1, STEEL achieves an average 9.6 × latency reduction over the prior state of the art (SotA), and delivers a 22.8 × speedup on average compared to a layer-bylayer attention implementation on XDNA™ 2. I. INTRODUCTION THEageinntcsreaintregperraattiionng of artificial intelligence (AI) system functions is a key driver in the design of modern laptop systems-on-chip (SoCs) [1] . These agents are typically implemented as large Transformerbased deep neural networks (DNNs) with several billion parameters [2] . While such models enable powerful capabilities, their inference demands impose substantial computational and data-movement overhead, making them inherently energyintensive. This energy cost has emerged as a fundamental bottleneck for embedded mobile platforms, where power and thermal budgets are tightly constrained [2] . As a result, most large language model (LLM) inference is currently offloaded to data-center graphics processing units (GPUs) . Although effective from a performance standpoint, this centralized approach introduces challenges in agentic workflows, including increased latency, reduced reliability, and heightened privacy risks [3] . To unlock the full potential of AI agents at the edge, recent laptop SoCs integrate neural processing units (NPUs) designed specifically for energy-efficient inference [4],[5],[6] . NPUs target the most computationally and energy expensive components of Transformer models, most notably the attention mechanism [7] . The prefill stage of Attention is a major contributor to inference latency and energy consumption at long-sequence length, a regime that is increasingly common in practical LLM deployments [7] . As a result, substantial effort has been made to optimize attention on both commercial [8] and academic hardware platforms [9] . The range of attention optimizations is broad, from algorithmic improvements like FlashAttention [8] to hardware enhancements including specialized non-linear units [10] . NPUs achieve high energy efficiency through spatial dataflow architectures and explicit data-movement programming models, whi

FAQs

Why is energy-efficient inference important for laptop LLM agent workflows?

Agentic LLM inference is energy-intensive due to computation and data movement, and edge platforms have tight power and thermal limits. Offloading to cloud can also increase latency and introduce reliability and privacy risks.

What is STEEL in this work?

STEEL is presented as the first open-source FlashAttention implementation targeting XDNA-like NPUs. It uses a prefill attention dataflow formulation designed for efficient execution on the NPU array.

How does STEEL improve performance and energy efficiency on XDNA NPUs?

STEEL exploits spatial parallelism and on-chip memory for the prefill stage, and uses a sparsity-aware pipeline placement to reduce causal-mask-induced load imbalance and synchronization overhead. Experiments report significant average energy and latency improvements versus CPU and GPU baselines and prior work.

Jul 16, 2026

Upload by Genevieve