Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention - arXiv 2608.19203v1

Sep 15, 2026

Uploaded by Levi

Standard multi-head attention assigns every head an identical causal context span, even though different heads can play distinct roles in capturing local lexical/syntactic cues or longer-range dependencies. This work introduces Asymmetric Attention Heads (AAH), a head-wise context allocation framework that treats context length as an explicit per-head/per-group variable. AAH groups and organizes heads hierarchically, assigns discrete causal local windows, and preserves the standard Transformer flat output interface. Experiments on 4096-token seed-0 show lower validation loss for AAH-style local allocation variants; ablations highlight the importance of stable local allocation and head-window structure. Attention Coverage Ratio (ACR) is used as a routing diagnostic, not a FLOPs reduction claim.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages30

Reading Time1 h 16 min

AccessFree to download

Abstract

1 Introduction

Method and core design

Diagnostics and claim boundaries

Contributions overview

arXiv : 2608 . 19203v 1 [cs .CL] 10 Jun 2026 Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention Zimu Zhao [zimmer061310@gmail.com](zimmer061310@gmail.com) § [github.com/Zimmer061310/Asymmetric-Attention-Heads](github.com/Zimmer061310/Asymmetric-Attention-Heads) Abstract Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise contextallocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096-token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic. (a) Standard MHA | | Input hidden states | | | | --- | --- | --- | --- | | | | | | | Shared Q/K/V projection | | | | |
Head | Head | |
Head | Head | | --- | --- | --- | --- | --- | | | | Full causal span
for every head | | | | --- | --- | --- | --- | --- | | | | | | | | | | Standard causal
attention | | | | | | | | | | | | Flat head concat | | | | | | | | | | | Output projection | | | | | | | | | | | Standard block output | | | | | Uniform full-span execution Figure 1: Standard MHA versus AAH-v3 . Standard MHA applies one full causal span to all heads before the flat output merge. AAH-v3 keeps the shared Q/K/V projection and flat Transformer interface, but adds a control branch that assigns per-head windows; the execution branch buckets heads by selected window and can use dense-masked, FlexAttention, or FlashAttention execution. 1 Introduction Transformers rely on multi-head attention (MHA) [1], where each head is typically executed with a uniform attention pattern and full attended span. This design is simple and robust, but it also hidesa structural question: different heads may not need the same amount of context to make useful contributions. This motivates the central question in this paper: can a Transformer allocate context windows unevenly across heads while preserving, or sometimes improving, language-modeling quality? We use the term Asymmetric Attention Heads (AAH) for this design direction. Attention heads remain part of the standard MHA interface, but they are allowed to receive unequal context windows. AAH-v3 controls MHA execution across heads and head groups by selecting discrete causal local windows, while preserving the standard Transformer block interface and flat output merge. The method changes the head-wise context-allocation policy, rather than the output topology of the Transformer block. Our evaluation scope is intentionally constrained. We treat Attention Coverage Ratio (ACR)1 as a selected-window coverage proxy, not a measured FLOPs metric. The dense-masked path does not establish skipped backend work; therefore the main rows omit backend work-accounting columns. True GPU FLOPs are measured only by Nsight-derived GPU floating-point-operation counters, and the current Nsight evidence does not support a FLOPs-reduction claim for the implementations in this paper. AAH is backend-agnostic in the narrow sense that it chooses per-head context windows and then

FAQs

How does Asymmetric Attention Heads (AAH) change standard multi-head attention?

AAH allows different heads (or head groups) to receive different-sized causal context windows. It controls per-head execution by selecting discrete causal local windows while keeping the standard flat Transformer interface for outputs.

What does Attention Coverage Ratio (ACR) measure in this work?

ACR is used as a selected-window routing diagnostic and a proxy for window coverage. The paper separates ACR from hardware FLOPs measurement and does not treat it as a compute-reduction metric.

Do the experiments show lower GPU FLOPs for AAH?

No. The paper reports Nsight and FLOPs-lab diagnostics indicating the current implementations do not establish measured GPU-FLOPs savings, and it treats quality/routing effects as the primary claim.

Sep 15, 2026

Uploaded by Levi