Simulated Attention Score (SAS) - Parameter-Efficient Attention Aggregation (PEAA)

Aug 27, 2026

Attention mechanisms are a core component of the Transformer architecture, and performance depends strongly on how attention scores are computed. Building on the observation that increasing the number of attention heads improves results when the per-head hidden size remains sufficiently large, this work introduces Simulated Attention Score (SAS). SAS simulates more heads and higher per-head feature dimensions without increasing parameter count by projecting compact head and embedding representations into larger simulated spaces. To further bound parameter and computation costs, Parameter-Efficient Attention Aggregation (PEAA) is proposed. Experiments across multiple datasets and tasks show consistent gains over attention variants.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

Transformer attention background and efficient variants

Motivation for head and feature simulation

Proposed method: SAS and PEAA

Contributions and experimental validation

arXiv :2507 .07694v2 [ cs .CL] 25 Nov 2025 SAS: Simulated Attention Score Chuanyang Zheng1 ∗ Jiankai Sun2,3 Yihang Gao4 Yuehao Wang5 Peihao Wang5 Jing Xiong6 Liliang Ren3 Hao Cheng3 Janardhan Kulkarni3 Yelong Shen3 Zhangyang Wang5 Mac Schwager2 Anderson Schneider1 Xiaodong Liu3 Jianfeng Gao3 1Morgan Stanley 2 Stanford 3Microsoft Research 4NUS 5UT Austin 6HKU Abstract The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multihead attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head. This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA). Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants. 1 Introduction The Transformer architecture [77] has emerged as the predominant framework in modern machine learning, demonstrating remarkable success across diverse domains. Transformer-based models consistently achieve state-of-the-art performance in numerous natural language processing tasks, including machine translation [5, 89], question answering [96, 27, 4], and commonsense reasoning [71, 106, 99, 92, 56] . The fundamental components of the Transformer comprise attention mechanisms and feed-forward networks, with attention score computation serving as a critical element. Beyond the original multi-head attention (MHA) formulation [77], researchers have developed various efficient sparse attention variants, such as sliding window approaches (e.g., Streaming LLMs [80]), linear transformers (e.g., Performer [14]), and sparse attention mechanisms (e.g., Reformer and Sparse Sinkhorn transformer [73, 21]) . Modern architectures employ diverse strategies for attention score computation. The standard MHA approach [77] maintains identical head counts across query, key, and value embeddings. To optimize memory efficiency, subsequent innovations have introduced parameter sharing schemes: Multi-Query Attention (MQA) [61] and Group-Query Attention (GQA) [3] share key and value projections across attention heads. The Multi-Latent Attention (MLA) approach [40] adopts low-rank compression for key and value projections, caching only latent representations. More recently, Tensor Product ∗ Contact Email: [cyzhengme@gmail.com](cyzhengme@gmail.com) 39th Conference on Neural Information Processing Systems (NeurIPS 2025) . Attention (TPA) [100] has introduced tensor decomposition techniques for compact representation of queries, keys, and values. We further analyze the MHA in Figure 1, and we find that the MHA performance gradually increases when the attention head number increases, as long as the hidden size per head is not too small. Hence, by increasing the number of attention heads and the hidden size per head at a minimal cost in terms of additional parameters, we can expect a substantial improvement in

FAQs

What is the main idea behind Simulated Attention Score (SAS)?

SAS simulates a larger number of attention heads and higher per-head feature dimensions by projecting low-dimensional head representations into higher-dimensional spaces, increasing attention capacity without increasing parameter count.

How does SAS expand head count and feature dimension for attention score computation?

It applies linear projections with nonlinear activations along the head dimension to expand heads from H to a larger simulated count, and similarly simulates more feature dimensions for query/key embeddings to expand from D to ˆD.

Why introduce Parameter-Efficient Attention Aggregation (PEAA)?

PEAA is used to control parameter cost and computation size while extending the simulation approach, keeping the overall model compact.

Aug 27, 2026

Upload by Ethan Miller