Adaptive Model Compression (AMC) Saliency-Driven Resource Allocation for Ultra-Low-Power Transformer Inference

Jul 16, 2026

Upload by Quinn

Deploying large-scale transformer models on resource-constrained edge devices remains difficult because static inference treats all tokens with equal compute, creating substantial energy and memory overhead. Adaptive Model Compression (AMC) introduces a saliency-driven, hardware-aware framework that assigns resources dynamically according to token importance. A multi-tier architecture processes high-saliency information at full precision, while compressing less important data via reduced rank and bit-width. On 45nm CMOS hardware, AMC cuts system energy by 59.2% and boosts throughput 2.24×, with only a 3.6% accuracy trade-off.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

I. Introduction

2. Challenges

3. Related Work

II. Methodology: Hardware-Software Co-Design

A. The Software Saliency Engine

Adaptive Model Compression (AMC): Saliency-Driven Resource Allocation for Ultra-Low-Power Transformer Inference Jiayin Hu, Kai Yuan, Vanessa Hu, Xuetao Yin, Jianhua Li, Sean Suchter Apple USA arXiv :2607 . 10 109v 1 [ cs .IR] 11 Jul 2026 Abstract—Deploying large-scale transformer models on resource-constrained edge devices remains a challenge due to the high energy and memory overhead inherent in static inference, which processes simple and complex tokens with uniform intensity. To address this, we propose Adaptive Model Compression (AMC), a saliency-driven framework that dynamically allocates hardware resources based on token importance. By implementing a multi-tier architecture, our system identifies critical high-saliency information for full-precision processing while aggressively reducing the rank and bit-width of less significant data. Experimental results demonstrate that AMC achieves a 59.2% reduction in system energy and a 2.24× increase in throughput on 45nm CMOS hardware. This approach effectively extends the battery life of mobile devices by utilizing high-definition compute only where necessary, maintaining robust performance with a marginal 3.6% accuracy trade-off. Index Terms—energy-saving, AMC, co-design I. INTRODUCTION Standard transformer inference is often bottle-necked by memory bandwidth and static computational overhead. The deployment of large-scale transformers on edge hardware is limited by the energy cost of SRAM access and multiplyaccumulate (MAC) operations. Conventional Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) are static, treating all input data with uniform precision regardless of complexity. We present a co-design approach where the software dynamically schedules hardware resource allocation based on real-time activation saliency. 1) . Background Modern Transformer-based LLMs are fundamentally bottle necked by the memory wall, where the energy cost of moving model weights from off-chip DRAM to on-chip SRAM exceeds the energy of the actual MultiplyAccumulate (MAC) operations by orders of magnitude. Standard inference pipelines operate in a static manner, allocating uniform computational precision and rank-dimensional capacity to every token in a sequence regardless of its linguistic entropy. This uniform allocation strategy fails to acknowledge that in natural language generation, the vast majority of tokens—such as syntactic glue words, repetitive punctuation, or semantically redundant filler—contribute negligibly to the final output probability distribution, yet consume the same energy budget as highly informative, context-defining tokens. 2) . Challenges Implementing adaptive resource scaling on silicon at the 45nm node introduces three primary engineering challenges are as follows. The Structural Alignment Dilemma: Raw Transformer hidden dimensions do not inherently exhibit ordered variance; blindly truncating dimensions leads to stochastic information loss. Our framework addresses this through offline structural calibration to enforce variance-based channel ordering.Controller Overhead: Introducing complex dynamic logic to guess token importance can consume more energy than the compression saves. The challenge is to maintain O (N) computational complexity for the saliency engine, ensuring the hardware-side clock-gating controller remains a negligible fraction of the total die area. Systolic Array Synchronicity: Aggressively gating physical execution elements can lead to pipeline stalls in systolic array architectures. AMC must ensure that token-level tier switching maintains data-flow throughput without inducing latency bottlenecks. 3) . Related Work Existing literature has pursued token compression through three primary lenses, each possessing distinct limitations that AMC aims to resolve. Dynamic Quantization: Methods such [as LLM.int8](as LLM.int8)() [14] and AWQ [15] reduce precision globally or per-layer. While effective for memory bandwidth

FAQs

What problem does Adaptive Model Compression (AMC) target in transformer inference on edge devices?

AMC targets the high energy and memory overhead of static inference pipelines that process all tokens with uniform precision, wasting compute on low-importance tokens.

How does AMC determine which tokens deserve higher precision?

AMC uses a saliency-driven software engine based on how activation magnitude correlates with influence on the final prediction, identifying information-rich tokens before hardware execution.

What hardware and performance benefits does AMC report on 45nm CMOS?

AMC reports a 59.2% system energy reduction and a 2.24× throughput increase on 45nm CMOS, while maintaining robust performance with a marginal 3.6% accuracy trade-off.

Jul 16, 2026

Upload by Quinn