Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

Jul 16, 2026

This paper proposes an improved structured pruning method for large language models (LLMs) that adapts Adaptive Feature Retention (AFR), originally an unstructured pruning technique, to structured pruning constraints. Direct score aggregation from AFR to neuron-level pruning causes distribution mismatch across heterogeneous scores, loss of sign information for optimization direction consistency, and outlier dominance. The method uses power transformation for nonlinear alignment, sign-preserving score aggregation, and percentile-based outlier removal to improve robustness and accuracy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Background: Adaptive Feature Retention

arXiv :2607 .08027v 1 [ cs .CL] 9 Jul 2026 Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention Ryota Kobayashi1, Tsubasa Hirakawa1, Takayoshi Yamashita1, Hironobu Fujiyoshi1, Yasunori Ishii2, Tomoyuki Okuno2, Kazuki Kozuka2 1 Chubu University 2 Panasonic Holdings Corporation Abstract This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning. 1 Introduction Large language models (LLMs) demonstrate remarkable performance across diverse tasks, yet their billions of parameters impose substantial computational costs and memory requirements that hinder practical deployment. Pruning, which removes redundant weights, offers a promising approach to model compression. Pruning methods are categorized into unstructured and structured pruning. Unstructured pruning removes individual weights independently, achieving high accuracy but providing limited practical speedup due to irregular sparsity patterns. Structured pruning removes entire neurons or channels, enabling efficient acceleration on standard hardware but often suffering from larger performance degradation due to reduced pruning granularity. Adaptive Feature Retention (AFR) (Nitta et al., 2025) is an unstructured pruning method that combines feature-based ReFer (Nitta et al., 2024) and gradient-based SNIP (Lee et al., 2019) scores through standardization and summation, balancing preservation of pre-trained feature representations with adaptation to downstream tasks. When adapting AFR to structured pruning by aggregating weight-level scores to neuron-level scores, several challenges arise: distribution mismatch between heterogeneous scores (ReFer exhibits wide value ranges while SNIP concentrates in [0, 1]), loss of sign information indicating optimization direction consistency, and influence of outliers that dominate simple averaging. To address these challenges, we propose an integrated approach combining power transformation for nonlinear distribution alignment, sign-preserving aggregation, and percentilebased outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B (Zheng et al., 2023), and LLaVA-v1.5-13B (Liu et al., 2024) demonstrate that our method significantly outperforms naive structured AFR and achieves comparable or superior performance to existing structured pruning methods while enabling 1.56-1.57× inference speedup at 50% pruning rate. Our main contributions are summarized as follows: • We identify three fundamental challenges in adapting AFR to structured pruning: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers during score aggregation. • We propose an integrated approach that addresses these challenges through (1) power transformation for nonlinear distribution alignment,(2) sign-preserving score aggregation to evaluate optimization direction consistency, and (3) percentilebased outlier removal for robust aggregation. • We conduct comprehensive experiments on three models (Llama-3-8B, Vicuna-v1.5- 13B, and LLaVA-v1.5-13B) demonstrating that our m

FAQs

What problem does the paper address when adapting Adaptive Feature Retention (AFR) to structured pruning?

Adapting AFR to structured pruning introduces distribution mismatch between heterogeneous pruning scores, loss of sign information needed for optimization direction consistency, and outlier influence that can dominate naive aggregation.

What three techniques are proposed to fix these adaptation issues?

The approach combines power transformation for nonlinear distribution alignment, sign-preserving score aggregation to keep optimization-direction information, and percentile-based outlier removal for robust scoring.

What results does the method achieve on models like Llama-3-8B and Vicuna-v1.5-13B?

Experiments show accuracy comparable to unstructured pruning while obtaining practical inference speedups using structured pruning, including about 1.56–1.57× speedup at a 50% pruning rate.

Jul 16, 2026

Upload by Genevieve