Large Separable Kernel Attention - Rethinking the Large Kernel Attention Design in CNN

Aug 25, 2026

Uploaded by Jiven

Visual Attention Networks (VAN) equipped with Large Kernel Attention (LKA) modules deliver strong results across vision tasks, but their depth-wise convolution inside LKA grows quadratically in computation and memory as kernel size increases. To make extremely large kernels practical, the work proposes Large Separable Kernel Attention (LSKA), which factorizes the 2D depth-wise kernel into cascaded horizontal and vertical 1D kernels. Experiments show LSKA reduces complexity and memory at larger kernels, matches LKA performance in VAN, and improves robustness across ImageNet-C corruptions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Motivation: large-kernel attention costs in VAN

Proposed idea: LSKA decomposition of depth-wise kernels

Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN Kin Wai Lau 1 ,2 , Lai-Man Po 1 , Yasar Abbas Ur Rehman2 City University of Hong Kong 1 TCL AI Lab2 arXiv :2309 .01439v3 [ cs .CV] 20 Oct 2023 Abstract—Visual Attention Networks (VAN) with Large Kernel Attention (LKA) modules have been shown to provide remarkable performance, that surpasses Vision Transformers (ViTs), on a range of vision-based tasks. However, the depth-wise convolutional layer in these LKA modules incurs a quadratic increase in the computational and memory footprints with increasing convolutional kernel size. To mitigate these problems and to enable the use of extremely large convolutional kernels in the attention modules of VAN, we propose a family of Large Separable Kernel Attention modules, termed LSKA. LSKA decomposes the 2D convolutional kernel of the depth-wise convolutional layer into cascaded horizontal and vertical 1-D kernels. In contrast to the standard LKA design, the proposed decomposition enables the direct use of the depth-wise convolutional layer with large kernels in the attention module, without requiring any extra blocks. We demonstrate that the proposed LSKA module in VAN can achieve comparable performance with the standard LKA module and incur lower computational complexity and memory footprints. We also find that the proposed LSKA design biases the VAN more toward the shape of the object than the texture with increasing kernel size. Additionally, we benchmark the robustness of the LKA and LSKA in VAN, ViTs, and the recent ConvNeXt on the five corrupted versions of the ImageNet dataset that are largely unexplored in the previous works. Our extensive experimental results show that the proposed LSKA module in VAN provides a significant reduction in computational complexity and memory footprints with increasing kernel size while outperforming ViTs, ConvNeXt, and providing similar performance compared to the LKA module in VAN on object recognition, object detection, semantic segmentation, and robustness tests. Codes are available at [https://github.com/](https://github.com/)[ ](https://github.com/)StevenLauHKHK/Large-Separable-Kernel-Attention. Index Terms—CNN, Attention block, Large kernel I. INTRODUCTION Over the past decade, Convolutional Neural Network (CNN) architectures and optimization techniques have expeditiously evolved. This evolution comes from the designing of activation functions [1]–[3], proposing regularization methods of CNN parameters [4]–[6], constructing new optimization methods [7]–[9], cost functions [10]–[12], and new network architectures [13]–[17] . Most of these breakthroughs in CNN are clustered around the human cognitive processes, specifically the human visual system. K.W. Lau is with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, and also with TCL AI Lab. Y.A.U. Rehman is with TCL AI Lab. (e-mail: [kinwailau6-c@my.cityu.edu.hk](kinwailau6-c@my.cityu.edu.hk), [yasar.abbas@my.cityu.edu.hk](yasar.abbas@my.cityu.edu.hk)) L.-M. Po is with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong (email: [eelmpo@cityu.edu.hk](eelmpo@cityu.edu.hk)) Fig. 1. Speed-accuracy trade-off under different large kernel decomposition methods and kernel sizes. Different markers represent different kernel sizesand VAN-Tiny is used in our comparison. We can see that the naive design of LKA (LKA-trivial) and the actual design of LKA in VAN result in higher GFLOPs with increasing kernel size. In comparison, our proposed LSKA-trivial and LSKA in VAN significantly reduce the GFLOPs without downgrading the performance. Owing to translational equivariance and locality properties, CNN is a common choice of a feature encoder for various vision-based tasks, including image classification [13]–[15], semantic segmentation [18]–[21], and object detection [22]–[25] . Further improvement in the performance of CNN, for these vision

FAQs

Why does standard LKA in VAN become expensive with larger kernel sizes?

The depth-wise convolution in LKA causes computational and memory footprints to increase quadratically as the kernel size grows.

How does LSKA reduce the cost while using large kernels?

LSKA decomposes the 2D depth-wise convolution kernel into cascaded horizontal and vertical 1D kernels, enabling large kernels without adding extra blocks.

What tasks and evaluations are used to compare LSKA with LKA and other backbones?

The study benchmarks object recognition, object detection, semantic segmentation, and robustness on five ImageNet-C corrupted versions across VAN, ViTs, and ConvNeXt.

Aug 25, 2026

Uploaded by Jiven