Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study

Jul 16, 2026

Upload by Aria

CorrNet offers a strong baseline for continuous sign language recognition (CSLR) by capturing inter-frame correlations within the visual encoding stage. This study tests two natural architectural extensions on a reproduced CorrNet system: swapping the BiLSTM temporal head for a Transformer encoder and adding motion cues after temporal pooling via a lightweight MotionGate module. Experiments show the Transformer head fails to beat the BiLSTM baseline with comparable runtime cost, while MotionGate converges to an identity-like, weak residual with lost motion selectivity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Related Work

Continuous Sign Language Recognition

CorrNet and VAC

arXiv :2607 .09890v1 [ cs .CV] 10 Jul 2026 Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study Lisi Wang 1 , Zhidong Xiao2 ,∗ , Jianjun Peng3 1University of New South Wales, Sydney, Australia 2National Centre for Computer Animation, Faculty of Media, Science and Technology, Bournemouth University, BH12 5BB, United Kingdom 3 School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China [lisiwang.tech@gmail.com](lisiwang.tech@gmail.com) [zxiao@bournemouth.ac.uk](zxiao@bournemouth.ac.uk), [pengjj@dlpu.edu.cn](pengjj@dlpu.edu.cn) ∗ Corresponding author Abstract CorrNet is a strong baseline for continuous sign language recognition (CSLR) because it models inter-frame correlations inside the visual encoding stage. In this paper, we study two natural extensions of a reproduced CorrNet system: replacing the BiLSTM temporal head with a Transformer encoder, and injecting motion cues after temporal pooling. We find that the Transformer head does not outperform the BiLSTM baseline, even with a training strategy adjusted for the Transformer, and the two heads have almost the same computational and runtime cost. For the second extension, we design a lightweight module called MotionGate. In our experiments, MotionGate consistently collapses to an identity-like mapping: the gate loses motion selectivity, and the injected residual becomes a weak, non-selective perturbation of the pooled features. These results suggest that explicit motion injection after CorrNet’s correlation-based encoding is largely redundant, and that natural-looking architectural extensions in CSLR should be tested carefully instead of being assumed to help. 1 Introduction Continuous sign language recognition (CSLR) aims to recognize a sequence of glosses from an input signing video. Unlike isolated sign recognition, CSLR has to model both visual appearance and temporal dependencies over long video sequences, and training usually only uses sentence-level gloss annotations. Because of this, most modern CSLR systems follow a CTC-based pipeline: a visual backbone extracts frame-wise features, a temporal module models the resulting sequence, and CTC supervision aligns the predicted gloss sequence without frame-level labels. A central challenge in CSLR is temporal modeling. Sign language contains rich motion patterns, such as hand trajectories, body movement, facial expressions, and transitions between signs. CorrNet handles this by modeling inter-frame correlations inside the visual encoder, which gives a strong representation for the downstream temporal module. This makes CorrNet a good basis for asking whether extra temporal or motion modules can still add useful information. We study two natural extensions on a reproduced CorrNet system. The first replaces the original BiLSTM temporal module with a lightweight Transformer encoder. Self-attention can model global dependencies directly, so it is often seen as a stronger temporal model than recurrent networks. However, in the CorrNet pipeline, the sequence after downsampling is already short, and the backbone has already encoded local motion correlations. It is therefore not clear whether a Transformer head can bring extra gain over the recurrent baseline. The second extension injects motion cues after temporal pooling. Early pooling may weaken fine-grained motion information, so post-pooling motion compensation looks like a reasonable fix. Based on this idea, we design a lightweight post-pooling motion gate, called MotionGate, which uses temporal feature differences to add residual motion information to the pooled sequence. We do not assume this module is helpful. Instead, we check whether the learned module actually contributes meaningful residual information in the reproduced CorrNet setting. Our experiments show that neither extension improves recognition accuracy in the reproduced CorrNet setting, w

FAQs

What two extensions does the paper evaluate on a reproduced CorrNet-based CSLR system?

The paper evaluates (1) replacing the BiLSTM temporal head with a Transformer encoder, and (2) injecting motion cues after temporal pooling using a lightweight MotionGate module.

Why does the paper consider post-pooling motion injection as a plausible improvement?

It argues that early pooling may weaken fine-grained motion information, so adding motion compensation after pooling could restore useful motion cues.

What do the experiments conclude about recognition accuracy and MotionGate behavior?

Neither extension improves recognition accuracy in the reproduced CorrNet setting, and MotionGate converges to an identity-like residual regime where the gate loses motion selectivity, becoming a weak, non-selective perturbation.

Jul 16, 2026

Upload by Aria