Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval

Jul 16, 2026

Sign Language Retrieval (SLRet) supports efficient access to sign language videos or text, yet struggles in fine-grained settings where visually similar signs must be distinguished. The work attributes this weakness not to insufficient model capacity, but to ineffective hard negative supervision. It models failures as a negative distribution mismatch: semantically different yet visually confusable signs are seldom used as hard negatives, while text-based mining cannot reflect visual ambiguity. A proposed Sign-Aware Hard Negative Mining (SAN) builds hard negatives using visual confusability in the sign embedding space, improving fine-grained retrieval on PHOENIX-2014T while maintaining coarse-grained accuracy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Background: Sign language retrieval tasks and data scarcity

Problem: Fine-grained failures and negative distribution mismatch

Proposed method: Sign-Aware Hard Negative Mining (SAN)

Evaluation setup and results

Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval Junmyeong Lee 1 Chan Hur4 ChangSu Choi2 Sukmin Cho 1 Fitsum Gaim 1 Eui Jun Hwang 1 Hoyun Song3 * KyungTae Lim2 ,3 * School of Computing 1 Graduate School of Culture Technology2 InnoCORE PRISM-AI Center3 Korea Advanced Institute of Science and Technology 1 , 2 , 3 ETRI Medical Informatics Laboratory4 [david516@kaist.ac.kr](david516@kaist.ac.kr) [chanhur@etri. re.kr](chanhur@etri. re.kr) {choics2623,nelllpic,fitsum.gaim,ehwa20,hysong, [ktlim}@kaist.ac.kr](ktlim}@kaist.ac.kr) arXiv :2607 .09263v1 [ cs .CV] 10 Jul 2026 Abstract Sign Language Retrieval (SLRet) enables efficient access to sign language content but remains fragile in fine-grained scenarios where visually similar signs must be distinguished. We show that this limitation does not stem from model capacity, but from ineffective hard negative supervision. Specifically, we formulate fine-grained retrieval failures as a negative distribution mismatch: semantically distinct yet visually confusable signs are rarely treated as hard negatives, while existing textbased mining strategies fail to capture such visual ambiguity. To address this issue, we propose Sign-Aware Hard Negative Mining (SAN), which constructs hard negatives based on visual confusability in the sign embedding space rather than linguistic similarity. Experiments on PHOENIX-2014T demonstrate that SAN substantially improves fine-grained retrieval performance while preserving coarse-grained accuracy, highlighting the importance of aligning negative supervision with visual ambiguity in sign language retrieval. Code is available at Github repository.1. 1 Introduction Sign languages are the primary means of communication for the Deaf community, expressed through hand, body, and facial movements. The unique grammar and visual complexity of sign languages often create communication barriers between signers and non-signers. To bridge this gap, prior research has focused on sign language understanding, particularly sign language recognition (Hu et al., 2021 ; Jiang et al., 2021 ; Zuo et al., 2023) and translation (Camgöz et al., 2020 ; Zhou et al., 2023 ; Gong et al., 2024) . However, the scarcity of sign language data leads to high error rates across both tasks (Cheng et al., 2023) . * Corresponding Author 1 [https://github.com/joonmy/SAN.git](https://github.com/joonmy/SAN.git) Figure 1: Illustration of fine-grained ambiguity in sign language retrieval. Semantically distinct words often correspond to visually similar signs, forming true hard negatives. SAN effectively targets these visually confusable instances that text-based mining methods frequently fail to address. Recently, Sign Language Retrieval (SLRet)(Duarte et al., 2022 ; Cheng et al., 2023 ; Wu et al., 2024) has emerged as a promising task. SLRet aims to retrieve relevant sign language videos or texts from a database given a query. It enables efficient access to sign language content and facilitates the use of unannotated sign videos, helping mitigate the data scarcity of sign language resources (Duarte et al., 2022) . However, we identify a core bottleneck that current SLRet models fail to address. Since sign languages construct meaning within a restricted visual space, even subtle differences in hand shape, position, or trajectory can alter meaning—a phenomenon known as sign confusability (Albanie et al., 2020 ; Zuo et al., 2023) . While existing models remain stable in coarse-grained retrieval, their performance degrades sharply in fine-grained retrieval scenarios where distinguishing these subtle distinctions is required. We argue that this degradation stems not from a lack of model capacity, but rather from training supervision that does not sufficiently expose the model to such cases. We formulate the cause of this failure as a negative distribution mismatch in contrastive learning, which is exacerbated in sign langu

FAQs

Why do SLRet models perform poorly in fine-grained scenarios?

Their training supervision fails to include enough true hard negatives. Semantically different but visually confusable signs are rarely treated as hard negatives, so models do not learn the subtle distinctions needed in fine-grained retrieval.

What is the key idea behind the negative distribution mismatch formulation?

Fine-grained retrieval failures are explained as a mismatch between the negative distribution used during contrastive learning and the visually confusable negatives actually required. Text-based mining emphasizes linguistic relation, not visual ambiguity.

How does SAN construct hard negatives differently from existing text-based strategies?

SAN uses visual confusability in the sign embedding space: it extracts high-confidence sign–word correspondences, finds visually close yet semantically distinct signs, and builds hard negative captions from the corresponding words. This aligns negative supervision with visual ambiguity rather than linguistic similarity.

Jul 16, 2026

Upload by Evangeline