Changing the Representation - Examining Language Representation for Neural Sign Language Production

Aug 23, 2026

Upload by Levi

Neural Sign Language Production (SLP) automatically converts spoken-language sentences into sign language videos. This work improves the first Text-to-Gloss step by applying NLP methods that enhance sentence-level embeddings using language models such as BERT and Word2Vec, along with multiple tokenization strategies for the low-resource Text to Gloss setting. The paper introduces Text to HamNoSys (T2H) translation and demonstrates benefits of phonetic HamNoSys over gloss-level representations. It further uses hand-shape extraction from HamNoSys as additional training supervision, achieving strong BLEU-4 results on MineDGS and PHOENIX14T.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Introduction

Related Work

Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology (SLTAT 7) , pages 117–124 Language Resources and Evaluation Conference (LREC 2022), Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC 4.0 Changing the Representation: Examining Language Representation for Neural Sign Language Production Harry Walsh, Ben Saunders, Richard Bowden University of Surrey {harry.walsh, b.saunders, [r.bowden](r.bowden}@surrey.ac.uk)[}](r.bowden}@surrey.ac.uk)[@surrey.ac.uk](r.bowden}@surrey.ac.uk) Abstract Neural Sign Language Production (SLP) aims to automatically translate from spoken language sentences to sign language videos. Historically the SLP task has been broken into two steps; Firstly, translating from a spoken language sentence to a gloss sequence and secondly, producing a sign language video given a sequence of glosses. In this paper we apply Natural Language Processing techniques to the first step of the SLP pipeline. We use language models such as BERT and Word2Vec to create better sentence level embeddings, and apply several tokenization techniques, demonstrating how these improve performance on the low resource translation task of Text to Gloss. We introduce Text to HamNoSys (T2H) translation, and show the advantages of using a phonetic representation for sign language translation rather than a sign level gloss representation. Furthermore, we use HamNoSys to extract the hand shape of a sign and use this as additional supervision during training, further increasing the performance on T2H. Assembling best practise, we achieve a BLEU-4 score of 26.99 on the MineDGS dataset and 25.09 on PHOENIX14T, two new state-of-the-art baselines. Keywords: Sign Language Translation (SLT), Natural Language Processing (NLP), Sign Language, Phonetic Representation 1. Introduction Sign languages are the dominant form of communication for Deaf communities, with 430 million users worldwide (WHO, 2021) . Sign languages are complex multichannel languages with their own grammatical structure and vocabulary (Stokoe, 1980) . For many people, sign language is their primary language, and written forms of spoken language are their secondary languages. Sign Language Production (SLP) aims to bridge the gap between hearing and Deaf communities, by translating from spoken language sentences to sign language sequences. This problem has historically been broken into two steps; 1) translation from spoken language to gloss 1 and 2) subsequent production of sign language sequences from a sequence of glosses, commonly using a graphical avatar (Elliott et al., 2008; Efthimiou et al., 2010; Efthimiou et al., 2009) or more recently, a photorealistic signer (Saunders et al., 2021a; Saunders et al., 2021b) . In this paper, we improve the SLP pipeline by focusing on the Text to Gloss (T2G) translation task of step 1 . Modern deep learning is heavily dependent upon data. However, the creation of sign language datasets is both time consuming and costly, restricting their size to orders of magnitude smaller than their spoken language counterparts. State-of-the-art datasets such as RWTH-PHOENIX-Weather-2014T (PHOENIX14T), and the newer MineDGS (mDGS), contain only 8,257 and 63,912 examples respectively (Koller et al., 2015; Hanke et al., 2020), compared to over 15 million exam- 1 Gloss is the written word associated with a sign ples for common spoken language datasets (Vrandei and Krtzsch, 2014) . Hence, sign languages can be considered as low resource languages. In this work, we take inspiration from NLP techniques to boost translation performance. We explore how language can be modeled using different tokenizers, more specifically Byte Pair Encoding (BPE), WordPiece, word and character level tokenizers. We show that finding the correct tokenizer for the task helps simplify the translation problem. Furthermore, to help tackle our low resource language task

FAQs

What does Neural Sign Language Production (SLP) aim to do?

SLP converts spoken-language sentences into sign language videos automatically. The task has traditionally been split into Text-to-Gloss translation and then video production from gloss sequences.

How does this paper improve the Text-to-Gloss step in SLP?

It applies NLP techniques in the first pipeline step, including sentence-level embeddings from models such as BERT and Word2Vec and multiple tokenization strategies to better handle the low-resource Text to Gloss setting.

Why introduce Text to HamNoSys (T2H), and what extra supervision is used?

The paper argues for using the phonetic HamNoSys representation instead of gloss-level representations. It also extracts hand shape from HamNoSys and uses it as additional supervision during training.

Aug 23, 2026

Upload by Levi