Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance - W-NUT 2018 Workshop Paper

Aug 28, 2026

Uploaded by Violet

Code-mixed data in NLP grows rapidly with social media, but it brings major challenges from grammatical inconsistencies and diverse spelling habits. This work introduces a seq2seq-based architecture for normalizing phonetic typing variants common in Roman-script Bengali-English code-mixed text. Beyond normalization, the intermediate back-transliterated output supports back-transliteration and word identification in relevant scenarios. Experiments report 90.27% accuracy on test data and the approach is designed to use ITRANS as the canonical form.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

AccessFree to download

Abstract

Introduction

Related Work

Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance Soumil Mandal, Karthick Nanmaran Department of Computer Science & Engineering SRM Institute of Science & Technology, Chennai, India fsoumil.mandal, [karthicknanmaran](karthicknanmarang@gmail.com)[g](karthicknanmarang@gmail.com)[@gmail.com](karthicknanmarang@gmail.com) Abstract Building tools for code-mixed data is rapidly gaining popularity in the NLP research community as such data is exponentially rising on social media. Working with code-mixed data contains several challenges, especially due to grammatical inconsistencies and spelling variations in addition to all the previous known challenges for social media scenarios. In this article, we present a novel architecture focusing on normalizing phonetic typing variations, which is commonly seen in code-mixed data. One of the main features of our architecture is that in addition to normalizing, it can also be utilized for back-transliteration and word identiﬁcation in some cases. Our model achieved an accuracy of 90.27% on the test data. 1 Introduction With rising popularity of social media, the amount of data is rising exponentially. If mined, this data can proof to be useful for various purposes. In countries where the number of bilinguals are high, we see that users tend to switch back and forth between multiple languages, a phenomenon known as code-mixing or code-switching. An interesting case is switching between languages which share different native scripts. On such occasions, one of the two languages is typed in it's phonetically transliterated form in order to use a common script. Though there are some standard transliteration rules, for example ITRANS 1 , ISO 2 , but it is extremely difﬁcult and un-realistic for people to follow them while typing. This indeed is the case as we see that identical words are being transliterated differently by different people based on their own phonetic judgment inﬂuenced by dialects, location, or sometimes even based on the informality or casualness of the situation. Thus, for cre- 1[https://en.wikipedia.org/wiki/ITRANS](https://en.wikipedia.org/wiki/ITRANS) 2[https://en.wikipedia.org/wiki/ISO](https://en.wikipedia.org/wiki/ISO) 15919 ating systems for code-mixed data, post language tagging, normalization of transliterated text is extremely important in order to identify the word and understand it's semantics. This would help a lot in systems like opinion mining, and is actually necessary for tasks like summarization, translation, etc. A normalizing module will also be of immense help while making word embeddings for code-mixed data. In this paper, we present an architecture for automatic normalization of phonetically transliterated words to their standard forms. The language pair we have worked on is Bengali-English (Bn-En), where both are typed in Roman script, thus the Bengali words are in their transliterated form. The canonical or normalized form we have considered is the Indian Languages Transliteration (ITRANS) form of the respective Bengali word. Bengali is an Indo-Aryan language of India where 8 . 10% 3 of the total population are 1st language speakers and is also the ofﬁcial language of Bangladesh. The native script of Bengali is the Eastern Nagari Script 4. Our architecture utilizes fully char based sequence to sequence learning in addition to Levenshtein distance to give the ﬁnal normalized form or as close to it as possible. Some additional advantages of our system is that at an intermediate stage, the back-transliterated form of the word can be fetched (i.e. word identiﬁcation), which will be very useful in several cases as original tools (i.e. tools using native script) can be utilized, for example emotion lexicons. Some other important contributions of our research are the new lexicons that have been prepared (discussed in Sec 3) which can be used for building various other tools for studying Beng

FAQs

What problem does the paper address in code-mixed social media text?

It addresses spelling variations and phonetic transliteration differences that make it difficult to identify words and recover correct semantics in code-mixed data.

What is the proposed normalization approach?

The approach uses fully character-based seq2seq learning combined with Levenshtein distance to map phonetic transliterated forms to a canonical representation.

What additional capability does the model provide besides normalization?

At an intermediate stage, it can generate a back-transliterated form to support back-transliteration and word identification in some cases.

Aug 28, 2026

Uploaded by Violet