LID - Language identiﬁcation of short text in transliterated space

Aug 28, 2026

Uploaded by Hazel

Huge volumes of social media content on platforms such as Twitter and Facebook use short text written in transliterated Roman form. Accurate language identification is needed to retrieve and organize information from these queries. This work focuses on transliterated Indian languages and evaluates Naive Bayes, Logistic Regression, and Support Vector Machine. Results show SVM with 4-gram features performs best, enabling integration with an IR platform to detect the language of incoming queries.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Introduction

Problem background

Data

Twitter corpus construction

Cleaning and preprocessing

Method and Models

Classifiers compared

SVM with 4-gram features

System Integration

Terrier-3.5 IR platform language detection

Language identiﬁcation of short text in transliterated space. Harsha Kokel DA-IICT, Gandhinagar [200901199@daiict.ac.in](200901199@daiict.ac.in) Abstract—Huge amount of data available on social media services like twitter, Facebook etc. comprises of short text in transliterated space. For such transliterated text there is a need to identify the language in order to access information. Language Identiﬁcation is a well studied problem. Various classiﬁers are used for language identiﬁcation but it is not evident how well they function with short text in Romanized Indian languages. We generated an annotated collection of Tweets in Roman script for three Indian languages: Bengali, Gujarati and Hindi. Three classiﬁers: Naive Bayes, Logistic Regression and Support Vector Machine were compared for language identiﬁcation of transliterated text. It was found that SVM performs the best with 4-gram features. Hence integrated SVM with the Terrier- 3.5 IR platform to detect language of the query provided. I. INTRODUCTION India is a multilingual and multiscript country. There are 23 ofﬁcially listed Indian languages and 12 ofﬁcially listed Indian scripts. Most of the Indian languages have a script of their own. For e.g. Hindi and Marathi use Devanagari Script, Gujarati uses Gujarati Script, Bangla and Assamese use Bengali Script, Kannada uses Kannada Script, Tamil uses Tamil Script, Telugu uses Telugu Script etc. There are multiple standard keyboards available for using Indian scripts like inscript keyboard, phonetic keyboard, unicode keyboard etc. But layman abstain from using such keyboards and rather use Transliteration. It has been observed that in most of the social media and discussion forums of Indian context, Indian languages have been written in Roman alphabets. It is evident from twitter and facebook that good amount of Hindi text is written in transliterated Roman script. Majority portion of the Indian song lyrics found online are in Roman Script and to search these lyrics people use search engine like Google or Bing. It becomes inevident for such search engines to identify the language of query to provide Cross Script information like lyrics of the Hindi song in Devanagari Script as well as Roman Script when query is in Roman Script. Query logs of search engines are full of transliterated texts and if only I can identify the language of these Transliterated text. In this paper I focus on the language identiﬁcation of Indian languages transliterated in Roman script only. Language identiﬁcation in such a transliterated space is a problem as 1) Indian languages have large character set, which is mapped to character set of size 26 in Roman Script while transliteration. This mapping is nor one to one neither many to one. for e.g., “f” and “q” both are represented as sha. “t” and “V” both are represented as ta. “j” can be represented as ˜j or z. like “mmdAr” (Majumadara) can be written as “Majumder” or “mazumdar”. 2) This transliteration do not abide by any standard rule and hence it has many spell variations according to the corresponding sound of the alphabets in both languages. Vowels in Hindi can be mapped to single or multiple alphabets in Roman Script. for e.g., “l”(Bh˜ula) can be written as “bhul” or “bhool”.“{h” can be written as “hai” or “hain”. 3) Humans have tendency to remove redundancies. Like in SMS Language, online users also skip vowels and replace alphabets with numbers for e.g., “nhF”(Na˜h˜i)˜ is written as “nhi” instead of “nahii”, “pFC􀀃”(Piche) is written as “pe6y” instead of “piche”. 4) This transliterated text is multilingual i.e., it is comprised of words from an Indian language and a few from English or sometimes from a third language. for e.g. Bengali: “shon besi deri hole Producer er ofﬁce giye unedited ﬁlm ta dekhe chole asbo” Gujarati: “Jab we met, movie na joie hoye to aaje jao, Must Watch!!!” Hindi : “Hey, kahan hai?? kabse wait kar rahe hai, come soon.” He

FAQs

Why is language identification important for transliterated short text on social media?

Because search and information access require knowing the language of short Romanized queries, which are common in platforms like Twitter and Facebook.

What dataset was created for this study?

A corpus of manually annotated bilingual tweets in Bengali, Gujarati, and Hindi, cleaned by removing punctuation, URLs, and digits.

Which classifier and feature setting performed best?

Support Vector Machine with 4-gram features provided the best language identification results for transliterated text.

Aug 28, 2026

Uploaded by Hazel