A Hybrid Approach for Transliterated Word-Level Language Identification - CRF with Post Processing Heuristics

Aug 28, 2026

Uploaded by Oliver

A hybrid system addresses word-level language (WLL) identification for Bangla words written in Roman script and mixed with English in the FIRE 2014 transliterated search shared task. The approach combines a CRF-based machine learning model with post-processing heuristics to label each token as English or Bangla transliterations. Two transliteration systems convert detected Bangla Roman-script tokens into native Bangla script, achieving reported token-level accuracy and leading scores across most evaluation metrics for the Bangla-English pair.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Abstract

Introduction

Task Definition

Datasets and Resources

System Description

WLL Classification Features

Character n-grams

A Hybrid Approach for Transliterated Word-Level Language Identiﬁcation: CRF with Post Processing Heuristics Somnath Banerjee CSE Department,JU, India [s.banerjee1980@gmail.com](s.banerjee1980@gmail.com)[ ](s.banerjee1980@gmail.com)Sudip Kumar Naskar CSE Department,JU, India [sudip.naskar@cse.jdvu.ac.in](sudip.naskar@cse.jdvu.ac.in) Aniruddha Roy CSE Department,JU, India [aniruddha@gmail.com](aniruddha@gmail.com) Alapan Kuila CSE Department,JU, India [alapan.cse@gmail.com](alapan.cse@gmail.com) Sivaji Bandyopadhyay CSE Department,JU, India sivaj[i_cse@yahoo.com](i_cse@yahoo.com) Paolo Rosso NLE Lab, UPV,Spain [prosso@dsic.upv.es](prosso@dsic.upv.es) ABSTRACT In this paper, we describe a hybrid approach for word-level language (WLL) identi􀀌cation of Bangla words written in Roman script and mixed with English words as part of our participation in the shared task on transliterated search at Forum for Information Retrieval Evaluation (FIRE) in 2014 . A CRF based machine learning model and post-processing heuristics are employed for the WLL identi􀀌cation task. In addition to language identi􀀌cation, two transliteration systems were built to transliterate detected Bangla words written in Roman script into native Bangla script. The system demonstrated an overall token level language identi􀀌cation accuracy of 0.905 . The token level Bangla and English language identi􀀌cation F-scores are 0.899, 0.920 respectively. The two transliteration systems achieved accuracies of 0.062 and 0.037 . The system presented in this paper resulted in the best scores across almost all metrics among all the participating systems for the Bangla-English language pair. Categories and Subject Descriptors 1.2.7 [Arti􀀌cial Intelligence]: Natural Language Processing, Language parsing and understanding General Terms Experimentation, Languages Keywords Word level language identi􀀌cation, Transliteration 1. INTRODUCTION In spite of having indigenous scripts, often Indian languages (e.g., Bangla, Hindi, Tamil etc.) are written in Roman script for user generated contents (such as blogs and tweets) due to various socio-cultural and technological reasons. This process of phonetically representing the words of Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or commercial advantage and that copies bear this notice and the full citation on the ﬁrst page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior speciﬁc permission and/or a fee. WOODSTOCK '97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00 . a language in a nonnative script is called (forward) transliteration. Especially the use of Roman script in transliteration for those languages presents serious challenges to understanding, search and (backward) transliteration. These challenges include handling spelling variations, diphthongs, doubled letters, reoccurring constructions, etc. Language identi􀀌cation for documents is a well-studied natural language problem [3] . King and Abney[9] presented the di􀀋erent aspects of this problem and focussed on the problem of labeling the language of individual words within a multilingual document. They proposed language identi􀀌 -cation at the word level in mixed language documents instead of sentence level identi􀀌cation. The last decade has seen the development of transliteration systems for Asian languages. Some notable transliteration systems were built for Chinese [14], Japanese [7], Korean [8], Arabic [1], etc. Transliteration systems were also developed for Indian languages [6, 16] . 2. TASK DEFINITION A query q : < w1 w2 w3 :::wn > is written in Roman script. The words, w1 ; w2 ; w3 ; :::; wn , could be standard English words or transliterated from Indian languages (IL), e.g. , Bangla, Hindi, etc. The objective of the task is to i

FAQs

What problem does the paper address in FIRE 2014?

It addresses word-level language identification for Bangla words written in Roman script mixed with English, followed by transliteration into native Bangla script.

How is word-level language identification performed?

A CRF-based machine learning model is used together with post-processing heuristics to decide whether each token is English or a transliterated Indian-language word.

What outputs does the system provide besides language labels?

For tokens identified as transliterated Bangla words, the system produces transliterations into native Bangla script using two transliteration systems.

Aug 28, 2026

Uploaded by Oliver