CoLI-Machine Learning Approaches for Code-mixed Language Identification at the Word Level in Kannada-English Texts

by   H. L. Shashirekha, et al.

The task of automatically identifying a language used in a given text is called Language Identification (LI). India is a multilingual country and many Indians especially youths are comfortable with Hindi and English, in addition to their local languages. Hence, they often use more than one language to post their comments on social media. Texts containing more than one language are called "code-mixed texts" and are a good source of input for LI. Languages in these texts may be mixed at sentence level, word level or even at sub-word level. LI at word level is a sequence labeling problem where each and every word in a sentence is tagged with one of the languages in the predefined set of languages. In order to address word level LI in code-mixed Kannada-English (Kn-En) texts, this work presents i) the construction of code-mixed Kn-En dataset called CoLI-Kenglish dataset, ii) code-mixed Kn-En embedding and iii) learning models using Machine Learning (ML), Deep Learning (DL) and Transfer Learning (TL) approaches. Code-mixed Kn-En texts are extracted from Kannada YouTube video comments to construct CoLI-Kenglish dataset and code-mixed Kn-En embedding. The words in CoLI-Kenglish dataset are grouped into six major categories, namely, "Kannada", "English", "Mixed-language", "Name", "Location" and "Other". The learning models, namely, CoLI-vectors and CoLI-ngrams based on ML, CoLI-BiLSTM based on DL and CoLI-ULMFiT based on TL approaches are built and evaluated using CoLI-Kenglish dataset. The performances of the learning models illustrated, the superiority of CoLI-ngrams model, compared to other models with a macro average F1-score of 0.64. However, the results of all the learning models were quite competitive with each other.


page 1

page 2

page 3

page 4


Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts

Using code-mixed data in natural language processing (NLP) research curr...

WLV-RIT at HASOC-Dravidian-CodeMix-FIRE2020: Offensive Language Identification in Code-switched YouTube Comments

This paper describes the WLV-RIT entry to the Hate Speech and Offensive ...

Labeling of Query Words using Conditional Random Field

This paper describes our approach on Query Word Labeling as an attempt i...

Measuring a Texts Fairness Dimensions Using Machine Learning Based on Social Psychological Factors

Fairness is a principal social value that can be observed in civilisatio...

Atypical lexical abbreviations identification in Russian medical texts

Abbreviation is a method of word formation that aims to construct the sh...

Code Word Detection in Fraud Investigations using a Deep-Learning Approach

In modern litigation, fraud investigators often face an overwhelming num...

De-Anonymizing Text by Fingerprinting Language Generation

Components of machine learning systems are not (yet) perceived as securi...

Please sign up or login with your details

Forgot password? Click here to reset