Language Identification of Hindi-English tweets using code-mixed BERT

Understand

Language identification of social media text has been an interesting problem of study in recent years.

• Social media messages are predominantly in code mixed in non-English speaking states.
• Prior knowledge by pre-training contextual embeddings have shown state of the art results for a range of downstream tasks.
• Recently, models such as BERT have shown that using a large amount of unlabeled data, the pretrained language models are even more beneficial for learning common language representations.