2016

A POS Tagger for Code Mixed Indian Social Media Text - ICON-2016 NLP Tools Contest Entry from Surukam

Ramesh, Sree Harsha, Kumar, Raveena R

Understand

Building Part-of-Speech (POS) taggers for code-mixed Indian languages is a particularly challenging problem in computational linguistics due to a dearth of accurately annotated training corpora.

ICON, as part of its NLP tools contest has organized this challenge as a shared task for the second consecutive year to improve the state-of-the-art.
This paper describes the POS tagger built at Surukam to predict the coarse-grained and fine-grained POS tags for three language pairs - Bengali-English, Telugu-English and Hindi-English, with the text spanning three popular social media platforms - Facebook, WhatsApp and Twitter.
We employed Conditional Random Fields as the sequence tagging algorithm and used a library called sklearn-crfsuite - a thin wrapper around CRFsuite for training our model.

Built on

Discourse strategies
John J. Gumperz · 1982
Earlier work this paper cites.
Part-of-speech tagging guidelines for the Penn Treebank Project (3rd revision)
Beatrice Santorini · 1990
Earlier work this paper cites.
Duelling languages: Grammatical structure in codeswitching
Carol Myers-Scotton · 1997
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
John Lafferty, Andrew McCallum, and Fernando Pereira · 2001
Earlier work this paper cites.
Building a treebank for French
Anne Abeillé, Lionel Clément, and François Toussenel · 2003
Earlier work this paper cites.
Feature-rich part-of-speech tagging with a cyclic dependency network
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer · 2003
Earlier work this paper cites.
Biomedical named entity recognition using conditional random fields and rich feature sets
Burr Settles · 2004
Earlier work this paper cites.
In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1051-1060. Association for Computational Linguistics, 2008
Thamar Solorio and Yang Liu · 2008
Earlier work this paper cites.

Similar

Coupling an Annotated Corpus and a Morphosyntactic Lexicon for State-of-the-Art POS Tagging with Less Human Effort
Pascal Denis, and Benoît Sagot · 2009
Cited alongside, same era.
A Universal Part-of-Speech Tagset
Slav Petrov, Dipanjan Das, and Ryan McDonald · 2011
Cited alongside, same era.
On Measuring the Complexity of Code-Mixing
Björn Gambäck, and Amitava Das · 2014
Cited alongside, same era.
Overview of FIRE 2014 Track on Transliterated Search
Monojit Choudhury, Gokul Chittaranjan, Parth Gupta, and Amitava Das · 2014
Cited alongside, same era.
Overview for the First Shared Task on Language Identification in Code-Switched Data
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang and Pascale Fung · 2014
Cited alongside, same era.
Code mixing: A challenge for language identification in the language of social media
Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster · 2014
Cited alongside, same era.
POS Tagging of English-Hindi Code-Mixed Social Media Content
Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury · 2014
Cited alongside, same era.
Part-of-Speech Tagging for CodeMixed English-Hindi Twitter and Facebook Chat Messages
Anupam Jamatia, Björn Gambäck, and Amitava Das · 2015
Cited alongside, same era.

Then

POS Tagging For Code-Mixed Indian Social Media Text : Systems from IIIT-H for ICON NLP Tools Contest
Arnav Sharma and Raveesh Motlani · 2015
Later among the works it cites.
AMRITA_CEN @ ICON-2015: Part-of-Speech Tagging on Indian Language Mixed Scripts in Social Media
K. P. Soman · 2015
Later among the works it cites.
TASK REPORT: TOOL CONTEST ON POS TAGGING FOR CODE-MIXED INDIAN SOCIAL MEDIA (FACEBOOK, TWITTER, AND WHATSAPP) TEXT @ ICON 2016
Anupam Jamatia and Amitava Das · 2016
Closest in time.
Transfer Learning for Low-Resource Neural Machine Translation
Original
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight · 2016
Closest in time.
Dynamic feature induction: The last gist to the state-of-the-art
Jinho D. Choi · 2016
Closest in time.
Experiments with POS Tagging Code-mixed Indian Social Media Text
Original
Prakash B. Pimpale, and Raj Nath Patel · 2016
Closest in time.
Automatically identify and label sections in scientific journals using conditional random fields
Sree Harsha Ramesh, Arnab Dhar, Raveena R. Kumar, V. Anjaly, K. S. Sarath, Jason Pearce, and Krishna R. Sundaresan · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…