2017

Language Modeling for Code-Switched Data: Challenges and Approaches

Sreeram, Ganji, Sinha, Rohit

Understand

Lately, the problem of code-switching has gained a lot of attention and has emerged as an active area of research.

In bilingual communities, the speakers commonly embed the words and phrases of a non-native language into the syntax of a native language in their day-to-day communications.
The code-switching is a global phenomenon among multilingual communities, still very limited acoustic and linguistic resources are available as yet.
For developing effective speech based applications, the ability of the existing language technologies to deal with the code-switched data can not be over emphasized.

“Hindi-English, Code Switching and Language choice in urban, uppermiddle-class Indian Families,”
Sunita Malhotra, · 1980
Earlier work this paper cites.
Discourse Strategies
John J Gumperz, · 1982
Earlier work this paper cites.
“Processing of sentences with intra-sentential code-switching,”
Aravind K Joshi, · 1982
Earlier work this paper cites.
Life with Two Languages: An Introduction to Bilingualism
François Grosjean, · 1982
Earlier work this paper cites.
“Comparing codeswitching and borrowing,”
Carol Myers Scotton, · 1992
Earlier work this paper cites.
“SRILM: an extensible language modeling toolkit,”
Andreas Stolcke, · 2002
Earlier work this paper cites.
“Factored language models and generalized parallel backoff,”
Jeff A Bilmes and Katrin Kirchhoff, · 2003
Earlier work this paper cites.
“Speech recognition on code-switching among the Chinese dialects,”
Dau Cheng Lyu, Ren Yuan Lyu, Yuang Chin Chiang, and Chun Nan Hsu, · 2006
Earlier work this paper cites.

“Factored Language Models Tutorial,”
Katrin Kirchhoff, Jeff Bilmes, and Kevin Duh, · 2007
Cited alongside, same era.
“Language identification on code-switching utterances using multiple cues,”
Dau Cheng Lyu and Ren Yuan Lyu, · 2008
Cited alongside, same era.
“Semantics-based language modeling for Cantonese-English code-mixing speech recognition,”
Houwei Cao, PC Ching, Tan Lee, and Yu Ting Yeung, · 2010
Cited alongside, same era.
“An integrated framework for transcribing Mandarin-English code-mixed lectures with improved acoustic and language modeling,”
Ching Feng Yeh, Chao Yu Huang, Liang Che Sun, Che Liang, and Lin Shan Lee, · 2010
Cited alongside, same era.
“Speech recognition on English-Mandarin code-switching data using factored language models,”
Jan Gebhardt, · 2011
Cited alongside, same era.
“Cross language POS taggers (and other tools) for Indian languages: An experiment with Kannada using Telugu resources,”
Siva Reddy and Serge Sharoff, · 2011
Cited alongside, same era.
“RNNLM: Recurrent neural network language modeling toolkit,”
Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky, · 2011
Cited alongside, same era.
“Mixed language speech recognition without explicit identification of language,”
Kiran Bhuvanagirir and Sunil Kumar Kopparapu, · 2012
Cited alongside, same era.

“Automatic speech recognition of code switching speech using 1-best rescoring,”
Basem HA Ahmed and Tien-Ping Tan, · 2012
Later among the works it cites.
“Performance analysis of neural networks in combination with n-gram language models,”
Ilya Oparin, Martin Sundermeyer, Hermann Ney, and Jean Luc Gauvain, · 2012
Later among the works it cites.
“Factored language model based on recurrent neural network,”
Youzheng Wu, Xugang Lu, Hitoshi Yamamoto, Shigeki Matsuda, Chiori Hori, and Hideki Kashioka, · 2012
Later among the works it cites.
“Efficient estimation of word representations in vector space,”
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, · 2013
Later among the works it cites.
“Recurrent neural network language modeling for code switching conversational speech,”
Heike Adel, Ngoc Thang Vu, Franziska Kraus, Tim Schlippe, Haizhou Li, and Tanja Schultz, · 2013
Later among the works it cites.
“A Hindi-English Code-Switching Corpus.,”
Anik Dey and Pascale Fung, · 2014
Later among the works it cites.
“CUED-RNNLM–An open-source toolkit for efficient training and evaluation of recurrent neural network language models,”
Xie Chen, Xunying Liu, Yanmin Qian, MJF Gales, and Philip C Woodland, · 2016
Later among the works it cites.
Accessed: 2017-09-30
“Hindi POS tagger,” [Online] http://sivareddy.in/downloads , · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…