2018

Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training

Gonen, Hila, Goldberg, Yoav

Understand

We focus on the problem of language modeling for code-switched language, in the context of automatic speech recognition (ASR).

Language modeling for code-switched language is challenging for (at least) three reasons: (1) lack of available large-scale code-switched data for training; (2) lack of a replicable evaluation setup that is ASR directed yet isolates language modeling performance from the other intricacies of the ASR system; and (3) the reliance on generative modeling.
We tackle these three issues: we propose an ASR-motivated evaluation setup which is decoupled from an ASR system and the choice of vocabulary, and provide an evaluation dataset for English-Spanish code-switching.
This setup lends itself to a discriminative training approach, which we demonstrate to work better than generative language modeling.

Built on

Sometimes i’ll start a sentence in spanish y termino en español: toward a typology of code-switching
Shana Poplack. 1980 · 1980
Earlier work this paper cites.
Class-based n-gram models of natural language
Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992 · 1992
Earlier work this paper cites.
An empirical study of smoothing techniques for language modeling
Stanley F Chen and Joshua Goodman. 1996 · 1996
Earlier work this paper cites.
From codeswitching via language mixing to fused lects: Toward a dynamic typology of bilingual speech
Peter Auer. 1999 · 1999
Earlier work this paper cites.
Bilingual speech: A typology of code-mixing , volume 11
Pieter Muysken. 2000 · 2000
Earlier work this paper cites.
Discriminative training of language models for speech recognition
Hong-Kwang Jeff Kuo, Eric Fosler-Lussier, Hui Jiang, and Chin-Hui Lee. 2002 · 2002
Earlier work this paper cites.
A neural probabilistic language model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003 · 2003
Earlier work this paper cites.
Hierarchical probabilistic neural network language model
Frederic Morin and Yoshua Bengio. 2005 · 2005
Earlier work this paper cites.
Discriminative n-gram language modeling
Brian Roark, Murat Saraclar, and Michael Collins. 2007 · 2007
Earlier work this paper cites.
Part-of-speech tagging for english-spanish code-switched text
Thamar Solorio and Yang Liu. 2008 · 2008
Earlier work this paper cites.
The Cambridge handbook of linguistic code-switching
Barbara E. Bullock and Almeida Jacqueline Toribio. 2009 · 2009
Earlier work this paper cites.
Automatic recognition of cantonese-english code-mixing speech
Joyce YC Chan, Houwei Cao, PC Ching, and Tan Lee. 2009 · 2009
Earlier work this paper cites.
News from OPUS - A collection of multilingual parallel corpora with tools and interfaces
Jörg Tiedemann. 2009 · 2009
Earlier work this paper cites.
Recurrent neural network based language model
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010 · 2010
Earlier work this paper cites.

Similar

Code-switch language model with inversion constraints for mixed language speech recognition
Ying Li and Pascale Fung. 2012 · 2012
Cited alongside, same era.
A first speech recognition system for mandarin-english code-switch conversational speech
Ngoc Thang Vu, Dau-Cheng Lyu, Jochen Weiner, Dominic Telaar, Tim Schlippe, Fabian Blaicher, Eng-Siong Chng, Tanja Schultz, and Haizhou Li. 2012 · 2012
Cited alongside, same era.
Classification and ranking approaches to discriminative language modeling for asr
Erinç Dikici, Murat Semerci, Murat Saraclar, and Ethem Alpaydin. 2013 · 2013
Cited alongside, same era.
Features for factored language models for code-switching speech
Heike Adel, Katrin Kirchhoff, Dominic Telaar, Ngoc Thang Vu, Tim Schlippe, and Tanja Schultz. 2014 · 2014
Cited alongside, same era.
An algerian arabic-french code-switched corpus
Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. 2014 · 2014
Cited alongside, same era.
Language modeling with functional head constraint for code switching speech recognition
Ying Li and Pascale Fung. 2014 · 2014
Cited alongside, same era.
Overview for the first shared task on language identification in code-switched data
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014 · 2014
Cited alongside, same era.
POS tagging of English-Hindi code-mixed social media content
Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014 · 2014
Cited alongside, same era.
Recurrent neural network regularization
Original
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014 · 2014
Cited alongside, same era.
Syntactic and semantic features for code-switching factored language models
Heike Adel, Ngoc Thang Vu, Katrin Kirchhoff, Dominic Telaar, and Tanja Schultz. 2015 · 2015
Cited alongside, same era.
Part-of-speech tagging of code-mixed social media content: Pipeline, stacking and joint modelling
Utsab Barman, Joachim Wagner, and Jennifer Foster. 2016 · 2016
Cited alongside, same era.
Challenges of computational processing of code-switching
Özlem Çetinoğlu, Sarah Schulz, and Ngoc Thang Vu. 2016 · 2016
Cited alongside, same era.

Then

Overview for the second shared task on language identification in code-switched data
Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016 · 2016
Later among the works it cites.
How grammatical is character-level neural machine translation? assessing MT quality with contrastive translation pairs
Original
Rico Sennrich. 2016 · 2016
Later among the works it cites.
Curriculum design for code-switching: Experiments with language identification and language modeling with deep neural networks
Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. 2017 · 2017
Later among the works it cites.
Intelligible language modeling with input switched affine networks
Original
Jakob N Foerster, Justin Gilmer, Jan Chorowski, Jascha Sohl-Dickstein, and David Sussillo. 2017 · 2017
Later among the works it cites.
Dual language models for code switched speech recognition
Original
Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. 2017 · 2017
Later among the works it cites.
On the state of the art of evaluation in neural language models
Original
Gábor Melis, Chris Dyer, and Phil Blunsom. 2017 · 2017
Later among the works it cites.
Regularizing and optimizing lstm language models
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017 · 2017
Later among the works it cites.
Dynet: The dynamic neural network toolkit
Original
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017 · 2017
Later among the works it cites.
Language modeling for code-switched data: Challenges and approaches
Original
Ganji Sreeram and Rohit Sinha. 2017 · 2017
Later among the works it cites.
Code-switched language models using dual rnns and same-source pretraining
Original
Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. 2018 · 2018
Closest in time.
Large margin neural language model
Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018 · 2018
Closest in time.
Language modeling for code-mixing: The role of linguistic theory based synthetic data
Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018 · 2018
Closest in time.
The importance of being recurrent for modeling hierarchical structure
Original
Ke Tran, Arianna Bisazza, and Christof Monz. 2018 · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…