2016

Demographic Dialectal Variation in Social Media: A Case Study of African-American English

Blodgett, Su Lin, Green, Lisa, O'Connor, Brendan

Understand

Though dialectal language is increasingly abundant on social media, few resources exist for developing NLP tools to handle such language.

We conduct a case study of dialectal language in online conversational text by investigating African-American English (AAE) on Twitter.
We propose a distantly supervised model to identify AAE-like language from demographics associated with geo-located messages, and we verify that this language follows well-known AAE linguistic phenomena.
In addition, we analyze the quality of existing language identification and dependency parsing tools on AAE-like text, demonstrating that they perform poorly on such text compared to text associated with white speakers.

Built on

African American Vernacular English: Features, Evolution, Educational Implications
John Russell Rickford · 1999
Earlier work this paper cites.
Mining the web to create minority language corpora
Rayid Ghani, Rosie Jones, and Dunja Mladenić · 2001
Earlier work this paper cites.
African American English: A Linguistic Introduction
Lisa J. Green · 2002
Earlier work this paper cites.
Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination
Marianne Bertrand and Sendhil Mullainathan · 2003
Earlier work this paper cites.
Finding scientific topics
T.L. Griffiths and M. Steyvers · 2004
Earlier work this paper cites.
Reconsidering language identification for written language resources
Baden Hughes, Timothy Baldwin, Steven Bird, Jeremy Nicholson, and Andrew MacKinlay · 2006
Earlier work this paper cites.
A tagging algorithm for mixed language identification in a noisy domain
Mike Rosner and Paulseph-John Farrugia · 2007
Earlier work this paper cites.
Stanford typed dependencies manual
M. C. de Marneffe and C. D. Manning · 2008
Earlier work this paper cites.
Topic models conditioned on arbitrary features with Dirichlet-Multinomial regression
David Mimno and Andrew McCallum · 2008
Earlier work this paper cites.
Fightin’ Words: Lexical feature selection and evaluation for identifying the content of political conflict
B. L. Monroe, M. P. Colaresi, and K. M. Quinn · 2008
Earlier work this paper cites.
Learning to predict code-switching points
Thamar Solorio and Yang Liu · 2008
Earlier work this paper cites.
On smoothing and inference for topic models
Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh · 2009
Earlier work this paper cites.
#hardtoparse: POS tagging and parsing the Twitterverse
Jennifer Foster, Ozlem Cetinoglu, Joachim Wagner, Joseph Le Roux, Stephen Hogan, Joakim Nivre, Deirdre Hogan, and Josef van Genabith · 2011
Earlier work this paper cites.
Part-of-speech tagging for Twitter: Annotation, features, and experiments
Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeff Flanigan, and Noah A. Smith · 2011
Earlier work this paper cites.
Partially labeled topic models for interpretable text mining
Daniel Ramage, Christopher D. Manning, and Susan Dumais · 2011
Earlier work this paper cites.
Language identification for creating language-specific Twitter collections
Shane Bergsma, Paul McNamee, Mossaab Bagdouri, Clayton Fink, and Theresa Wilson · 2012
Earlier work this paper cites.

Similar

langid. py: An off-the-shelf language identification tool
M. Lui and T. Baldwin · 2012
Cited alongside, same era.
How noisy social media text, how diffrnt social media sources?
Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang · 2013
Cited alongside, same era.
Phonological factors in social media writing
Jacob Eisenstein · 2013
Cited alongside, same era.
Labeling the languages of words in mixed-language documents using weakly supervised methods
Ben King and Steven P Abney · 2013
Cited alongside, same era.
Is the sample good enough? Comparing data from twitter’s streaming api with Twitter’s Firehose
Fred Morstatter, Jürgen Pfeffer, Huan Liu, and Kathleen Carley · 2013
Cited alongside, same era.
Improved part-of-speech tagging for online conversational text with word clusters
Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith · 2013
Cited alongside, same era.
The structural topic model and applied social science
Margaret E Roberts, Brandon M Stewart, Dustin Tingley, and Edoardo M Airoldi · 2013
Cited alongside, same era.
Discrimination in online ad delivery
Latanya Sweeney · 2013
Cited alongside, same era.
Multinomial inverse regression for text analysis
Matt Taddy · 2013
Cited alongside, same era.
N-gram language models and pos distribution for the identification of Spanish varieties
Marcos Zampieri, Binyam Gebrekidan Gebre, and Sascha Diwersy · 2013
Cited alongside, same era.
A fast and accurate dependency parser using neural networks
Danqi Chen and Christopher Manning · 2014
Cited alongside, same era.
An Algerian Arabic-French code-switched corpus
Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch · 2014
Cited alongside, same era.

Then

Overview for the first shared task on language identification in code-switched data
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung · 2014
Later among the works it cites.
Now we stronger than ever: African-American syntax in Twitter
Ian Stewart · 2014
Later among the works it cites.
Overview of TweetLID: Tweet language identification at SEPLN 2014
Arkaitz Zubiaga, Inaki San Vincente, Pablo Gamallo, Jose Ramom Pichel, Inaki Algeria, Nora Aranberri, Aitzol Ezeiza, and Victor Fresno · 2014
Later among the works it cites.
Identifying regional dialects in online social media
Jacob Eisenstein · 2015
Later among the works it cites.
Understanding US regional linguistic variation with Twitter data analysis
Yuan Huang, Diansheng Guo, Alice Kasakoff, and Jack Grieve · 2015
Later among the works it cites.
Toward a description of African American Vernacular English dialect regions using “Black Twitter”
Taylor Jones · 2015
Later among the works it cites.
Challenges of studying and processing dialects in social media
Anna Katrine Jørgensen, Dirk Hovy, and Anders Søgaard · 2015
Later among the works it cites.
Minority language Twitter: Part-of-speech tagging and analysis of Irish tweets
Teresa Lynn, Kevin Scannell, and Eimear Maguire · 2015
Later among the works it cites.
Developing language-tagged corpora for code-switching tweets
Suraj Maharjan, Elizabeth Blair, Steven Bethard, and Thamar Solorio · 2015
Later among the works it cites.
Arabic dialect identification using a parallel multidialectal corpus
Shervin Malmasi, Eshrag Refaee, and Mark Dras · 2015
Later among the works it cites.
Confounds and consequences in geotagged Twitter data
Umashanthi Pavalanathan and Jacob Eisenstein · 2015
Later among the works it cites.
Globally normalized transition-based neural networks
Original
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins · 2016
Closest in time.
Learning a POS tagger for AAVE-like language
Anna Jørgensen, Dirk Hovy, and Anders Søgaard · 2016
Closest in time.
Modeling factuality judgments in social media text
Sandeep Soni, Tanushree Mitra, Eric Gilbert, and Jacob Eisenstein · 2068
Closest in time.
The social impact of natural language processing
Dirk Hovy and L. Shannon Spruit · 2096
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…