Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection

Understand

Language models increasingly rely on massive web dumps for diverse text data.

• However, these sources are rife with undesirable content.
• As such, resources like Wikipedia, books, and newswire often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering.
• Using a new dataset of U.S.