Fetching the paper…

Reading the bibliography…

2019

Partially Shuffling the Training Data to Improve Language Models

Press, Ofir

Original
PDF
Post on X

Understand

Although SGD requires shuffling the training data between epochs, currently none of the word-level language modeling systems do this.

Naively shuffling all sentences in the training data would not permit the model to learn inter-sentence dependencies.
Here we present a method that partially shuffles the training data between epochs.
This method makes each batch random, while keeping most sentence ordering intact.

Partially Shuffling the Training Data to Improve Language Models · Around

Built on

Building a large annotated corpus of english: The penn treebank
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993 · 1993
Earlier work this paper cites.
Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017 · 2017
Earlier work this paper cites.

Similar

Regularizing and optimizing LSTM language models
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018 · 2018
Cited alongside, same era.
Direct output connection for a high-rank language model
Sho Takase, Jun Suzuki, and Masaaki Nagata. 2018 · 2018
Cited alongside, same era.

Then

Breaking the softmax bottleneck: A high-rank RNN language model
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018 · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…