2020

Blank Language Models

Shen, Tianxiao, Quach, Victor, Barzilay, Regina et al.

Understand

We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks.

The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks.
The model can start from a single blank or partially completed text with blanks at specified locations.
It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill.

Built on

Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901
Earlier work this paper cites.
Text infilling
Original
Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019 · 1901
Earlier work this paper cites.
Insertion transformer: Flexible sequence generation via insertion operations
Original
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019 · 1902
Earlier work this paper cites.
Bert has a mouth, and it must speak: Bert as a markov random field language model
Original
Alex Wang and Kyunghyun Cho. 2019 · 1902
Earlier work this paper cites.
Constant-time machine translation with conditional masked language models
Original
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019 · 1904
Earlier work this paper cites.
The curious case of neural text degeneration
Original
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019 · 1904
Earlier work this paper cites.
Kermit: Generative insertion-based modeling for sequences
Original
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019 · 1906
Earlier work this paper cites.
A hierarchical reinforced sequence operation method for unsupervised text style transfer
Original
Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019a · 1906
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b · 1907
Earlier work this paper cites.
Xl-editor: Post-editing sentences with xlnet
Original
Yong-Siang Shih, Wei-Cheng Chang, and Yiming Yang. 2019 · 1910
Earlier work this paper cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019 · 1910
Earlier work this paper cites.
Enabling language models to fill in the blanks
Original
Chris Donahue, Mina Lee, and Percy Liang. 2020 · 2005
Earlier work this paper cites.

Similar

Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010 · 2010
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
A neural attention model for abstractive sentence summarization
Original
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015 · 2015
Cited alongside, same era.
Recurrent neural network grammars
Original
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016 · 2016
Cited alongside, same era.
Improving neural language models with a continuous cache
Original
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016 · 2016
Cited alongside, same era.
Pointer sentinel mixture models
Original
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016
Cited alongside, same era.
Non-autoregressive neural machine translation
Original
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2017 · 2017
Cited alongside, same era.
Regularizing and optimizing lstm language models
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017 · 2017
Cited alongside, same era.
Style transfer from non-parallel text by cross-alignment
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017 · 2017
Cited alongside, same era.
Bidirectional beam search: Forward-backward inference in neural sequence models for fill-in-the-blank image captioning
Qing Sun, Stefan Lee, and Dhruv Batra. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Improved variational autoencoders for text modeling using dilated convolutions
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017 · 2017
Cited alongside, same era.

Then

Adaptive input representations for neural language modeling
Original
Alexei Baevski and Michael Auli. 2018 · 2018
Later among the works it cites.
An empirical evaluation of generic convolutional and recurrent networks for sequence modeling
Original
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018 · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Later among the works it cites.
Maskgan: better text generation via filling in the _
Original
William Fedus, Ian Goodfellow, and Andrew M Dai. 2018 · 2018
Later among the works it cites.
Delete, retrieve, generate: a simple approach to sentiment and style transfer
Juncen Li, Robin Jia, He He, and Percy Liang. 2018 · 2018
Later among the works it cites.
Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018 · 2018
Later among the works it cites.
Style transfer as unsupervised machine translation
Original
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018 · 2018
Later among the works it cites.
Restoring ancient text using deep learning: a case study on Greek epigraphy
Yannis Assael, Thea Sommerschield, and Jonathan Prag. 2019 · 2019
Later among the works it cites.
fairseq: A fast, extensible toolkit for sequence modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019 · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 · 2019
Later among the works it cites.
Spanbert: Improving pre-training by representing and predicting spans
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020 · 2020
Closest in time.
Decoding as dynamic programming for recurrent autoregressive models
Najam Zaidi, Trevor Cohn, and Gholamreza Haffari. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…