Understand
We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation.
- The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase.
- The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis.
- The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output.
Built on
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014 · 2014
Earlier work this paper cites.
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Deliberation networks: Sequence generation beyond one-pass decoding
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017 · 2017
Earlier work this paper cites.
Similar
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
KERMIT: Generative Insertion-Based Modeling for Sequences
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019 · 2019
Cited alongside, same era.
EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019 · 2019
Cited alongside, same era.
Insertion-based Decoding with Automatically Inferred Generation Order
Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019a
Cited in the paper.
Levenshtein Transformer
Jiatao Gu, Changhan Wang, and Jake Zhao. 2019b
Cited in the paper.
Then
Optimal Completion Distillation for Sequence Learning
Sara Sabour, William Chan, and Mohammad Norouzi. 2019 · 2019
Later among the works it cites.
Insertion Transformer: Flexible Sequence Generation via Insertion Operations
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019 · 2019
Later among the works it cites.
Non-Monotonic Sequential Text Generation
Sean Welleck, Kiante Brantley, Hal Daume, and Kyunghyun Cho. 2019 · 2019
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…