Sequence-to-Sequence Learning as Beam-Search Optimization

Understand

Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks.

• Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions.
• In this work, we introduce a model and beam-search training scheme, based on the work of Daume III and Marcu (2005), that extends seq2seq to learn global sequence scores.
• This structured approach avoids classical biases associated with local training and unifies the training loss with the test-time usage, while preserving the proven model architecture of seq2seq and its efficient training approach.