Attending to Future Tokens For Bidirectional Sequence Generation

Understand

Neural sequence generation is typically performed token-by-token and left-to-right.

• Whenever a token is generated only previously produced tokens are taken into consideration.
• In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better.
• We propose to make the sequence generation process bidirectional by employing special placeholder tokens.