Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences

Understand

We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents.

• Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state.
• We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence.
• In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons).