LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

Understand

Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings.

• However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface.
• Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions.
• We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user.