VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions

Understand

Predicting future trajectories for other road agents is an essential task for autonomous vehicles.

• Established trajectory prediction methods primarily use agent tracks generated by a detection and tracking system and HD map as inputs.
• In this work, we propose a novel method that also incorporates visual input from surround-view cameras, allowing the model to utilize visual cues such as human gazes and gestures, road conditions, vehicle turn signals, etc, which are typically hidden from the model in prior methods.
• Furthermore, we use textual descriptions generated by a Vision-Language Model (VLM) and refined by a Large Language Model (LLM) as supervision during training to guide the model on what to learn from the input data.