VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformers

Understand

In autonomous driving, dynamic environment and corner cases pose significant challenges to the robustness of ego vehicle's decision-making.

• To address these challenges, commencing with the representation of state-action mapping in the end-to-end autonomous driving paradigm, we introduce a novel pipeline, VDT-Auto.
• Leveraging the advancement of the state understanding of Visual Language Model (VLM), incorporating with diffusion Transformer-based action generation, our VDT-Auto parses the environment geometrically and contextually for the conditioning of the diffusion process.
• Geometrically, we use a bird's-eye view (BEV) encoder to extract feature grids from the surrounding images.