Vision-Based High Speed Driving with a Deep Dynamic Observer

Understand

In this paper we present a framework for combining deep learning-based road detection, particle filters, and Model Predictive Control (MPC) to drive aggressively using only a monocular camera, IMU, and wheel speed sensors.

• This framework uses deep convolutional neural networks combined with LSTMs to learn a local cost map representation of the track in front of the vehicle.
• A particle filter uses this dynamic observation model to localize in a schematic map, and MPC is used to drive aggressively using this particle filter based state estimate.
• We show extensive real world testing results, and demonstrate reliable operation of the vehicle at the friction limits on a complex dirt track.