Two-stream convolutional networks for end-to-end learning of self-driving cars

Understand

We propose a methodology to extend the concept of Two-Stream Convolutional Networks to perform end-to-end learning for self-driving cars with temporal cues.

• The system has the ability to learn spatiotemporal features by simultaneously mapping raw images and pre-calculated optical flows directly to steering commands.
• Although optical flows encode temporal-rich information, we found that 2D-CNNs are prone to capturing features only as spatial representations.
• We show how the use of Multitask Learning favors the learning of temporal features via inductive transfer from a shared spatiotemporal representation.