A Multi-level Alignment Training Scheme for Video-and-Language Grounding

Understand

To solve video-and-language grounding tasks, the key is for the network to understand the connection between the two modalities.

• For a pair of video and language description, their semantic relation is reflected by their encodings' similarity.
• A good multi-modality encoder should be able to well capture both inputs' semantics and encode them in the shared feature space where embedding distance gets properly translated into their semantic similarity.
• In this work, we focused on this semantic connection between video and language, and developed a multi-level alignment training scheme to directly shape the encoding process.