Weak Supervision and Referring Attention for Temporal-Textual Association Learning

Understand

A system capturing the association between video frames and textual queries offer great potential for better video analysis.

• However, training such a system in a fully supervised way inevitably demands a meticulously curated video dataset with temporal-textual annotations.
• Therefore we provide a Weak-Supervised alternative with our proposed Referring Attention mechanism to learn temporal-textual association (dubbed WSRA).
• The weak supervision is simply a textual expression (e.g., short phrases or sentences) at video level, indicating this video contains relevant frames.