Efficient Video Action Detection with Token Dropout and Context Refinement

Understand

Streaming video clips with large-scale video tokens impede vision transformers (ViTs) for efficient recognition, especially in video action detection where sufficient spatiotemporal representations are required for precise actor identification.

• In this work, we propose an end-to-end framework for efficient video action detection (EVAD) based on vanilla ViTs.
• Our EVAD consists of two specialized designs for video action detection.
• First, we propose a spatiotemporal token dropout from a keyframe-centric perspective.