Don't Look Twice: Faster Video Transformers with Run-Length Tokenization

Understand

Transformers are slow to train on videos due to extremely large numbers of input tokens, even though many video tokens are repeated over time.

• Existing methods to remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples.
• We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression.
• RLT efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length.