Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers

Understand

Position Embeddings (PEs), an arguably indispensable component in Vision Transformers (ViTs), have been shown to improve the performance of ViTs on many vision tasks.

• However, PEs have a potentially high risk of privacy leakage since the spatial information of the input patches is exposed.
• This caveat naturally raises a series of interesting questions about the impact of PEs on the accuracy, privacy, prediction consistency, etc.
• To tackle these issues, we propose a Masked Jigsaw Puzzle (MJP) position embedding method.