PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers

Understand

This paper explores a better prediction target for BERT pre-training of vision transformers.

• We observe that current prediction targets disagree with human perception judgment.This contradiction motivates us to learn a perceptual prediction target.
• We argue that perceptually similar images should stay close to each other in the prediction target space.
• We surprisingly find one simple yet effective idea: enforcing perceptual similarity during the dVAE training.