EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones

Understand

The superior performance of modern deep networks usually comes with a costly training procedure.

• This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers).
• Our work is inspired by the inherent learning dynamics of deep networks: we experimentally show that at an earlier training stage, the model mainly learns to recognize some 'easier-to-learn' discriminative patterns within each example, e.g., the lower-frequency components of images and the original information before data augmentation.
• Driven by this phenomenon, we propose a curriculum where the model always leverages all the training data at each epoch, while the curriculum starts with only exposing the 'easier-to-learn' patterns of each example, and introduces gradually more difficult patterns.