Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation

Understand

We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks.

• Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively.
• The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates.
• It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model.