DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Understand

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks.

• A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning.
• However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks.
• As a result, they struggle to scale to a large number of tasks without significant overhead.