ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks

Understand

Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well-studied in the literature.

• On the other hand, low-rank training, as an alternative way to train low-rank CNNs from scratch, has been exploited little yet.
• Unlike low-rank compression, low-rank training does not need pre-trained full-rank models, and the entire training phase is always performed on the low-rank structure, bringing attractive benefits for practical applications.
• However, the existing low-rank training solutions still face several challenges, such as a considerable accuracy drop and/or still needing to update full-size models during the training.