Learning Low-Rank Approximation for CNNs

Understand

Low-rank approximation is an effective model compression technique to not only reduce parameter storage requirements, but to also reduce computations.

• For convolutional neural networks (CNNs), however, well-known low-rank approximation methods, such as Tucker or CP decomposition, result in degraded model accuracy because decomposed layers hinder training convergence.
• In this paper, we propose a new training technique that finds a flat minimum in the view of low-rank approximation without a decomposed structure during training.
• By preserving the original model structure, 2-dimensional low-rank approximation demanding lowering (such as im2col) is available in our proposed scheme.