TRP: Trained Rank Pruning for Efficient Deep Neural Networks

Understand

To enable DNNs on edge devices like mobile phones, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations.

• Several previous works attempted to directly approximate a pretrained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss.
• As a result, performance usually drops significantly and a sophisticated effort on fine-tuning is required to recover accuracy.
• Apparently, it is not optimal to separate low-rank approximation from training.