CGaP: Continuous Growth and Pruning for Efficient Deep Learning

Understand

Today a canonical approach to reduce the computation cost of Deep Neural Networks (DNNs) is to pre-define an over-parameterized model before training to guarantee the learning capacity, and then prune unimportant learning units (filters and neurons) during training to improve model compactness.

• We argue it is unnecessary to introduce redundancy at the beginning of the training but then reduce redundancy for the ultimate inference model.
• In this paper, we propose a Continuous Growth and Pruning (CGaP) scheme to minimize the redundancy from the beginning.
• CGaP starts the training from a small network seed, then expands the model continuously by reinforcing important learning units, and finally prunes the network to obtain a compact and accurate model.