Structured Transforms for Small-Footprint Deep Learning

Understand

We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices.

• We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank.
• Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured.
• Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques.