Look-ups are not (yet) all you need for deep learning inference

Understand

Fast approximations to matrix multiplication have the potential to dramatically reduce the cost of neural network inference.

• Recent work on approximate matrix multiplication proposed to replace costly multiplications with table-lookups by fitting a fast hash function from training data.
• In this work, we propose improvements to this previous work, targeted to the deep learning inference setting, where one has access to both training data and fixed (already learned) model weight matrices.
• We further propose a fine-tuning procedure for accelerating entire neural networks while minimizing loss in accuracy.