Online Embedding Compression for Text Classification using Low Rank Matrix Factorization

Understand

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints.

• Existing compression methods are either lossy or introduce significant latency.
• We propose a compression method that leverages low rank matrix factorization during training,to compress the word embedding layer which represents the size bottleneck for most NLP models.
• Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size.