Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation

Understand

Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU.

• However, it is unclear how such pruning techniques affect a model's learned representations.
• By probing Transformers with more and more low-magnitude weights pruned away, we find that complex semantic information is first to be degraded.
• Analysis of internal activations reveals that higher layers diverge most over the course of pruning, gradually becoming less complex than their dense counterparts.