On the Prunability of Attention Heads in Multilingual BERT

Understand

Large multilingual models, such as mBERT, have shown promise in crosslingual transfer.

• In this work, we employ pruning to quantify the robustness and interpret layer-wise importance of mBERT.
• On four GLUE tasks, the relative drops in accuracy due to pruning have almost identical results on mBERT and BERT suggesting that the reduced attention capacity of the multilingual models does not affect robustness to pruning.
• For the crosslingual task XNLI, we report higher drops in accuracy with pruning indicating lower robustness in crosslingual transfer.