What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

Understand

Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge.

• Previous work largely focused on what these models learn at the representation level.
• We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks.
• We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t.