Interpretability in Activation Space Analysis of Transformers: A Focused Survey

What does BERT look at? an analysis of bert’s attention,

Original

K. Clark, U. Khandelwal, O. Levy, C. D. Manning, · 1906

Earlier work this paper cites.

Augmenting self-attention with persistent memory,

Original

S. Sukhbaatar, E. Grave, G. Lample, H. Jégou, A. Joulin, · 1907

Earlier work this paper cites.

One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques,

Original

V. Arya, R. K. E. Bellamy, P. Chen, A. Dhurandhar, M. Hind, S. C. Hoffman, S. Houde, Q. V. Liao, R. Luss, A. Mojsilovic, S. Mourad, P. Pedemonte, R. Raghavendra, J. T. Richards, P. Sattigeri, K. Shanmugam, M. Singh, K. R. Varshney, D. Wei, Y. Zhang, · 1909

Earlier work this paper cites.

On the linguistic representational power of neural machine translation models,

Original

Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, J. R. Glass, · 1911

Earlier work this paper cites.

Y. Hao, L. Dong, F. Wei, K. Xu, Self-attention attribution: Interpreting information interactions inside transformer, 2020. URL: https://arxiv.org/abs/2004.11207 . doi: 10.48550/ARXIV.2004.11207

Original

2004

Earlier work this paper cites.

Opportunities and challenges in explainable artificial intelligence (XAI): A survey,

Original

A. Das, P. Rad, · 2006

Earlier work this paper cites.

Transformer with depth-wise LSTM,

Original

H. Xu, Q. Liu, D. Xiong, J. van Genabith, · 2007

Earlier work this paper cites.

Analyzing individual neurons in pre-trained language models,

Original

N. Durrani, H. Sajjad, F. Dalvi, Y. Belinkov, · 2010

Earlier work this paper cites.

Transformer feed-forward layers are key-value memories,

Original

M. Geva, R. Schuster, J. Berant, O. Levy, · 2012

Earlier work this paper cites.

D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, 2014. URL: https://arxiv.org/abs/1409.0473 . doi: 10.48550/ARXIV.1409.0473

Original

2014

Earlier work this paper cites.

Examples are not enough, learn to criticize! criticism for interpretability,

B. Kim, R. Khanna, O. O. Koyejo, · 2016

Earlier work this paper cites.

Attention is all you need,

Original

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, · 2017

Earlier work this paper cites.

Explanation in artificial intelligence: Insights from the social sciences,

Original

T. Miller, · 2017

Earlier work this paper cites.

Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure (2017). URL: https://arxiv.org/abs/1711.10203 . doi: 10.48550/ARXIV.1711.10203

Original

D. Hupkes, S. Veldhoen, W. Zuidema, · 2017

Earlier work this paper cites.

Axiomatic attribution for deep networks,

M. Sundararajan, A. Taly, Q. Yan, · 2017

Earlier work this paper cites.

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav) (2017). URL: https://arxiv.org/abs/1711.11279 . doi: 10.48550/ARXIV.1711.11279

Original

B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, R. Sayres, · 2017

Earlier work this paper cites.

BERT: pre-training of deep bidirectional transformers for language understanding,

Original

J. Devlin, M. Chang, K. Lee, K. Toutanova, · 2018

Earlier work this paper cites.

Peeking inside the black-box: A survey on explainable artificial intelligence (xai),

A. Adadi, M. Berrada, · 2018

Cited alongside, same era.

A survey of methods for explaining black box models,

R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, · 2018

Cited alongside, same era.

Explaining explanations: An approach to evaluating interpretability of machine learning,

Original

L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. A. Specter, L. Kagal, · 2018

Cited alongside, same era.

Explainable prediction of medical codes from clinical text,

J. Mullenbach, S. Wiegreffe, J. Duke, J. Sun, J. Eisenstein, · 2018

Cited alongside, same era.

What you can cram into a single vector: Probing sentence embeddings for linguistic properties,

Original

A. Conneau, G. Kruszewski, G. Lample, L. Barrault, M. Baroni, · 2018

Cited alongside, same era.

BERT rediscovers the classical NLP pipeline,

I. Tenney, D. Das, E. Pavlick, · 2019

Later among the works it cites.

Improving transformer models by reordering their sublayers,

O. Press, N. A. Smith, O. Levy, · 2020

Later among the works it cites.

A survey of the state of explainable AI for natural language processing,

M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, P. Sen, · 2020

Later among the works it cites.

What happens to BERT embeddings during fine-tuning?,

A. Merchant, E. Rahimtoroghi, E. Pavlick, I. Tenney, · 2020

Later among the works it cites.

On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers,

M. Mosbach, A. Khokhlova, M. A. Hedderich, D. Klakow, · 2020

Later among the works it cites.

Unsupervised domain clusters in pretrained language models,

R. Aharoni, Y. Goldberg, · 2020