2020

Multi-Head Attention: Collaborate Instead of Concatenate

Cordonnier, Jean-Baptiste, Loukas, Andreas, Jaggi, Martin

Understand

Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures.

Training very large transformer models allowed significant improvement in both fields, but once trained, these networks show symptoms of over-parameterization.
For instance, it is known that many attention heads can be pruned without impacting accuracy.
This work aims to enhance current understanding on how multiple heads interact.

Some mathematical notes on three-mode factor analysis
Tucker, L · 1966
Earlier work this paper cites.
Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis
Harshman, R. A · 1970
Earlier work this paper cites.
Tensor decompositions and applications
Kolda, T. G. and Bader, B. W · 2009
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate, 2014
Bahdanau, D., Cho, K., and Bengio, Y · 2014
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2015
Earlier work this paper cites.
Compression of deep convolutional neural networks for fast and low power mobile applications, 2016
Kim, Y.-D., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D · 2016
Earlier work this paper cites.
Automatic differentiation in pytorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
GNU Parallel 2018
Tange, O · 2018
Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S · 2018
Cited alongside, same era.
Attention augmented convolutional networks
Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q. V · 2019
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
Tensorly: Tensor learning in python
Kossaifi, J., Panagakis, Y., Anandkumar, A., and Pantic, M · 2019
Cited alongside, same era.
Are sixteen heads really better than one?
Michel, P., Levy, O., and Neubig, G · 2019
Cited alongside, same era.
fairseq: A fast, extensible toolkit for sequence modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M · 2019
Cited alongside, same era.
Stand-alone self-attention in vision models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J · 2019
Cited alongside, same era.
Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter
Original
Sanh, V., Debut, L., Chaumond, J., and Wolf, T · 2019
Cited alongside, same era.
Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I · 2019
Cited alongside, same era.

Huggingface’s transformers: State-of-the-art natural language processing
Original
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J · 2019
Later among the works it cites.
Low-rank bottleneck in multi-head attention models, 2020
Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S. J., and Kumar, S · 2020
Closest in time.
Experiment tracking with weights and biases, 2020
Biewald, L · 2020
Closest in time.
On the relationship between self-attention and convolutional layers
Cordonnier, J.-B., Loukas, A., and Jaggi, M · 2020
Closest in time.
Albert: A lite bert for self-supervised learning of language representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R · 2020
Closest in time.
Talking-heads attention, 2020
Shazeer, N., Lan, Z., Cheng, Y., Ding, N., and Hou, L · 2020
Closest in time.
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N · 2021
Closest in time.
Training data-efficient image transformers & distillation through attention, 2021
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H · 2021
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…