Fetching the paper…

Reading the bibliography…

2019

A Multiscale Visualization of Attention in the Transformer Model

Vig, Jesse

Original
PDF
Post on X

Understand

The Transformer is a sequence model that forgoes traditional recurrent architectures in favor of a fully attention-based approach.

Besides improving performance, an advantage of using attention is that it can also help to interpret a model by showing how the model assigns weight to different input elements.
However, the multi-layer, multi-head attention mechanism in the Transformer model can be difficult to decipher.
To make the model more accessible, we introduce an open-source tool that visualizes attention at multiple scales, each of which provides a unique perspective on the attention mechanism.

A Multiscale Visualization of Attention in the Transformer Model · Around

Built on

Attention is not explanation
Original
Sarthak Jain and Byron C. Wallace. 2019 · 1902
Earlier work this paper cites.
Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned
Original
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 1905
Earlier work this paper cites.
Envisioning Information
Edward Tufte. 1990 · 1990
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
A neural attention model for abstractive sentence summarization
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015 · 2015
Earlier work this paper cites.
Reasoning about entailment with neural attention
Original
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016 · 2016
Earlier work this paper cites.
Tensor2tensor transformer visualization
Llion Jones. 2017 · 2017
Earlier work this paper cites.

Similar

Interactive visualization and manipulation of attention-based neural machine translation
Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Original
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Visual interrogation of attention-based models for natural language inference and machine comprehension
Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018 · 2018
Cited alongside, same era.
Gender bias in neural natural language processing
Original

Then

Tensor2tensor for neural machine translation
Original
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018
Later among the works it cites.
Gender bias in coreference resolution: Evaluation and debiasing methods
Original
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018 · 2018
Later among the works it cites.
Identifying and controlling important neurons in neural machine translation
Original
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019 · 2019
Closest in time.
Analysis methods in neural language processing: A survey
Original
Yonatan Belinkov and James Glass. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018 · 2018

Cited alongside, same era.

An analysis of encoder representations in transformer-based machine translation

Alessandro Raganato and Jörg Tiedemann. 2018 · 2018

Cited alongside, same era.

Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models

H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush. 2018 · 2018

Cited alongside, same era.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Closest in time.

BertViz: A tool for visualizing multi-head self-attention in the BERT model

Jesse Vig. 2019 · 2019

Closest in time.

Analyzing the structure of attention in a transformer language model

Jesse Vig and Yonatan Belinkov. 2019 · 2019

Closest in time.