2016

LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

Strobelt, Hendrik, Gehrmann, Sebastian, Pfister, Hanspeter et al.

Understand

Recurrent neural networks, and in particular long short-term memory (LSTM) networks, are a remarkably effective tool for sequence modeling that learn a dense black-box hidden representation of their sequential input.

Researchers interested in better understanding these models have studied the changes in hidden state representations over time and noticed some interpretable patterns but also significant noise.
In this work, we present LSTMVIS, a visual analysis tool for recurrent neural networks with a focus on understanding these hidden state dynamics.
The tool allows users to select a hypothesis input range to focus on local state changes, to match these states changes to similar patterns in a large data set, and to align these results with structural annotations from their domain.

Built on

Finding structure in time
J. L. Elman · 1990
Earlier work this paper cites.
Building a large annotated corpus of english: The penn treebank
M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini · 1993
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Recurrent neural networks can learn to implement symbol-sensitive counting
P. R. J. Wiles · 1998
Earlier work this paper cites.
Lstm recurrent networks learn simple context-free and context-sensitive languages
F. A. Gers and E. Schmidhuber · 2001
Earlier work this paper cites.
Introduction to the conll-2003 shared task: Language-independent named entity recognition
E. F. Tjong Kim Sang and F. De Meulder · 2003
Earlier work this paper cites.
Opening the black box-data driven visualization of neural networks
F.-Y. Tzeng and K.-L. Ma · 2005
Earlier work this paper cites.
Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies
B. Shneiderman and C. Plaisant · 2006
Earlier work this paper cites.
Interactive optimization for steering machine classification
A. Kapoor, B. Lee, D. Tan, and E. Horvitz · 2010
Earlier work this paper cites.
Recurrent neural network based language model
T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur · 2010
Earlier work this paper cites.
Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription
N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent · 2012
Earlier work this paper cites.
Recurrent continuous translation models
N. Kalchbrenner and P. Blunsom · 2013
Earlier work this paper cites.

Similar

Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Cited alongside, same era.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
K. Simonyan, A. Vedaldi, and A. Zisserman · 2013
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Original
D. Bahdanau, K. Cho, and Y. Bengio · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. V. Le · 2014
Cited alongside, same era.
Recurrent Neural Network Regularization
Original
W. Zaremba, I. Sutskever, and O. Vinyals · 2014
Cited alongside, same era.
Visualizing and understanding convolutional networks
M. D. Zeiler and R. Fergus · 2014
Cited alongside, same era.
Deep speech 2: End-to-end speech recognition in english and mandarin
Original
D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. C. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, E. Elsen, J. Engel, L. Fan, C. Fougner, T. Han, A. Y. Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, A. Y. Ng, S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, J. Zhan, and Z. Zhu · 2015
Cited alongside, same era.
Semi-supervised sequence learning
A. M. Dai and Q. V. Le · 2015
Cited alongside, same era.
Teaching machines to read and comprehend
K. M. Hermann, T. Kociský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom · 2015
Cited alongside, same era.
Lingusitic analysis of multi-modal recurrent neural networks
A. Kádár, G. Chrupała, and A. Alishahi · 2015
Cited alongside, same era.
Visualizing and understanding recurrent networks
Original
A. Karpathy, J. Johnson, and F.-F. Li · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio · 2015
Cited alongside, same era.

Then

Predicting effects of noncoding variants with deep learning-based sequence model
J. Zhou and O. G. Troyanskaya · 2015
Later among the works it cites.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Original
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al · 2016
Closest in time.
Representation of linguistic form and function in recurrent neural networks
Original
Á. Kádár, G. Chrupała, and A. Alishahi · 2016
Closest in time.
Interacting with predictions: Visual inspection of black-box machine learning models
J. Krause, A. Perer, and K. Ng · 2016
Closest in time.
Visualizing and understanding neural models in nlp
J. Li, X. Chen, E. Hovy, and D. Jurafsky · 2016
Closest in time.
Modeling language vagueness in privacy policies using deep neural networks
F. Liu, N. L. Fella, and K. Liao · 2016
Closest in time.
Danq: a hybrid convolutional and recurrent deep neural network for quantifying the function of dna sequences
D. Quang and X. Xie · 2016
Closest in time.
Visualizing the hidden activity of artificial neural networks
P. E. Rauber, S. G. Fadel, A. X. Falcão, and A. C. Telea · 2016
Closest in time.
Opportunities and obstacles for deep learning in biology and medicine
T. Ching, D. S. Himmelstein, B. K. Beaulieu-Jones, A. A. Kalinin, B. T. Do, G. P. Way, E. Ferrero, P.-M. Agapow, W. Xie, G. L. Rosen, B. J. Lengerich, J. Israeli, J. Lanchantin, S. Woloszynek, A. E. Carpenter, A. Shrikumar, J. Xu, E. M. Cofer, D. J. Harris, D. DeCaprio, Y. Qi, A. Kundaje, Y. Peng, L. K. Wiley, M. H. S. Segler, A. Gitter, and C. S. Greene · 2017
Closest in time.
Predicting process behaviour using deep learning
J. Evermann, J.-R. Rehse, and P. Fettke · 2017
Closest in time.
Towards better analysis of deep convolutional neural networks
M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…