Built on

Distributed Representations of Words and Phrases and their Compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013 · 2013
Earlier work this paper cites.
Semantic Tagging with Deep Residual Networks
Johannes Bjerva, Barbara Plank, and Johan Bos. 2016 · 2016
Earlier work this paper cites.
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Original
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016 · 2016
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016 · 2016
Earlier work this paper cites.
Encoding of phonology in a recurrent neural model of grounded speech
Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017 · 2017
Earlier work this paper cites.
What do Neural Machine Translation Models Learn about Morphology?
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017 · 2017
Earlier work this paper cites.
The groningen meaning bank
Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017 · 2017
Earlier work this paper cites.

Similar

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Original
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017 · 2017
Cited alongside, same era.
Opening the Black Box of Deep Neural Networks via Information
Original
Ravid Shwartz-Ziv and Naftali Tishby. 2017 · 2017
Cited alongside, same era.
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
Original
Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2018 · 2018
Cited alongside, same era.
Deep RNNs Encode Soft Hierarchical Syntax
Original
Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018 · 2018
Cited alongside, same era.

Then

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
Original
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018 · 2018
Closest in time.
Indicatements that character language models learn English morpho-syntactic units and regularities
Yova Kementchedjhieva and Adam Lopez. 2018 · 2018
Closest in time.
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Original
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018 · 2018
Closest in time.
Insights on representational similarity in neural networks with canonical correlation
Original
Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018 · 2018
Closest in time.
Deep contextualized word representations

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2018

Understanding Learning Dynamics Of Language Models with SVCCA

Saphra, Naomi, Lopez, Adam

Understand

Research has shown that neural models implicitly encode linguistic features, but there has been no research showing \emph{how} these encodings arise as the models are trained.

We present the first study on the learning dynamics of neural language models, using a simple and flexible analysis method called Singular Vector Canonical Correlation Analysis (SVCCA), which enables us to compare learned representations across time and across models, without the need to evaluate directly on annotated data.
We probe the evolution of syntactic, semantic, and topic representations and find that part-of-speech is learned earlier than topic; that recurrent layers become more similar to those of a tagger during training; and embedding layers less similar.
Our results and methods could inform better learning algorithms for NLP models, possibly to incorporate linguistic information more effectively.

Built on

Distributed Representations of Words and Phrases and their Compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013 · 2013
Earlier work this paper cites.
Semantic Tagging with Deep Residual Networks
Johannes Bjerva, Barbara Plank, and Johan Bos. 2016 · 2016
Earlier work this paper cites.
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Original
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016 · 2016
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016 · 2016
Earlier work this paper cites.
Encoding of phonology in a recurrent neural model of grounded speech
Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017 · 2017
Earlier work this paper cites.
What do Neural Machine Translation Models Learn about Morphology?
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017 · 2017
Earlier work this paper cites.
The groningen meaning bank
Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017 · 2017
Earlier work this paper cites.

Similar

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Original
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017 · 2017
Cited alongside, same era.
Opening the Black Box of Deep Neural Networks via Information
Original
Ravid Shwartz-Ziv and Naftali Tishby. 2017 · 2017
Cited alongside, same era.
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
Original
Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2018 · 2018
Cited alongside, same era.
Deep RNNs Encode Soft Hierarchical Syntax
Original
Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018 · 2018
Cited alongside, same era.

Then

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
Original
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018 · 2018
Closest in time.
Indicatements that character language models learn English morpho-syntactic units and regularities
Yova Kementchedjhieva and Adam Lopez. 2018 · 2018
Closest in time.
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Original
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018 · 2018
Closest in time.
Insights on representational similarity in neural networks with canonical correlation
Original
Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018 · 2018
Closest in time.
Deep contextualized word representations

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

On the Learning Dynamics of Deep Neural Networks

Original

Remi Tachet des Combes, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. 2018 · 2018

Cited alongside, same era.

The Lottery Ticket Hypothesis: Training Pruned Neural Networks

Original

Jonathan Frankle and Michael Carbin. 2018 · 2018

Cited alongside, same era.

Original

Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018

Closest in time.

Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

Original

Kelly W. Zhang and Samuel R. Bowman. 2018 · 2018

Closest in time.