2019

Capacity, Bandwidth, and Compositionality in Emergent Language Learning

Resnick, Cinjon, Gupta, Abhinav, Foerster, Jakob et al.

Understand

Many recent works have discussed the propensity, or lack thereof, for emergent languages to exhibit properties of natural languages.

A favorite in the literature is learning compositionality.
We note that most of those works have focused on communicative bandwidth as being of primary importance.
While important, it is not the only contributing factor.

Built on

Word-order biases in deep-agent emergent communication
Original
Rahma Chaabouni, Eugene Kharitonov, Alessandro Lazaric, Emmanuel Dupoux, and Marco Baroni. 2019b · 1905
Earlier work this paper cites.
Convention: A philosophical study
David Lewis. 1969 · 1969
Earlier work this paper cites.
Iconicity and the Emergence of Combinatorial Structure in Language
Tessa Verhoef, Simon Kirby, and Bart de Boer. 2016 · 1994
Earlier work this paper cites.
Long Short-Term Memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
The Synthetic Modeling of Language Origins
Luc Steels. 1997 · 1997
Earlier work this paper cites.
AIBO’s first words: The social learning of language and meaning
Luc Steels and Frédéric Kaplan. 2000 · 2000
Earlier work this paper cites.
Auto-encoding Variational Bayes. In International Conference on Learning Representations
Diederik P Kingma and Max Welling. 2014 · 2014
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations
Diederik P. Kingma and Jimmy Ba. 2015 · 2015
Earlier work this paper cites.
Compression and Communication in the Cultural Evolution of Linguistic Structure
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. 2015 · 2015
Earlier work this paper cites.
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. 2016 · 2016
Earlier work this paper cites.
Learning Multiagent Communication with Backpropagation
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2016 · 2016
Earlier work this paper cites.
Capacity and Trainability in Recurrent Neural Networks. In International Conference on Learning Representations
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. 2017 · 2017
Earlier work this paper cites.
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
Serhii Havrylov and Ivan Titov. 2017 · 2017
Earlier work this paper cites.

Similar

Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations
Eric Jang, Shixiang Gu, and Ben Poole. 2017 · 2017
Cited alongside, same era.
Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017 · 2017
Cited alongside, same era.
Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017 · 2017
Cited alongside, same era.
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In International Conference on Learning Representations
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017 · 2017
Cited alongside, same era.
Minimal Requirements for the Emergence of Learned Signaling. In Cognitive Science
Matthew Spike, Kevin Stadler, Simon Kirby, and Kenny Smith. 2017 · 2017
Cited alongside, same era.
Hierarchical Models for the Evolution of Compositional Language
Jeffrey A. Barrett, Brian Skyrms, and Calvin Cochran. 2018 · 2018
Cited alongside, same era.
Recurrent Neural Networks as Weighted Language Recognizers. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018 · 2018
Cited alongside, same era.
Emergent Communication in a Multi-Modal, Multi-Step Referential Game. In International Conference on Learning Representations
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. 2018 · 2018
Cited alongside, same era.
Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input. In International Conference on Learning Representations
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. 2018 · 2018
Cited alongside, same era.
Emergent Translation in Multi-Agent Communication. In International Conference on Learning Representations
Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2018 · 2018
Cited alongside, same era.
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data. In Advances in Neural Information Processing Systems 31
Yuanzhi Li and Yingyu Liang. 2018 · 2018
Cited alongside, same era.
Memorize or generalize? Searching for a compositional RNN in a haystack
Original
Adam Liska, Germán Kruszewski, and Marco Baroni. 2018 · 2018
Cited alongside, same era.

Then

The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models. In Proceedings of the Workshop on Generalization in the Age of Deep Learning
Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018 · 2018
Later among the works it cites.
Efficient compression in color naming and its evolution
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018 · 2018
Later among the works it cites.
Measuring Compositionality in Representation Learning. In International Conference on Learning Representations
Jacob Andreas. 2019 · 2019
Closest in time.
Systematic Generalization: What Is Required and Can It Be Learned?. In International Conference on Learning Representations
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2019 · 2019
Closest in time.
Linguistic generalization and compositionality in modern artificial neural networks
Marco Baroni. 2020 · 2019
Closest in time.
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. 2019 · 2019
Closest in time.
Gradient Descent Provably Optimizes Over-parameterized Neural Networks. In International Conference on Learning Representations
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 2019 · 2019
Closest in time.
Compositional generalization through meta sequence-to-sequence learning
Brenden M Lake. 2019 · 2019
Closest in time.
Countering Language Drift via Visual Grounding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019 · 2019
Closest in time.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa, et al · 2019
Closest in time.
The compositionality of neural networks: integrating symbolism and connectionism
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020 · 2020
Closest in time.
On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations
Ryan Lowe*, Abhinav Gupta*, Jakob Foerster, Douwe Kiela, and Joelle Pineau. 2020 · 2020
Closest in time.
Deep Double Descent: Where Bigger Models and More Data Hurt. In International Conference on Learning Representations
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…