2020

Compositionality and Generalization in Emergent Languages

Chaabouni, Rahma, Kharitonov, Eugene, Bouchacourt, Diane et al.

Understand

Natural language allows us to refer to novel composite concepts by combining expressions denoting their parts according to systematic rules, a property known as \emph{compositionality}.

In this paper, we study whether the language emerging in deep multi-agent simulations possesses a similar ability to refer to novel primitive combinations, and whether it accomplishes this feat by strategies akin to human-language compositionality.
Equipped with new ways to measure compositionality in emergent languages inspired by disentanglement in representation learning, we establish three main results.
First, given sufficiently large input spaces, the emergent language will naturally develop the ability to refer to novel composite concepts.

Built on

Emergence of compositional language with deep generational transmission
Original
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. 2019 · 1904
Earlier work this paper cites.
Ease-of-teaching and language structure from emergent communication
Original
Fushan Li and Michael Bowling. 2019 · 1906
Earlier work this paper cites.
The compositionality of neural networks: integrating symbolism and connectionism
Original
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2019 · 1908
Earlier work this paper cites.
The emergence of compositional languages for numeric concepts through iterated learning in neural agents
Original
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith. 2019 · 1910
Earlier work this paper cites.
Capacity, bandwidth, and compositionality in emergent language learning
Original
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M Dai, and Kyunghyun Cho. 2019 · 1910
Earlier work this paper cites.
Evaluating combinatorial generalization in variational autoencoders
Original
Alican Bozkurt, Babak Esmaeili, Dana H Brooks, Jennifer G Dy, and Jan-Willem van de Meent. 2019 · 1911
Earlier work this paper cites.
Convention
David Lewis. 1969 · 1969
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams. 1992 · 1992
Earlier work this paper cites.
Iconicity and the emergence of combinatorial structure in language
Tessa Verhoef, Simon Kirby, and Bart de Boer. 2016 · 1994
Earlier work this paper cites.
The evolution of language
Martin Nowak and David Krakauer. 1999 · 1999
Earlier work this paper cites.
The evolution of syntactic communication
Martin A Nowak, Joshua B Plotkin, and Vincent AA Jansen. 2000 · 2000
Earlier work this paper cites.
Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity
Simon Kirby. 2001 · 2001
Earlier work this paper cites.
The Compositionality Papers
Jerry Fodor and Ernest Lepore. 2002 · 2002
Earlier work this paper cites.
The Algebraic Mind
Gary Marcus. 2003 · 2003
Earlier work this paper cites.
Emergent language generalization and acquisition speed are not tied to compositionality
Original
Eugene Kharitonov and Marco Baroni. 2020 · 2004
Earlier work this paper cites.
Compositionality in Formal Semantics
Barbara Partee. 2004 · 2004
Earlier work this paper cites.

Similar

Lack of combinatorial productivity in language processing with simple recurrent networks
Frank van der Velde, Gwendid van der Voort van der Kleij, and Marc de Kamps. 2004 · 2004
Cited alongside, same era.
Understanding linguistic evolution by visualizing the emergence of topographic mappings
Henry Brighton and Simon Kirby. 2006 · 2006
Cited alongside, same era.
Strong systematicity in sentence processing by simple recurrent networks
Philémon Brakel and Stefan Frank. 2009 · 2009
Cited alongside, same era.
Compositionality II: Arguments and problems
Peter Pagin and Dag Westerståhl. 2010 · 2010
Cited alongside, same era.
The communicative function of ambiguity in language
Steven Piantadosi, Harry Tily, and Edward Gibson. 2012 · 2012
Cited alongside, same era.
Representation learning: A review and new perspectives
Y. Bengio, A. Courville, and P. Vincent. 2013 · 2013
Cited alongside, same era.
More than Nature Needs: Language, Mind, and Evolution
Derek Bickerton. 2014 · 2014
Cited alongside, same era.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Original
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Cited alongside, same era.
The Origins of Language
James Hurford. 2014 · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba. 2014 · 2014
Cited alongside, same era.
Compression and communication in the cultural evolution of linguistic structure
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. 2015 · 2015
Cited alongside, same era.
Gradient estimation using stochastic computation graphs
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. 2015 · 2015
Cited alongside, same era.

Then

Natural language does not emerge ‘naturally’ in multi-agent dialog
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017 · 2017
Later among the works it cites.
Isolating sources of disentanglement in variational autoencoders
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018 · 2018
Later among the works it cites.
Disentangling by factorising
Original
Hyunjik Kim and Andriy Mnih. 2018 · 2018
Later among the works it cites.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Brenden Lake and Marco Baroni. 2018 · 2018
Later among the works it cites.
Emergence of linguistic communication from referential games with symbolic and pixel input
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. 2018 · 2018
Later among the works it cites.
Emergence of grounded compositional language in multi-agent populations
Igor Mordatch and Pieter Abbeel. 2018 · 2018
Later among the works it cites.
Compositionality in animals and humans
Simon Townsend, Sabrina Engesser, Sabine Stoll, Klaus Zuberbühler, and Balthasar Bickel. 2018 · 2018
Later among the works it cites.
Bias and generalization in deep generative models: An empirical study
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. 2018 · 2018
Later among the works it cites.
Measuring compositionality in representation learning
Jacob Andreas. 2019 · 2019
Later among the works it cites.
On the realization of compositionality in neural networks
Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes, and Elia Bruni. 2019 · 2019
Later among the works it cites.
Anti-efficient encoding in emergent communication
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. 2019 · 2019
Later among the works it cites.
Explain Me This: Creativity, Competition, and the Partial Productivity of Constructions
Adele Goldberg. 2019 · 2019
Later among the works it cites.
EGG: a toolkit for research on emergence of language in games
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. 2019 · 2019
Later among the works it cites.
Challenging common assumptions in the unsupervised learning of disentangled representations
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. 2019 · 2019
Later among the works it cites.
Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. 2019 · 2019
Later among the works it cites.
A puzzle concerning compositionality in machines
Ryan Nefdt. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…