2020

How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context

Lau, Jey Han, Armendariz, Carlos S., Lappin, Shalom et al.

Understand

We study the influence of context on sentence acceptability.

First we compare the acceptability ratings of sentences judged in isolation, with a relevant context, and with an irrelevant context.
Our results show that context induces a cognitive load for humans, which compresses the distribution of ratings.
Moreover, in relevant contexts we observe a discourse coherence effect which uniformly raises acceptability.

Built on

Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019 · 1901
Earlier work this paper cites.
Translationese in machine translation evaluation
Original
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019 · 1906
Earlier work this paper cites.
XLNet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019 · 1906
Earlier work this paper cites.
Judgments of grammaticality
Dwight Bolinger. 1968 · 1968
Earlier work this paper cites.
The cognitive basis for linguistic structures
Thomas G. Bever. 1970 · 1970
Earlier work this paper cites.
Contextual influence on acceptability judgements
Sidney Greenbaum. 1976 · 1976
Earlier work this paper cites.
The Semantics of Definite and Indefinite Noun Phrases
Irene Heim. 1982 · 1982
Earlier work this paper cites.
Context and distance-to-disambiguation effects in ambiguity resolution: Evidence from grammaticality judgments of garden path sentences
John Warner and Arnold L Glass. 1987 · 1987
Earlier work this paper cites.
Rhetorical structure theory: Toward a functional theory of text organization
William Mann and Sandra Thompson. 1988 · 1988
Earlier work this paper cites.
Cognitive load during problem solving: Effects on learning
John Sweller. 1988 · 1988
Earlier work this paper cites.
Dynamic montague grammar
Jeroen Groenendijk and Martin Stokhof. 1990 · 1990
Earlier work this paper cites.
From Discourse To Logic
Hans Kamp and Uwe Reyle. 1993 · 1993
Earlier work this paper cites.
Anchoring and grammar effects in judgments of sentence acceptability
Wayne Cowart. 1994 · 1994
Earlier work this paper cites.
ACT: A simple theory of complex cognition
John R. Anderson. 1996 · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Mallet: A machine learning for language toolkit
Andrew Kachites McCallum. 2002 · 2002
Earlier work this paper cites.
Core syntax: A minimalist approach
David Adger. 2003 · 2003
Earlier work this paper cites.

Similar

Logics of Conversation
Nicholas Asher and Alex Lascarides. 2003 · 2003
Cited alongside, same era.
A neural probabilistic language model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003 · 2003
Cited alongside, same era.
Fuzzy Grammar: A Reader
David Denison. 2004 · 2004
Cited alongside, same era.
Gradience in linguistic data
Antonella Sorace and Frank Keller. 2005 · 2005
Cited alongside, same era.
Context and well-formedness: the dynamics of ellipsis
Ronnie Cann, Ruth Kempson, and Matthew Purver. 2007 · 2007
Cited alongside, same era.
Continuous acceptability, categorical grammaticality, and experimental syntax
Jon Sprouse. 2007 · 2007
Cited alongside, same era.
Recurrent neural network based language model
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010 · 2010
Cited alongside, same era.
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer. 2011 · 2011
Cited alongside, same era.
A computational cognitive model of syntactic priming
Daivd Reitter, Frank Keller, and Johanna D. Moore. 2011 · 2011
Cited alongside, same era.
The Interactive Stance: Meaning for Conversation
Jonathan Ginzburg. 2012 · 2012
Cited alongside, same era.
Large-scale syntactic language modeling with treelets
Adam Pauls and Dan Klein. 2012 · 2012
Cited alongside, same era.
Analysis of cognitive load for language processing based on brain activities
Hyangsook Park, Jun-Su Kang, Sungmook Choi, and Minho Lee. 2013 · 2013
Cited alongside, same era.

Then

Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Later among the works it cites.
High working memory load impairs language processing during a simulated piloting task: An erp and pupillometry study
Mickaël Causse, Vsevolod Peysakhovich, and Eve F. Fabre. 2016 · 2016
Later among the works it cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016 · 2016
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
The influence of context on sentence acceptability judgements
Jean-Philippe Bernardy, Shalom Lappin, and Jey Han Lau. 2018 · 2018
Later among the works it cites.
Structural priming in sentence comprehension: A single prime is enough
Maria Giavazzi, Sara Sambin, Ruth de Diego-Balaguer, Lorna Le Stanc, Anne-Catherine Bachoud-Lévi, and Charlotte Jacquemot. 2018 · 2018
Later among the works it cites.
A cognitive load delays predictive eye movements similarly during l1 and l2 comprehension
Aine Ito, Martin Corley, and Martin J. Pickering. 2018 · 2018
Later among the works it cites.
Sharp nearby, fuzzy far away: How neural language models use context
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018 · 2018
Later among the works it cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Later among the works it cites.
Neural network acceptability judgments
Original
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018 · 2018
Later among the works it cites.
Effects of linguistic context on the acceptability of co-speech gestures
Christina Zlogar and Kathryn Davidson. 2018 · 2018
Later among the works it cites.
The effect of context on metaphor paraphrase aptness judgments
Yuri Bizzoni and Shalom Lappin. 2019 · 2019
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Later among the works it cites.
BERT has a mouth, and it must speak: BERT as a Markov random field language model
Alex Wang and Kyunghyun Cho. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…