2020

Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

Jeretic, Paloma, Warstadt, Alex, Bhooshan, Suvrat et al.

Understand

Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer whether a sentence entails another.

However, the ability of NLI models to make pragmatic inferences remains understudied.
We create an IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of >25k semiautomatically generated sentence pairs illustrating well-studied pragmatic inference types.
We use IMPPRES to evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI (Williams et al., 2018) learn to make pragmatic inferences.

Built on

On denoting
Bertrand Russell. 1905 · 1905
Earlier work this paper cites.
RoBERTa: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
BLiMP: The benchmark of linguistic minimal pairs for English
Original
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2019b · 1912
Earlier work this paper cites.
Presuppositions of compound sentences
Lauri Karttunen. 1973 · 1973
Earlier work this paper cites.
Pragmatic presuppositions
Robert Stalnaker. 1974 · 1974
Earlier work this paper cites.
Logic and conversation
H Paul Grice. 1975 · 1975
Earlier work this paper cites.
Pragmatics, implicature, presuposition and logical form
Gerald Gazdar. 1979 · 1979
Earlier work this paper cites.
Scorekeeping in a language game
David Lewis. 1979 · 1979
Earlier work this paper cites.
On the projection problem for presuppositions
Irene Heim. 1983 · 1983
Earlier work this paper cites.
A natural history of negation
Laurence Horn. 1989 · 1989
Earlier work this paper cites.
FraCaS: A framework for computational semantics
Robin Cooper, Richard Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, Stephen Pulman, et al. 1994 · 1994
Earlier work this paper cites.
Using language
Herbert H Clark. 1996 · 1996
Earlier work this paper cites.
Presupposition
David I. Beaver. 1997 · 1997
Earlier work this paper cites.
Presumptive meanings: The theory of generalized conversational implicature
Stephen C Levinson. 2000 · 2000
Earlier work this paper cites.
The handbook of pragmatics
Laurence R Horn and Gregory L Ward. 2004 · 2004
Earlier work this paper cites.
Local textual inference: Can it be defined or circumscribed?
Annie Zaenen, Lauri Karttunen, and Richard Crouch. 2005 · 2005
Earlier work this paper cites.
Circumscribing is not excluding: A reply to Manning
Richard Crouch, Lauri Karttunen, and Annie Zaenen. 2006 · 2006
Earlier work this paper cites.
The PASCAL recognising textual entailment challenge
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006 · 2006
Earlier work this paper cites.
Applied textual entailment
Oren Glickman. 2006 · 2006
Earlier work this paper cites.
Local textual inference: It’s hard to circumscribe, but you know it when you see it – and NLP needs it
Christopher D. Manning. 2006 · 2006
Earlier work this paper cites.
Addressing discourse and document structure in the RTE search task
Shachar Mirkin, Roy Bar-Haim, Ido Dagan, Eyal Shnarch, Asher Stern, Idan Szpektor, and Jonathan Berant. 2009 · 2009
Earlier work this paper cites.

Similar

Recognizing textual entailment: Models and applications
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013 · 2013
Cited alongside, same era.
SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014 · 2014
Cited alongside, same era.
GloVe: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 · 2014
Cited alongside, same era.
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
Investigating the distribution of some (but not all) implicatures using corpora and web-based methods
Judith Degen. 2015 · 2015
Cited alongside, same era.
Presupposition and implicature
Christopher Potts. 2015 · 2015
Cited alongside, same era.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
Probing for semantic evidence of composition by means of simple classification tasks
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016 · 2016
Cited alongside, same era.
Natural language inference by tree-based convolution and heuristic matching
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016 · 2016
Cited alongside, same era.
Supervised learning of universal sentence representations from natural language inference data
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017 · 2017
Cited alongside, same era.
Natural language inference from multiple premises
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017 · 2017
Cited alongside, same era.
Inference is everything: Recasting semantic resources into a unified evaluation framework
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017 · 2017
Cited alongside, same era.

Then

Event2Mind: Commonsense inference on events, intents, and reactions
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018 · 2018
Later among the works it cites.
The role of veridicality and factivity in clause selection
Aaron Steven White and Kyle Rawlins. 2018 · 2018
Later among the works it cites.
Lexicosyntactic inference in neural models
Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018 · 2018
Later among the works it cites.
A broad-coverage challenge corpus for sentence understanding through inference
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018
Later among the works it cites.
SWAG: A large-scale adversarial dataset for grounded commonsense inference
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018 · 2018
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commitment
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019 · 2019
Later among the works it cites.
Probing what different NLP tasks teach machines about function word comprehension
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019 · 2019
Later among the works it cites.
The CommitmentBank: Investigating projection in naturally occurring discourse
Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019 · 2019
Later among the works it cites.
Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019 · 2019
Later among the works it cites.
Analyzing compositionality-sensitivity of NLI models
Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019 · 2019
Later among the works it cites.
SherLIiC: A typed event-focused lexical inference benchmark for evaluating natural language inference
Martin Schmitt and Hinrich Schütze. 2019 · 2019
Later among the works it cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019 · 2019
Later among the works it cites.
Dialogue natural language inference
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019 · 2019
Later among the works it cites.
Can neural networks understand monotonicity reasoning?
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a · 2019
Later among the works it cites.
HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b · 2019
Later among the works it cites.
HellaSwag: Can a machine really finish your sentence?
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019
Later among the works it cites.
Abductive commonsense reasoning
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020 · 2020
Closest in time.
Probing natural language inference models through semantic fragments
Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2020 · 2020
Closest in time.
Harnessing the richness of the linguistic signal in predicting pragmatic inferences
Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…