2022

State-of-the-art generalisation research in NLP: A taxonomy and review

Hupkes, Dieuwke, Giulianelli, Mario, Dankers, Verna et al.

Understand

The ability to generalise well is one of the primary desiderata of natural language processing (NLP).

Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation.
In this paper, we lay the groundwork to address both of these issues.
We present a taxonomy for characterising and understanding generalisation research in NLP.

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Learning and evaluating general linguistic intelligence
Original
Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019 · 1901
Earlier work this paper cites.
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
Original
Mathijs Mul and Willem Zuidema. 2019 · 1906
Earlier work this paper cites.
RoBERTa: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b · 1907
Earlier work this paper cites.
The child’s learning of English morphology
Jean Berko. 1958 · 1958
Earlier work this paper cites.
Cross-lingual name tagging and linking for 282 languages
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017 · 1958
Earlier work this paper cites.
Universal grammar
Richard Montague. 1970 · 1970
Earlier work this paper cites.
Connectionism and cognitive architecture: A critical analysis
Jerry A Fodor and Zenon W Pylyshyn. 1988 · 1988
Earlier work this paper cites.
Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate
Christo Kirov and Ryan Cotterell. 2018 · 1988
Earlier work this paper cites.
Towards compositional learning in dynamic networks
Jürgen Schmidhuber. 1990 · 1990
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn Treebank
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993 · 1993
Earlier work this paper cites.
Improved backing-off for m-gram language modeling
Reinhard Kneser and Hermann Ney. 1995 · 1995
Earlier work this paper cites.
Statistical decision-tree models for parsing
David M. Magerman. 1995 · 1995
Earlier work this paper cites.
German inflection: The exception that proves the rule
Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995 · 1995
Earlier work this paper cites.
A new statistical parser based on bigram lexical dependencies
Michael John Collins. 1996 · 1996
Earlier work this paper cites.
A maximum entropy approach to adaptive statistical language modelling
R ROSENFELD. 1996 · 1996
Earlier work this paper cites.
Rethinking eliminative connectionism
Gary F. Marcus. 1998 · 1998
Earlier work this paper cites.
Connectionism: with or without rules?: Response to jl mcclelland and dc plaut (1999)
Gary F Marcus. 1999 · 1999
Earlier work this paper cites.
Does generalization in infant learning implicate abstract algebra-like rules?
James L McClelland and David C Plaut. 1999 · 1999
Earlier work this paper cites.
The algebraic mind: Integrating connectionism and cognitive science
Gary F Marcus. 2003 · 2003
Earlier work this paper cites.
Domain adaptation with structural correspondence learning
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006 · 2006
Earlier work this paper cites.
Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification
John Blitzer, Mark Dredze, and Fernando Pereira. 2007 · 2007
Earlier work this paper cites.
Frustratingly easy domain adaptation
Hal Daumé III. 2007 · 2007
Earlier work this paper cites.
Experiments in domain adaptation for statistical machine translation
Philipp Koehn and Josh Schroeder. 2007 · 2007
Earlier work this paper cites.
Improved inference for unlexicalized parsing
Slav Petrov and Dan Klein. 2007 · 2007
Earlier work this paper cites.
Generalisation towards combinatorial productivity in language acquisition by simple recurrent networks
Francis CK Wong and William SY Wang. 2007 · 2007
Earlier work this paper cites.
A unified architecture for natural language processing: deep neural networks with multitask learning
Ronan Collobert and Jason Weston. 2008 · 2008
Earlier work this paper cites.
Domain adaptation for statistical machine translation with monolingual resources
Nicola Bertoldi and Marcello Federico. 2009 · 2009
Earlier work this paper cites.
When training and test sets are different: characterizing learning transfer
Amos Storkey. 2009 · 2009
Earlier work this paper cites.
Domain adaptation via pseudo in-domain data selection
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011 · 2011
Earlier work this paper cites.
On achieving and evaluating language-independence in nlp
Emily M Bender. 2011 · 2011
Earlier work this paper cites.
Accurate parsing with compact tree-substitution grammars: Double-DOP
Federico Sangati and Willem Zuidema. 2011 · 2011
Earlier work this paper cites.
A unifying view on dataset shift in classification
Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. 2012 · 2012
Earlier work this paper cites.
Crowdsourcing and annotating NER for Twitter #drift
Hege Fromreide, Dirk Hovy, and Anders Søgaard. 2014 · 2014
Earlier work this paper cites.
A SICK cure for the evaluation of compositional distributional semantic models
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014 · 2014
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a · 2015
Earlier work this paper cites.
How to avoid unwanted pregnancies: Domain adaptation using neural network models
Shafiq Joty, Hassan Sajjad, Nadir Durrani, Kamla Al-Mannai, Ahmed Abdelali, and Stephan Vogel. 2015 · 2015
Earlier work this paper cites.
Stanford neural machine translation systems for spoken language domains
Minh-Thang Luong and Christopher Manning. 2015 · 2015
Earlier work this paper cites.
Demographic dialectal variation in social media: A case study of African-American English
Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016 · 2016
Earlier work this paper cites.
Broad Twitter corpus: A diverse named entity recognition resource
Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016 · 2016
Earlier work this paper cites.
Fast domain adaptation for neural machine translation
Original
Markus Freitag and Yaser Al-Onaizan. 2016 · 2016
Earlier work this paper cites.
A corpus and cloze evaluation for deeper understanding of commonsense stories
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016 · 2016
Earlier work this paper cites.
What to do about non-standard (or non-canonical) language in NLP
Original
Barbara Plank. 2016 · 2016
Earlier work this paper cites.
Diagnostic classifiers: Revealing how neural networks process hierarchical structure
Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. 2016 · 2016
Earlier work this paper cites.
A dataset and classifier for recognizing social media English
Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2017 · 2017
Earlier work this paper cites.
An empirical comparison of domain adaptation methods for neural machine translation
Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017 · 2017
Earlier work this paper cites.
Detecting negation scope is easy, except when it isn’t
Federico Fancellu, Adam Lopez, Bonnie Webber, and Hangfeng He. 2017 · 2017
Earlier work this paper cites.
Building machines that learn and think like people
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017 · 2017
Earlier work this paper cites.
Abstractive morphological learning with a recurrent neural network
Robert Malouf. 2017 · 2017
Earlier work this paper cites.
Instance weighting for neural machine translation domain adaptation
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017b · 2017
Earlier work this paper cites.
diaNED: Time-aware named entity disambiguation for diachronic corpora
Prabal Agarwal, Jannik Strötgen, Luciano del Corro, Johannes Hoffart, and Gerhard Weikum. 2018 · 2018
Earlier work this paper cites.
A survey of domain adaptation for neural machine translation
Chenhui Chu and Rui Wang. 2018 · 2018
Earlier work this paper cites.
XNLI: Evaluating cross-lingual sentence representations
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018
Earlier work this paper cites.
Measuring and mitigating unintended bias in text classification
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018 · 2018
Earlier work this paper cites.
Improving text-to-SQL evaluation methodology
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018 · 2018
Earlier work this paper cites.
Colorless green recurrent networks dream hierarchically
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018 · 2018
Earlier work this paper cites.
Annotation artifacts in natural language inference data
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018 · 2018
Earlier work this paper cites.
Time-evolving text classification with deep neural networks
Yu He, Jianxin Li, Yangqiu Song, Mutian He, and Hao Peng. 2018 · 2018
Earlier work this paper cites.
Universal language model fine-tuning for text classification
Jeremy Howard and Sebastian Ruder. 2018 · 2018
Earlier work this paper cites.
Examining temporality in document classification
Xiaolei Huang and Michael J. Paul. 2018 · 2018
Earlier work this paper cites.
Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure
Dieuwke Hupkes, Sara , and Willem Zuidema. 2018 · 2018
Earlier work this paper cites.
How much reading does reading comprehension require? a critical investigation of popular benchmarks
Divyansh Kaushik and Zachary C. Lipton. 2018 · 2018
Earlier work this paper cites.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Brenden Lake and Marco Baroni. 2018 · 2018
Earlier work this paper cites.
Sentiment analysis under temporal shift
Jan Lukes and Anders Søgaard. 2018 · 2018
Earlier work this paper cites.
Deep learning: A critical appraisal
Original
Gary Marcus. 2018 · 2018
Earlier work this paper cites.
The natural language decathlon: Multitask learning as question answering
Original
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018 · 2018
Earlier work this paper cites.
MTNT: A testbed for machine translation of noisy text
Paul Michel and Graham Neubig. 2018 · 2018
Earlier work this paper cites.
Reducing gender bias in abusive language detection
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018 · 2018
Earlier work this paper cites.
Deep contextualized word representations
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Earlier work this paper cites.
Hypothesis only baselines in natural language inference
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018 · 2018
Earlier work this paper cites.
Out-of-domain detection based on generative adversarial network
Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Geunbae Lee. 2018 · 2018
Earlier work this paper cites.
Bridging the gaps: Multi task learning for domain transfer of hate speech detection
Zeerak Talat, James Thorne, and Joachim Bingel. 2018 · 2018
Earlier work this paper cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018
Earlier work this paper cites.

Similar

A broad-coverage challenge corpus for sentence understanding through inference
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018
Cited alongside, same era.
SWAG: A large-scale adversarial dataset for grounded commonsense inference
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018 · 2018
Cited alongside, same era.
Massively parallel cross-lingual learning in low-resource target language translation
Original
Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2018 · 2018
Cited alongside, same era.
Massively multilingual neural machine translation
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019 · 2019
Cited alongside, same era.
Consistency by agreement in zero-shot neural machine translation
Maruan Al-Shedivat and Ankur Parikh. 2019 · 2019
Cited alongside, same era.
Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers) . Association for Computational Linguistics, Florence, Italy
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors. 2019 · 2019
Cited alongside, same era.
Are we there yet? encoder-decoder neural networks as cognitive models of English past tense inflection
Maria Corkery, Yevgen Matusevych, and Sharon Goldwater. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Cross-domain generalization of neural constituency parsers
Daniel Fried, Nikita Kitaev, and Dan Klein. 2019 · 2019
Cited alongside, same era.
Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019 · 2019
Cited alongside, same era.
We need to talk about standard splits
Kyle Gorman and Steven Bedrick. 2019 · 2019
Cited alongside, same era.
Domain adaptation of neural machine translation by lexicon induction
Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019 · 2019
Cited alongside, same era.

Then

On the dangers of stochastic parrots: Can language models be too big?
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021 · 2021
Later among the works it cites.
Generalization in NLI: Ways (not) to go beyond simple heuristics
Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021 · 2021
Later among the works it cites.
Not quite there yet: Combining analogical patterns and encoder-decoder networks for cognitively plausible inflection
Basilio Calderone, Nabil Hathout, and Olivier Bonami. 2021 · 2021
Later among the works it cites.
Can transformers jump around right in natural language? assessing performance transfer from SCAN
Rahma Chaabouni, Roberto Dessì, and Eugene Kharitonov. 2021 · 2021
Later among the works it cites.
One semantic parser to parse them all: Sequence to sequence multi-task learning on semantic parsing datasets
Marco Damonte and Emilio Monti. 2021 · 2021
Later among the works it cites.
Generalising to German plural noun classes, from the perspective of a recurrent neural network
Verna Dankers, Anna Langedijk, Kate McCurdy, Adina Williams, and Dieuwke Hupkes. 2021 · 2021
Later among the works it cites.
Disentangling document topic and author gender in multiple languages: Lessons for adversarial debiasing
Erenay Dayanik and Sebastian Padó. 2021 · 2021
Later among the works it cites.
Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021 · 2021
Later among the works it cites.
Back to square one: Artifact detection, training and commonsense disentanglement in the Winograd schema
Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021 · 2021
Later among the works it cites.
Beyond english-centric multilingual machine translation
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021 · 2021
Later among the works it cites.
Datasheets for datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021 · 2021
Later among the works it cites.
XL-sum: Large-scale multilingual abstractive summarization for 44 languages
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021 · 2021
Later among the works it cites.
Language models use monotonicity to assess NPI licensing
Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021 · 2021
Later among the works it cites.
Dynabench: Rethinking benchmarking in NLP
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021 · 2021
Later among the works it cites.
A survey of generalisation in deep reinforcement learning
Original
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. 2021 · 2021
Later among the works it cites.
Wilds: A benchmark of in-the-wild distribution shifts
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. 2021 · 2021
Later among the works it cites.
Mind the gap: Assessing temporal generalization in neural language models
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021 · 2021
Later among the works it cites.
Question and answer test-train overlap in open-domain question answering datasets
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021 · 2021
Later among the works it cites.
Leveraging paradigmatic information in inflection acceptability prediction: The JHU-SFU submission to SIGMORPHON shared task 0.2
Jane S.Y. Li and Colin Wilson. 2021 · 2021
Later among the works it cites.
Few-shot learning with multilingual language models
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2021 · 2021
Later among the works it cites.
Challenges in generalization in open domain question answering
Original
Linqing Liu, Patrick S. H. Lewis, Sebastian Riedel, and Pontus Stenetorp. 2021 · 2021
Later among the works it cites.
MKQA: A linguistically diverse benchmark for multilingual open domain question answering
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021 · 2021
Later among the works it cites.
Shifts: A dataset of real distributional shift across multiple large-scale tasks
Andrey Malinin, Neil Band, Yarin Gal, Mark J. F. Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panagiotis Tigas, and Boris Yangel. 2021 · 2021
Later among the works it cites.
Cross-lingual genqa: Open-domain question answering with answer sentence generation
Benjamin Muller, Luca Soldaini, Rik Koncel-Kedziorski, Eric Lind, and Alessandro Moschitti. 2021 · 2021
Later among the works it cites.
Sigmorphon 2021 shared task on morphological reinflection: Generalization across languages
Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean-Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aelita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraś, Marcin Woliński, Totok Suhardijanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prud’hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Judit Ács, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vylomova. 2021a · 2021
Later among the works it cites.
Scaling language models: Methods, analysis & insights from training gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021 · 2021
Later among the works it cites.
XTREME-R: Towards more challenging and nuanced multilingual evaluation
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021 · 2021
Later among the works it cites.
Winogrande: An adversarial Winograd schema challenge at scale
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021 · 2021
Later among the works it cites.
Exploiting cloze-questions for few-shot text classification and natural language inference
Timo Schick and Hinrich Schütze. 2021 · 2021
Later among the works it cites.
Compositional generalization and natural language variation: Can a semantic parsing approach handle both?
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021 · 2021
Later among the works it cites.
Towards out-of-distribution generalization: A survey
Original
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021 · 2021
Later among the works it cites.
Masked language modeling and the distributional hypothesis: Order word matters pre-training for little
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021 · 2021
Later among the works it cites.
We need to talk about random splits
Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021 · 2021
Later among the works it cites.
Bme submission for sigmorphon 2021 shared task 0. a three step training approach with data augmentation for morphological inflection
Gábor Szolnok, Botond Barta, Dorina Lakatos, and Judit Ács. 2021 · 2021
Later among the works it cites.
Diagnosing the first-order logical reasoning ability through LogicNLI
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021 · 2021
Later among the works it cites.
MassiveSumm: a very large-scale, very multilingual, news summarisation dataset
Daniel Varab and Natalie Schluter. 2021 · 2021
Later among the works it cites.
Meta-learning for domain generalization in semantic parsing
Bailin Wang, Mirella Lapata, and Ivan Titov. 2021 · 2021
Later among the works it cites.
Frequency effects on syntactic rule learning in transformers
Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021 · 2021
Later among the works it cites.
A targeted assessment of incremental processing in neural language models and humans
Ethan Wilcox, Pranali Vani, and Roger Levy. 2021 · 2021
Later among the works it cites.
Were we there already? applying minimal generalization to the sigmorphon-unimorph shared task on cognitively plausible morphological inflection
Colin Wilson and Jane S.Y. Li. 2021 · 2021
Later among the works it cites.
Exploring transitivity in neural NLI models through veridicality
Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. 2021 · 2021
Later among the works it cites.
Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021 · 2021
Later among the works it cites.
Hidden biases in unreliable news detection datasets
Xiang Zhou, Heba Elfardy, Christos Christodoulopoulos, Thomas Butler, and Mohit Bansal. 2021 · 2021
Later among the works it cites.
Ext5: Towards extreme multi-task scaling for transfer learning
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022 · 2022
Closest in time.
PromptSource: An integrated development environment and repository for natural language prompts
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022 · 2022
Closest in time.
Bias mitigation in machine translation quality estimation
Hanna Behnke, Marina Fomicheva, and Lucia Specia. 2022 · 2022
Closest in time.
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022 · 2022
Closest in time.
No language left behind: Scaling human-centered machine translation
Original
Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022 · 2022
Closest in time.
Generalized quantifiers as a source of error in multilingual NLU benchmarks
Ruixiang Cui, Daniel Hershcovich, and Anders Søgaard. 2022 · 2022
Closest in time.
The paradox of the compositionality of natural language: A neural machine translation case study
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022 · 2022
Closest in time.
Multilingualism encourages recursion: a transfer study with mBERT
Andrea De Varda and Roberto Zamparelli. 2022 · 2022
Closest in time.
AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022 · 2022
Closest in time.
Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022 · 2022
Closest in time.
Are prompt-based models clueless?
Pride Kavumba, Ryo Takahashi, and Yusuke Oda. 2022 · 2022
Closest in time.
Using linguistic typology to enrich multilingual lexicons: the case of lexical gaps in kinship
Original
Temuulen Khishigsuren, Gábor Bella, Khuyagbaatar Batsuren, Abed Alhakim Freihat, Nandu Chandran Nair, Amarsanaa Ganbold, Hadi Khalilia, Yamini Chandrashekar, and Fausto Giunchiglia. 2022 · 2022
Closest in time.
SIGMORPHON–UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection
Jordan Kodner, Salam Khalifa, Khuyagbaatar Batsuren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Aryaman Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Goldman, David Guriel, Simon Guriel, Silvia Guriel-Agiashvili, Witold Kieraś, Andrew Krizhanovsky, Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomniashchaya, Daria Rodionova, Karina Scheifer, Alexandra Sorova, Anastasia Yemelina, Jeremiah Young, and Ekaterina Vylomova. 2022 · 2022
Closest in time.
Quantifying adaptability in pre-trained language models with 500 tasks
Belinda Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. 2022 · 2022
Closest in time.
Why don’t people use character-level machine translation?
Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2022 · 2022
Closest in time.
Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models
Ling Liu and Mans Hulden. 2022 · 2022
Closest in time.
Improving generalization of hate speech detection systems to novel target groups via domain adaptation
Florian Ludwig, Klara Dolos, Torsten Zesch, and Eleanor Hobley. 2022 · 2022
Closest in time.
Cross-task generalization via natural language crowdsourcing instructions
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022 · 2022
Closest in time.
Impact of pretraining term frequencies on few-shot reasoning
Original
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022 · 2022
Closest in time.
Multitask prompted training enables zero-shot task generalization
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022 · 2022
Closest in time.
Bloom: A 176b-parameter open-access multilingual language model
Original
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022 · 2022
Closest in time.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models
Original
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022 · 2022
Closest in time.
Benchmarking generalization via in-context instructions on 1,600+ language tasks
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022 · 2022
Closest in time.
Finetuned language models are zero-shot learners
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022 · 2022
Closest in time.
Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models
Original
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022 · 2022
Closest in time.
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022 · 2022
Closest in time.
Adversarial examples for evaluating reading comprehension systems
Robin Jia and Percy Liang. 2017 · 2031
Closest in time.
Language modelling as a multi-task problem
Lucas Weber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. 2021 · 2060
Closest in time.
BBQ: A hand-built bias benchmark for question answering
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022 · 2086
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…