Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Generating expressions that refer to visible objects

Margaret Mitchell, Kees van Deemter, and Ehud Reiter · 2013

Cited alongside, same era.

Playing Atari with deep reinforcement learning, 2013

Original

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013

Cited alongside, same era.

"The things that we have to do": Ethics and instrumentality in humanitarian communication

David Nolan and Akina Mikami · 2013

Cited alongside, same era.

Recursive deep models for semantic compositionality over a sentiment treebank

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts · 2013

Cited alongside, same era.

Non-linear mapping for improved identification of 1300+ languages

Ralf Brown · 2014

Cited alongside, same era.

Eliciting good teaching from humans for machine learners

Maya Cakmak and Andrea L. Thomaz · 2014

Cited alongside, same era.

Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek · 2019

Later among the works it cites.

MELD: A multimodal multi-party dataset for emotion recognition in conversations

Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea · 2019

Later among the works it cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019

Later among the works it cites.

Folk psychology as a theory

Ian Ravenscroft · 2019

Later among the works it cites.

CoQA: A conversational question answering challenge

Siva Reddy, Danqi Chen, and Christopher D. Manning · 2019

Later among the works it cites.

How well do NLI models capture verb veridicality?

Alexis Ross and Ellie Pavlick · 2019

Later among the works it cites.

Social IQa: Commonsense reasoning about social interactions

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi · 2019

Later among the works it cites.

Revisiting low-resource neural machine translation: A case study

Rico Sennrich and Biao Zhang · 2019

Later among the works it cites.

The woman worked as a babysitter: On biases in language generation

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng · 2019

Later among the works it cites.

Mining discourse markers for unsupervised sentence representation learning

Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller · 2019

Later among the works it cites.

Patching gender: Non-binary utopias in HCI

Katta Spiel, Os Keyes, and Pınar Barlas · 2019

Later among the works it cites.

Evaluating gender bias in machine translation

Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer · 2019

Later among the works it cites.

Executing instructions in situated collaborative interactions

Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi · 2019

Later among the works it cites.

The bitter lesson

Rich Sutton · 2019

Later among the works it cites.

CommonsenseQA: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant · 2019

Later among the works it cites.

The teaching size: Computable teachers and learners for universal languages

Jan Arne Telle, José Hernández-Orallo, and Cèsar Ferri · 2019

Later among the works it cites.

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver · 2019

Later among the works it cites.

SuperGLUE: A stickier benchmark for general-purpose language understanding systems

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman · 2019

Later among the works it cites.

TalkDown: A corpus for condescension detection in context

Zijian Wang and Christopher Potts · 2019

Later among the works it cites.

Humor detection: A transformer gets the last laugh

Orion Weller and Kevin Seppi · 2019

Later among the works it cites.

Some additional experiments extending the tech report “assessing BERT’s syntactic abilities” by Yoav Goldberg, 2019

Thomas Wolf · 2019

Later among the works it cites.

CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases

Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev · 2019

Later among the works it cites.

Learning the Dyck language with attention-based Seq2Seq models

Xiang Yu, Ngoc Thang Vu, and Jonas Kuhn · 2019

Later among the works it cites.

The gap of semantic parsing: A survey on automatic math word problem solvers

Dongxiang Zhang, Lei Wang, Luming Zhang, Bing Tian Dai, and Heng Tao Shen · 2019

Later among the works it cites.

Irony detection via sentiment-based transfer learning

Shiwei Zhang, Xiuzhen Zhang, Jeffrey Chan, and Paolo Rosso · 2019

Later among the works it cites.

Learning to ask unanswerable questions for machine reading comprehension

Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu · 2019

Later among the works it cites.

A very unlikely chess game, 2020

Scott Alexander · 2020

Later among the works it cites.

Structural language models of code

Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav · 2020

Later among the works it cites.

A survey on approaches to computational humor generation

Miriam Amin and Manuel Burghardt · 2020

Later among the works it cites.

On the cross-lingual transferability of monolingual representations

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama · 2020

Later among the works it cites.

Generating fact checking explanations

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein · 2020

Later among the works it cites.

The relationship between inference skills and reading comprehension

Nihat Bayat and Gökhan Çetinkaya · 2020

Later among the works it cites.

Climbing towards NLU: On meaning, form, and understanding in the age of data

Emily M. Bender and Alexander Koller · 2020

Later among the works it cites.

On the ability and limitations of transformers to recognize formal languages

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal · 2020

Later among the works it cites.

On the practical ability of recurrent neural networks to recognize hierarchical languages

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal · 2020

Later among the works it cites.

PIQA: reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi · 2020

Later among the works it cites.

Language (technology) is power: A critical survey of “bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach · 2020

Later among the works it cites.

GPT-3 creative fiction

Gwern Branwen · 2020

Later among the works it cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020

Later among the works it cites.

Developing self-awareness in robots via inner speech

Antonio Chella, Arianna Pipitone, Alain Morin, and Famira Racy · 2020

Later among the works it cites.

Generative pretraining from pixels

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever · 2020

Later among the works it cites.

Transformers play chess, 2020

Ricson Chen · 2020

Later among the works it cites.

Abstraction and reasoning challenge, 2020

François Chollet · 2020

Later among the works it cites.

Automated data transformation with inductive programming and dynamic background knowledge

Lidia Contreras-Ochando, Cèsar Ferri, José Hernández-Orallo, Fernando Martínez-Plumed, María José Ramírez-Quintana, and Susumu Katayama · 2020

Later among the works it cites.

Learning higher-order logic programs

Andrew Cropper, Rolf Morel, and Stephen Muggleton · 2020

Later among the works it cites.

When redundancy is useful: A Bayesian approach to “overinformative” referring expressions

Judith Degen, Robert D. Hawkins, Caroline Graf, Elisa Kreiss, and Noah D. Goodman · 2020

Later among the works it cites.

Calibration of pre-trained transformers

Shrey Desai and Greg Durrett · 2020

Later among the works it cites.

On measuring and mitigating biased inferences of word embeddings

Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar · 2020

Later among the works it cites.

RoFT: A tool for evaluating human detection of machine-generated text

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch · 2020

Later among the works it cites.

To test machine comprehension, start by defining comprehension

Jesse Dunietz, Greg Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and Dave Ferrucci · 2020

Later among the works it cites.

FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization

Esin Durmus, He He, and Mona Diab · 2020

Later among the works it cites.

Humor detection via an internal and external neural network

Xiaochao Fan, Hongfei Lin, Liang Yang, Yufeng Diao, Chen Shen, Yonghe Chu, and Yanbo Zou · 2020

Later among the works it cites.

SyntaxGym: An online platform for targeted evaluation of language models

Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy · 2020

Later among the works it cites.

RealToxicityPrompts: Evaluating neural toxic degeneration in language models

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith · 2020

Later among the works it cites.

Conversational implicatures in English dialogue: Annotated dataset

Elizabeth Jasmi George and Radhika Mamidi · 2020

Later among the works it cites.

Injecting numerical reasoning skills into language models

Mor Geva, Ankit Gupta, and Jonathan Berant · 2020

Later among the works it cites.

Irony detection in a multilingual context

Bilal Ghanem, Jihen Karoui, Farah Benamara, Paolo Rosso, and Véronique Moriceau · 2020

Later among the works it cites.

Are neural open-domain dialog systems robust to speech recognition errors in the dialog history? An empirical study

Karthik Gopalakrishnan, Behnam Hedayatnia, Longshaokan Wang, Yang Liu, and Dilek Hakkani-Tür · 2020

Later among the works it cites.

Theoretical limitations of self-attention in neural sequence models

Michael Hahn · 2020

Later among the works it cites.

Policy-driven neural response generation for knowledge-grounded dialog systems

Behnam Hedayatnia, Karthik Gopalakrishnan, Seokhwan Kim, Yang Liu, Mihail Eric, and Dilek Hakkani-Tur · 2020

Later among the works it cites.

3D-DEEP: 3-dimensional deep-learning based on elevation patterns for road scene interpretation

Álvaro Hernandez, Suhan Woo, Héctor Corrales, Ignacio Parra, Euntai Kim, D. Fernández Llorca, and Miguel A. Sotelo · 2020

Later among the works it cites.

TaPas: Weakly supervised table parsing via pre-training

Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos · 2020

Later among the works it cites.

Bridging anaphora resolution as question answering

Yufang Hou · 2020

Later among the works it cites.

Compositionality decomposed: How do neural networks generalise?

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni · 2020

Later among the works it cites.

Automatic detection of generated text is easiest when humans are fooled

Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck · 2020

Later among the works it cites.

Learning to execute instructions in a Minecraft dialogue

Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier · 2020

Later among the works it cites.

Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition

Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams · 2020

Later among the works it cites.

How can we know what language models know?

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig · 2020

Later among the works it cites.

Template guided text generation for task-oriented dialogue

Mihir Kale and Abhinav Rastogi · 2020

Later among the works it cites.

UNIFIEDQA: Crossing format boundaries with a single QA system

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi · 2020

Later among the works it cites.

Evaluating approaches to personalizing language models

Milton King and Paul Cook · 2020

Later among the works it cites.

All the news that’s fit to fabricate: AI-generated text as a tool of media misinformation

Sarah E. Kreps, Miles McCain, and Miles Brundage · 2020

Later among the works it cites.

Evaluating the factual consistency of abstractive text summarization

Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher · 2020

Later among the works it cites.

Giving GPT-3 a Turing test

Kevin Lacker · 2020

Later among the works it cites.

Language models as fact checkers?

Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa · 2020

Later among the works it cites.

MLQA: Evaluating cross-lingual extractive question answering

Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk · 2020

Later among the works it cites.

UNQOVERing stereotyping biases via underspecified questions

Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar · 2020

Later among the works it cites.

Towards debiasing sentence representations

Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency · 2020

Later among the works it cites.

Learning to contrast the counterfactual samples for robust visual question answering

Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu · 2020

Later among the works it cites.

Birds have four legs?! NumerSense: probing numerical commonsense knowledge of pre-trained language models

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren · 2020

Later among the works it cites.

Gender bias in neural natural language processing

Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta · 2020

Later among the works it cites.

GPT-3, bloviator: OpenAI’s language generator has no idea what it’s talking about

Gary Marcus and Ernest Davis · 2020

Later among the works it cites.

OpenAI API alchemy: Emoji storytelling

Andrew Mayne · 2020

Later among the works it cites.

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

R. Thomas McCoy, Robert Frank, and Tal Linzen · 2020

Later among the works it cites.

USR: An unsupervised and reference free evaluation metric for dialog generation

Shikib Mehri and Maxine Eskenazi · 2020

Later among the works it cites.

A framework for the computational linguistic analysis of dehumanization

Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky · 2020

Later among the works it cites.

The effect of natural distribution shift on question answering models

John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt · 2020

Later among the works it cites.

The deep bootstrap framework: Good online learners are good offline generalizers

Original

Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi · 2020

Later among the works it cites.

Participatory research for low-resourced machine translation: A case study in African languages

Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir · 2020

Later among the works it cites.

iSarcasm: A dataset of intended sarcasm

Silviu Oprea and Walid Magdy · 2020

Later among the works it cites.

Don’t patronize me! An annotated dataset with patronizing and condescending language towards vulnerable communities

Carla Perez Almendros, Luis Espinosa Anke, and Steven Schockaert · 2020

Later among the works it cites.

Data cleaning: A case study with OpenRefine and Trifacta Wrangler

Dessislava Petrova-Antonova and Rumyana Tancheva · 2020

Later among the works it cites.

Fleet system, 2020

Steve Piantadosi · 2020

Later among the works it cites.

A transformer-based approach to irony and sarcasm detection

Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis · 2020

Later among the works it cites.

A survey on computational metaphor processing

Sunny Rai and Shampa Chakraverty · 2020

Later among the works it cites.

Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset

Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan · 2020

Later among the works it cites.

Comparing conventions

Rachel Etta Rudolph and Alexander W. Kocurek · 2020

Later among the works it cites.

The child as hacker: Building more human-like models of learning

Joshua S. Rule · 2020

Later among the works it cites.

The child as hacker

Joshua S. Rule, Joshua B. Tenenbaum, and Steven T. Piantadosi · 2020

Later among the works it cites.

PuzzLing Machines: A challenge on learning from small data

Gözde Gül Şahin, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych · 2020

Later among the works it cites.

WINOGRANDE: An adversarial Winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi · 2020

Later among the works it cites.

Masked language model scoring

Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff · 2020

Later among the works it cites.

Social bias frames: Reasoning about social and power implications of language

Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi · 2020

Later among the works it cites.

All your questions answered

Janelle Shane · 2020

Later among the works it cites.

EmoTag1200: Understanding the association between emojis and emotions

Abu Awal Md Shoeb and Gerard de Melo · 2020

Later among the works it cites.

DiscSense: Automated semantic analysis of discourse markers

Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller · 2020

Later among the works it cites.

Evolution and impact of bias in human and machine learning algorithm interaction

Wenlong Sun, Olfa Nasraoui, and Patrick Shafto · 2020

Later among the works it cites.

Correlation-based network analysis combined with machine learning techniques highlight the role of the gaba shunt in brachypodium sylvaticum freezing tolerance

David Toubiana, Nir Sade, Lifeng Liu, Maria del Mar Rubio Wilhelmi, Yariv Brotman, Urszula Luzarowska, John P. Vogel, and Eduardo Blumwald · 2020

Later among the works it cites.

Temporal reasoning in natural language inference

Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White · 2020

Later among the works it cites.

Fill in the BLANC: Human-free quality estimation of document summaries

Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon · 2020

Later among the works it cites.

Does GPT-2 know your phone number?

Eric Wallace, Florian Tramèr, Matthew Jagielski, and Ariel Herbert-Voss · 2020

Later among the works it cites.

Asking and answering questions to evaluate the factual consistency of summaries

Alex Wang, Kyunghyun Cho, and Mike Lewis · 2020

Later among the works it cites.

Continuity of topic, interaction, and query: Learning to quote in online conversations

Lingzhi Wang, Jing Li, Xingshan Zeng, Haisong Zhang, and Kam-Fai Wong · 2020

Later among the works it cites.

AutoQA: From databases to QA semantic parsers with only synthetic training data

Silei Xu, Sina Semnani, Giovanni Campagna, and Monica Lam · 2020

Later among the works it cites.

Figure me out: A gold standard dataset for metaphor interpretation

Omnia Zayed, John Philip McCrae, and Paul Buitelaar · 2020

Later among the works it cites.

WinoWhy: A deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge

Hongming Zhang, Xinran Zhao, and Yangqiu Song · 2020

Later among the works it cites.

Reasoning about goals, steps, and temporal ordering with WikiHow

Li Zhang, Qing Lyu, and Chris Callison-Burch · 2020

Later among the works it cites.

Persistent anti-Muslim bias in large language models, 2021

Original

Abubakar Abid, Maheen Farooqi, and James Zou · 2021

Later among the works it cites.

Efficient large scale language modeling with mixtures of experts, 2021

Original

Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Ves Stoyanov · 2021

Later among the works it cites.

A general language assistant as a laboratory for alignment, 2021

Original

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan · 2021

Later among the works it cites.

Program synthesis with large language models, 2021

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton · 2021

Later among the works it cites.

Explaining neural scaling laws, 2021

Original

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma · 2021

Later among the works it cites.

On the dangers of stochastic parrots: Can language models be too big?

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell · 2021

Later among the works it cites.

Think you have solved direct-answer question answering? Try ARC-DA, the direct-answer AI2 reasoning challenge, 2021

Original

Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark · 2021

Later among the works it cites.

Multimodal datasets: Misogyny, pornography, and malignant stereotypes, 2021

Original

Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe · 2021

Later among the works it cites.

On the opportunities and risks of foundation models, 2021

Original

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang · 2021

Later among the works it cites.

What will it take to fix benchmarking in natural language understanding?

Samuel R. Bowman and George Dahl · 2021

Later among the works it cites.

Extracting training data from large language models

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel · 2021

Later among the works it cites.

Evaluating large language models trained on code, 2021

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021

Later among the works it cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman · 2021

Later among the works it cites.

White Chicago cops use force more often than Black officers

Jim Daley · 2021

Later among the works it cites.

NL-Augmenter: A framework for task-sensitive natural language augmentation, 2021

Original

Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel-Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Pfister, Richard Plant, Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji, Pawan Kumar Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos P. H. S., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom, Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen, Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Taylor Sorensen, William Soto, Aman Srivastava, KV Aditya Srivatsa, Tony Sun, Mukund Varma T, A Tabassum, Fiona Anting Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn, Athena Wang, Zijian Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang · 2021

Later among the works it cites.

GLaM: Efficient scaling of language models with mixture-of-experts, 2021

Original

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui · 2021

Later among the works it cites.

Neural path hunter: Reducing hallucination in dialogue systems via path grounding, 2021

Original

Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose · 2021

Later among the works it cites.

Cryptonite: A cryptic crossword benchmark for extreme ambiguity in language

Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy · 2021

Later among the works it cites.

Measuring and improving consistency in pretrained language models, 2021

Original

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg · 2021

Later among the works it cites.

Beyond English-centric multilingual machine translation

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin · 2021

Later among the works it cites.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021

Original

William Fedus, Barret Zoph, and Noam Shazeer · 2021

Later among the works it cites.

The pile: An 800GB dataset of diverse text for language modeling, 2021

Original

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy · 2021

Later among the works it cites.

The GEM benchmark: Natural language generation, its evaluation and metrics

Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou · 2021

Later among the works it cites.

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant · 2021

Later among the works it cites.

ePiC: Employing proverbs in context as a benchmark for abstract language understanding, 2021

Original

Sayan Ghosh and Shashank Srivastava · 2021

Later among the works it cites.

Disfl-QA: A benchmark dataset for understanding disfluencies in question answering

Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui · 2021

Later among the works it cites.

A survey on recent approaches for natural language processing in low-resource scenarios

Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow · 2021

Later among the works it cites.

Scaling laws for transfer, 2021

Original

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish · 2021

Later among the works it cites.

National name report 2018

China Household Management Research Center, Ministry of Public Security · 2021

Later among the works it cites.

National name report 2019

China Household Management Research Center, Ministry of Public Security · 2021

Later among the works it cites.

National name report 2020

China Household Management Research Center, Ministry of Public Security · 2021

Later among the works it cites.

Alignment of language agents, 2021

Original

Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving · 2021

Later among the works it cites.

Dynabench: Rethinking benchmarking in NLP

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams · 2021

Later among the works it cites.

MultiEmo: Multilingual, multilevel, multidomain sentiment analysis corpus of consumer reviews

Jan Kocoń, Piotr Miłkowski, and Kamil Kanclerz · 2021

Later among the works it cites.

Counterlogicals as counterconventionals

Alexander W. Kocurek and Ethan Jerzak · 2021

Later among the works it cites.

Hurdles to progress in long-form question answering, 2021

Original

Kalpesh Krishna, Aurko Roy, and Mohit Iyyer · 2021

Later among the works it cites.

Mechanisms for handling nested dependencies in neural-network language models and humans

Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene · 2021

Later among the works it cites.

Towards few-shot fact-checking via perplexity

Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung · 2021

Later among the works it cites.

Investigating memorization of conspiracy theories in text generation, 2021

Original

Sharon Levy, Michael Saxon, and William Yang Wang · 2021

Later among the works it cites.

Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark, 2021

Original

Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi · 2021

Later among the works it cites.

What’s in the box? An analysis of undesirable content in the Common Crawl corpus

Alexandra Luccioni and Joseph Viviano · 2021

Later among the works it cites.

EventPlus: A temporal event understanding pipeline, 2021

Original

Mingyu Derek Ma, Jiao Sun, Mu Yang, Kung-Hsiang Huang, Nuan Wen, Shikhar Singh, Rujun Han, and Nanyun Peng · 2021

Later among the works it cites.

Few-shot bot: Prompt-based learning for dialogue systems, 2021

Original

Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung · 2021

Later among the works it cites.

Inclusive data visualization for people with disabilities: A call to action

Kim Marriott, Bongshin Lee, Matthew Butler, Ed Cutrell, Kirsten Ellis, Cagatay Goncu, Marti Hearst, Kathleen McCoy, and Danielle Albers Szafir · 2021

Later among the works it cites.

Research community dynamics behind popular AI benchmarks

Fernando Martínez-Plumed, Pablo Barredo, Seán Ó hÉigeartaigh, and José Hernández-Orallo · 2021

Later among the works it cites.

Acquisition of chess knowledge in AlphaZero, 2021

Original

Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik · 2021

Later among the works it cites.

National name statistical analysis, 2018

Republic of China Ministry of the Interior · 2021

Later among the works it cites.

Cross-task generalization via natural language crowdsourcing instructions, 2021

Original

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi · 2021

Later among the works it cites.

Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations

Karl Mulligan, Robert Frank, and Tal Linzen · 2021

Later among the works it cites.

Deep double descent: Where bigger models and more data hurt

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever · 2021

Later among the works it cites.

Show your work: Scratchpads for intermediate computation with language models

Original

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al · 2021

Later among the works it cites.

Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics

Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov · 2021

Later among the works it cites.

A review of speaker diarization: Recent advances with deep learning, 2021

Original

Tae Jin Park, Naoyuki Kanda, Dimitrios Dimitriadis, Kyu J. Han, Shinji Watanabe, and Shrikanth Narayanan · 2021

Later among the works it cites.

BBQ: A hand-built bias benchmark for question answering, 2021

Original

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman · 2021

Later among the works it cites.

Are NLP models really able to solve simple math word problems?

Arkil Patel, Satwik Bhattamishra, and Navin Goyal · 2021

Later among the works it cites.

Carbon emissions and large neural network training, 2021

Original

David Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean · 2021

Later among the works it cites.

True few-shot learning with language models, 2021

Original

Ethan Perez, Douwe Kiela, and Kyunghyun Cho · 2021

Later among the works it cites.

Few-shot instruction prompts for pretrained language models to detect social biases, 2021

Shrimai Prabhumoye, Rafal Kocielnik, Mohammad Shoeybi, Anima Anandkumar, and Bryan Catanzaro · 2021

Later among the works it cites.

What are the most popular names chinese parents give their babies? a perspective from big data

Qimingtong · 2021

Later among the works it cites.

TIMEDIAL: Temporal commonsense reasoning in dialog

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui · 2021

Later among the works it cites.

Scaling language models: Methods, analysis & insights from training Gopher

Original

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving · 2021

Later among the works it cites.

Med-BERT: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi · 2021

Later among the works it cites.

Decrypting cryptic crosswords: Semantically complex wordplay puzzles as a target for NLP, 2021

Original

Joshua Rozner, Christopher Potts, and Kyle Mahowald · 2021

Later among the works it cites.

XTREME-R: Towards more challenging and nuanced multilingual evaluation

Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson · 2021

Later among the works it cites.

Symbolic behaviour in artificial intelligence, 2021

Original

Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, and David Raposo · 2021

Later among the works it cites.

Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP, 2021

Original

Timo Schick, Sahana Udupa, and Hinrich Schütze · 2021

Later among the works it cites.

Get your vitamin C! Robust fact verification with contrastive evidence

Tal Schuster, Adam Fisch, and Regina Barzilay · 2021

Later among the works it cites.

Towards causal representation learning, 2021

Original

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio · 2021

Later among the works it cites.

Does he wink or does he nod? A challenging benchmark for evaluating word understanding of language models, 2021

Original

Lutfi Kerem Senel and Hinrich Schütze · 2021

Later among the works it cites.

Counterfactual learning in networks: An empirical study of model dependence, 2021

Usman Shahid and Elena Zheleva · 2021

Later among the works it cites.

Retrieval augmentation reduces hallucination in conversation, 2021

Original

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston · 2021

Later among the works it cites.

COM2SENSE: A commonsense reasoning benchmark with complementary sentences

Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng · 2021

Later among the works it cites.

Early detection of freeze damage in navel orange fruit using nondestructive low intensity ultrasound coupled with machine learning

Mahmoud Soltani Firouz, Ali Farahmandi, and Soleiman Hosseinpour · 2021

Later among the works it cites.

Watching a language model learning chess

Andreas Stöckl · 2021

Later among the works it cites.

ChePT – applying deep neural transformer models to chess move prediction and self-commentary, 2021

Colton Swingle, Henry Mellsop, and Alex Langshur · 2021

Later among the works it cites.

Understanding the capabilities, limitations, and societal impact of large language models, 2021

Original

Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli · 2021

Later among the works it cites.

Representing numbers in NLP: A survey and a vision

Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely · 2021

Later among the works it cites.

Metaphor paraphrasing and word sense disambiguation: Toward a new approach to automated metaphor, 2021

Xiaoyu Tong · 2021

Later among the works it cites.

Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective

Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis · 2021

Later among the works it cites.

Learning chess blindfolded: Evaluating language models on state tracking, 2021

Original

Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel · 2021

Later among the works it cites.

GPT-J-6B: A 6 billion parameter autoregressive language model, May 2021

Ben Wang and Aran Komatsuzaki · 2021

Later among the works it cites.

Finetuned language models are zero-shot learners, 2021

Original

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V. Le · 2021

Later among the works it cites.

Language models are few-shot multilingual learners

Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung · 2021

Later among the works it cites.

An embedding method for unseen words considering contextual information and morphological information

Min-Sub Won, YunSeok Choi, Samuel Kim, CheolWon Na, and Jee-Hyong Lee · 2021

Later among the works it cites.

The causal-neural connection: Expressiveness, learnability, and inference, 2021

Original

Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim · 2021

Later among the works it cites.

On hallucination and predictive uncertainty in conditional language generation, 2021

Original

Yijun Xiao and William Yang Wang · 2021

Later among the works it cites.

mT5: A massively multilingual pre-trained text-to-text transformer

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel · 2021

Later among the works it cites.

Calibrate before use: Improving few-shot performance of language models, 2021

Original

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh · 2021

Later among the works it cites.

Neural language models are effective plagiarists, 2022

Original

Stella Biderman and Edward Raff · 2022

Closest in time.

GPT-NeoX-20B: An open-source autoregressive language model

Original

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach · 2022

Closest in time.

PaLM: Scaling language modeling with pathways, 2022

Original

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel · 2022

Closest in time.

Unified scaling laws for routed language models, 2022

Original

Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan · 2022

Closest in time.

Predictability and surprise in large generative models, 2022

Original

Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Dario Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Chris Olah, and Jack Clark · 2022

Closest in time.

EleutherAI/lm-evaluation-harness: v0.2.0, March 2022

Leo Gao, Jonathan Tow, Stella Biderman, Jason Phang, Anish Thite, Fazz, Thomas Wang, Niklas Muennighoff, sdtblck, Jeffrey Hsu, Eric Tang, Charles Foster, uyhcire, Andy Zou, Ben Wang, researcher2, igor0, Kevin Wang, Nicholas Kross, and silentv0x · 2022

Closest in time.

Training compute-optimal large language models, 2022

Original

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre · 2022

Closest in time.

Quality at a glance: An audit of web-crawled multilingual datasets

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi · 2022

Closest in time.

Documenting geographically and contextually diverse data sources: The BigScience catalogue of language data and resources, 2022

Original

Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilić, Nurulaqilla Khamis, Colin Leong, Maraim Masoud, Aitor Soroa, Pedro Ortiz Suarez, Zeerak Talat, Daniel van Strien, and Yacine Jernite · 2022

Closest in time.

Multitask prompted training enables zero-shot task generalization

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush · 2022

Closest in time.

Zero-shot recommendation as language modeling

Damien Sileo, Wout Vossen, and Robbe Raymaekers · 2022

Closest in time.

You reap what you sow: On the challenges of bias evaluation under multilingual settings

Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal · 2022

Closest in time.

LaMDA: Language models for dialog applications, 2022

Original

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc V. Le · 2022

Closest in time.

Chain of thought prompting elicits reasoning in large language models

Original

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc V. Le, and Denny Zhou · 2022

Closest in time.

Designing effective sparse expert models, 2022

Original

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus · 2022

Closest in time.

Against conventional wisdom

Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph · 2027

Closest in time.