Fetching the paper…
Reading the bibliography…
Prompting is now a dominant method for evaluating the linguistic knowledge of large language models (LLMs).
Language Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901
Earlier work this paper cites.
A Model and an Hypothesis for Language Structure
Victor H. Yngve. 1960 · 1960
Earlier work this paper cites.
Aspects of the Theory of Syntax
Noam Chomsky. 1965 · 1965
Earlier work this paper cites.
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016 · 2016
Earlier work this paper cites.
On Calibration of Modern Neural Networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017 · 2017
Earlier work this paper cites.
Toward a universal decoder of linguistic meaning from brain activation
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018 · 2018
Earlier work this paper cites.
Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality
Paolo Vassallo, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2018 · 2018
Earlier work this paper cites.
Neural language models as psycholinguistic subjects: Representations of syntactic state
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019 · 2019
Earlier work this paper cites.
Performance vs. competence in human–machine comparisons
Chaz Firestone. 2020 · 2020
Earlier work this paper cites.
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020 · 2020
Earlier work this paper cites.
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020 · 2020
Earlier work this paper cites.
Masked Language Model Scoring
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020 · 2020
Earlier work this paper cites.
BLiMP: The Benchmark of Linguistic Minimal Pairs for English
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020 · 2020
Earlier work this paper cites.
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models
Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021 · 2021
Earlier work this paper cites.
(What) Can Deep Learning Contribute to Theoretical Linguistics?
Gabe Dupre. 2021 · 2021
Earlier work this paper cites.
Revisiting the Calibration of Modern Neural Networks
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021 · 2021
Earlier work this paper cites.
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021 · 2021
Earlier work this paper cites.
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models
Yiwen Wang, Jennifer Hu, Roger Levy, and Peng Qian. 2021 · 2021
Earlier work this paper cites.
On the proper role of linguistically-oriented deep net analysis in linguistic theorizing
Marco Baroni. 2022 · 2022
Cited alongside, same era.
LMPriors: Pre-Trained Language Models as Task-Specific Priors
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. 2022 · 2022
Cited alongside, same era.
Scaling Instruction-Finetuned Language Models
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022
Cited alongside, same era.
A resource-rational model of human processing of recursive linguistic structure
Michael Hahn, Richard Futrell, Roger P. Levy, and Edward Gibson. 2022 · 2022
Cited alongside, same era.
Mysteries of mode collapse
Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs
Gašper Beguš, Maksymilian Dąbkowski, and Ryan Rhodes. 2023 · 2023
Closest in time.
Large Language Models Demonstrate the Potential of Statistical Learning in Language
Pablo Contreras Kallens, Ross Deans Kristensen-McLachlan, and Morten H. Christiansen. 2023 · 2023
Closest in time.
Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. 2023 · 2023
Closest in time.
Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi (2023)
Roni Katzir. 2023 · 2023
Closest in time.
Andrew Kyle Lampinen. 2023 · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Janus. 2022 · 2022
Cited alongside, same era.
Language Models (Mostly) Know What They Know
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022 · 2022
Cited alongside, same era.
Event knowledge in large language models: the gap between the impossible and the unlikely
Carina Kauf, Anna A. Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S. She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022 · 2022
Cited alongside, same era.
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022 · 2022
Cited alongside, same era.
Large Language Models are Zero-Shot Reasoners
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022
Cited alongside, same era.
Large Language Models and the Argument From the Poverty of the Stimulus
Nur Lan, Emmanuel Chemla, and Roni Katzir. 2022 · 2022
Cited alongside, same era.
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022 · 2022
Cited alongside, same era.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022
Cited alongside, same era.
Closest in time.
LaMPP: Language Models as Probabilistic Priors for Perception and Action
Belinda Z. Li, William Chen, Pratyusha Sharma, and Jacob Andreas. 2023 · 2023
Closest in time.
Evaluating statistical language models as pragmatic reasoners
Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B. Tenenbaum. 2023 · 2023
Closest in time.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. 2023 · 2023
Closest in time.
A Response to Piantadosi (2023)
Daniel Milway. 2023 · 2023
Closest in time.
The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023 · 2023
Closest in time.
Notes on Large Language Models and Linguistic Theory
Elliot Murphy. 2023 · 2023
Closest in time.
Modern language models refute Chomsky’s approach to language
Steven T. Piantadosi. 2023 · 2023
Closest in time.
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023 · 2023
Closest in time.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023 · 2023
Closest in time.
Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
Tomer Ullman. 2023 · 2023
Closest in time.
Emergent Analogical Reasoning in Large Language Models
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023 · 2023
Closest in time.
Are Language Models Worse than Humans at Following Prompts? It’s Complicated
Albert Webson, Alyssa Marie Loo, Qinan Yu, and Ellie Pavlick. 2023 · 2023
Closest in time.