Fetching the paper…
Reading the bibliography…
Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.
Syntax in universal translation
Itiroo Sakai · 1961
Earlier work this paper cites.
Trainable grammars for speech recognition
James K Baker · 1979
Earlier work this paper cites.
Backward feature correction: How deep learning performs deep learning
Zeyuan Allen-Zhu and Yuanzhi Li · 2001
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn Treebank
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz · 2004
Earlier work this paper cites.
Introduction to the Theory of Computation
Michael Sipser · 2012
Earlier work this paper cites.
Explicit and implicit syntactic features for text classification
Matt Post and Shane Bergsma · 2013
Earlier work this paper cites.
Do gans actually learn the distribution? an empirical study
Sanjeev Arora and Yi Zhang · 2017
Earlier work this paper cites.
A convergence theory for deep learning via over-parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2019
Earlier work this paper cites.
A structural probe for finding syntax in word representations
John Hewitt and Christopher D. Manning · 2019
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Earlier work this paper cites.
Deberta: Decoding-enhanced bert with disentangled attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen · 2020
Earlier work this paper cites.
Emergent linguistic structure in artificial neural networks trained by self-supervision
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy · 2020
Earlier work this paper cites.
An empirical study on robustness to spurious correlations using pre-trained language models
Lifu Tu, Garima Lalwani, Spandana Gella, and He He · 2020
Earlier work this paper cites.
Parsing as pretraining
David Vilares, Michalina Strzyz, Anders Søgaard, and Carlos Gómez-Rodríguez · 2020
Cited alongside, same era.
Perturbed masking: Parameter-free probing for analyzing and interpreting bert
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu · 2020
Cited alongside, same era.
A mathematical framework for transformer circuits
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al · 2021
Cited alongside, same era.
Do syntactic probes probe syntax? experiments with jabberwocky probing
Rowan Hall Maudslay and Ryan Cotterell · 2021
Cited alongside, same era.
Evaluating the robustness of neural language models to input perturbations
Milad Moradi and Matthias Samwald · 2021
Cited alongside, same era.
Neural networks and the chomsky hierarchy
Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, et al · 2023
Closest in time.
Characterizing intrinsic compositionality in transformers with tree projections
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning · 2023
Closest in time.
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt · 2023
Closest in time.
Gpt-4 technical report, 2023
OpenAI · 2023
Closest in time.
Can transformers learn to solve problems recursively?
Shizhuo Dylan Zhang, Curt Tigges, Stella Biderman, Maxim Raginsky, and Talia Ringer · 2023
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Ofir Press, Noah A Smith, and Mike Lewis · 2021
Cited alongside, same era.
Roformer: Enhanced transformer with rotary position embedding, 2021
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu · 2021
Cited alongside, same era.
Probing for constituency structure in neural language models
David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad · 2022
Cited alongside, same era.
GPT-NeoX-20B: An open-source autoregressive language model
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach · 2022
Cited alongside, same era.
Learning hierarchical structures with differentiable nondeterministic stacks
Brian DuSell and David Chiang · 2022
Cited alongside, same era.
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al · 2022
Cited alongside, same era.
Learning bounded context-free-grammar via lstm and the transformer: Difference and the explanations
Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, and Jishen Zhao · 2022
Cited alongside, same era.
Haoyu Zhao, Abhishek Panigrahi, Rong Ge, and Sanjeev Arora · 2023
Closest in time.
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
Zeyuan Allen-Zhu and Yuanzhi Li · 2024
Closest in time.
Repeat after me: Transformers are better than state space models at copying
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach · 2024
Closest in time.
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Zeyuan Allen-Zhu · 2025
Closest in time.
Physics of Language Models: Part 3.2, Knowledge Manipulation
Zeyuan Allen-Zhu and Yuanzhi Li · 2025
Closest in time.
Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
Zeyuan Allen-Zhu and Yuanzhi Li · 2025
Closest in time.
Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu · 2025
Closest in time.
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu · 2025
Closest in time.