Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020 · 1901

Earlier work this paper cites.

Frequency principle: Fourier analysis sheds light on deep neural networks

Original

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. 2019 · 1901

Earlier work this paper cites.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

Original

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 1905

Earlier work this paper cites.

A multiscale visualization of attention in the transformer model

Original

Jesse Vig. 2019 · 1906

Earlier work this paper cites.

Revealing the dark secrets of bert

Original

Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019 · 1908

Earlier work this paper cites.

Clutrr: A diagnostic benchmark for inductive reasoning from text

Original

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. 2019 · 1908

Earlier work this paper cites.

Transformers as soft reasoners over language

Original

Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020 · 2002

Earlier work this paper cites.

Attention is not only a weight: Analyzing transformers with vector norms

Original

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020 · 2004

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

A convergence analysis of gradient descent for deep linear neural networks

Original

Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018 · 2018

Earlier work this paper cites.

Generating wikipedia by summarizing long sequences

Original

Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018 · 2018

Earlier work this paper cites.

How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective

Lei Wu, Chao Ma, and 1 others. 2018 · 2018

Earlier work this paper cites.

The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects

Original

Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. 2018 · 2018

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019 · 2019

Earlier work this paper cites.

Implicit regularization of random feature models

Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. 2020 · 2020

Earlier work this paper cites.

Investigating gender bias in language models using causal mediation analysis

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020 · 2020

Earlier work this paper cites.

Advancing mathematics by guiding human intuition with ai

Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, and 1 others. 2021 · 2021

Earlier work this paper cites.

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021 · 2021

Earlier work this paper cites.

On the validity of modeling sgd with stochastic differential equations (sdes)

Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. 2021 · 2021

Earlier work this paper cites.

Phase diagram for two-layer relu neural networks at infinite-width limit

Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. 2021 · 2021

Earlier work this paper cites.

Transformers can do bayesian inference

Original

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. 2021 · 2021

Earlier work this paper cites.

Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization

Original

Chiyuan Zhang, Maithra Raghu, Jon Kleinberg, and Samy Bengio. 2021 · 2021

Earlier work this paper cites.

Learning to reason with neural networks: Generalization, unseen data and boolean measures

Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, and Chiyuan Zhang. 2022 · 2022

Earlier work this paper cites.

Understanding gradient descent on the edge of stability in deep learning

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. 2022 · 2022

Earlier work this paper cites.

Multi-step deductive reasoning over natural language: An empirical study on out-of-distribution generalisation

Original

Qiming Bao, Alex Yuxuan Peng, Tim Hartill, Neset Tan, Zhenyun Deng, Michael Witbrock, and Jiamou Liu. 2022 · 2022

Earlier work this paper cites.

Analyzing transformers in embedding space

Original

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022 · 2022

Cited alongside, same era.

A survey on in-context learning

Original

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022 · 2022

Cited alongside, same era.

What can transformers learn in-context? a case study of simple function classes

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022 · 2022

Cited alongside, same era.

What changed? investigating debiasing methods using causal mediation analysis

Original

Sullam Jeoung and Jana Diesner. 2022 · 2022

Cited alongside, same era.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

The implicit regularization of dynamical stability in stochastic gradient descent

Lei Wu and Weijie J Su. 2023 · 2023

Later among the works it cites.

Beam retrieval: General end-to-end retrieval for multi-hop question answering

Original

Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Yong Liu, and Shen Huang. 2023 · 2023

Later among the works it cites.

How far can transformers reason? the locality barrier and inductive scratchpad

Original

Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Colin Sandon, and Omid Saremi. 2024 · 2024

Closest in time.

Phi-3 technical report: A highly capable language model locally on your phone

Original

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, and 1 others. 2024 · 2024

Closest in time.

On the benefits of rank in attention layers