Fetching the paper…

Reading the bibliography…

2022

GPT-Neo for commonsense reasoning -- a theoretical and practical lens

Kashyap, Rohan, Kashyap, Vivek, P., Narendra C.

Original
PDF
Post on X

Understand

Recent work has demonstrated substantial gains in pre-training large-language models (LLMs) followed by supervised fine-tuning on the downstream task.

In this paper, we evaluate the performance of the GPT-neo model using $6$ commonsense reasoning benchmark tasks.
We aim to examine the performance of smaller models using the GPT-neo models against several larger model baselines such as GPT-$3$, Llama-$2$, MPT and Falcon.
Upon fine-tuning with the appropriate set of hyperparameters, our model achieves competitive accuracy on several tasks.

GPT-Neo for commonsense reasoning -- a theoretical and practical lens · Around

Built on

Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901
1901
Earlier work this paper cites.
Mostafazadeh, Nasrin, et al. "A corpus and evaluation framework for deeper understanding of commonsense stories." arXiv preprint arXiv:1604.01696 (2016)
Original
2016
Earlier work this paper cites.
Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017)
2017
Earlier work this paper cites.
Madry, Aleksander, et al. "Towards deep learning models resistant to adversarial attacks." arXiv preprint arXiv:1706.06083 (2017)
Original
2017
Earlier work this paper cites.
Poggio, Tomaso, et al. "Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review." International Journal of Automation and Computing 14.5 (2017): 503-519
2017
Earlier work this paper cites.
Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." arXiv preprint arXiv:1701.06538 (2017)
Original
2017
Earlier work this paper cites.
Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018)
Original
2018
Earlier work this paper cites.
Zellers, Rowan, et al. "Swag: A large-scale adversarial dataset for grounded commonsense inference." arXiv preprint arXiv:1808.05326 (2018)
Original
2018
Earlier work this paper cites.
Chollet, François. "On the measure of intelligence." arXiv preprint arXiv:1911.01547 (2019)
Original
2019
Earlier work this paper cites.
Rajani, Nazneen Fatema, et al. "Explain yourself! leveraging language models for commonsense reasoning." arXiv preprint arXiv:1906.02361 (2019)
Original
2019
Earlier work this paper cites.
Zellers, Rowan, et al. "HellaSwag: Can a machine really finish your sentence?." arXiv preprint arXiv:1905.07830 (2019)
Original
2019
Earlier work this paper cites.
Huang, Lifu, et al. "Cosmos QA: Machine reading comprehension with contextual commonsense reasoning." arXiv preprint arXiv:1909.00277 (2019)
Original
2019
Earlier work this paper cites.
Radford, Alec, et al. "Language models are unsupervised multitask learners." OpenAI blog 1.8 (2019): 9
2019
Earlier work this paper cites.
Ott, Myle, et al. "fairseq: A fast, extensible toolkit for sequence modeling." arXiv preprint arXiv:1904.01038 (2019)
Original
2019
Earlier work this paper cites.
Ho, Jonathan, et al. "Axial attention in multidimensional transformers." arXiv preprint arXiv:1912.12180 (2019)
Original
2019
Earlier work this paper cites.
Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020)
Original
2020
Earlier work this paper cites.
Bisk, Yonatan, et al. "Piqa: Reasoning about physical commonsense in natural language." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020
2020
Earlier work this paper cites.
Zhou, Xuhui, et al. "Evaluating commonsense in pre-trained language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020
2020
Earlier work this paper cites.
Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020)
Original
2020
Earlier work this paper cites.

Similar

Sagawa, Shiori, et al. "An investigation of why overparameterization exacerbates spurious correlations." International Conference on Machine Learning. PMLR, 2020
2020
Cited alongside, same era.
Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." The Journal of Machine Learning Research 21.1 (2020): 5485-5551
2020
Cited alongside, same era.
Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150 (2020)
Original
2020
Cited alongside, same era.
Zhou, Xuhui, et al. "Evaluating commonsense in pre-trained language models." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020
2020
Cited alongside, same era.
Sakaguchi, Keisuke, et al. "Winogrande: An adversarial winograd schema challenge at scale." Communications of the ACM 64.9 (2021): 99-106

Then

Li, Xiang Lorraine, et al. "A systematic investigation of commonsense knowledge in large language models." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022
2022
Closest in time.
Arora, Simran, et al. "Ask me anything: A simple strategy for prompting language models." arXiv preprint arXiv:2210.02441 (2022)
Original
2022
Closest in time.
Chung, Hyung Won, et al. "Scaling instruction-finetuned language models." arXiv preprint arXiv:2210.11416 (2022)
Original
2022
Closest in time.
Olsson, Catherine, et al. "In-context learning and induction heads." arXiv preprint arXiv:2209.11895 (2022)
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2021

Cited alongside, same era.

Wei, Jason, et al. "Finetuned language models are zero-shot learners." arXiv preprint arXiv:2109.01652 (2021)

Original

2021

Cited alongside, same era.

Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003

2021

Cited alongside, same era.

Bubeck, Sébastien, and Mark Sellke. "A universal law of robustness via isoperimetry." Advances in Neural Information Processing Systems 34 (2021): 28811-28822

2021

Cited alongside, same era.

Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021)

2021

Cited alongside, same era.

Geva, Mor, et al. "Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies." Transactions of the Association for Computational Linguistics 9 (2021): 346-361

2021

Cited alongside, same era.

Poth, Clifton, et al. "What to pre-train on? efficient intermediate task selection." arXiv preprint arXiv:2104.08247 (2021)

2021

Cited alongside, same era.

Elhage, Nelson, et al. "A mathematical framework for transformer circuits." Transformer Circuits Thread 1 (2021)

2021

Cited alongside, same era.

Razeghi, Yasaman, et al. "Impact of pretraining term frequencies on few-shot numerical reasoning." Findings of the Association for Computational Linguistics: EMNLP 2022. 2022

2022

Closest in time.

Huang, Jie, and Kevin Chen-Chuan Chang. "Towards reasoning in large language models: A survey." arXiv preprint arXiv:2212.10403 (2022)

2022

Closest in time.

Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023)

2023

Closest in time.

Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023)

2023

Closest in time.

Bai, Yu, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. "Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection." arXiv preprint arXiv:2306.04637 (2023)

2023

Closest in time.

Allen-Zhu, Zeyuan, and Yuanzhi Li. "Physics of Language Models: Part 1, Context-Free Grammar." arXiv preprint arXiv:2305.13673 (2023)

2023

Closest in time.

Arora, Sanjeev, and Anirudh Goyal. "A theory for emergence of complex skills in language models." arXiv preprint arXiv:2307.15936 (2023)

2023

Closest in time.

Von Oswald, Johannes, et al. "Transformers learn in-context by gradient descent." International Conference on Machine Learning. PMLR, 2023

2023

Closest in time.

Penedo, Guilherme, et al. "The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only." arXiv preprint arXiv:2306.01116 (2023)

2023

Closest in time.

Angelica Chen, et al. "Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs." arXiv:2309.07311 (2023)

2023

Closest in time.

Refinetti, Maria, Alessandro Ingrosso, and Sebastian Goldt. "Neural networks trained with SGD learn distributions of increasing complexity." International Conference on Machine Learning. PMLR, 2023

2023

Closest in time.

Mitchell, Melanie, and David C. Krakauer. "The debate over understanding in AI’s large language models." Proceedings of the National Academy of Sciences 120.13 (2023)

2023

Closest in time.

Mitchell, Melanie. "How do we know how smart AI systems are?." Science 381.6654 (2023)

2023

Closest in time.

Wu, Zhaofeng, et al. "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks." arXiv preprint arXiv:2307.02477 (2023)

2023

Closest in time.