Fetching the paper…

Reading the bibliography…

2023

Improving Length-Generalization in Transformers via Task Hinting

Awasthi, Pranjal, Gupta, Anupam

Original
PDF
Post on X

Understand

It has been observed in recent years that transformers have problems with length generalization for certain types of reasoning and arithmetic tasks.

In particular, the performance of a transformer model trained on tasks (say addition) up to a certain length (e.g., 5 digit numbers) drops sharply when applied to longer instances of the same problem.
This work proposes an approach based on task hinting towards addressing length generalization.
Our key idea is that while training the model on task-specific data, it is helpful to simultaneously train the model to solve a simpler but related auxiliary task as well.

Improving Length-Generalization in Transformers via Task Hinting · Around

Built on

Constant depth reducibility
Ashok K. Chandra, Larry J. Stockmeyer, and Uzi Vishkin · 1984
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
Ilya Loshchilov and Frank Hutter · 2016
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
Location attention for extrapolation to longer sequences
Original
Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni · 2019
Earlier work this paper cites.
Albert: A lite BERT for self-supervised learning of language representations
Original
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
The EOS decision and length extrapolation
Original
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning · 2020
Earlier work this paper cites.

Similar

Multi-task learning with deep neural networks: A survey
Original
Michael Crawshaw · 2020
Cited alongside, same era.
Show your work: Scratchpads for intermediate computation with language models
Original
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al · 2021
Cited alongside, same era.
Train short, test long: Attention with linear biases enables input length extrapolation
Original
Ofir Press, Noah A Smith, and Mike Lewis · 2021
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Original
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al · 2022
Cited alongside, same era.

Then

Saturated transformers are constant-depth threshold circuits
William Merrill, Ashish Sabharwal, and Noah A Smith · 2022
Later among the works it cites.
Towards generalist biomedical AI
Original
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al · 2023
Closest in time.
Length generalization in arithmetic transformers
Original
Samy Jelassi, Stéphane d’Ascoli, Carles Domingo-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton · 2023
Closest in time.
Generalization on the unseen, logic reasoning and degree curriculum
Original
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk · 2023
Closest in time.
Auto-regressive next-token predictors are universal learners
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Pali: A jointly-scaled multilingual language-image model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al · 2022

Cited alongside, same era.

Exploring length generalization in large language models

Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur · 2022

Cited alongside, same era.

Unveiling transformers with LEGO: a synthetic reasoning task

Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022

Cited alongside, same era.

Emergent world representations: Exploring a sequence model trained on a synthetic task

Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg · 2022

Cited alongside, same era.

Progress measures for grokking via mechanistic interpretability

Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt

Cited in the paper.

Emergent linear representations in world models of self-supervised sequence models, 2023b

Neel Nanda, Andrew Lee, and Martin Wattenberg

Cited in the paper.

Eran Malach · 2023

Closest in time.

Goat: Fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks

Tiedong Liu and Bryan Kian Hsiang Low · 2023

Closest in time.

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023

Closest in time.

Flax: A neural network library and ecosystem for JAX, 2023

Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee · 2023

Closest in time.