Fetching the paper…

Reading the bibliography…

2024

Loop Neural Networks for Parameter Sharing

Ng, Kei-Sing, Wang, Qingchen

Original
PDF
Post on X

Understand

The success of large-scale language models like GPT can be attributed to their ability to efficiently predict the next token in a sequence.

However, these models rely on constant computational effort regardless of the complexity of the token they are predicting, lacking the capacity for iterative refinement.
In this paper, we introduce a novel Loop Neural Network, which achieves better performance by utilizing longer computational time without increasing the model size.
Our approach revisits the input multiple times, refining the prediction by iteratively looping over a subset of the model with residual connections.

Loop Neural Networks for Parameter Sharing · Around

Built on

Depth-adaptive transformer
Original
Elbayad, M., Zeghidour, N., and Usunier, N. (2019) · 1910
Earlier work this paper cites.
Adaptive computation time for recurrent neural networks
Original
Graves, A. (2016) · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J. (2016) · 2016
Earlier work this paper cites.

Similar

Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017) · 2017
Cited alongside, same era.
Universal transformers
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, L. (2019) · 2019
Cited alongside, same era.
Openwebtext corpus
Gokaslan, A. and Cohen, V. (2019) · 2019
Cited alongside, same era.

Then

Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019) · 2019
Later among the works it cites.
Lessons on parameter sharing across layers in transformers
Takase, S. and Kiyono, S. (2023) · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…