Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models

Compressive transformers for long-range sequence modelling

Original

J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap · 1911

Earlier work this paper cites.

Decoupled weight decay regularization

Original

I. Loshchilov and F. Hutter · 2017

Earlier work this paper cites.

Attention is all you need

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017

Earlier work this paper cites.

Dynamic evaluation of neural sequence models

B. Krause, E. Kahembwe, I. Murray, and S. Renals · 2018

Earlier work this paper cites.

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Original

T. Kudo and J. Richardson · 2018

Cited alongside, same era.

Transformer-xl: Attentive language models beyond a fixed-length context

Original

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov · 2019

Cited alongside, same era.

Dynamic evaluation of transformer language models

Original

B. Krause, E. Kahembwe, I. Murray, and S. Renals · 2019

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2020

Later among the works it cites.

On layer normalization in the transformer architecture

R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu · 2020

Later among the works it cites.

Lora: Low-rank adaptation of large language models

Original

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen · 2021

Later among the works it cites.