Fetching the paper…

Reading the bibliography…

2023

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Yu, Lili, Simig, Dániel, Flaherty, Colin et al.

Original
PDF
Post on X

Understand

Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books.

We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end differentiable modeling of sequences of over one million bytes.
Megabyte segments sequences into patches and uses a local submodel within patches and a global model between patches.
This enables sub-quadratic self-attention, much larger feedforward layers for the same compute, and improved parallelism during decoding -- unlocking better performance at reduced cost for both training and generation.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers · Around

Built on

Transformer-xl: Attentive language models beyond a fixed-length context, 2019
Original
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R · 1901
Earlier work this paper cites.
Compressive transformers for long-range sequence modelling
Original
Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P · 1911
Earlier work this paper cites.
Compressive transformers for long-range sequence modelling
Original
Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P · 1911
Earlier work this paper cites.
Efficient content-based sparse attention with routing transformers, 2020
Original
Roy, A., Saffar, M., Vaswani, A., and Grangier, D · 2003
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Sennrich, R., Haddow, B., and Birch, A · 2015
Earlier work this paper cites.
Pixel Recurrent Neural Networks
Original
Oord, A. v. d., Kalchbrenner, N., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Original
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Mixed precision training
Original
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al · 2017
Earlier work this paper cites.
Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications
Original
Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P · 2017
Earlier work this paper cites.
Parallel wavenet: Fast high-fidelity speech synthesis
Original
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo, L. C., Stimberg, F., Casagrande, N., Grewe, D., Noury, S., Dieleman, S., Elsen, E., Kalchbrenner, N., Zen, H., Graves, A., King, H., Walters, T., Belov, D., and Hassabis, D · 2017
Earlier work this paper cites.
Efficient neural audio synthesis
Original
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., van den Oord, A., Dieleman, S., and Kavukcuoglu, K · 2018
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Original
Kudo, T. and Richardson, J · 2018
Earlier work this paper cites.
A simple method for commonsense reasoning
Original
Trinh, T. H. and Le, Q. V · 2018
Earlier work this paper cites.
Generating long sequences with sparse transformers
Original
Child, R., Gray, S., Radford, A., and Sutskever, I · 2019
Earlier work this paper cites.
PyTorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al · 2019
Earlier work this paper cites.

Similar

Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019
Cited alongside, same era.
Longformer: The long-document transformer
Original
Beltagy, I., Peters, M. E., and Cohan, A · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Rethinking attention with performers
Original
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al · 2020

Then

Linear transformers are secretly fast weight programmers
Schlag, I., Irie, K., and Schmidhuber, J · 2021
Later among the works it cites.
Roformer: Enhanced transformer with rotary position embedding
Original
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y · 2021
Later among the works it cites.
Do long-range language models actually use long-range context?
Original
Sun, S., Krishna, K., Mattarella-Micke, A., and Iyyer, M · 2021
Later among the works it cites.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al · 2022
Later among the works it cites.
Canine: Pre-training an efficient tokenization-free encoder for language representation
Clark, J. H., Garrette, D., Turc, I., and Wieting, J · 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

The pile: An 800gb dataset of diverse text for language modeling, 2020

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C · 2020

Cited alongside, same era.

Scaling laws for neural language models

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020

Cited alongside, same era.

Transformers are rnns: Fast autoregressive transformers with linear attention

Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F · 2020

Cited alongside, same era.

Reformer: The efficient transformer

Kitaev, N., Kaiser, Ł., and Levskaya, A · 2020

Cited alongside, same era.

Shortformer: Better language modeling using shorter inputs

Press, O., Smith, N. A., and Lewis, M · 2020

Cited alongside, same era.

FairScale: A general purpose modular PyTorch library for high performance and large scale training

Baines, M., Bhosale, S., Caggiano, V., Goyal, N., Goyal, S., Ott, M., Lefaudeux, B., Liptchinsky, V., Rabbat, M., Sheiffer, S., Sridhar, A., and Xu, M · 2021

Cited alongside, same era.

Efficiently modeling long sequences with structured state spaces

Gu, A., Goel, K., and Ré, C · 2021

Cited alongside, same era.

Later among the works it cites.

Temporal latent bottleneck: Synthesis of fast and slow processing mechanisms in sequence learning, 2022

Didolkar, A., Gupta, K., Goyal, A., Gundavarapu, N. B., Lamb, A., Ke, N. R., and Bengio, Y · 2022

Later among the works it cites.

Subword-delimited downsampling for better character-level translation, 2022

Edman, L., Toral, A., and van Noord, G · 2022

Later among the works it cites.

General-purpose, long-context autoregressive modeling with perceiver ar

Hawthorne, C., Jaegle, A., Cangea, C., Borgeaud, S., Nash, C., Malinowski, M., Dieleman, S., Vinyals, O., Botvinick, M., Simon, I., et al · 2022

Later among the works it cites.

Training compute-optimal large language models

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al · 2022

Later among the works it cites.

Block-recurrent transformers

Hutchins, D., Schlag, I., Wu, Y., Dyer, E., and Neyshabur, B · 2022

Later among the works it cites.

Mega: moving average equipped gated attention

Original

Ma, X., Zhou, C., Kong, X., He, J., Gui, L., Neubig, G., May, J., and Zettlemoyer, L · 2022

Later among the works it cites.

Simplified state space layers for sequence modeling

Smith, J. T., Warrington, A., and Linderman, S. W · 2022

Later among the works it cites.

Memorizing transformers

Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C · 2022

Later among the works it cites.

A systematic analysis of vocabulary and bpe settings for optimal fine-tuning of nmt: A case study of in-domain translation

Sharami, J., Shterionov, D., and Spronck, P · 2023

Closest in time.

Llama: Open and efficient foundation language models

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al · 2023

Closest in time.