Zamba: A Compact 7B SSM Hybrid Model

Adaptive mixtures of local experts

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991) · 1991

Earlier work this paper cites.

Learning and development in neural networks: The importance of starting small

Elman, J. L. (1993) · 1993

Earlier work this paper cites.

Scaling laws for neural language models

Original

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020) · 2001

Earlier work this paper cites.

Language models are few-shot learners

Original

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020) · 2005

Earlier work this paper cites.

Curriculum learning

Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009) · 2009

Earlier work this paper cites.

An image is worth 16x16 words: Transformers for image recognition at scale

Original

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2020) · 2010

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2016) · 2016

Earlier work this paper cites.

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017) · 2017

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018) · 2018

Earlier work this paper cites.

Universal transformers

Original

Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. (2018) · 2018

Earlier work this paper cites.

Curriculum learning for natural answer generation

Liu, C., He, S., Liu, K., and Zhao, J. (2018) · 2018

Earlier work this paper cites.

The pile: An 800gb dataset of diverse text for language modeling

Original

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2020) · 2020

Earlier work this paper cites.

Hippo: Recurrent memory with optimal polynomial projections

Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. (2020) · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020) · 2020

Earlier work this paper cites.

Going in circles is the way forward: the role of recurrence in visual inference

van Bergen, R. S. and Kriegeskorte, N. (2020) · 2020

Earlier work this paper cites.

The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation

Whittington, J. C. R., Muller, T. H., Mark, S., Barry, C., Burgess, N., Behrens, T. E. E., Chen, G., Barry, C., Burgess, N., and Behrens, T. E. E. (2020) · 2020

Earlier work this paper cites.

Measuring massive multitask language understanding

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021) · 2021

Earlier work this paper cites.

Relating transformers to models and neural representations of the hippocampal formation

Whittington, J. C. R., Warren, J., and Behrens, T. E. J. (2021) · 2021

Earlier work this paper cites.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

Fedus, W., Zoph, B., and Shazeer, N. (2022) · 2022

Earlier work this paper cites.

Training compute-optimal large language models

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. (2022) · 2022

Earlier work this paper cites.

Scaling language models: Methods, analysis & insights from training gopher

Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. (2022) · 2022

Earlier work this paper cites.

Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. (2022) · 2022

Cited alongside, same era.

Scaling mlps: A tale of inductive bias

Bachmann, G., Anagnostidis, S., and Hofmann, T. (2023) · 2023

Cited alongside, same era.

Pythia: A suite for analyzing large language models across training and scaling

Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · 2023

Cited alongside, same era.

Flashattention-2: Faster attention with better parallelism and work partitioning

Dao, T. (2023) · 2023

Cited alongside, same era.

A framework for few-shot language model evaluation

Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2023) · 2023

Cited alongside, same era.

Phi-3 technical report: A highly capable language model locally on your phone

Original

Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. (2024) · 2024

Closest in time.

Blackmamba: Mixture of experts for state-space models

Original

Anthony, Q., Tokpanov, Y., Glorioso, P., and Millidge, B. (2024) · 2024

Closest in time.

Griffin: Mixing gated linear recurrences with local attention for efficient language models

De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., Freitas, N. D., and Gulcehre, C. (2024) · 2024

Closest in time.

Gemma: Open models based on gemini research and technology

Original

Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024) · 2024

Closest in time.