Optimised Grouped-Query Attention Mechanism for Transformers

Measuring massive multitask language understanding

Original

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2009

Earlier work this paper cites.

Recursive deep models for semantic compositionality over a sentiment treebank

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C · 2013

Earlier work this paper cites.

Decoupled weight decay regularization

Original

Loshchilov, I. and Hutter, F · 2017

Earlier work this paper cites.

A broad-coverage challenge corpus for sentence understanding through inference

Original

Williams, A., Nangia, N., and Bowman, S. R · 2017

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S · 2018

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019

Cited alongside, same era.

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020

Cited alongside, same era.

Lora: Low-rank adaptation of large language models, 2021

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W · 2021

Cited alongside, same era.

Llama: Open and efficient foundation language models, 2023a

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G

Cited in the paper.

Llama 2: Open foundation and fine-tuned chat models, 2023b

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T

Cited in the paper.

Opt: Open pre-trained transformer language models, 2022

Original

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L · 2022

Later among the works it cites.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S · 2023

Later among the works it cites.

Attention is all you need, 2023

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2023

Later among the works it cites.