Fetching the paper…
Reading the bibliography…
Large Language Models (LLMs) face limitations due to the high demand on GPU memory and computational resources when handling long contexts.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, March 2020
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B · 1909
Earlier work this paper cites.
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection, December 2019
Zhao, G., Lin, J., Zhang, Z., Ren, X., Su, Q., and Sun, X · 1912
Earlier work this paper cites.
Big Bird: Transformers for Longer Sequences, January 2021
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A · 2007
Earlier work this paper cites.
An Introduction to Convolutional Neural Networks, December 2015
O’Shea, K. and Nash, R · 2015
Earlier work this paper cites.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J · 2020
Earlier work this paper cites.
Training Verifiers to Solve Math Word Problems, November 2021
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J · 2021
Cited alongside, same era.
Memory-efficient Transformers via Top-$k$ Attention, June 2021
Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J · 2021
Cited alongside, same era.
Learning to Reason and Memorize with Self-Notes, October 2023
Lanchantin, J., Toshniwal, S., Weston, J., Szlam, A., and Sukhbaatar, S · 2023
Cited alongside, same era.
Learning to Compress Prompts with Gist Tokens, July 2023
Mu, J., Li, X. L., and Goodman, N · 2023
Cited alongside, same era.
SparQ Attention: Bandwidth-Efficient LLM Inference, December 2023
Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D · 2023
Later among the works it cites.
Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T · 2023
Later among the works it cites.
Baichuan 2: Open Large-scale Language Models, September 2023
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., Yang, F., Deng, F., Wang, F., Liu, F., Ai, G., Dong, G., Zhao, H., Xu, H., Sun, H., Zhang, H., Liu, H., Ji, J., Xie, J., Dai, J., Fang, K., Su, L., Song, L., Liu, L., Ru, L., Ma, L., Wang, M., Liu, M., Lin, M., Nie, N., Guo, P., Sun, R., Zhang, T., Li, T., Li, T., Cheng, W., Chen, W., Zeng, X., Wang, X., Chen, X., Men, X., Yu, X., Pan, X., Shen, Y., Wang, Y., Li, Y., Jiang, Y., Gao, Y., Zhang, Y., Zhou, Z., and Wu, Z · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E
Cited in the paper.
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models, December 2023b
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L
Cited in the paper.
Later among the works it cites.
H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, July 2023
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B · 2023
Later among the works it cites.