AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design

B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal brain surgeon and general network pruning,” IEEE International Conference on Neural Networks , pp. 293–299 vol.1, 1993

1993

Earlier work this paper cites.

S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Commun. ACM , vol. 52, no. 4, p. 65–76, apr 2009

2009

Earlier work this paper cites.

P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient transfer learning,” ArXiv , vol. abs/1611.06440, 2016

Original

2016

Earlier work this paper cites.

S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” ArXiv , vol. abs/1609.07843, 2016

Original

2016

Earlier work this paper cites.

Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” 2017 IEEE International Conference on Computer Vision (ICCV) , pp. 1398–1406, 2017

2017

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019

2019

Earlier work this paper cites.

C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res. , vol. 21, pp. 140:1–140:67, 2019

2019

Earlier work this paper cites.

M. Chen et al. , “Evaluating large language models trained on code,” ArXiv , vol. abs/2107.03374, 2021

Original

2021

Earlier work this paper cites.

T. J. Ham, Y. Lee, S. H. Seo, S.-U. Kim, H. Choi, S. Jung, and J. W. Lee, “Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,” 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) , pp. 692–705, 2021

2021

Earlier work this paper cites.

L. Lu, Y. Jin, H. Bi, Z. Luo, P. Li, T. Wang, and Y. Liang, “Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture,” MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture , 2021

2021

Earlier work this paper cites.

O. Press, N. A. Smith, and M. Lewis, “Train short, test long: Attention with linear biases enables input length extrapolation,” ArXiv , vol. abs/2108.12409, 2021

Original

2021

Earlier work this paper cites.

A. K. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius, “Accelerating sparse deep neural networks,” ArXiv , vol. abs/2104.08378, 2021

Original

2021

Earlier work this paper cites.

S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi, “Metaicl: Learning to learn in context,” ArXiv , vol. abs/2110.15943, 2021

Original

2021

Earlier work this paper cites.

J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” ArXiv , vol. abs/2109.01652, 2021

Original

2021

Earlier work this paper cites.

S.-C. Kao, S. Subramanian, G. Agrawal, A. Yazdanbakhsh, and T. Krishna, “Flat: An optimized dataflow for mitigating attention bottlenecks,” Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , 2021

2021

Earlier work this paper cites.

Z. Liu, K.-T. Cheng, D. Huang, E. P. Xing, and Z. Shen, “Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 4932–4942, 2021

2021

Earlier work this paper cites.

J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” ArXiv , vol. abs/2106.09685, 2021

Original

2021

Earlier work this paper cites.

S. Zhang et al. , “Opt: Open pre-trained transformer language models,” ArXiv , vol. abs/2205.01068, 2022

Original

2022

Earlier work this paper cites.

R. Y. Aminabadi et al. , “Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale,” SC22: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1–15, 2022

2022

Earlier work this paper cites.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R’e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” ArXiv , vol. abs/2205.14135, 2022

Original

2022

Earlier work this paper cites.

R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” ArXiv , vol. abs/2211.05102, 2022

Original

2022

Earlier work this paper cites.

G. Xiao, J. Lin, M. Seznec, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” ArXiv , vol. abs/2211.10438, 2022

Original

2022

Cited alongside, same era.

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” ArXiv , vol. abs/2205.05638, 2022

Original

2022

Cited alongside, same era.

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” ArXiv , vol. abs/2210.17323, 2022

Original

2022

Cited alongside, same era.

S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, and J.-Y. Kim, “Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation,” 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) , pp. 616–630, 2022

2022

Cited alongside, same era.

X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” ArXiv , vol. abs/2305.11627, 2023

Original

2023

Later among the works it cites.

Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Ré, and B. Chen, “Deja vu: Contextual sparsity for efficient llms at inference time,” in International Conference on Machine Learning , 2023

2023

Later among the works it cites.

Y. Li, Y. Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao, “Losparse: Structured compression of large language models based on low-rank and sparse approximation,” in International Conference on Machine Learning , 2023

2023

Later among the works it cites.

W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. J. Qiao, and P. Luo, “Omniquant: Omnidirectionally calibrated quantization for large language models,” ArXiv , vol. abs/2308.13137, 2023

Original

2023

Later among the works it cites.