Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

Pareto, V.: Cours D’économie Politique vol. 1. Librairie Droz, ??? (1964)

1964

Earlier work this paper cites.

Burton, F.W.: Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers 100

1985

Earlier work this paper cites.

LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Advances in neural information processing systems 2

1989

Earlier work this paper cites.

Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artificial intelligence review 18

2002

Earlier work this paper cites.

Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)

Original

2015

Earlier work this paper cites.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (2016)

2016

Earlier work this paper cites.

Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

Original

2016

Earlier work this paper cites.

Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)

Original

2016

Earlier work this paper cites.

Teerapittayanon, S., McDanel, B., Kung, H.-T.: Branchynet: Fast inference via early exiting from deep neural networks. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464–2469 (2016). IEEE

2016

Earlier work this paper cites.

Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Yepes, A.J., Koehn, P., Logacheva, V., Monz, C., et al

2016

Earlier work this paper cites.

Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)

Original

2016

Earlier work this paper cites.

Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al.: Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016)

Original

2016

Earlier work this paper cites.

Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: Vision and challenges. IEEE internet of things journal 3

2016

Earlier work this paper cites.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30

2017

Earlier work this paper cites.

Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)

Original

2017

Earlier work this paper cites.

Gunantara, N.: A review of multi-objective optimization: Methods and its applications. Cogent Engineering 5

2018

Earlier work this paper cites.

Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al.: Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems 31

2018

Earlier work this paper cites.

Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al.: Mixed precision training. The International Conference on Learning Representation (2018)

2018

Earlier work this paper cites.

Johnson, T.B., Guestrin, C.: Training deep models faster with robust, approximate importance sampling. Advances in Neural Information Processing Systems 31

2018

Earlier work this paper cites.

Katharopoulos, A., Fleuret, F.: Not all samples are created equal: Deep learning with importance sampling. In: International Conference on Machine Learning, pp. 2525–2534 (2018). PMLR

2018

Earlier work this paper cites.

Su, D., Zhang, H., Chen, H., Yi, J., Chen, P.-Y., Gao, Y.: Is robustness the cost of accuracy?–a comprehensive study on the robustness of 18 deep image classification models. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 631–648 (2018)

2018

Earlier work this paper cites.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 (2018)

Original

2018

Earlier work this paper cites.

Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 (2018)

Original

2018

Earlier work this paper cites.

Ying, X.: An overview of overfitting and its solutions. In: Journal of Physics: Conference Series, vol. 1168, p. 022022 (2019). IOP Publishing

2019

Earlier work this paper cites.

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

Original

2019

Earlier work this paper cites.

Long, X., Ben, Z., Liu, Y.: A survey of related research on compression and acceleration of deep neural networks. In: Journal of Physics: Conference Series, vol. 1213, p. 052003 (2019). IOP Publishing

2019

Earlier work this paper cites.

Kenton, J.D.M.-W.C., Toutanova, L.K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of naacL-HLT, vol. 1, p. 2 (2019)

2019

Earlier work this paper cites.

Jang, H., Kim, J., Jo, J.-E., Lee, J., Kim, J.: Mnnfast: A fast and scalable system architecture for memory-augmented neural networks. In: Proceedings of the 46th International Symposium on Computer Architecture, pp. 250–263 (2019)

2019

Earlier work this paper cites.

Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q.V., Wu, Y., et al.: Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32

2019

Earlier work this paper cites.

Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N.R., Ganger, G.R., Gibbons, P.B., Zaharia, M.: Pipedream: Generalized pipeline parallelism for dnn training. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 1–15 (2019)

2019

Earlier work this paper cites.

Song, K., Tan, X., Qin, T., Lu, J., Liu, T.-Y.: Mass: Masked sequence to sequence pre-training for language generation. In: International Conference on Machine Learning, pp. 5926–5936 (2019). PMLR

2019

Earlier work this paper cites.

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, pp. 2790–2799 (2019). PMLR

2019

Earlier work this paper cites.

Bapna, A., Firat, O.: Simple, scalable adaptation for neural machine translation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1538–1548 (2019)

2019

Earlier work this paper cites.

Jawahar, G., Sagot, B., Seddah, D.: What does bert learn about the structure of language? In: ACL 2019-57th Annual Meeting of the Association for Computational Linguistics (2019)

2019

Earlier work this paper cites.

Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., Liu, Q.: Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

Original

2019

Earlier work this paper cites.

Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243 (2019)

Original

2019

Earlier work this paper cites.

Lacoste, A., Luccioni, A., Schmidt, V., Dandres, T.: Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019)

Original

2019

Earlier work this paper cites.

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32

2019

Earlier work this paper cites.

Xie, Y., Chen, S., Ni, Q., Wu, H.: Integration of resource allocation and task assignment for optimizing the cost and maximum throughput of business processes. Journal of Intelligent Manufacturing 30

2019

Earlier work this paper cites.

Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. The Journal of Machine Learning Research 20

2019

Earlier work this paper cites.

Floridi, L., Chiriatti, M.: Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30

2020

Earlier work this paper cites.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

Original

2020

Earlier work this paper cites.

Rajbhandari, S., Rasley, J., Ruwase, O., He, Y.: Zero: Memory optimizations toward training trillion parameter models. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16 (2020). IEEE

2020

Earlier work this paper cites.

Deng, L., Li, G., Han, S., Shi, L., Xie, Y.: Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE 108

2020

Earlier work this paper cites.

Tay, Y., Dehghani, M., Bahri, D., Metzler, D.: Efficient transformers: A survey. arXiv e-prints, 2009 (2020)

2020

Earlier work this paper cites.

Capra, M., Bussolino, B., Marchisio, A., Shafique, M., Masera, G., Martina, M.: An updated survey of efficient hardware architectures for accelerating deep convolutional neural networks. Future Internet 12

2020

Earlier work this paper cites.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21

2020

Earlier work this paper cites.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

2020

Earlier work this paper cites.

Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=rkgNKkHtvB

2020

Earlier work this paper cites.

Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are rnns: Fast autoregressive transformers with linear attention. In: International Conference on Machine Learning, pp. 5156–5165 (2020). PMLR

2020

Earlier work this paper cites.

Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., Chen, Z.: Gshard: Scaling giant models with conditional computation and automatic sharding. In: International Conference on Learning Representations (2020)

2020

Earlier work this paper cites.

Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Zhao, T.: SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2177–2190 (2020)

2020

Earlier work this paper cites.

Xin, J., Tang, R., Lee, J., Yu, Y., Lin, J.: Deebert: Dynamic early exiting for accelerating bert inference. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2246–2251 (2020)

2020

Earlier work this paper cites.

Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., Smith, N.A.: The right tool for the job: Matching model and instance complexities. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6640–6651 (2020)

2020

Earlier work this paper cites.

Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., Wei, F.: Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems 33

2020

Earlier work this paper cites.

Iandola, F., Shaw, A., Krishna, R., Keutzer, K.: Squeezebert: What can computer vision teach nlp about efficient neural networks? In: Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 124–135 (2020)

2020

Earlier work this paper cites.

Wu, Z., Liu, Z., Lin, J., Lin, Y., Han, S.: Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886 (2020)

Original

2020

Earlier work this paper cites.

Anthony, L.F.W., Kanding, B., Selvan, R.: Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051 (2020)

Original

2020

Earlier work this paper cites.

Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., Pineau, J.: Towards the systematic reporting of the energy and carbon footprints of machine learning. The Journal of Machine Learning Research 21

2020

Earlier work this paper cites.

Hooker, S., Moorosi, N., Clark, G., Bengio, S., Denton, E.: Characterising bias in compressed models. arXiv preprint arXiv:2010.03058 (2020)

Original

2020

Earlier work this paper cites.

Barrault, L., Biesialska, M., Bojar, O., Costa-jussà, M.R., Federmann, C., Graham, Y., Grundkiewicz, R., Haddow, B., Huck, M., Joanis, E., et al

2020

Earlier work this paper cites.

Wang, A., Wolf, T.: Overview of the sustainlp 2020 shared task. In: Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 174–178 (2020)

2020

Earlier work this paper cites.

Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., Metzler, D.: Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 (2020)

Original

2020

Earlier work this paper cites.

Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al.: Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)

Original

2021

Earlier work this paper cites.

Menghani, G.: Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys (2021) https://doi.org/10.1145/3578938

2021

Earlier work this paper cites.

Shen, Z., Zhang, M., Zhao, H., Yi, S., Li, H.: Efficient attention: Attention with linear complexities. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3531–3539 (2021)

2021

Earlier work this paper cites.

Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., Susskind, J.: An attention free transformer. arXiv preprint arXiv:2105.14103 (2021)

Original

2021

Earlier work this paper cites.

Wang, X., Xiong, Y., Wei, Y., Wang, M., Li, L.: Lightseq: A high performance inference library for transformers. NAACL-HLT 2021, 113 (2021)

2021

Earlier work this paper cites.

NVIDIA: FasterTransformer: A Faster Transformer Framework. https://github.com/NVIDIA/FasterTransformer (2021)

2021

Earlier work this paper cites.

Baines, M., Bhosale, S., Caggiano, V., Goyal, N., Goyal, S., Ott, M., Lefaudeux, B., Liptchinsky, V., Rabbat, M., Sheiffer, S., et al.: Fairscale: A general purpose modular pytorch library for high performance and large scale training (2021)

2021

Earlier work this paper cites.

Paul, M., Ganguli, S., Dziugaite, G.K.: Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems 34

2021

Earlier work this paper cites.

Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., Huang, F.: Raise a child in large language model: Towards effective and generalizable fine-tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9514–9528 (2021)

2021

Earlier work this paper cites.

Zhang, R., Zheng, Y., Mao, X., Huang, M.: Unsupervised domain adaptation with adapter. arXiv preprint arXiv:2111.00667 (2021)

Original

2021

Earlier work this paper cites.

Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

2021

Earlier work this paper cites.

Aghajanyan, A., Gupta, S., Zettlemoyer, L.: Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319–7328 (2021)

2021

Earlier work this paper cites.

Evans, R.D., Aamodt, T.: Ac-gc: Lossy activation compression with guaranteed convergence. Advances in Neural Information Processing Systems 34

2021

Earlier work this paper cites.

Muhamed, A., Keivanloo, I., Perera, S., Mracek, J., Xu, Y., Cui, Q., Rajagopalan, S., Zeng, B., Chilimbi, T.: Ctr-bert: Cost-effective knowledge distillation for billion-parameter teacher models. In: NeurIPS Efficient Natural Language and Speech Processing Workshop (2021)

2021

Earlier work this paper cites.

Chen, P., Yu, H.-F., Dhillon, I., Hsieh, C.-J.: Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems 34

2021

Earlier work this paper cites.

Tahaei, M.S., Charlaix, E., Nia, V.P., Ghodsi, A., Rezagholizadeh, M.: Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation. arXiv preprint arXiv:2109.06243 (2021)

Original

2021

Earlier work this paper cites.

Edalati, A., Tahaei, M., Rashid, A., Nia, V.P., Clark, J.J., Rezagholizadeh, M.: Kronecker decomposition for gpt compression. arXiv preprint arXiv:2110.08152 (2021)

Original

2021

Earlier work this paper cites.

Reid, M., Marrese-Taylor, E., Matsuo, Y.: Subformer: Exploring weight sharing for parameter efficiency in generative transformers. arXiv preprint arXiv:2101.00234 (2021)

Original

2021

Earlier work this paper cites.

Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics 8

2021

Earlier work this paper cites.

Wang, H., Zhang, Z., Han, S.: Spatten: Efficient sparse attention architecture with cascade token and head pruning. In: 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 97–110 (2021). IEEE

2021

Earlier work this paper cites.

Ye, D., Lin, Y., Huang, Y., Sun, M.: Tr-bert: Dynamic token reduction for accelerating bert inference. arXiv preprint arXiv:2105.11618 (2021)

Original

2021

Earlier work this paper cites.

Sankar, C., Ravi, S., Kozareva, Z.: Proformer: Towards on-device lsh projection based transformers. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2823–2828 (2021)

2021

Earlier work this paper cites.

Huang, Z., Hou, L., Shang, L., Jiang, X., Chen, X., Liu, Q.: Ghostbert: Generate more features with cheap operations for bert. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6512–6523 (2021)

2021

Earlier work this paper cites.

Ma, Z., Ethayarajh, K., Thrush, T., Jain, S., Wu, L., Jia, R., Potts, C., Williams, A., Kiela, D.: Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking. Advances in Neural Information Processing Systems 34

2021

Earlier work this paper cites.

Schmidt, V., Goyal, K., Joshi, A., Feld, B., Conell, L., Laskaris, N., Blank, D., Wilson, J., Friedler, S., Luccioni, S.: Codecarbon: estimate and track carbon emissions from machine learning computing. Cited on, 20 (2021)

2021

Earlier work this paper cites.

Xu, C., Zhou, W., Ge, T., Xu, K., McAuley, J., Wei, F.: Beyond preserved accuracy: Evaluating loyalty and robustness of bert compression. arXiv preprint arXiv:2109.03228 (2021)

Original

2021

Earlier work this paper cites.

Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A.A., Wilson, A.G.: Does knowledge distillation really work? Advances in Neural Information Processing Systems 34

2021

Earlier work this paper cites.

Liu, X., Sun, T., He, J., Wu, J., Wu, L., Zhang, X., Jiang, H., Cao, Z., Huang, X., Qiu, X.: Towards efficient nlp: A standard evaluation and a strong baseline. arXiv preprint arXiv:2110.07038 (2021)

Original

2021

Earlier work this paper cites.

Min, S., Boyd-Graber, J., Alberti, C., Chen, D., Choi, E., Collins, M., Guu, K., Hajishirzi, H., Lee, K., Palomaki, J., et al

2021

Earlier work this paper cites.

Ford, B.W., Zong, Z.: Portauthority: Integrating energy efficiency analysis into cross-platform development cycles via dynamic program analysis. Sustainable Computing: Informatics and Systems 30

2021

Earlier work this paper cites.

Shamshirband, S., Joloudari, J.H., Shirkharkolaie, S.K., Mojrian, S., Rahmani, F., Mostafavi, S., Mansor, Z.: Game theory and evolutionary optimization approaches applied to resource allocation problems in computing environments: A survey. Mathematical Biosciences and Engineering (2021)

2021

Earlier work this paper cites.

Murshed, M.S., Murphy, C., Hou, D., Khan, N., Ananthanarayanan, G., Hussain, F.: Machine learning at the network edge: A survey. ACM Computing Surveys (CSUR) 54

2021

Earlier work this paper cites.

Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., Gao, Y.: A survey on federated learning. Knowledge-Based Systems 216

2021

Earlier work this paper cites.

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

Original

2022

Earlier work this paper cites.

Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., Gallé, M., et al.: Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022)

Original

2022

Earlier work this paper cites.

Tirumala, K., Markosyan, A., Zettlemoyer, L., Aghajanyan, A.: Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems 35

2022

Cited alongside, same era.

Aminabadi, R.Y., Rajbhandari, S., Awan, A.A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al

2022

Cited alongside, same era.

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., Clark, A., et al.: Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)

Original

2022

Cited alongside, same era.

Dhilleswararao, P., Boppu, S., Manikandan, M.S., Cenkeramaddi, L.R.: Efficient hardware architectures for accelerating deep neural networks: Survey. IEEE Access (2022)

2022

Cited alongside, same era.

Rabe, M.N., Staats, C.: Self-attention does not need o ⁡ ( n 2 ) o(n^{2}) memory. arXiv preprint arXiv:2112.05682 (2022)

Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M.W., Keutzer, K.: Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629 (2023)

Original

2023

Later among the works it cites.

Lee, C., Jin, J., Kim, T., Kim, H., Park, E.: Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02272 (2023)

Original

2023

Later among the works it cites.

Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., Zhu, Y.: Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In: Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1–15 (2023)

2023

Later among the works it cites.

Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., Chandra, V.: Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 (2023)

Original

2023