CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks

H. C. Jiang, Z. Y. Weng, and T. Xiang, Accurate determination of tensor network state of quantum lattice models in two dimensions, Phys. Rev. Lett. 101

2008

Earlier work this paper cites.

M. Jaderberg, A. Vedaldi, and A. Zisserman, Speeding up convolutional neural networks with low rank expansions, ArXiv abs/1405.3866

Original

2014

Earlier work this paper cites.

R. Orús, A practical introduction to tensor networks: Matrix product states and projected entangled pair states, Annals of Physics 349

Original

2014

Earlier work this paper cites.

G. E. Hinton, O. Vinyals, and J. Dean, Distilling the knowledge in a neural network, ArXiv abs/1503.02531

Original

2015

Earlier work this paper cites.

S. Han, J. Pool, J. Tran, and W. J. Dally, Learning both weights and connections for efficient neural network, in Neural Information Processing Systems (2015)

2015

Earlier work this paper cites.

A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov, Tensorizing neural networks (2015), arXiv:1509.06569 [cs.LG]

Original

2015

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems , Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., 2017)

2017

Earlier work this paper cites.

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. G. Howard, H. Adam, and D. Kalenichenko, Quantization and training of neural networks for efficient integer-arithmetic-only inference, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2704 (2017)

2017

Cited alongside, same era.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 10.48550/arXiv.1810.04805 (2018), arXiv:1810.04805

Original

2018

Cited alongside, same era.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI Technical Report (2019)

2019

Cited alongside, same era.

R. Orús, Tensor networks for complex quantum systems, Nature Reviews Physics 1

2019

Cited alongside, same era.

S. Lock, What is AI chatbot phenomenon ChatGPT and could it replace humans?, The Guardian (2022)

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, LlaMA: Open and Efficient Foundation Language Models 10.48550/arXiv.2302.13971 (2023), arXiv:2302.13971

Original

2023

Later among the works it cites.

The bigger-is-better approach to AI is running out of road, The Economist (2023)

2023

Later among the works it cites.

See also, Multiverse Computing CompactifAI (2023)

2023

Later among the works it cites.

M. Wang, Y. Pan, Z. Xu, X. Yang, G. Li, and A. Cichocki, Tensor networks meet neural networks: A survey and future perspectives (2023), arXiv:2302.09019 [cs.LG]

Original

2023

Later among the works it cites.

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, A framework for few-shot language model evaluation (2023)