Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” in Proceedings of the fifth annual workshop on Computational learning theory , 1992, pp. 287–294

1992

Earlier work this paper cites.

J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y. Ng et al. , “Multimodal deep learning.” in ICML , vol. 11, 2011, pp. 689–696

2011

Earlier work this paper cites.

A. Graves and A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks , pp. 37–45, 2012

2012

Earlier work this paper cites.

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3156–3164

2015

Earlier work this paper cites.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature , vol. 521, no. 7553, pp. 436–444, 2015

2015

Earlier work this paper cites.

I. Goodfellow, “Deep learning,” 2016

2016

Earlier work this paper cites.

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017

2017

Earlier work this paper cites.

J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

M. Ott, S. Edunov, D. Grangier, and M. Auli, “Scaling neural machine translation,” arXiv preprint arXiv:1806.00187 , 2018

Original

2018

Earlier work this paper cites.

T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 2, pp. 423–443, 2018

2018

Earlier work this paper cites.

Z. Lan, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942 , 2019

Original

2019

Earlier work this paper cites.

Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” in Proceedings of the conference. Association for computational linguistics. Meeting , vol. 2019. NIH Public Access, 2019, p. 6558

2019

Earlier work this paper cites.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020

2020

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Earlier work this paper cites.

R. Dale, “Gpt-3: What is it good for,” Nat. Lang. Eng , vol. 27, pp. 113–118, 2021

2021

Earlier work this paper cites.

G. Soyalp, A. Alar, K. Ozkanli, and B. Yildiz, “Improving text classification with transformer,” in 2021 6th International Conference on Computer Science and Engineering (UBMK) . IEEE, 2021, pp. 707–712

2021

Earlier work this paper cites.

S. Chaudhari, V. Mithal, G. Polatkan, and R. Ramanath, “An attentive survey of attention models,” ACM Transactions on Intelligent Systems and Technology (TIST) , vol. 12, no. 5, pp. 1–32, 2021

2021

Earlier work this paper cites.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning . PMLR, 2021, pp. 10 347–10 357

2021

Earlier work this paper cites.

Z. Liu, Y. Xu, T. Yu, W. Dai, Z. Ji, S. Cahyawijaya, A. Madotto, and P. Fung, “Crossner: Evaluating cross-domain named entity recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 15, 2021, pp. 13 452–13 460

2021

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

2021

Earlier work this paper cites.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258 , 2021

Original

2021

Earlier work this paper cites.

J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” Advances in neural information processing systems , vol. 34, pp. 9694–9705, 2021

2021

Earlier work this paper cites.

Y. Guan, Z. Li, Z. Lin, Y. Zhu, J. Leng, and M. Guo, “Block-skim: Efficient question answering for transformer,” in Proceedings of the AAAI conference on artificial intelligence , vol. 36, no. 10, 2022, pp. 10 710–10 719

2022

Earlier work this paper cites.

A. de Santana Correia and E. L. Colombini, “Attention, please! a survey of neural attention models in deep learning,” Artificial Intelligence Review , vol. 55, no. 8, pp. 6037–6124, 2022

2022

Earlier work this paper cites.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

2022

Earlier work this paper cites.

OpenAI and Microsoft, “Chatgpt,” https://chatgpt.com/ , 2022, accessed: date-of-access

2022

Earlier work this paper cites.

J. Zhang, R. Krishna, A. H. Awadallah, and C. Wang, “Ecoassistant: Using llm assistant more affordably and accurately,” arXiv preprint arXiv:2310.03046 , 2023

Original

2023

Earlier work this paper cites.

L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176 , 2023

Original

2023

Earlier work this paper cites.

K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou, “Routing to the expert: Efficient reward-guided ensemble of large language models,” arXiv preprint arXiv:2311.08692 , 2023

Original

2023

Earlier work this paper cites.

X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” arXiv preprint arXiv:2312.15234 , 2023

Original

2023

Earlier work this paper cites.

T. Ding, T. Chen, H. Zhu, J. Jiang, Y. Zhong, J. Zhou, G. Wang, Z. Zhu, I. Zharkov, and L. Liang, “The efficiency spectrum of large language models: An algorithmic survey,” arXiv preprint arXiv:2312.00678 , 2023

Original

2023

Earlier work this paper cites.

K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal et al. , “Towards expert-level medical question answering with large language models,” arXiv preprint arXiv:2305.09617 , 2023

Original

2023

Earlier work this paper cites.

K. Sanderson, “Gpt-4 is here: what scientists think,” Nature , vol. 615, no. 7954, p. 773, 2023

2023

Earlier work this paper cites.

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Earlier work this paper cites.

E. Foundation, “Xtext language engineering for everyone,” https://eclipse.dev/Xtext/ , 2023, accessed: date-of-access

2023

Earlier work this paper cites.

S. N. Hari and M. Thomson, “Tryage: Real-time, intelligent routing of user prompts to large language model,” arXiv preprint arXiv:2308.11601 , 2023

Original

2023

Cited alongside, same era.

Amazon, “Amazon titan in amazon bedrock,” https://aws.amazon.com/bedrock/amazon-models/ , 2023, accessed: date-of-access

2023

Cited alongside, same era.

Anthropic, “Claude,” https://claude.ai/ , 2023, accessed: date-of-access

2023

Cited alongside, same era.

MetaAI, “Llama,” https://www.llama.com/ , 2023, accessed: date-of-access

2023

Cited alongside, same era.

G. Ramírez, M. Lindemann, A. Birch, and I. Titov, “Cache & distil: Optimising api calls to large language models,” arXiv preprint arXiv:2310.13561 , 2023

Original

2023

Cited alongside, same era.

A. MS, J. VG, and D. PS, “Efficient hybrid inference for llms: Reward-based token modelling with selective cloud assistance,” arXiv preprint arXiv:2409.13757 , 2024

Original

2024

Later among the works it cites.

G. Ramírez, A. Birch, and I. Titov, “Optimising calls to large language models with uncertainty-based two-tier selection,” arXiv preprint arXiv:2405.02134 , 2024

Original

2024

Later among the works it cites.

J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, and Z. Ling, “On-device language models: A comprehensive review,” arXiv preprint arXiv:2409.00088 , 2024

Original

2024

Later among the works it cites.

C. He, R. Luo, S. Hu, Y. Zhao, J. Zhou, H. Wu, J. Zhang, X. Han, Z. Liu, and M. Sun, “Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms,” arXiv preprint arXiv:2404.07584 , 2024

Original

2024

Later among the works it cites.