2024

Mixture-of-Agents Enhances Large Language Model Capabilities

Wang, Junlin, Wang, Jue, Athiwaratkun, Ben et al.

Understand

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks.

With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction.
Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology.
In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents.

Built on

Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and Zhu, W · 2002
Earlier work this paper cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J · 2017
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
Measuring mathematical problem solving with the math dataset
Original
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J · 2021
Earlier work this paper cites.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al · 2022
Earlier work this paper cites.
Complexity-based prompting for multi-step reasoning
Original
Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T · 2022
Earlier work this paper cites.
Large language models are zero-shot reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al · 2022
Earlier work this paper cites.
Self-consistency improves chain of thought reasoning in language models
Original
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al · 2022
Earlier work this paper cites.
Qwen technical report
Original
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al · 2023
Earlier work this paper cites.

Similar

Chateval: Towards better llm-based evaluators through multi-agent debate
Original
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z · 2023
Cited alongside, same era.
Active prompting with chain-of-thought for large language models
Original
Diao, S., Wang, P., Lin, Y., and Zhang, T · 2023
Cited alongside, same era.
Improving factuality and reasoning in language models through multiagent debate
Original
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I · 2023
Cited alongside, same era.
LLM-blender: Ensembling large language models with pairwise ranking and generative fusion
Jiang, D., Ren, X., and Lin, B. Y · 2023
Cited alongside, same era.
Encouraging divergent thinking in large language models through multi-agent debate
Original
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S · 2023
Cited alongside, same era.
Deductive verification of chain-of-thought reasoning
Original
Ling, Z., Fang, Y., Li, X., Huang, Z., Lee, M., Memisevic, R., and Su, H · 2023
Cited alongside, same era.
Routing to the expert: Efficient reward-guided ensemble of large language models, 2023
Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J · 2023
Cited alongside, same era.
Gpt-4 technical report, 2023
OpenAI · 2023
Cited alongside, same era.
python-levenshtein by rapidfuzz
RapidFuzz · 2023
Cited alongside, same era.
Code llama: Open foundation models for code
Original
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al · 2023
Cited alongside, same era.
Gemini: a family of highly capable multimodal models
Original
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al · 2023
Cited alongside, same era.
Reconcile: Round-table conference improves reasoning via consensus among diverse llms
Original
Chen, J. C.-Y., Saha, S., and Bansal, M
Cited in the paper.

Then

Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models
Original
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P · 2023
Later among the works it cites.
Flask: Fine-grained language model evaluation based on alignment skill sets
Original
Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., and Seo, M · 2023
Later among the works it cites.
Exploring collaboration mechanisms for llm agents: A social psychology view
Original
Zhang, J., Xu, X., and Deng, S · 2023
Later among the works it cites.
Judging llm-as-a-judge with mt-bench and chatbot arena
Original
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I · 2023
Later among the works it cites.
Length-controlled alpacaeval: A simple way to debias automatic evaluators
Original
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B · 2024
Closest in time.
Deepseek-coder: When the large language model meets programming–the rise of code intelligence
Original
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al · 2024
Closest in time.
Enabling ensemble learning for heterogeneous large language models with deep parallel collaboration
Original
Huang, Y., Feng, X., Li, B., Xiang, Y., Wang, H., Qin, B., and Liu, T · 2024
Closest in time.
Mixtral of experts
Original
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E · 2024
Closest in time.
Large language model routing with benchmark datasets, 2024
Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y., Solomon, J., Thompson, N., and Yurochkin, M · 2024
Closest in time.
Introducing dbrx: A new state-of-the-art open llm
The Mosaic Research Team · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…