2024

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Chiang, Wei-Lin, Zheng, Lianmin, Sheng, Ying et al.

Understand

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges.

To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences.
Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing.
The platform has been operational for several months, amassing over 240K votes.

Built on

Statistical methods for research workers
Fisher, R. A · 1928
Earlier work this paper cites.
Rank analysis of incomplete block designs: I. the method of paired comparisons
Bradley, R. A. and Terry, M. E · 1952
Earlier work this paper cites.
The proposed uscf rating system, its development, theory, and applications
Elo, A. E · 1967
Earlier work this paper cites.
The behavior of maximum likelihood estimates under nonstandard conditions
Huber, P. J. et al · 1967
Earlier work this paper cites.
Ties in paired-comparison experiments: A generalization of the bradley-terry model
Rao, P. V. and Kupper, L. L · 1967
Earlier work this paper cites.
Maximum likelihood estimation of misspecified models
White, H · 1982
Earlier work this paper cites.
Sequential Design of Experiments , pp. 345–360
Chernoff, H · 1992
Earlier work this paper cites.
Bootstrap confidence intervals
DiCiccio, T. J. and Efron, B · 1996
Earlier work this paper cites.
Crafting papers on machine learning
Langley, P · 2000
Earlier work this paper cites.
MM algorithms for generalized Bradley-Terry models
Hunter, D. R · 2004
Earlier work this paper cites.
On the so-called “huber sandwich estimator”’ and “robust standard errors”’
Freedman, D. A · 2006
Earlier work this paper cites.
Learning to rank for information retrieval
Liu, T.-Y. et al · 2009
Earlier work this paper cites.
Online rank elicitation for plackett-luce: A dueling bandits approach
Szörényi, B., Busa-Fekete, R., Paul, A., and Hüllermeier, E · 2015
Earlier work this paper cites.
Probability: theory and examples , volume 49
Durrett, R · 2019
Earlier work this paper cites.
Hellaswag: Can a machine really finish your sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y · 2019
Earlier work this paper cites.
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2020
Earlier work this paper cites.

Similar

Time-uniform chernoff bounds via nonnegative supermartingales
Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J · 2020
Cited alongside, same era.
Umap: Uniform manifold approximation and projection for dimension reduction, 2020
McInnes, L., Healy, J., and Melville, J · 2020
Cited alongside, same era.
Estimating means of bounded random variables by betting
Original
Waudby-Smith, I. and Ramdas, A · 2020
Cited alongside, same era.
Evaluating large language models trained on code
Original
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al · 2021
Cited alongside, same era.
Training verifiers to solve math word problems
Original
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al · 2021
Cited alongside, same era.
Online active model selection for pre-trained classifiers
Karimi, M. R., Gürel, N. M., Karlaš, B., Rausch, J., Zhang, C., and Krause, A · 2021
Cited alongside, same era.
The perils of using Mechanical Turk to evaluate open-ended text generation
Karpinska, M., Akoury, N., and Iyyer, M · 2021
Cited alongside, same era.
Dynabench: Rethinking benchmarking in nlp
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al · 2021
Cited alongside, same era.
E-values: Calibration, combination and applications
Vovk, V. and Wang, R · 2021
Cited alongside, same era.
Training a helpful and harmless assistant with reinforcement learning from human feedback
Original
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al · 2022
Cited alongside, same era.
Bertopic: Neural topic modeling with a class-based tf-idf procedure
Original
Grootendorst, M · 2022
Cited alongside, same era.
Competition-level code generation with alphacode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al · 2022
Cited alongside, same era.

Then

Ultrafeedback: Boosting language models with high-quality feedback, 2023
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M · 2023
Later among the works it cites.
Gemini: a family of highly capable multimodal models
Original
Gemini, T., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al · 2023
Later among the works it cites.
Koala: A dialogue model for academic research
Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D · 2023
Later among the works it cites.
Competition-level problems are effective llm evaluators
Original
Huang, Y., Lin, Z., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., et al · 2023
Later among the works it cites.
Openassistant conversations–democratizing large language model alignment
Original
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al · 2023
Later among the works it cites.
Alpacaeval: An automatic evaluator of instruction-following models
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Later among the works it cites.
ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation
Lin, Z., Wang, Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J · 2023
Later among the works it cites.
Gpt-4 technical report
Original
OpenAI · 2023
Later among the works it cites.
Proving test set contamination in black box language models
Original
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B · 2023
Later among the works it cites.
Game-theoretic statistics and safe anytime-valid inference
Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G · 2023
Later among the works it cites.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Original
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al · 2023
Later among the works it cites.
Self-instruct: Aligning language models with self-generated instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H · 2023
Later among the works it cites.
Rethinking benchmark and contamination for language models with rephrased samples
Original
Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I · 2023
Later among the works it cites.
Agieval: A human-centric benchmark for evaluating foundation models
Original
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N · 2023
Later among the works it cites.
Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023
Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…