TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models

A new measure of rank correlation

Maurice G Kendall. 1938 · 1938

Earlier work this paper cites.

A simple sequentially rejective multiple test procedure

Sture Holm. 1979 · 1979

Earlier work this paper cites.

K-sample Anderson–Darling tests

Fritz W Scholz and Michael A Stephens. 1987 · 1987

Earlier work this paper cites.

Item clusters and computerized adaptive testing: A case for testlets

Howard Wainer and Gerard L Kiely. 1987 · 1987

Earlier work this paper cites.

Individual comparisons by ranking methods

Frank Wilcoxon. 1992 · 1992

Earlier work this paper cites.

Table for conversion of Kendall’s Tau to Spearman’s Rho within the context of measures of magnitude of effect for meta-analysis

Andrew R Gilpin. 1993 · 1993

Earlier work this paper cites.

Interpretation of Kappa and B statistics measures of agreement

Sergio R Munoz and Shrikant I Bangdiwala. 1997 · 1997

Earlier work this paper cites.

The basics of item response theory

Frank B Baker. 2001 · 2001

Earlier work this paper cites.

Planning poker or how to avoid analysis paralysis while release planning

James Grenning. 2002 · 2002

Earlier work this paper cites.

The origins of testlet response theory – three alternatives

Howard Wainer, Eric T. Bradlow, and Xiaohui Wang. 2007 · 2007

Earlier work this paper cites.

Effect size estimates: current use, calculations, and interpretation

Catherine O Fritz, Peter E Morris, and Jennifer J Richler. 2012 · 2012

Earlier work this paper cites.

Building an evaluation scale using item response theory. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , Vol. 2016. NIH Public Access, 648

John P Lalor, Hao Wu, and Hong Yu. 2016 · 2016

Earlier work this paper cites.

An application of item response theory to psychological test development

Cristian Zanon, Claudio S Hutz, Hanwook Henry Yoo, and Ronald K Hambleton. 2016 · 2016

Earlier work this paper cites.

hdbscan: Hierarchical density based clustering

Leland McInnes, John Healy, Steve Astels, et al · 2017

Earlier work this paper cites.

Metamorphic testing: A review of challenges and opportunities

Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018 · 2018

Earlier work this paper cites.

Umap: Uniform manifold approximation and projection for dimension reduction

Original

Leland McInnes, John Healy, and James Melville. 2018 · 2018

Earlier work this paper cites.

Correlation coefficients: appropriate use and interpretation

Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018 · 2018

Earlier work this paper cites.

β 3 \beta^{3} -IRT: A New Item Response Model and its Applications. In The 22nd International Conference on Artificial Intelligence and Statistics . PMLR, 1013–1021

Yu Chen, Telmo Silva Filho, Ricardo B Prudencio, Tom Diethe, and Peter Flach. 2019 · 2019

Earlier work this paper cites.

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 2019

Earlier work this paper cites.

Codebleu: a method for automatic evaluation of code synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2020

Earlier work this paper cites.

Item Response Theory for Efficient Human Evaluation of Chatbots. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems , Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, and Eduard Hovy (Eds.). Association for Computational Linguistics, Online, 21–33

João Sedoc and Lyle Ungar. 2020 · 2020

Earlier work this paper cites.

Playing planning poker in crowds: human computation of software effort estimates. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1–12

Mohammed Alhamed and Tim Storer. 2021 · 2021

Earlier work this paper cites.

Program Synthesis with Large Language Models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021 · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021

Earlier work this paper cites.

A fine-grained analysis of BERTScore. In Proceedings of the Sixth Conference on Machine Translation . 507–517

Michael Hanna and Ondřej Bojar. 2021 · 2021

Earlier work this paper cites.

Measuring coding challenge competence with apps

Original

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al · 2021

Earlier work this paper cites.

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Original

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, et al · 2021

Earlier work this paper cites.

Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 4486–4503

Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021b · 2021

Earlier work this paper cites.

Comparing test sets with item response theory

Original

Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R Bowman. 2021a · 2021

Earlier work this paper cites.

Comparing test sets with item response theory

Original

Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R Bowman. 2021b · 2021

Earlier work this paper cites.

Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

Original

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021 · 2021

Cited alongside, same era.

Multi-lingual Evaluation of Code Generation Models

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2022 · 2022

Cited alongside, same era.

Predicting Difficulty and Discrimination of Natural Language Questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 119–130

Matthew Byrd and Shashank Srivastava. 2022 · 2022

Cited alongside, same era.

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

Original

ClassEval Leaderboard

2024 · 2024

Closest in time.

EvalPlus Leaderboard

2024 · 2024

Closest in time.

Models on Hugging Face

2024 · 2024

Closest in time.

Rep-Package

2024 · 2024

Closest in time.

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Original

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwan, Yoshua Bengio, Danqi Chen, Philip H. S. Torr, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. 2024 · 2024

Closest in time.