mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904

Earlier work this paper cites.

Unsupervised cross-lingual representation learning at scale

Original

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911

Earlier work this paper cites.

Mapping language to code in programmatic context

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018 · 2018

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 2020

Earlier work this paper cites.

The state and fate of linguistic diversity and inclusion in the nlp world

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020 · 2020

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021 · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. 2021 · 2021

Earlier work this paper cites.

Larger-scale transformers for multilingual masked language modeling

Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021 · 2021

Earlier work this paper cites.

Systematic inequalities in language technology performance across the world’s languages

Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022 · 2022

Earlier work this paper cites.

No language left behind: Scaling human-centered machine translation

Original

Marta R Costa-jussà, James Cross, Onur Çelebi, et al. 2022 · 2022

Cited alongside, same era.

Aixbench: A code generation benchmark dataset

Original

Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, et al. 2022 · 2022

Cited alongside, same era.

Training compute-optimal large language models

Original

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, et al. 2022 · 2022

Cited alongside, same era.

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, et al. 2023 · 2023

Cited alongside, same era.

Multipl-e: a scalable and polyglot approach to benchmarking neural code generation

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, et al. 2023 · 2023

Measuring the impact of programming language distribution

Gabriel Orlanski, Kefan Xiao, Xavier Garcia, et al. 2023 · 2023

Later among the works it cites.

Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task

Ricardo Rei, Nuno M Guerreiro, Daan van Stigt, Marcos Treviso, et al. 2023 · 2023

Later among the works it cites.

Claude 3: A next-generation ai assistant

Anthropic. 2024 · 2024

Closest in time.

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

Original

Damai Dai, Chengqi Deng, Chenggang Zhao, et al. 2024 · 2024

Closest in time.

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024 · 2024

Closest in time.