Output Format Biases in the Evaluation of Large Language Models for Code Translation

HuggingFace’s Transformers: State-of-the-art Natural Language Processing

Original

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020 · 1910

Earlier work this paper cites.

BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02 . Association for Computational Linguistics, Philadelphia, Pennsylvania, 311

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001 · 2001

Earlier work this paper cites.

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009

Earlier work this paper cites.

Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering . 651–654

Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2013 · 2013

Earlier work this paper cites.

Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software . 173–184

Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014 · 2014

Earlier work this paper cites.

Migrating code with statistical machine translation. In Companion Proceedings of the 36th International Conference on Software Engineering . 544–547

Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2014 · 2014

Earlier work this paper cites.

Using machine translation for converting python 2 to python 3 code

Karan Aggarwal, Mohammad Salameh, and Abram Hindle. 2015 · 2015

Earlier work this paper cites.

chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (Eds.). Association for Computational Linguistics, Lisbon, Portugal, 392–395

Maja Popović. 2015 · 2015

Earlier work this paper cites.

Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16) . Association for Computing Machinery, New York, NY, USA, 120–131

Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016 · 2016

Earlier work this paper cites.

Tree-to-tree neural networks for program translation

Xinyun Chen, Chang Liu, and Dawn Song. 2018 · 2018

Earlier work this paper cites.

Codebert: A pre-trained model for programming and natural languages

Original

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al · 2020

Earlier work this paper cites.

Unsupervised translation of programming languages

Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020 · 2020

Earlier work this paper cites.

DOBF: A deobfuscation pre-training objective for programming languages

Marie-Anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021 · 2021

Earlier work this paper cites.

Codexglue: A machine learning benchmark dataset for code understanding and generation

Original

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al · 2021

Earlier work this paper cites.

CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks

Original

Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021 · 2021

Earlier work this paper cites.

Leveraging automated unit tests for unsupervised code translation

Original

Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021 · 2021

Cited alongside, same era.

Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

Original

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021 · 2021

Cited alongside, same era.

Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces . ACM, College Station TX USA, 402–412

Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021 · 2021

Cited alongside, same era.

CrystalBLEU: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1–12

Aryaz Eghbali and Michael Pradel. 2022 · 2022

Cited alongside, same era.

Code Translation with Compiler Representations

Original

Marc Szafraniec, Baptiste Roziere, Hugh Leather, Francois Charton, Patrick Labatut, and Gabriel Synnaeve. 2023 · 2023

Later among the works it cites.

Magicoder: Source Code Is All You Need

Original

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023 · 2023

Later among the works it cites.

Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

Original

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al · 2023

Later among the works it cites.

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code

Original

Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023 · 2023

Later among the works it cites.

Airoboros 13B HF fp16