Preliminary WMT24 Ranking of General MT Systems and LLMs

Re-evaluating the role of Bleu in machine translation research

Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006 · 2006

Earlier work this paper cites.

To ship or not to ship: An extensive evaluation of automatic metrics for machine translation

Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021 · 2021

Earlier work this paper cites.

DEMETR: Diagnosing evaluation metrics for translation

Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Iyyer. 2022 · 2022

Earlier work this paper cites.

Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023 · 2023

Cited alongside, same era.

MetricX-23: The Google submission to the WMT 2023 metrics shared task

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023 · 2023

Cited alongside, same era.

Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task

Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023 · 2023

Cited alongside, same era.

High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics

Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a

Cited in the paper.

Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b

Cited in the paper.

Error span annotation: A balanced approach for human evaluation of machine translation

Original

Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024b

Cited in the paper.

Navigating the metrics maze: Reconciling score magnitudes and accuracies

Original

Tom Kocmi, Vilém Zouhar, Christian Federmann, and Matt Post. 2024a · 2024

Closest in time.

Machine translation meta evaluation through translation accuracy challenge sets

Original

Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sennrich, and Liane Guillou. 2024 · 2024

Closest in time.