PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation

THE TREATMENT OF TIES IN RANKING PROBLEMS

M. G. Kendall. 1945 · 1945

Earlier work this paper cites.

Making monolingual sentence embeddings multilingual using knowledge distillation

Nils Reimers and Iryna Gurevych. 2020 · 2020

Earlier work this paper cites.

A statistical analysis of summarization evaluation metrics using resampling methods

Daniel Deutsch, Rotem Dror, and Dan Roth. 2021 · 2021

Earlier work this paper cites.

SummEval: Re-evaluating summarization evaluation

Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021 · 2021

Earlier work this paper cites.

Bartscore: Evaluating generated text as text generation

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

LangChain

Harrison Chase. 2022 · 2022

Earlier work this paper cites.

Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

Do prompt-based models really understand the meaning of their prompts?

Albert Webson and Ellie Pavlick. 2022 · 2022

Earlier work this paper cites.

Findings of the WMT 2022 shared task on quality estimation

Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022 · 2022

Earlier work this paper cites.

UScore: An effective approach to fully unsupervised evaluation metrics for machine translation

Jonas Belouadi and Steffen Eger. 2023 · 2023

Earlier work this paper cites.

SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation

Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. 2023 · 2023

Cited alongside, same era.

Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration

Daniel Deutsch, George Foster, and Markus Freitag. 2023 · 2023

Cited alongside, same era.

The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023 · 2023

Cited alongside, same era.

Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023 · 2023

Cited alongside, same era.

Gptscore: Evaluate as you desire

Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

Original

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023 · 2023

Later among the works it cites.

The icl consistency test

Original

Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023 · 2023

Later among the works it cites.

Nllg quarterly arxiv report 09/23: What are the most influential current ai papers?

Original

Ran Zhang, Aida Kostikova, Christoph Leiter, Jonas Belouadi, Daniil Larionov, Yanran Chen, Vivian Fresen, and Steffen Eger. 2023 · 2023

Later among the works it cites.

Llama 3 model card

AI@Meta. 2024 · 2024

Closest in time.

Tower: An open multilingual large language model for translation-related tasks

Original