An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Evaluating gender bias in machine translation

Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019 · 2019

Earlier work this paper cites.

Language (technology) is power: A critical survey of “bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020 · 2020

Earlier work this paper cites.

COMET: A Neural Framework for MT Evaluation

Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020 · 2020

Earlier work this paper cites.

E-mimic: Empowering multilingual inclusive communication

Giuseppe Attanasio, Salvatore Greco, Moreno La Quatra, Luca Cagliero, Michela Tonti, Tania Cerquitelli, and Rachele Raus. 2021 · 2021

Earlier work this paper cites.

Harms of gender exclusivity and challenges in non-binary representation in language technologies

Original

Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, and Kai-Wei Chang. 2021 · 2021

Earlier work this paper cites.

NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives

Eva Vanmassenhove, Chris Emmery, and Dimitar Shterionov. 2021 · 2021

Earlier work this paper cites.

Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender

Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022 · 2022

Earlier work this paper cites.

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023 · 2023

Earlier work this paper cites.

MISGENDERED: Limits of large language models in understanding pronouns

Tamanna Hossain, Sunipa Dev, and Sameer Singh. 2023 · 2023

Earlier work this paper cites.

Gender-fair post-editing: A case study beyond the binary

Manuel Lardelli and Dagmar Gromann. 2023 · 2023

Earlier work this paper cites.

G-eval: Nlg evaluation using gpt-4 with better human alignment

Original

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Earlier work this paper cites.

Hi guys or hi folks? benchmarking gender-neutral machine translation with the GeNTE corpus

Andrea Piergentili, Beatrice Savoldi, Dennis Fucci, Matteo Negri, and Luisa Bentivogli. 2023b · 2023

Earlier work this paper cites.

Neutral is not fair enough: testing the efficiency of different language gender-fair strategies

Elsa Spinelli, Jean-Pierre Chevrot, and Léo Varnet. 2023 · 2023

Cited alongside, same era.

Evaluating the impact of stereotypes and language combinations on gender bias occurrence in NMT generic systems

Bertille Triboulet and Pierrette Bouillon. 2023 · 2023

Cited alongside, same era.

A rewriting approach for gender inclusivity in Portuguese

Leonor Veloso, Luisa Coheur, and Rui Ribeiro. 2023 · 2023

Cited alongside, same era.

Is chatgpt a good nlg evaluator? a preliminary study

Original

Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 · 2023

Cited alongside, same era.

Efficient guided generation for llms

Original

Brandon T Willard and Rémi Louf. 2023 · 2023

Cited alongside, same era.

Fine-grained gender control in machine translation with large language models

Minwoo Lee, Hyukhun Koh, Minsung Kim, and Kyomin Jung. 2024 · 2024

Later among the works it cites.

PrExMe! large scale prompt exploration of open source LLMs for machine translation and summarization evaluation

Christoph Leiter and Steffen Eger. 2024 · 2024

Later among the works it cites.

Error analysis prompting enables human-like translation evaluation in large language models

Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024 · 2024

Later among the works it cites.

Scaling neural machine translation to 200 languages

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2024 · 2024

Later among the works it cites.