Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

The silent language

Edward T. Hall. 1959 · 1959

Earlier work this paper cites.

A coefficient of agreement for nominal scales

Jacob Cohen. 1960 · 1960

Earlier work this paper cites.

WordNet: A lexical database for English

George A. Miller. 1994 · 1994

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997

Earlier work this paper cites.

Towards a human-like open-domain chatbot

Original

Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020 · 2001

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

Satanjeev Banerjee and Alon Lavie. 2005 · 2005

Earlier work this paper cites.

Communication style and cultural features in high/low context communication cultures: A case study of finland, japan and india

Shoji Nishimura, Anne Nevgi, and Seppo Tella. 2008 · 2008

Earlier work this paper cites.

Emora: An inquisitive social chatbot who cares for you

Original

Sarah E. Finch, James D. Finch, Ali Ahmadvand, Ingyu, Choi, Xiangjue Dong, Ruixiang Qi, Harshita Sahijwani, Sergey Volokhin, Zihan Wang, Zihao Wang, and Jinho D. Choi. 2020 · 2009

Earlier work this paper cites.

Sequence to sequence learning with neural networks

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014

Earlier work this paper cites.

Neural responding machine for short-text conversation

Lifeng Shang, Zhengdong Lu, and Hang Li. 2015 · 2015

Earlier work this paper cites.

How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation

Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016 · 2016

Earlier work this paper cites.

Key-value memory networks for directly reading documents

Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016 · 2016

Earlier work this paper cites.

Building end-to-end dialogue systems using generative hierarchical neural network models

Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016 · 2016

Earlier work this paper cites.

DailyDialog: A manually labelled multi-turn dialogue dataset

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions

Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018 · 2018

Earlier work this paper cites.

Personalizing dialogue agents: I have a dog, do you have pets too?

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 · 2018

Earlier work this paper cites.

Topical-chat: Towards knowledge-grounded open-domain conversations

Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anushree Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019 · 2019

Earlier work this paper cites.

ChatEval: A tool for chatbot evaluation

João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019 · 2019

Earlier work this paper cites.

Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems

Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Earlier work this paper cites.

Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining

Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020 · 2020

Earlier work this paper cites.

DIALOGPT : Large-scale generative pre-training for conversational response generation

Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020 · 2020

Cited alongside, same era.

Integrated taxonomy of errors in chat-oriented dialogue systems

Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2021 · 2021

Cited alongside, same era.

Lora: Low-rank adaptation of large language models

Original

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Cited alongside, same era.

I like fish, especially dolphins: Addressing contradictions in dialogue modeling

Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021 · 2021

Cited alongside, same era.

Recipes for building an open-domain chatbot

Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021 · 2021

LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models

Yen-Ting Lin and Yun-Nung Chen. 2023 · 2023

Later among the works it cites.

G-eval: NLG evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Later among the works it cites.

Simple LLM prompting is state-of-the-art for robust and multilingual dialogue evaluation

John Mendonça, Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho, Alon Lavie, and Isabel Trancoso. 2023 · 2023

Later among the works it cites.

Learning from free-text human feedback – collect new datasets or extend existing ones?

Dominic Petrak, Nafise Moosavi, Ye Tian, Nikolai Rozanov, and Iryna Gurevych. 2023 · 2023

Later among the works it cites.

Overview of robust and multilingual automatic evaluation metricsfor open-domain dialogue systems at DSTC 11 track 4

Mario Rodríguez-Cantelar, Chen Zhang, Chengguang Tang, Ke Shi, Sarik Ghazarian, João Sedoc, Luis Fernando D’Haro, and Alexander I. Rudnicky. 2023 · 2023