Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries

Fine-tuning language models from human preferences

Original

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019 · 1909

Earlier work this paper cites.

Ambiguous requests: implications for retrieval tests, systems and theories

Karen Spärck-Jones, Stephen E. Robertson, and Mark Sanderson. 2007 · 2007

Earlier work this paper cites.

An effectiveness measure for ambiguous and underspecified queries

Charles LA Clarke, Maheedhar Kolla, and Olga Vechtomova. 2009 · 2009

Earlier work this paper cites.

The weirdest people in the world?

Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010 · 2010

Earlier work this paper cites.

Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information

Sudha Rao and Hal Daumé III. 2018 · 2018

Earlier work this paper cites.

Asking clarifying questions in open-domain information-seeking conversations

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019 · 2019

Earlier work this paper cites.

Returning the N to NLP: Towards contextually personalized classification models

Lucie Flek. 2020 · 2020

Earlier work this paper cites.

ClarQ: A large-scale and diverse dataset for clarification question generation

Vaibhav Kumar and Alan W Black. 2020 · 2020

Earlier work this paper cites.

UNQOVERing stereotyping biases via underspecified questions

Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020 · 2020

Earlier work this paper cites.

Interactive classification by asking informative questions

Lili Yu, Howard Chen, Sida I. Wang, Tao Lei, and Yoav Artzi. 2020 · 2020

Earlier work this paper cites.

Generating clarifying questions for information retrieval

Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020 · 2020

Earlier work this paper cites.

Ask what’s missing and what’s useful: Improving clarification question generation using global knowledge

Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley. 2021 · 2021

Earlier work this paper cites.

Open-domain clarification question generation without question examples

Julia White, Gabriel Poesia, Robert Hawkins, Dorsa Sadigh, and Noah Goodman. 2021 · 2021

Earlier work this paper cites.

How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges

Kimiya Keyvan and Jimmy Xiangji Huang. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Everyone deserves a reward: Learning customized human preferences

Original

Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. 2023 · 2023

Earlier work this paper cites.

Can large language models be an alternative to human evaluations?

Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023

Earlier work this paper cites.

Toxicity in chatgpt: Analyzing persona-assigned language models

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023 · 2023

Earlier work this paper cites.

Aligning language models to user opinions

EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023 · 2023

Cited alongside, same era.

Personalized soups: Personalized large language model alignment via post-hoc parameter merging

Original

Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023 · 2023

Cited alongside, same era.

Diversify and disambiguate: Learning from underspecified data

Original

Yoonho Lee, Huaxiu Yao, and Chelsea Finn. 2023 · 2023

Cited alongside, same era.

Alpacaeval: An automatic evaluator of instruction-following models

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023

Cited alongside, same era.

G-eval: NLG evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Gemma 2: Improving open language models at a practical size

Original

Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024 · 2024

Closest in time.

On overcoming miscalibrated conversational priors in llm-based chatbots

Original

Christine Herlihy, Jennifer Neville, Tobias Schnabel, and Adith Swaminathan. 2024 · 2024

Closest in time.

Jamba-1.5: Hybrid transformer-mamba models at scale

Original

Team Jamba, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. 2024 · 2024

Closest in time.

ExpertQA: Expert-curated questions and attributed answers

Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024 · 2024

Closest in time.

Llm evaluators recognize and favor their own generations