Built on

Perplexity—a measure of the difficulty of speech recognition tasks
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker · 1977
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Unsupervised quality estimation for neural machine translation
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia · 2020
Earlier work this paper cites.
Language models (mostly) know what they know
Original
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al · 2022
Earlier work this paper cites.
Teaching models to express their uncertainty in words
Original
Stephanie Lin, Jacob Hilton, and Owain Evans · 2022
Earlier work this paper cites.
Lm-polygraph: Uncertainty estimation for language models
Original
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al · 2023
Earlier work this paper cites.
Mistral 7b
Original
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023
Earlier work this paper cites.
Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation
Original
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar · 2023
Earlier work this paper cites.
Large language models sensitivity to the order of options in multiple-choice questions
Original
Pouya Pezeshkpour and Estevam Hruschka · 2023
Earlier work this paper cites.
Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback
Original
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning · 2023
Earlier work this paper cites.

Similar

Llama 2: Open foundation and fine-tuned chat models
Original
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Cited alongside, same era.
Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms
Original
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi · 2023
Cited alongside, same era.
Evaluating large language models at evaluating instruction following
Original
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen · 2023
Cited alongside, same era.
Instruction-following evaluation for large language models
Original
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou · 2023
Cited alongside, same era.

Then

Personal llm agents: Insights and survey about the capability, efficiency and security
Original
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al · 2024
Closest in time.
Uncertainty estimation and quantification for llms: A simple supervised approach
Original
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen · 2024
Closest in time.
Infobench: Evaluating instruction following ability in large language models
Original
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu · 2024
Closest in time.
Conifer: Improving complex constrained instruction-following ability of large language models
Original
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Do LLMs estimate uncertainty well in instruction-following?

Heo, Juyeon, Xiong, Miao, Heinze-Deml, Christina et al.

Understand

Large language models (LLMs) could be valuable personal AI agents across various domains, provided they can precisely follow user instructions.

However, recent studies have shown significant limitations in LLMs' instruction-following capabilities, raising concerns about their reliability in high-stakes applications.
Accurately estimating LLMs' uncertainty in adhering to instructions is critical to mitigating deployment risks.
We present, to our knowledge, the first systematic evaluation of the uncertainty estimation abilities of LLMs in the context of instruction-following.

Built on

Perplexity—a measure of the difficulty of speech recognition tasks
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker · 1977
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Unsupervised quality estimation for neural machine translation
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia · 2020
Earlier work this paper cites.
Language models (mostly) know what they know
Original
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al · 2022
Earlier work this paper cites.
Teaching models to express their uncertainty in words
Original
Stephanie Lin, Jacob Hilton, and Owain Evans · 2022
Earlier work this paper cites.
Lm-polygraph: Uncertainty estimation for language models
Original
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al · 2023
Earlier work this paper cites.
Mistral 7b
Original
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023
Earlier work this paper cites.
Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation
Original
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar · 2023
Earlier work this paper cites.
Large language models sensitivity to the order of options in multiple-choice questions
Original
Pouya Pezeshkpour and Estevam Hruschka · 2023
Earlier work this paper cites.
Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback
Original
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning · 2023
Earlier work this paper cites.

Similar

Llama 2: Open foundation and fine-tuned chat models
Original
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Cited alongside, same era.
Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms
Original
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi · 2023
Cited alongside, same era.
Evaluating large language models at evaluating instruction following
Original
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen · 2023
Cited alongside, same era.
Instruction-following evaluation for large language models
Original
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou · 2023
Cited alongside, same era.

Then

Personal llm agents: Insights and survey about the capability, efficiency and security
Original
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al · 2024
Closest in time.
Uncertainty estimation and quantification for llms: A simple supervised approach
Original
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen · 2024
Closest in time.
Infobench: Evaluating instruction following ability in large language models
Original
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu · 2024
Closest in time.
Conifer: Improving complex constrained instruction-following ability of large language models
Original
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Phi-3 technical report: A highly capable language model locally on your phone

Original

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al · 2024

Cited alongside, same era.

Distinguishing the knowable from the unknowable with language models

Original

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman · 2024

Cited alongside, same era.

Semantically diverse language generation for uncertainty estimation in language models

Original

Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielanskyi, and Sepp Hochreiter · 2024

Cited alongside, same era.

Detecting hallucinations in large language models using semantic entropy

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal · 2024

Cited alongside, same era.

From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models

Original

Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao · 2024

Cited alongside, same era.

Evalverse: Unified and accessible library for large language model evaluation

Original

Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, and Chanjun Park · 2024

Cited alongside, same era.

Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with llms

Original

Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong

Cited in the paper.

Large language models are not fair evaluators

Original

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui

Cited in the paper.

Towards conversational diagnostic ai

Original

Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al · 2024

Closest in time.

Fofo: A benchmark to evaluate llms’ format-following capability

Original

Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong · 2024

Closest in time.

Refutebench: Evaluating refuting instruction-following for large language models

Original

Jianhao Yan, Yun Luo, and Yue Zhang · 2024

Closest in time.

Benchmarking llms via uncertainty quantification

Original

Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu · 2024

Closest in time.

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2024

Closest in time.