A Survey of Calibration Process for Black-Box LLMs

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904

Earlier work this paper cites.

Learning to count objects with few exemplar annotations

Original

Jianfeng Wang, Rong Xiao, Yandong Guo, and Lei Zhang. 2019 · 1905

Earlier work this paper cites.

Eli5: Long form question answering

Original

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019 · 1907

Earlier work this paper cites.

Sentence-bert: Sentence embeddings using siamese bert-networks

Original

N Reimers. 2019 · 1908

Earlier work this paper cites.

Evaluating the factual consistency of abstractive text summarization

Original

Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019 · 1910

Earlier work this paper cites.

Multicalibration: Calibration for the (computationally-identifiable) masses

Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. 2018 · 1948

Earlier work this paper cites.

Verification of forecasts expressed in terms of probability

Glenn W Brier. 1950 · 1950

Earlier work this paper cites.

A bayesian approach to calibration

David N DeJong, Beth Fisher Ingram, and Charles H Whiteman. 1996 · 1996

Earlier work this paper cites.

Axiomatic characterization of the quadratic scoring rule

Reinhard Selten. 1998 · 1998

Earlier work this paper cites.

Learning and making decisions when costs and probabilities are both unknown

Bianca Zadrozny and Charles Elkan. 2001 · 2001

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B Brown. 2020 · 2005

Earlier work this paper cites.

Some comparisons among quadratic, spherical, and logarithmic scoring rules

J Eric Bickel. 2007 · 2007

Earlier work this paper cites.

Strictly proper scoring rules, prediction, and estimation

Tilmann Gneiting and Adrian E Raftery. 2007 · 2007

Earlier work this paper cites.

Sampling uncertainty and confidence intervals for the brier score and brier skill score

A Allen Bradley, Stuart S Schwartz, and Tempei Hashino. 2008 · 2008

Earlier work this paper cites.

Pearson correlation coefficient

Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009 · 2009

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Accuracy-rejection curves (arcs) for comparing classification methods with a reject option

Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar. 2009 · 2009

Earlier work this paper cites.

Smooth isotonic regression: a new method to calibrate predictive models

Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. 2011 · 2011

Earlier work this paper cites.

Area under the precision-recall curve: point estimates and confidence intervals

Kendrick Boyd, Kevin H Eng, and C David Page. 2013 · 2013

Earlier work this paper cites.

Bigbench: Towards an industry standard benchmark for big data analytics

Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013 · 2013

Earlier work this paper cites.

Question answering with subgraph embeddings

Original

Antoine Bordes, Sumit Chopra, and Jason Weston. 2014 · 2014

Earlier work this paper cites.

Area under precision-recall curves for weighted and unweighted data

Jens Keilwagen, Ivo Grosse, and Jan Grau. 2014 · 2014

Earlier work this paper cites.

Spearman’s rank correlation coefficient

Philip Sedgwick. 2014 · 2014

Earlier work this paper cites.

Obtaining well calibrated probabilities using bayesian binning

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015 · 2015

Earlier work this paper cites.

Abstractive text summarization using sequence-to-sequence rnns and beyond

Original

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016 · 2016

Earlier work this paper cites.

Rethinking the inception architecture for computer vision

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016 · 2016

Earlier work this paper cites.

On calibration of modern neural networks

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017 · 2017

Earlier work this paper cites.

Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017 · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin. 2018 · 2018

Earlier work this paper cites.

A survey of methods for explaining black box models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018 · 2018

Earlier work this paper cites.

To trust or not to trust a classifier

Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018 · 2018

Earlier work this paper cites.

Commonsenseqa: A question answering challenge targeting commonsense knowledge

Original

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018 · 2018

Earlier work this paper cites.

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Original

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018

Earlier work this paper cites.

Retrieval augmented language model pre-training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021 · 2021

Earlier work this paper cites.

Domain-specific language model pretraining for biomedical natural language processing

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021 · 2021

Earlier work this paper cites.

Top-label calibration and multiclass-to-binary reductions

Original

Chirag Gupta and Aaditya Ramdas. 2021 · 2021

Earlier work this paper cites.

Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

Original

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

How can we know when language models know? on the calibration of language models for question answering

Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021 · 2021

Cited alongside, same era.

What disease does this patient have? a large-scale open domain question answering dataset from medical exams

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021 · 2021

Cited alongside, same era.

Bayesian confidence calibration for epistemic uncertainty modelling

Fabian Küppers, Jan Kronenberger, Jonas Schneider, and Anselm Haselhoff. 2021 · 2021

Cited alongside, same era.

Truthfulqa: Measuring how models mimic human falsehoods

Original

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021 · 2021

Cited alongside, same era.

Are nlp models really able to solve simple math word problems?

Original

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021 · 2021

Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

Original

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023 · 2023

Later among the works it cites.

A comprehensive capability analysis of gpt-3 and gpt-3.5 series models

Original

Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023 · 2023

Later among the works it cites.

Sac: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency

Original

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A Malin, and Sricharan Kumar. 2023 · 2023

Later among the works it cites.

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 · 2023

Later among the works it cites.