Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904

Earlier work this paper cites.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

ExplainCPE: A free-text explanation benchmark of Chinese pharmacist examination

Dongfang Li, Jindi Yu, Baotian Hu, Zhenran Xu, and Min Zhang. 2023a · 1940

Earlier work this paper cites.

Communications through limited-response questioning

E. M. BENNETT, R. ALPERT, and A. C. GOLDSTEIN. 1954 · 1954

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

The elements of statistical learning: data mining, inference, and prediction , volume 2

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009 · 2009

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Medalign: A clinician-generated dataset for instruction following with electronic medical records

Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo Pontes Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju S. Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott J. Adams, Oluseyi Fayanju, Shreya J. Shah, Thomas Savage, Ethan Goh, Akshay S. Chaudhari, Nima Aghaeepour, Christopher D. Sharp, Michael A. Pfeffer, Percy Liang, Jonathan H. Chen, Keith E. Morse, Emma P. Brunskill, Jason A. Fries, and Nigam H. Shah. 2024 · 2014

Earlier work this paper cites.

Teaching machines to read and comprehend

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015

Earlier work this paper cites.

Overview of the medical question answering task at trec 2017 liveqa

Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017 · 2017

Earlier work this paper cites.

emrqa: A large corpus for question answering on electronic medical records

Original

Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018 · 2018

Earlier work this paper cites.

A question-entailment approach to question answering

Asma Ben Abacha and Dina Demner-Fushman. 2019 · 2019

Earlier work this paper cites.

Bridging the gap between consumers’ medication questions and trusted answers

Asma Ben Abacha, Yassine Mrabet, Mark E. Sharp, Travis R. Goodwin, Sonya E. Shooshan, and Dina Demner-Fushman. 2019 · 2019

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Large-scale, diverse, paraphrastic bitexts via sampling and clustering

J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019 · 2019

Earlier work this paper cites.

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020

Earlier work this paper cites.

BLEURT: Learning robust metrics for text generation

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020 · 2020

Earlier work this paper cites.

Compression, transduction, and creation: A unified framework for evaluating natural language generation

Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021 · 2021

Earlier work this paper cites.

What disease does this patient have? a large-scale open domain question answering dataset from medical exams

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021 · 2021

Earlier work this paper cites.

Bartscore: evaluating generated text as text generation

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering

Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon M. Lin, and Huan Sun. 2020 · 2021

Earlier work this paper cites.

Meta-learning via language model in-context tuning

Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022 · 2022

Cited alongside, same era.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Cited alongside, same era.

Can large language models reason about medical questions?

Original

Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022 · 2022

Cited alongside, same era.

Data contamination: From memorization to exploitation

Inbal Magar and Roy Schwartz. 2022 · 2022

Cited alongside, same era.

Rethinking the role of demonstrations: What makes in-context learning work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Cited alongside, same era.

Gpt-4 technical report

OpenAI. 2023 · 2023

Later among the works it cites.

Longbox: Evaluating transformers on long-sequence clinical tasks

Original

Mihir Parmar, Aakanksha Naik, Himanshu Gupta, Disha Agrawal, and Chitta Baral. 2023 · 2023

Later among the works it cites.

NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023 · 2023

Later among the works it cites.

Use of gpt-4 to analyze medical records of patients with extensive investigations and delayed diagnosis

Yat-Fung Shea, Cynthia Min Yao Lee, Whitney Chin Tung Ip, Dik Wai Anderson Luk, and Stephanie Sze Wing Wong. 2023 · 2023

Later among the works it cites.

Towards expert-level medical question answering with large language models

Original

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023 · 2023