Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

What disease does this patient have? a large-scale open domain question answering dataset from medical exams

Original

Di Jin et al. 2020b · 2009

Earlier work this paper cites.

Mimic-iii, a freely accessible critical care database

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016 · 2016

Earlier work this paper cites.

Diverse beam search: Decoding diverse solutions from neural sequence models

Original

Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016 · 2016

Earlier work this paper cites.

Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering

Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019 · 2019

Earlier work this paper cites.

PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin et al. 2019 · 2019

Earlier work this paper cites.

Meddialog: Large-scale medical dialogue datasets

Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. 2020 · 2020

Earlier work this paper cites.

Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

Ankit Pal et al. 2022 · 2022

Earlier work this paper cites.

Biored: a rich biomedical relation extraction dataset

Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022 · 2022

Earlier work this paper cites.

Evaluating unsupervised text classification: Zero-shot and similarity-based approaches

Tim Schopf, Daniel Braun, and Florian Matthes. 2023 · 2022

Earlier work this paper cites.

Challenging big-bench tasks and whether chain-of-thought can solve them

Original

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Creating trustworthy llms: Dealing with hallucinations in healthcare ai

Original

Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023 · 2023

Earlier work this paper cites.

An empirical study of clinical note generation from doctor-patient encounters

Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023 · 2023

Earlier work this paper cites.

Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations

Original

Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023 · 2023

Earlier work this paper cites.

Camels in a changing climate: Enhancing lm adaptation with tulu 2

Original

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023 · 2023

Earlier work this paper cites.

Evaluating open-domain question answering in the era of large language models

Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023 · 2023

Cited alongside, same era.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Nlg evaluation metrics beyond correlation analysis: An empirical metric preference checklist

Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023 · 2023

Cited alongside, same era.

Large language models sensitivity to the order of options in multiple-choice questions

Original

Pouya Pezeshkpour and Estevam Hruschka. 2023 · 2023

Cited alongside, same era.

Med-halt: Medical domain hallucination test for large language models

Original

Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023 · 2023

Clue: A clinical language understanding evaluation for llms

Original

Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Koraş, Constantin Marc Seibold, Kaleb E Smith, and Jens Kleesiek. 2024 · 2024

Later among the works it cites.

Medalign: A clinician-generated dataset for instruction following with electronic medical records

Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2024 · 2024

Later among the works it cites.

Athene-70b: Redefining the boundaries of post-training for open models

Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. 2024 · 2024

Later among the works it cites.

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. 2024 · 2024

Later among the works it cites.

Olaph: Improving factuality in biomedical long-form question answering