Fetching the paper…

Reading the bibliography…

2023

Almanac: Retrieval-Augmented Language Models for Clinical Medicine

Zakka, Cyril, Chaurasia, Akash, Shad, Rohan et al.

Original
PDF
Post on X

Understand

Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering.

Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements.
In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations.
Performance on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety.

Almanac: Retrieval-Augmented Language Models for Clinical Medicine · Around

Built on

Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text (2019). https://doi.org/10.48550/ARXIV.1903.10676
1903
Earlier work this paper cites.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling Laws for Neural Language Models. arXiv (2020). https://doi.org/10.48550/ARXIV.2001.08361 . https://arxiv.org/abs/2001.08361
Original
2001
Earlier work this paper cites.
Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Annual Meeting of the Association for Computational Linguistics (2002)
2002
Earlier work this paper cites.
Papanikolaou, Y., Pierleoni, A.: DARE: Data Augmented Relation Extraction with GPT-2. arXiv (2020). https://doi.org/10.48550/ARXIV.2004.13845 . https://arxiv.org/abs/2004.13845
Original
2004
Earlier work this paper cites.
Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language Models are Few-Shot Learners. arXiv (2020). https://doi.org/10.48550/ARXIV.2005.14165 . https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
Hoogendam, A., Stalenhoef, A.F., de Vries Robbé, P.F., Overbeke, A.J.P.: Answers to Questions Posed During Daily Patient Care Are More Likely to Be Answered by UpToDate Than PubMed. J Med Internet Res 10
2008
Earlier work this paper cites.
Shin, H.-C., Zhang, Y., Bakhturina, E., Puri, R., Patwary, M., Shoeybi, M., Mani, R.: BioMegatron: Larger Biomedical Domain Language Model. arXiv (2020). https://doi.org/10.48550/ARXIV.2010.06060 . https://arxiv.org/abs/2010.06060
Original
2010
Earlier work this paper cites.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., Raffel, C.: Extracting Training Data from Large Language Models. arXiv (2020). https://doi.org/10.48550/ARXIV.2012.07805 . https://arxiv.org/abs/2012.07805
Original
2012
Earlier work this paper cites.
Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. arXiv (2016). https://doi.org/10.48550/ARXIV.1603.09320 . https://arxiv.org/abs/1603.09320
Original
2016
Earlier work this paper cites.
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv (2018). https://doi.org/10.48550/ARXIV.1810.04805 . https://arxiv.org/abs/1810.04805
Original
2018
Earlier work this paper cites.
Swinger, N., De-Arteaga, M., Heffernan, N.T., Leiserson, M.D., Kalai, A.T.: What are the biases in my word embedding? arXiv (2018). https://doi.org/10.48550/ARXIV.1812.08769 . https://arxiv.org/abs/1812.08769
Original
2018
Earlier work this paper cites.
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36
2019
Earlier work this paper cites.
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., Lu, X.: PubMedQA: A dataset for biomedical research question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567–2577. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1259 . https://aclanthology.org/D19-1259
2019
Earlier work this paper cites.
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams (2020)
Original
2020
Earlier work this paper cites.

Similar

Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W.H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A.N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W.: Evaluating large language models trained on code. CoRR abs/2107.03374
Original
2021
Cited alongside, same era.
Wei, C., Xie, S.M., Ma, T.: Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning. arXiv (2021). https://doi.org/10.48550/ARXIV.2106.09226 . https://arxiv.org/abs/2106.09226
Original
2021
Cited alongside, same era.
Raunak, V., Menezes, A., Junczys-Dowmunt, M.: The Curious Case of Hallucinations in Neural Machine Translation. arXiv (2021). https://doi.org/10.48550/ARXIV.2104.06683 . https://arxiv.org/abs/2104.06683
Original
2021
Cited alongside, same era.

Then

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H.S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhao, V., Zhou, Y., Chang, C.-C., Krivokon, I., Rusch, W., Pickett, M., Srinivasan, P., Man, L., Meier-Hellstern, K., Morris, M.R., Doshi, T., Santos, R.D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., Le, Q.: LaMDA: Language Models for Dialog Applications. arXiv (2022). https://doi.org/10.48550/ARXIV.2201.08239 . https://arxiv.org/abs/2201.08239
Original
2022
Later among the works it cites.
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., Kamar, E.: ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. arXiv (2022). https://doi.org/10.48550/ARXIV.2203.09509 . https://arxiv.org/abs/2203.09509
Original
2022
Later among the works it cites.
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., Liu, T.-Y.: BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics 23
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Liang, P.P., Wu, C., Morency, L.-P., Salakhutdinov, R.: Towards Understanding and Mitigating Social Biases in Language Models. arXiv (2021). https://doi.org/10.48550/ARXIV.2106.13219 . https://arxiv.org/abs/2106.13219

2021

Cited alongside, same era.

Reynolds, L., McDonell, K.: Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv (2021). https://doi.org/10.48550/ARXIV.2102.07350 . https://arxiv.org/abs/2102.07350

2021

Cited alongside, same era.

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W.H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A.N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W.: Evaluating Large Language Models Trained on Code. arXiv (2021). https://doi.org/10.48550/ARXIV.2107.03374 . https://arxiv.org/abs/2107.03374

2021

Cited alongside, same era.

Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 3

2021

Cited alongside, same era.

Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., Schulman, J.: WebGPT: Browser-assisted question-answering with human feedback. arXiv (2021). https://doi.org/10.48550/ARXIV.2112.09332 . https://arxiv.org/abs/2112.09332

Original

2021

Cited alongside, same era.

Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., Fedus, W.: Emergent Abilities of Large Language Models. arXiv (2022). https://doi.org/10.48550/ARXIV.2206.07682 . https://arxiv.org/abs/2206.07682

2022

Cited alongside, same era.

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G.v.d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J.W., Vinyals, O., Sifre, L.: Training Compute-Optimal Large Language Models. arXiv (2022). https://doi.org/10.48550/ARXIV.2203.15556 . https://arxiv.org/abs/2203.15556

2022

Cited alongside, same era.

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys (2022). https://doi.org/10.1145/3571730

2022

Cited alongside, same era.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. arXiv (2022). https://doi.org/10.48550/ARXIV.2203.02155 . https://arxiv.org/abs/2203.02155

2022

Cited alongside, same era.

Hong, Z., Ajith, A., Pauloski, G., Duede, E., Malamud, C., Magoulas, R., Chard, K., Foster, I.: ScholarBERT: Bigger is Not Always Better. arXiv (2022). https://doi.org/10.48550/ARXIV.2205.11342 . https://arxiv.org/abs/2205.11342

2022

Later among the works it cites.

Yang, X., Chen, A., PourNejatian, N., Shin, H.C., Smith, K.E., Parisien, C., Compas, C., Martin, C., Flores, M.G., Zhang, Y., Magoc, T., Harle, C.A., Lipori, G., Mitchell, D.A., Hogan, W.R., Shenkman, E.A., Bian, J., Wu, Y.: GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records. arXiv (2022). https://doi.org/10.48550/ARXIV.2203.03540 . https://arxiv.org/abs/2203.03540

2022

Later among the works it cites.

Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P., Arcas, B.A.y., Webster, D., Corrado, G.S., Matias, Y., Chou, K., Gottweis, J., Tomasev, N., Liu, Y., Rajkomar, A., Barral, J., Semturs, C., Karthikesalingam, A., Natarajan, V.: Large Language Models Encode Clinical Knowledge. arXiv (2022). https://doi.org/10.48550/ARXIV.2212.13138 . https://arxiv.org/abs/2212.13138

2022

Later among the works it cites.

Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., Stojnic, R.: Galactica: A Large Language Model for Science. arXiv (2022). https://doi.org/10.48550/ARXIV.2211.09085 . https://arxiv.org/abs/2211.09085

2022

Later among the works it cites.

Pal, A., Umapathi, L.K., Sankarasubbu, M.: Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering (2022). https://doi.org/10.48550/ARXIV.2203.14371

2022

Later among the works it cites.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv (2022). https://doi.org/10.48550/ARXIV.2201.11903 . https://arxiv.org/abs/2201.11903

2022

Later among the works it cites.

Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C.D., Liang, P., Leskovec, J.: Deep Bidirectional Language-Knowledge Graph Pretraining (2022)

2022

Later among the works it cites.

Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences (2023)

2023

Closest in time.

Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M.J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., Alsentzer, E.: Do We Still Need Clinical Language Models? arXiv (2023). https://doi.org/10.48550/ARXIV.2302.08091 . https://arxiv.org/abs/2302.08091

2023

Closest in time.

Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., Scialom, T.: Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023). https://doi.org/10.48550/ARXIV.2302.04761 . https://arxiv.org/abs/2302.04761

2023

Closest in time.

Liévin, V., Hother, C.E., Winther, O.: Can large language models reason about medical questions? (2023)

2023

Closest in time.