Built on

Wilcoxon signed-rank test
Robert F Woolson. 2005 · 2005
Earlier work this paper cites.
Ms marco: A human-generated machine reading comprehension dataset
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Earlier work this paper cites.
Natural questions: a benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al · 2019
Earlier work this paper cites.
Generation-augmented retrieval for open-domain question answering
Original
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020 · 2020
Earlier work this paper cites.
On the opportunities and risks of foundation models
Original
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Earlier work this paper cites.
A dataset of information-seeking questions and answers anchored in research papers
Original
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021 · 2021
Earlier work this paper cites.
Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering
Original
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, and Suranga Nanayakkara. 2021 · 2021
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Self-rag: Learning to retrieve, generate, and critique through self-reflection
Original
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023 · 2023
Earlier work this paper cites.
Ragas: Automated evaluation of retrieval augmented generation
Original
Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023 · 2023
Earlier work this paper cites.
Retrieval-augmented generation for large language models: A survey
Original
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 · 2023
Earlier work this paper cites.
Large language models for software engineering: A systematic literature review
Original
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023 · 2023
Earlier work this paper cites.
Gpteval: Nlg evaluation using gpt-4 with better human alignment
Original
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023
Earlier work this paper cites.

Similar

Query rewriting for retrieval-augmented large language models
Original
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023 · 2023
Cited alongside, same era.
GPT-4 Technical Report
Original
OpenAI. 2023 · 2023
Cited alongside, same era.
Evaluating LLMs on Document-Based QA: Exact Answer Selection and Numerical Extraction using Cogtale dataset
Original
Zafaryab Rasool, Scott Barnett, Stefanus Kurniawan, Sherwin Balugo, Rajesh Vasa, Courtney Chesser, and Alex Bahar-Fuchs. 2023 · 2023
Cited alongside, same era.
Ares: An automated evaluation framework for retrieval-augmented generation systems
Original
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023 · 2023
Cited alongside, same era.
Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design

Then

Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 11661–11665
Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2024 · 2024
Closest in time.
Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity
Original
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024 · 2024
Closest in time.
Piperag: Fast retrieval-augmented generation via algorithm-system co-design
Original
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. 2024 · 2024
Closest in time.
From Matching to Generation: A Survey on Generative Information Retrieval
Original
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

RAGProbe: An Automated Approach for Evaluating RAG Applications

Sivasothy, Shangeetha, Barnett, Scott, Kurniawan, Stefanus et al.

Understand

Retrieval Augmented Generation (RAG) is increasingly being used when building Generative AI applications.

Evaluating these applications and RAG pipelines is mostly done manually, via a trial and error process.
Automating evaluation of RAG pipelines requires overcoming challenges such as context misunderstanding, wrong format, incorrect specificity, and missing content.
Prior works therefore focused on improving evaluation metrics as well as enhancing components within the pipeline using available question and answer datasets.

Built on

Wilcoxon signed-rank test
Robert F Woolson. 2005 · 2005
Earlier work this paper cites.
Ms marco: A human-generated machine reading comprehension dataset
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Earlier work this paper cites.
Natural questions: a benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al · 2019
Earlier work this paper cites.
Generation-augmented retrieval for open-domain question answering
Original
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020 · 2020
Earlier work this paper cites.
On the opportunities and risks of foundation models
Original
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Earlier work this paper cites.
A dataset of information-seeking questions and answers anchored in research papers
Original
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021 · 2021
Earlier work this paper cites.
Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering
Original
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, and Suranga Nanayakkara. 2021 · 2021
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Self-rag: Learning to retrieve, generate, and critique through self-reflection
Original
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023 · 2023
Earlier work this paper cites.
Ragas: Automated evaluation of retrieval augmented generation
Original
Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023 · 2023
Earlier work this paper cites.
Retrieval-augmented generation for large language models: A survey
Original
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 · 2023
Earlier work this paper cites.
Large language models for software engineering: A systematic literature review
Original
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023 · 2023
Earlier work this paper cites.
Gpteval: Nlg evaluation using gpt-4 with better human alignment
Original
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023
Earlier work this paper cites.

Similar

Query rewriting for retrieval-augmented large language models
Original
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023 · 2023
Cited alongside, same era.
GPT-4 Technical Report
Original
OpenAI. 2023 · 2023
Cited alongside, same era.
Evaluating LLMs on Document-Based QA: Exact Answer Selection and Numerical Extraction using Cogtale dataset
Original
Zafaryab Rasool, Scott Barnett, Stefanus Kurniawan, Sherwin Balugo, Rajesh Vasa, Courtney Chesser, and Alex Bahar-Fuchs. 2023 · 2023
Cited alongside, same era.
Ares: An automated evaluation framework for retrieval-augmented generation systems
Original
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023 · 2023
Cited alongside, same era.
Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design

Then

Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 11661–11665
Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2024 · 2024
Closest in time.
Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity
Original
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024 · 2024
Closest in time.
Piperag: Fast retrieval-augmented generation via algorithm-system co-design
Original
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. 2024 · 2024
Closest in time.
From Matching to Generation: A Survey on Generative Information Retrieval
Original
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. 2023 · 2023

Cited alongside, same era.

A survey of large language models

Original

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al · 2023

Cited alongside, same era.

Large language models can learn rules

Original

Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. 2023 · 2023

Cited alongside, same era.

Automated unit test improvement using large language models at meta

Original

Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024 · 2024

Cited alongside, same era.

RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O Nunes, et al · 2024

Cited alongside, same era.

Seven Failure Points When Engineering a Retrieval Augmented Generation System

Original

Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024 · 2024

Cited alongside, same era.

The Power of Noise: Redefining Retrieval for RAG Systems

Original

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024 · 2024

Cited alongside, same era.

InspectorRAGet: An Introspection Platform for RAG Evaluation

Original

Kshitij Fadnis, Siva Sankalp Patel, Odellia Boni, Yannis Katsis, Sara Rosenthal, Benjamin Sznajder, and Marina Danilevsky. 2024 · 2024

Cited alongside, same era.

LLMs for Test Input Generation for Semantic Caches

Original

Zafaryab Rasool, Scott Barnett, David Willie, Stefanus Kurniawan, Sherwin Balugo, Srikanth Thudumu, and Mohamed Abdelrazek. 2024 · 2024

Closest in time.

CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

Original

Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. 2024 · 2024