Built on

Revisiting self-training for neural sequence generation
Original
Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019 · 1909
Earlier work this paper cites.
Language models as knowledge bases?
Original
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019 · 1909
Earlier work this paper cites.
Rank analysis of incomplete block designs: I. the method of paired comparisons
Ralph Allan Bradley and Milton E Terry. 1952 · 1952
Earlier work this paper cites.
A broad-coverage challenge corpus for sentence understanding through inference
Original
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017 · 2017
Earlier work this paper cites.
Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing
Original
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 · 2021
Earlier work this paper cites.
Towards continual knowledge learning of language models
Original
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021 · 2021
Earlier work this paper cites.
Large language models can self-improve
Original
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022 · 2022
Earlier work this paper cites.
Explanations from large language models make small reasoners better
Original
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022 · 2022
Earlier work this paper cites.
Self-instruct: Aligning language models with self-generated instructions
Original
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Earlier work this paper cites.
Seven pillars for the future of artificial intelligence
Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho. 2023 · 2023
Earlier work this paper cites.
Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios
Original
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023 · 2023
Earlier work this paper cites.
Dola: Decoding by contrasting layers improves factuality in large language models
Original
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023 · 2023
Earlier work this paper cites.

Similar

Inspecting and editing knowledge representations in language models
Original
Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023 · 2023
Cited alongside, same era.
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes
Original
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023 · 2023
Cited alongside, same era.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions
Original
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023 · 2023
Cited alongside, same era.
Continual pre-training of language models
Original
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023 · 2023
Cited alongside, same era.

Then

Can ai assistants know what they don’t know?
Original
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Kai Chen, and Xipeng Qiu. 2024 · 2024
Closest in time.
Statistical knowledge assessment for large language models
Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2024 · 2024
Closest in time.
Into the unknown: Self-learning large language models
Original
Teddy Ferdinan, Jan Kocoń, and Przemysław Kazienko. 2024 · 2024
Closest in time.
Plausible extractive rationalization through semi-supervised entailment signal
Original
Yeo Wei Jie, Ranjan Satapathy, and Erik Cambria. 2024 · 2024
Closest in time.
Post hoc explanations of language models can improve language models
Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. 2024 · 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Self-training Large Language Models through Knowledge Detection

Yeo, Wei Jie, Ferdinan, Teddy, Kazienko, Przemyslaw et al.

Understand

Large language models (LLMs) often necessitate extensive labeled datasets and training compute to achieve impressive performance across downstream tasks.

This paper explores a self-training paradigm, where the LLM autonomously curates its own labels and selectively trains on unknown data samples identified through a reference-free consistency method.
Empirical evaluations demonstrate significant improvements in reducing hallucination in generation across multiple subjects.
Furthermore, the selective training framework mitigates catastrophic forgetting in out-of-distribution benchmarks, addressing a critical limitation in training LLMs.

Built on

Revisiting self-training for neural sequence generation
Original
Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019 · 1909
Earlier work this paper cites.
Language models as knowledge bases?
Original
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019 · 1909
Earlier work this paper cites.
Rank analysis of incomplete block designs: I. the method of paired comparisons
Ralph Allan Bradley and Milton E Terry. 1952 · 1952
Earlier work this paper cites.
A broad-coverage challenge corpus for sentence understanding through inference
Original
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017 · 2017
Earlier work this paper cites.
Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing
Original
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 · 2021
Earlier work this paper cites.
Towards continual knowledge learning of language models
Original
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021 · 2021
Earlier work this paper cites.
Large language models can self-improve
Original
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022 · 2022
Earlier work this paper cites.
Explanations from large language models make small reasoners better
Original
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022 · 2022
Earlier work this paper cites.
Self-instruct: Aligning language models with self-generated instructions
Original
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Earlier work this paper cites.
Seven pillars for the future of artificial intelligence
Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho. 2023 · 2023
Earlier work this paper cites.
Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios
Original
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023 · 2023
Earlier work this paper cites.
Dola: Decoding by contrasting layers improves factuality in large language models
Original
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023 · 2023
Earlier work this paper cites.

Similar

Inspecting and editing knowledge representations in language models
Original
Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023 · 2023
Cited alongside, same era.
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes
Original
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023 · 2023
Cited alongside, same era.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions
Original
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023 · 2023
Cited alongside, same era.
Continual pre-training of language models
Original
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023 · 2023
Cited alongside, same era.

Then

Can ai assistants know what they don’t know?
Original
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Kai Chen, and Xipeng Qiu. 2024 · 2024
Closest in time.
Statistical knowledge assessment for large language models
Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2024 · 2024
Closest in time.
Into the unknown: Self-learning large language models
Original
Teddy Ferdinan, Jan Kocoń, and Przemysław Kazienko. 2024 · 2024
Closest in time.
Plausible extractive rationalization through semi-supervised entailment signal
Original
Yeo Wei Jie, Ranjan Satapathy, and Erik Cambria. 2024 · 2024
Closest in time.
Post hoc explanations of language models can improve language models
Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. 2024 · 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

An empirical study of catastrophic forgetting in large language models during continual fine-tuning

Original

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023 · 2023

Cited alongside, same era.

Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

Original

Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023 · 2023

Cited alongside, same era.

Factscore: Fine-grained atomic evaluation of factual precision in long form text generation

Original

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023 · 2023

Cited alongside, same era.

Orca: Progressive learning from complex explanation traces of gpt-4

Original

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023 · 2023

Cited alongside, same era.

Fine-tuning language models for factuality

Original

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2023 · 2023

Cited alongside, same era.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Cited alongside, same era.

Zephyr: Direct distillation of lm alignment

Original

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023 · 2023

Cited alongside, same era.

Assessing the reliability of large language model knowledge

Original

Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2023 · 2023

Cited alongside, same era.

Closest in time.

Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation

Original

Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. 2024 · 2024

Closest in time.

Meta llama 3

Llama-3. 2024 · 2024