Fetching the paper…
Reading the bibliography…
This paper presents a compute-efficient approach to pre-training a Language Model-the "1.5-Pints"-in only 9 days, while outperforming state-of-the-art models as an instruction-following assistant.Based on MT-Bench (a benchmark that emulates human judgments), 1.5-Pints outperforms Apple's OpenELM and Microsoft's Phi.This is achieved by a carefully curated pre-training dataset of 57 billion tokens, using a mix of automated workflows and manual human review.
Europarl: A parallel corpus for statistical machine translation
Philipp Koehn · 2005
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
A 3 {A}^{3} : Accelerating attention mechanisms in neural networks with approximation
Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee, and Deog-Kyoon Jeong · 2020
Earlier work this paper cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al · 2020
Earlier work this paper cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela · 2020
Earlier work this paper cites.
An optimal resource allocator of elastic training for deep learning jobs on cloud
Liang Hu, Jiangcheng Zhu, Zirui Zhou, Ruiqing Cheng, Xiaolong Bai, and Yong Zhang · 2021
Earlier work this paper cites.
Combiner: Full attention transformer with sparse computation cost
Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai · 2021
Earlier work this paper cites.
Retrieval augmentation reduces hallucination in conversation, 2021
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston · 2021
Earlier work this paper cites.
A software-defined tensor streaming multiprocessor for large-scale machine learning
Dennis Abts, Garrin Kimmell, Andrew Ling, John Kim, Matt Boyd, Andrew Bitar, Sahil Parmar, Ibrahim Ahmed, et al · 2022
Earlier work this paper cites.
Transformer quality in linear time
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le · 2022
Earlier work this paper cites.
Holistic evaluation of language models
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda · 2022
Earlier work this paper cites.
XRICL: Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-SQL semantic parsing
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou · 2022
Earlier work this paper cites.
GQA: Training generalized multi-query transformer models from multi-head checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai · 2023
Earlier work this paper cites.
Self-RAG: Self-reflective retrieval augmented generation
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi · 2023
Earlier work this paper cites.
Scaling mlps: A tale of inductive bias
Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann · 2023
Earlier work this paper cites.
Unlimiformer: Long-range transformers with unlimited length input
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley · 2023
Earlier work this paper cites.
Why LLMs hallucinate, and how to get (evidential) closure: Perceptual, intensional, and extensional learning for faithful natural language generation
Adam Bouyamourn · 2023
Earlier work this paper cites.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen · 2023
Earlier work this paper cites.
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, 2023
Tri Dao · 2023
Earlier work this paper cites.
ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu · 2023
Earlier work this paper cites.
Survey of hallucination in natural language generation
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung · 2023
Earlier work this paper cites.
Prospector: Improving LLM agents with self-asking and trajectory ranking
Byoungjip Kim, Youngsoo Jang, Lajanugen Logeswaran, Geon-Hyeong Kim, Yu Jin Kim, Honglak Lee, and Moontae Lee · 2023
Earlier work this paper cites.
Making large language models better data creators
Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W White, and Sujay Kumar Jauhar · 2023
Cited alongside, same era.
Camel: Communicative agents for "mind" exploration of large language model society
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem · 2023
Cited alongside, same era.
Textbooks are all you need ii: phi-1.5 technical report, 2023
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee · 2023
Cited alongside, same era.
When less is more: Investigating data pruning for pretraining LLMs at scale
Max Marion, Ahmet Ustun, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker · 2023
Cited alongside, same era.
Orca: Progressive learning from complex explanation traces of gpt-4, 2023
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah · 2023
Cited alongside, same era.
LLM augmented hierarchical agents
Retrieval-augmented generation for large language models: A survey, 2024
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang · 2024
Closest in time.
Textbooks are all you need
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Conti Kauffmann, Gustavo Henrique de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Behl, Xin Wang, Sebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li · 2024
Closest in time.
PhilArchive
Travis Hoppe · 2024
Closest in time.
Compression represents intelligence linearly, 2024
Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He · 2024
Closest in time.
Ever: Mitigating hallucination in large language models through real-time verification and rectification, 2024
Haoqiang Kang, Juntong Ni, and Huaxiu Yao · 2024
Closest in time.
https://x.com/karpathy/status/1621578354024677377?lang=en
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Bharat Prakash, Tim Oates, and Tinoosh Mohsenin · 2023
Cited alongside, same era.
Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn · 2023
Cited alongside, same era.
Replug: Retrieval-augmented black-box language models, 2023
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih · 2023
Cited alongside, same era.
falcon-refinedweb (revision 184df75)
Technology Innovation Institute · 2023
Cited alongside, same era.
The alignment handbook
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf · 2023
Cited alongside, same era.
Knowledge graph prompting for multi-document question answering
Yu Wang, Nedim Lipka, Ryan Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr · 2023
Cited alongside, same era.
Gpt4tools: Teaching large language model to use tools via self-instruction
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan · 2023
Cited alongside, same era.
Andrej Karpathy · 2024
Closest in time.
Tinystories: How small can language models be and still speak coherent english
Yuanzhi Li and Ronen Eldan · 2024
Closest in time.
At which training stage does code data help LLMs reasoning?
YINGWEI MA, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li · 2024
Closest in time.
US-PD-Books Dataset
Sebastian Majstorovic · 2024
Closest in time.
Introducing meta llama 3: The most capable openly available llm to date
Meta AI · 2024
Closest in time.
SciPhi Textbooks Are All You Need Lite
Donald Della Pietra Owen Colegrove, Chunqiu Steven Xia · 2024
Closest in time.
Tiny Orca Textbooks
Nam Pham · 2024
Closest in time.
Tiny Textbooks
Nam Pham · 2024
Closest in time.
Toolformer: Language models can teach themselves to use tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom · 2024
Closest in time.
Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang · 2024
Closest in time.
Biomedical knowledge graph-optimized prompt generation for large language models, 2024
Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk-Jackson, Sharat Israni, Charlotte A Nelson, Sui Huang, and Sergio E Baranzini · 2024
Closest in time.
Fidelis: Faithful reasoning in large language model for knowledge graph question answering, 2024
Yuan Sui, Yufei He, Nian Liu, Xiaoxin He, Kun Wang, and Bryan Hooi · 2024
Closest in time.
wiki40b/en
Tensorflow · 2024
Closest in time.
LESS: Selecting influential data for targeted instruction tuning
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen · 2024
Closest in time.
WizardLM: Empowering large pre-trained language models to follow complex instructions
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang · 2024
Closest in time.
Tinyllama: An open-source small language model, 2024
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu · 2024
Closest in time.
One token can help! learning scalable and pluggable virtual tokens for retrieval-augmented large language models, 2024
Yutao Zhu, Zhaoheng Huang, Zhicheng Dou, and Ji-Rong Wen · 2024
Closest in time.