KMMLU: Measuring Massive Multitask Language Understanding in Korean

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 1905

Earlier work this paper cites.

Korquad1. 0: Korean qa dataset for machine reading comprehension

Original

Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019 · 1909

Earlier work this paper cites.

Unsupervised cross-lingual representation learning at scale

Original

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Beep! korean corpus of online news comments for toxic speech detection

Original

Jihyung Moon, Won Ik Cho, and Junbum Lee. 2020 · 2005

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Squad: 100,000+ questions for machine comprehension of text

Original

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

Superglue: A stickier benchmark for general-purpose language understanding systems

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a · 2019

Earlier work this paper cites.

Glue: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b · 2019

Earlier work this paper cites.

Generalized data augmentation for low-resource translation

Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019 · 2019

Earlier work this paper cites.

KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding

Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

Larger-scale transformers for multilingual masked language modeling

Original

Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021 · 2021

Earlier work this paper cites.

Measuring mathematical problem solving with the math dataset

Original

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Understanding by understanding not: Modeling negation in language models

Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021 · 2021

Earlier work this paper cites.

Truthfulqa: Measuring how models mimic human falsehoods

Original

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021 · 2021

Earlier work this paper cites.

Klue: Korean language understanding evaluation

Original

Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021 · 2021

Earlier work this paper cites.

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021 · 2021

Earlier work this paper cites.

Scaling instruction-finetuned language models

Original

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022 · 2022

Earlier work this paper cites.

Training compute-optimal large language models

Original

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022 · 2022

Cited alongside, same era.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Cited alongside, same era.

Lifting the curse of multilinguality by pre-training modular transformers

Original

Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022 · 2022

Cited alongside, same era.

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Original

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022 · 2022

Cited alongside, same era.

Arb: Advanced reasoning benchmark for large language models

Original

Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023 · 2023

Later among the works it cites.

Hae-rae bench: Evaluation of korean knowledge in language models

Original

Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, and Songseong Kim. 2023 · 2023

Later among the works it cites.

Gemini: a family of highly capable multimodal models

Original

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Later among the works it cites.