2020

Measuring Massive Multitask Language Understanding

Hendrycks, Dan, Burns, Collin, Basart, Steven et al.

Understand

We propose a new test to measure a text model's multitask accuracy.

The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.
We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average.

Built on

Natural adversarial examples
Original
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song · 1907
Earlier work this paper cites.
Computing machinery and intelligence
A. Turing · 1950
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents (extended abstract)
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling · 2013
Earlier work this paper cites.
MCTest: A challenge dataset for the open-domain machine comprehension of text
M. Richardson, C. J. Burges, and E. Renshaw · 2013
Earlier work this paper cites.
On calibration of modern neural networks
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger · 2017
Earlier work this paper cites.
Race: Large-scale reading comprehension dataset from examinations, 2017
G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy · 2017
Earlier work this paper cites.
Think you have solved question answering? try arc, the ai2 reasoning challenge
Original
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord · 2018
Earlier work this paper cites.
Can a suit of armor conduct electricity? a new dataset for open book question answering
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal · 2018
Earlier work this paper cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman · 2018
Earlier work this paper cites.
Piqa: Reasoning about physical commonsense in natural language, 2019
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi · 2019
Earlier work this paper cites.
From ’f’ to ’a’ on the n.y. regents science exams: An overview of the aristo project
Original
P. Clark, O. Etzioni, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, N. Tandon, S. Bhakthavatsalam, D. Groeneveld, M. Guerquin, and M. Schmitz · 2019
Earlier work this paper cites.

Similar

Bert: Pre-training of deep bidirectional transformers for language understanding
Original
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2019
Cited alongside, same era.
Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019
L. Huang, R. L. Bras, C. Bhagavatula, and Y. Choi · 2019
Cited alongside, same era.
Qasc: A dataset for question answering via sentence composition, 2019
T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal · 2019
Cited alongside, same era.
Verified uncertainty calibration, 2019
A. Kumar, P. Liang, and T. Ma · 2019
Cited alongside, same era.
Roberta: A robustly optimized bert pretraining approach
Original
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov · 2019
Cited alongside, same era.
Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek · 2019
Cited alongside, same era.
Language models as knowledge bases?, 2019
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever · 2019
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer, 2019
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2019
Cited alongside, same era.
Superglue: A stickier benchmark for general-purpose language understanding systems, 2019
A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman · 2019
Cited alongside, same era.
Deep anomaly detection with outlier exposure
D. Hendrycks, M. Mazeika, and T. Dietterich
Cited in the paper.

Then

Hellaswag: Can a machine really finish your sentence?, 2019
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi · 2019
Later among the works it cites.
Experience grounds language, 2020
Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto, and J. Turian · 2020
Closest in time.
Language models are few-shot learners, 2020
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei · 2020
Closest in time.
Shortcut learning in deep neural networks, 2020
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann · 2020
Closest in time.
Aligning ai with shared human values, 2020
D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt · 2020
Closest in time.
Scaling laws for neural language models, 2020
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei · 2020
Closest in time.
Unifiedqa: Crossing format boundaries with a single qa system, 2020
D. Khashabi, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi · 2020
Closest in time.
Albert: A lite bert for self-supervised learning of language representations
Original
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut · 2020
Closest in time.
A survey of evaluation metrics used for nlg systems
A. B. Sai, A. K. Mohankumar, and M. M. Khapra · 2020
Closest in time.
Evaluating machines by their real-world language use, 2020
R. Zellers, A. Holtzman, E. Clark, L. Qin, A. Farhadi, and Y. Choi · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…